Sep 17 17:56:43.753932 ip-10-0-137-95 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 17:56:47.171890 ip-10-0-137-95 kubenswrapper[2553]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:56:47.171890 ip-10-0-137-95 kubenswrapper[2553]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 17:56:47.171890 ip-10-0-137-95 kubenswrapper[2553]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:56:47.171890 ip-10-0-137-95 kubenswrapper[2553]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 17:56:47.171890 ip-10-0-137-95 kubenswrapper[2553]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:56:47.172935 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.172839    2553 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 17:56:47.177579 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177555    2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:56:47.177579 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177577    2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:56:47.177579 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177581    2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:56:47.177708 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177584    2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:56:47.177708 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177588    2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:56:47.177708 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177591    2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:56:47.177708 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177594    2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:56:47.177708 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177597    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:56:47.177708 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177600    2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:56:47.177708 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177604    2553 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:56:47.177708 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177607    2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:56:47.177708 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177612    2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:56:47.177708 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177629    2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:56:47.177708 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177632    2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:56:47.177708 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177635    2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:56:47.177708 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177638    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:56:47.177708 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177642    2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:56:47.177708 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177650    2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:56:47.177708 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177653    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:56:47.177708 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177656    2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:56:47.177708 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177659    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:56:47.177708 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177662    2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:56:47.178167 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177665    2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:56:47.178167 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177667    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:56:47.178167 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177671    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:56:47.178167 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177673    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:56:47.178167 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177676    2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:56:47.178167 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177679    2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:56:47.178167 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177682    2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:56:47.178167 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177684    2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:56:47.178167 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177687    2553 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:56:47.178167 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177690    2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:56:47.178167 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177693    2553 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:56:47.178167 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177696    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:56:47.178167 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177698    2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:56:47.178167 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177701    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:56:47.178167 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177706    2553 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:56:47.178167 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177709    2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:56:47.178167 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177712    2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:56:47.178167 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177715    2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:56:47.178167 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177718    2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:56:47.178167 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177721    2553 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:56:47.178711 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177724    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:56:47.178711 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177727    2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:56:47.178711 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177730    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:56:47.178711 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177733    2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:56:47.178711 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177735    2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:56:47.178711 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177738    2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:56:47.178711 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177740    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:56:47.178711 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177743    2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:56:47.178711 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177746    2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:56:47.178711 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177749    2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:56:47.178711 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177752    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:56:47.178711 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177754    2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:56:47.178711 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177757    2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:56:47.178711 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177761    2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:56:47.178711 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177765    2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:56:47.178711 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177769    2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:56:47.178711 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177772    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:56:47.178711 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177775    2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:56:47.178711 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177778    2553 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:56:47.179172 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177781    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:56:47.179172 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177784    2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:56:47.179172 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177787    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:56:47.179172 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177790    2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:56:47.179172 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177792    2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:56:47.179172 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177795    2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:56:47.179172 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177798    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:56:47.179172 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177801    2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:56:47.179172 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177804    2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:56:47.179172 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177808    2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:56:47.179172 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177811    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:56:47.179172 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177814    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:56:47.179172 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177817    2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:56:47.179172 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177820    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:56:47.179172 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177823    2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:56:47.179172 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177826    2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:56:47.179172 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177829    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:56:47.179172 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177832    2553 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:56:47.179172 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177834    2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:56:47.179686 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177837    2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:56:47.179686 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177840    2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:56:47.179686 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177843    2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:56:47.179686 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177846    2553 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:56:47.179686 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177848    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:56:47.179686 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177851    2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:56:47.179686 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177854    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:56:47.179686 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177856    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:56:47.179686 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177859    2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:56:47.179686 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177861    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:56:47.179686 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177864    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:56:47.179686 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177866    2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:56:47.179686 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177869    2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:56:47.179686 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177871    2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:56:47.179686 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177874    2553 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:56:47.179686 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177876    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:56:47.179686 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177879    2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:56:47.179686 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177881    2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:56:47.179686 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.177885    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:56:47.179686 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178685    2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:56:47.180223 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178692    2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:56:47.180223 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178695    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:56:47.180223 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178699    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:56:47.180223 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178702    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:56:47.180223 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178707    2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:56:47.180223 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178710    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:56:47.180223 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178714    2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:56:47.180223 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178718    2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:56:47.180223 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178721    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:56:47.180223 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178724    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:56:47.180223 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178727    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:56:47.180223 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178730    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:56:47.180223 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178733    2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:56:47.180223 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178735    2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:56:47.180223 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178738    2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:56:47.180223 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178741    2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:56:47.180223 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178743    2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:56:47.180223 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178746    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:56:47.180223 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178749    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:56:47.180757 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178751    2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:56:47.180757 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178754    2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:56:47.180757 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178756    2553 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:56:47.180757 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178759    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:56:47.180757 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178762    2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:56:47.180757 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178764    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:56:47.180757 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178767    2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:56:47.180757 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178771    2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:56:47.180757 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178774    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:56:47.180757 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178777    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:56:47.180757 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178780    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:56:47.180757 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178783    2553 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:56:47.180757 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178786    2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:56:47.180757 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178789    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:56:47.180757 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178791    2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:56:47.180757 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178794    2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:56:47.180757 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178797    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:56:47.180757 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178800    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:56:47.180757 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178802    2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:56:47.181217 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178805    2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:56:47.181217 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178808    2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:56:47.181217 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178810    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:56:47.181217 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178813    2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:56:47.181217 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178816    2553 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:56:47.181217 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178819    2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:56:47.181217 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178821    2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:56:47.181217 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178824    2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:56:47.181217 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178826    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:56:47.181217 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178829    2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:56:47.181217 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178831    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:56:47.181217 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178834    2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:56:47.181217 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178836    2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:56:47.181217 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178839    2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:56:47.181217 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178841    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:56:47.181217 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178844    2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:56:47.181217 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178847    2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:56:47.181217 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178849    2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:56:47.181217 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178853    2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:56:47.181217 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178855    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:56:47.181737 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178857    2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:56:47.181737 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178861    2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:56:47.181737 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178865    2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:56:47.181737 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178867    2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:56:47.181737 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178871    2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:56:47.181737 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178874    2553 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:56:47.181737 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178877    2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:56:47.181737 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178880    2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:56:47.181737 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178882    2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:56:47.181737 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178885    2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:56:47.181737 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178888    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:56:47.181737 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178891    2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:56:47.181737 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178893    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:56:47.181737 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178896    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:56:47.181737 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178898    2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:56:47.181737 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178901    2553 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:56:47.181737 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178904    2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:56:47.181737 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178907    2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:56:47.181737 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178910    2553 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:56:47.181737 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178913    2553 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:56:47.182233 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178916    2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:56:47.182233 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178918    2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:56:47.182233 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178921    2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:56:47.182233 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178924    2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:56:47.182233 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178927    2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:56:47.182233 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178929    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:56:47.182233 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178932    2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:56:47.182233 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178935    2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:56:47.182233 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178937    2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:56:47.182233 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178940    2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:56:47.182233 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178942    2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:56:47.182233 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178945    2553 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:56:47.182233 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178947    2553 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:56:47.182233 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178950    2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:56:47.182233 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178952    2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:56:47.182233 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178955    2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:56:47.182233 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178958    2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:56:47.182233 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178961    2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:56:47.182233 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178964    2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:56:47.182233 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.178966    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:56:47.182739 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179391    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:56:47.182739 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179397    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:56:47.182739 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179401    2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:56:47.182739 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179405    2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:56:47.182739 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179409    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:56:47.182739 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179413    2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:56:47.182739 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179416    2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:56:47.182739 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179419    2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:56:47.182739 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179421    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:56:47.182739 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179425    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:56:47.182739 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179428    2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:56:47.182739 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179431    2553 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:56:47.182739 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179434    2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:56:47.182739 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179438    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:56:47.182739 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179441    2553 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:56:47.182739 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179443    2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:56:47.182739 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179446    2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:56:47.182739 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179450    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:56:47.182739 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179452    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:56:47.183273 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179455    2553 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:56:47.183273 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179458    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:56:47.183273 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179461    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:56:47.183273 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179464    2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:56:47.183273 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179467    2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:56:47.183273 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179469    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:56:47.183273 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179472    2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:56:47.183273 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179475    2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:56:47.183273 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179478    2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:56:47.183273 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179481    2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:56:47.183273 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179483    2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:56:47.183273 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179486    2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:56:47.183273 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179489    2553 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:56:47.183273 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179491    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:56:47.183273 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179494    2553 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:56:47.183273 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179497    2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:56:47.183273 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179499    2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:56:47.183273 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179502    2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:56:47.183273 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179505    2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:56:47.183273 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179507    2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:56:47.183795 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179510    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:56:47.183795 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179512    2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:56:47.183795 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179515    2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:56:47.183795 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179519    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:56:47.183795 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179521    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:56:47.183795 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179524    2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:56:47.183795 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179526    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:56:47.183795 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179529    2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:56:47.183795 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179531    2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:56:47.183795 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179534    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:56:47.183795 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179537    2553 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:56:47.183795 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179540    2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:56:47.183795 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179542    2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:56:47.183795 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179545    2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:56:47.183795 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179550    2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:56:47.183795 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179553    2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:56:47.183795 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179556    2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:56:47.183795 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179559    2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:56:47.183795 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179562    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:56:47.183795 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179564    2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:56:47.184279 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179567    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:56:47.184279 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179570    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:56:47.184279 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179574    2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:56:47.184279 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179577    2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:56:47.184279 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179579    2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:56:47.184279 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179582    2553 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:56:47.184279 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179585    2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:56:47.184279 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179587    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:56:47.184279 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179590    2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:56:47.184279 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179593    2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:56:47.184279 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179596    2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:56:47.184279 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179600    2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:56:47.184279 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179603    2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:56:47.184279 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179606    2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:56:47.184279 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179608    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:56:47.184279 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179612    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:56:47.184279 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179629    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:56:47.184279 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179632    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:56:47.184279 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179634    2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:56:47.184761 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179637    2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:56:47.184761 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179640    2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:56:47.184761 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179642    2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:56:47.184761 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179646    2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:56:47.184761 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179649    2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:56:47.184761 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179652    2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:56:47.184761 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179655    2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:56:47.184761 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179658    2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:56:47.184761 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179661    2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:56:47.184761 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179664    2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:56:47.184761 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179667    2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:56:47.184761 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179669    2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:56:47.184761 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179672    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:56:47.184761 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179674    2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:56:47.184761 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179677    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:56:47.184761 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179680    2553 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:56:47.184761 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179682    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:56:47.184761 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179685    2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:56:47.184761 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179687    2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:56:47.184761 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179690    2553 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:56:47.185245 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.179693    2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:56:47.185245 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.179781    2553 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 17:56:47.185245 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.179992    2553 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 17:56:47.185245 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180002    2553 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 17:56:47.185245 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180006    2553 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 17:56:47.185245 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180011    2553 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 17:56:47.185245 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180015    2553 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 17:56:47.185245 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180020    2553 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 17:56:47.185245 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180024    2553 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 17:56:47.185245 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180027    2553 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 17:56:47.185245 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180031    2553 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 17:56:47.185245 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180035    2553 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 17:56:47.185245 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180038    2553 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 17:56:47.185245 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180042    2553 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 17:56:47.185245 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180045    2553 flags.go:64] FLAG: --cgroup-root=""
Sep 17 17:56:47.185245 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180048    2553 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 17:56:47.185245 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180051    2553 flags.go:64] FLAG: --client-ca-file=""
Sep 17 17:56:47.185245 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180054    2553 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 17:56:47.185245 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180058    2553 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 17:56:47.185245 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180062    2553 flags.go:64] FLAG: --cluster-domain=""
Sep 17 17:56:47.185245 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180065    2553 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 17:56:47.185245 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180068    2553 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 17:56:47.185245 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180071    2553 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 17:56:47.185245 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180075    2553 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 17:56:47.185879 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180079    2553 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 17:56:47.185879 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180082    2553 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 17:56:47.185879 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180085    2553 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 17:56:47.185879 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180089    2553 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 17:56:47.185879 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180092    2553 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 17:56:47.185879 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180095    2553 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 17:56:47.185879 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180098    2553 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 17:56:47.185879 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180101    2553 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 17:56:47.185879 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180104    2553 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 17:56:47.185879 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180109    2553 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 17:56:47.185879 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180112    2553 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 17:56:47.185879 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180115    2553 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 17:56:47.185879 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180118    2553 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 17:56:47.185879 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180121    2553 flags.go:64] FLAG: --enable-server="true"
Sep 17 17:56:47.185879 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180124    2553 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 17:56:47.185879 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180130    2553 flags.go:64] FLAG: --event-burst="100"
Sep 17 17:56:47.185879 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180133    2553 flags.go:64] FLAG: --event-qps="50"
Sep 17 17:56:47.185879 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180136    2553 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 17:56:47.185879 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180139    2553 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 17:56:47.185879 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180143    2553 flags.go:64] FLAG: --eviction-hard=""
Sep 17 17:56:47.185879 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180147    2553 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 17:56:47.185879 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180150    2553 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 17:56:47.185879 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180153    2553 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 17:56:47.185879 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180157    2553 flags.go:64] FLAG: --eviction-soft=""
Sep 17 17:56:47.185879 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180160    2553 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 17:56:47.186505 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180163    2553 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 17:56:47.186505 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180166    2553 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 17:56:47.186505 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180169    2553 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 17:56:47.186505 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180173    2553 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 17:56:47.186505 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180176    2553 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 17:56:47.186505 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180179    2553 flags.go:64] FLAG: --feature-gates=""
Sep 17 17:56:47.186505 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180183    2553 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 17:56:47.186505 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180186    2553 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 17:56:47.186505 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180190    2553 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 17:56:47.186505 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180193    2553 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 17:56:47.186505 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180196    2553 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 17:56:47.186505 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180199    2553 flags.go:64] FLAG: --help="false"
Sep 17 17:56:47.186505 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180202    2553 flags.go:64] FLAG: --hostname-override="ip-10-0-137-95.ec2.internal"
Sep 17 17:56:47.186505 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180206    2553 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 17:56:47.186505 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180209    2553 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 17:56:47.186505 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180212    2553 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 17:56:47.186505 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180216    2553 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 17:56:47.186505 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180220    2553 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 17:56:47.186505 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180223    2553 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 17:56:47.186505 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180226    2553 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 17:56:47.186505 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180229    2553 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 17:56:47.186505 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180232    2553 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 17:56:47.186505 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180235    2553 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 17:56:47.187121 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180239    2553 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 17:56:47.187121 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180242    2553 flags.go:64] FLAG: --kube-reserved=""
Sep 17 17:56:47.187121 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180245    2553 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 17:56:47.187121 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180248    2553 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 17:56:47.187121 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180252    2553 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 17:56:47.187121 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180256    2553 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 17:56:47.187121 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180259    2553 flags.go:64] FLAG: --lock-file=""
Sep 17 17:56:47.187121 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180262    2553 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 17:56:47.187121 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180265    2553 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 17:56:47.187121 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180268    2553 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 17:56:47.187121 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180274    2553 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 17:56:47.187121 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180277    2553 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 17:56:47.187121 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180280    2553 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 17:56:47.187121 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180283    2553 flags.go:64] FLAG: --logging-format="text"
Sep 17 17:56:47.187121 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180286    2553 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 17:56:47.187121 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180289    2553 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 17:56:47.187121 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180292    2553 flags.go:64] FLAG: --manifest-url=""
Sep 17 17:56:47.187121 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180295    2553 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 17:56:47.187121 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180301    2553 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 17:56:47.187121 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180304    2553 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 17:56:47.187121 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180308    2553 flags.go:64] FLAG: --max-pods="110"
Sep 17 17:56:47.187121 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180311    2553 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 17:56:47.187121 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180315    2553 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 17:56:47.187121 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180318    2553 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 17:56:47.187121 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180321    2553 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 17:56:47.187794 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180324    2553 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 17:56:47.187794 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180327    2553 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 17:56:47.187794 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180330    2553 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 17:56:47.187794 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180338    2553 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 17:56:47.187794 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180341    2553 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 17:56:47.187794 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180344    2553 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 17:56:47.187794 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180347    2553 flags.go:64] FLAG: --pod-cidr=""
Sep 17 17:56:47.187794 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180350    2553 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 17:56:47.187794 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180359    2553 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 17:56:47.187794 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180362    2553 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 17:56:47.187794 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180365    2553 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 17:56:47.187794 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180368    2553 flags.go:64] FLAG: --port="10250"
Sep 17 17:56:47.187794 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180372    2553 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 17:56:47.187794 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180375    2553 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e9a9942cf95f4738"
Sep 17 17:56:47.187794 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180378    2553 flags.go:64] FLAG: --qos-reserved=""
Sep 17 17:56:47.187794 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180383    2553 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 17:56:47.187794 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180387    2553 flags.go:64] FLAG: --register-node="true"
Sep 17 17:56:47.187794 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180390    2553 flags.go:64] FLAG: --register-with-taints=""
Sep 17 17:56:47.187794 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180394    2553 flags.go:64] FLAG: --registry-burst="10"
Sep 17 17:56:47.187794 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180397    2553 flags.go:64] FLAG: --registry-qps="5"
Sep 17 17:56:47.187794 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180400    2553 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 17:56:47.187794 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180402    2553 flags.go:64] FLAG: --reserved-memory=""
Sep 17 17:56:47.187794 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180407    2553 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 17:56:47.187794 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180410    2553 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 17:56:47.187794 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180413    2553 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 17:56:47.188590 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180416    2553 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 17:56:47.188590 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180419    2553 flags.go:64] FLAG: --runonce="false"
Sep 17 17:56:47.188590 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180422    2553 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 17:56:47.188590 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180425    2553 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 17:56:47.188590 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180428    2553 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 17:56:47.188590 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180431    2553 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 17:56:47.188590 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180434    2553 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 17:56:47.188590 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180438    2553 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 17:56:47.188590 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180441    2553 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 17:56:47.188590 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180444    2553 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 17:56:47.188590 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180447    2553 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 17:56:47.188590 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180450    2553 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 17:56:47.188590 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180452    2553 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 17:56:47.188590 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180456    2553 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 17:56:47.188590 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180459    2553 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 17:56:47.188590 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180463    2553 flags.go:64] FLAG: --system-cgroups=""
Sep 17 17:56:47.188590 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180466    2553 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 17:56:47.188590 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180471    2553 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 17:56:47.188590 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180474    2553 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 17:56:47.188590 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180477    2553 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 17:56:47.188590 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180482    2553 flags.go:64] FLAG: --tls-min-version=""
Sep 17 17:56:47.188590 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180485    2553 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 17:56:47.188590 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180489    2553 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 17:56:47.188590 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180492    2553 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 17:56:47.188590 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180495    2553 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 17:56:47.189585 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180498    2553 flags.go:64] FLAG: --v="2"
Sep 17 17:56:47.189585 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180503    2553 flags.go:64] FLAG: --version="false"
Sep 17 17:56:47.189585 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180507    2553 flags.go:64] FLAG: --vmodule=""
Sep 17 17:56:47.189585 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180511    2553 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 17:56:47.189585 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.180514    2553 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 17:56:47.189585 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180681    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:56:47.189585 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180686    2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:56:47.189585 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180689    2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:56:47.189585 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180692    2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:56:47.189585 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180695    2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:56:47.189585 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180698    2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:56:47.189585 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180701    2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:56:47.189585 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180703    2553 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:56:47.189585 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180706    2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:56:47.189585 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180709    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:56:47.189585 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180712    2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:56:47.189585 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180715    2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:56:47.189585 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180717    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:56:47.189585 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180720    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:56:47.189585 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180724    2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:56:47.189585 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180726    2553 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:56:47.190156 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180729    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:56:47.190156 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180734    2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:56:47.190156 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180736    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:56:47.190156 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180739    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:56:47.190156 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180743    2553 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:56:47.190156 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180745    2553 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:56:47.190156 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180748    2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:56:47.190156 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180751    2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:56:47.190156 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180755    2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:56:47.190156 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180760    2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:56:47.190156 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180764    2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:56:47.190156 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180766    2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:56:47.190156 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180769    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:56:47.190156 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180772    2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:56:47.190156 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180775    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:56:47.190156 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180778    2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:56:47.190156 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180780    2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:56:47.190156 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180783    2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:56:47.190156 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180786    2553 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:56:47.190156 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180789    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:56:47.190688 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180791    2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:56:47.190688 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180794    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:56:47.190688 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180797    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:56:47.190688 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180799    2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:56:47.190688 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180802    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:56:47.190688 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180804    2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:56:47.190688 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180807    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:56:47.190688 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180810    2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:56:47.190688 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180813    2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:56:47.190688 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180815    2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:56:47.190688 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180818    2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:56:47.190688 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180821    2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:56:47.190688 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180823    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:56:47.190688 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180827    2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:56:47.190688 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180830    2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:56:47.190688 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180833    2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:56:47.190688 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180835    2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:56:47.190688 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180838    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:56:47.190688 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180841    2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:56:47.190688 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180843    2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:56:47.191177 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180848    2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:56:47.191177 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180851    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:56:47.191177 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180854    2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:56:47.191177 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180857    2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:56:47.191177 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180860    2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:56:47.191177 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180864    2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:56:47.191177 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180868    2553 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:56:47.191177 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180871    2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:56:47.191177 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180874    2553 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:56:47.191177 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180877    2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:56:47.191177 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180880    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:56:47.191177 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180883    2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:56:47.191177 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180886    2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:56:47.191177 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180888    2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:56:47.191177 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180891    2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:56:47.191177 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180894    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:56:47.191177 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180896    2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:56:47.191177 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180899    2553 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:56:47.191177 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180902    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:56:47.191655 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180904    2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:56:47.191655 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180907    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:56:47.191655 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180910    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:56:47.191655 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180912    2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:56:47.191655 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180916    2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:56:47.191655 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180920    2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:56:47.191655 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180925    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:56:47.191655 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180928    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:56:47.191655 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180931    2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:56:47.191655 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180934    2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:56:47.191655 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180936    2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:56:47.191655 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180939    2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:56:47.191655 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180941    2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:56:47.191655 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180945    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:56:47.191655 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180948    2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:56:47.191655 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180951    2553 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:56:47.191655 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180954    2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:56:47.191655 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180957    2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:56:47.191655 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180959    2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:56:47.191655 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180962    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:56:47.192147 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180964    2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:56:47.192147 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180967    2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:56:47.192147 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180970    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:56:47.192147 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.180972    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:56:47.192147 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.181496    2553 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:56:47.192147 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.190453    2553 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 17:56:47.192147 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.190472    2553 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 17:56:47.192147 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.190493    2553 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 17:56:47.192147 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.190503    2553 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 17:56:47.192147 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190564    2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:56:47.192147 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190569    2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:56:47.192147 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190572    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:56:47.192147 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190575    2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:56:47.192147 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190578    2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:56:47.192498 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190581    2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:56:47.192498 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190584    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:56:47.192498 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190587    2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:56:47.192498 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190590    2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:56:47.192498 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190594    2553 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:56:47.192498 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190597    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:56:47.192498 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190599    2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:56:47.192498 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190602    2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:56:47.192498 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190605    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:56:47.192498 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190607    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:56:47.192498 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190610    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:56:47.192498 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190613    2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:56:47.192498 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190630    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:56:47.192498 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190633    2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:56:47.192498 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190635    2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:56:47.192498 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190638    2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:56:47.192498 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190641    2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:56:47.192498 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190646    2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:56:47.192498 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190650    2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:56:47.192988 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190653    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:56:47.192988 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190656    2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:56:47.192988 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190659    2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:56:47.192988 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190663    2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:56:47.192988 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190666    2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:56:47.192988 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190670    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:56:47.192988 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190672    2553 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:56:47.192988 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190675    2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:56:47.192988 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190678    2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:56:47.192988 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190680    2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:56:47.192988 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190683    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:56:47.192988 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190686    2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:56:47.192988 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190689    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:56:47.192988 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190692    2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:56:47.192988 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190694    2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:56:47.192988 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190697    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:56:47.192988 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190700    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:56:47.192988 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190704    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:56:47.192988 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190706    2553 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:56:47.193511 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190709    2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:56:47.193511 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190712    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:56:47.193511 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190715    2553 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:56:47.193511 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190718    2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:56:47.193511 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190721    2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:56:47.193511 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190723    2553 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:56:47.193511 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190726    2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:56:47.193511 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190729    2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:56:47.193511 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190731    2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:56:47.193511 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190734    2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:56:47.193511 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190737    2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:56:47.193511 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190739    2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:56:47.193511 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190742    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:56:47.193511 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190744    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:56:47.193511 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190747    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:56:47.193511 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190750    2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:56:47.193511 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190752    2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:56:47.193511 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190755    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:56:47.193511 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190757    2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:56:47.193511 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190760    2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:56:47.194164 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190763    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:56:47.194164 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190766    2553 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:56:47.194164 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190769    2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:56:47.194164 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190771    2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:56:47.194164 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190774    2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:56:47.194164 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190776    2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:56:47.194164 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190779    2553 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:56:47.194164 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190782    2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:56:47.194164 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190784    2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:56:47.194164 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190787    2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:56:47.194164 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190790    2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:56:47.194164 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190793    2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:56:47.194164 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190795    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:56:47.194164 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190798    2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:56:47.194164 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190801    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:56:47.194164 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190803    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:56:47.194164 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190807    2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:56:47.194164 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190810    2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:56:47.194164 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190812    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:56:47.194164 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190815    2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:56:47.194675 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190818    2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:56:47.194675 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190820    2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:56:47.194675 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190823    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:56:47.194675 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190825    2553 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:56:47.194675 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190828    2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:56:47.194675 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190830    2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:56:47.194675 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190833    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:56:47.194675 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190836    2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:56:47.194675 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190839    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:56:47.194675 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190841    2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:56:47.194675 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190844    2553 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:56:47.194675 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190847    2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:56:47.194675 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190851    2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:56:47.194675 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190854    2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:56:47.194675 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190857    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:56:47.194675 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190860    2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:56:47.195113 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.190865    2553 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:56:47.195113 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190981    2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:56:47.195113 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190986    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:56:47.195113 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190989    2553 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:56:47.195113 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190992    2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:56:47.195113 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190994    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:56:47.195113 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.190997    2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:56:47.195113 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191000    2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:56:47.195113 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191003    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:56:47.195113 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191005    2553 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:56:47.195113 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191008    2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:56:47.195113 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191010    2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:56:47.195113 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191013    2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:56:47.195113 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191017    2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:56:47.195113 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191019    2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:56:47.195506 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191022    2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:56:47.195506 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191025    2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:56:47.195506 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191028    2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:56:47.195506 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191031    2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:56:47.195506 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191033    2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:56:47.195506 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191036    2553 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:56:47.195506 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191039    2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:56:47.195506 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191041    2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:56:47.195506 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191044    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:56:47.195506 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191046    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:56:47.195506 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191049    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:56:47.195506 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191052    2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:56:47.195506 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191055    2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:56:47.195506 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191057    2553 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:56:47.195506 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191060    2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:56:47.195506 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191063    2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:56:47.195506 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191065    2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:56:47.195506 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191068    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:56:47.195506 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191070    2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:56:47.195506 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191073    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:56:47.196000 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191075    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:56:47.196000 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191078    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:56:47.196000 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191081    2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:56:47.196000 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191084    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:56:47.196000 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191086    2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:56:47.196000 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191089    2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:56:47.196000 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191091    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:56:47.196000 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191094    2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:56:47.196000 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191098    2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:56:47.196000 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191102    2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:56:47.196000 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191105    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:56:47.196000 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191108    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:56:47.196000 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191111    2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:56:47.196000 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191114    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:56:47.196000 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191117    2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:56:47.196000 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191119    2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:56:47.196000 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191122    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:56:47.196000 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191124    2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:56:47.196000 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191127    2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:56:47.196497 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191129    2553 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:56:47.196497 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191132    2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:56:47.196497 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191135    2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:56:47.196497 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191137    2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:56:47.196497 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191140    2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:56:47.196497 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191143    2553 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:56:47.196497 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191145    2553 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:56:47.196497 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191148    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:56:47.196497 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191150    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:56:47.196497 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191154    2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:56:47.196497 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191158    2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:56:47.196497 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191160    2553 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:56:47.196497 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191163    2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:56:47.196497 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191166    2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:56:47.196497 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191168    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:56:47.196497 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191171    2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:56:47.196497 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191175    2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:56:47.196497 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191178    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:56:47.196497 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191182    2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:56:47.196967 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191185    2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:56:47.196967 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191188    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:56:47.196967 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191191    2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:56:47.196967 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191193    2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:56:47.196967 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191196    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:56:47.196967 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191199    2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:56:47.196967 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191203    2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:56:47.196967 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191206    2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:56:47.196967 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191209    2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:56:47.196967 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191211    2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:56:47.196967 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191214    2553 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:56:47.196967 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191216    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:56:47.196967 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191219    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:56:47.196967 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191222    2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:56:47.196967 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191224    2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:56:47.196967 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191227    2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:56:47.196967 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191230    2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:56:47.196967 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191232    2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:56:47.196967 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191235    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:56:47.196967 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191238    2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:56:47.197509 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191240    2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:56:47.197509 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191243    2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:56:47.197509 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191245    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:56:47.197509 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191248    2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:56:47.197509 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191251    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:56:47.197509 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191256    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:56:47.197509 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:47.191259    2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:56:47.197509 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.191264    2553 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:56:47.197509 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.191393    2553 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 17:56:47.197509 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.193632    2553 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 17:56:47.197509 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.194676    2553 server.go:1019] "Starting client certificate rotation"
Sep 17 17:56:47.197509 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.194773    2553 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 17:56:47.197509 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.194805    2553 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 17:56:47.198015 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.197283    2553 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 17:56:47.198665 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.198644    2553 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 17:56:47.203481 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.203463    2553 log.go:25] "Validated CRI v1 runtime API"
Sep 17 17:56:47.205063 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.205046    2553 log.go:25] "Validated CRI v1 image API"
Sep 17 17:56:47.206373 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.206358    2553 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 17:56:47.209565 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.209546    2553 fs.go:135] Filesystem UUIDs: map[21655467-0f0f-4de0-9c3b-2f4297c94145:/dev/nvme0n1p4 2e4eb121-65d0-470c-ab4e-e7793e421da3:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Sep 17 17:56:47.209612 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.209565    2553 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 17:56:47.214986 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.214614    2553 manager.go:216] Machine: {Timestamp:2026-09-17 17:56:47.213868974 +0000 UTC m=+1.909959651 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3094630 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2db98106b218fe1d573315b9fc83a7 SystemUUID:ec2db981-06b2-18fe-1d57-3315b9fc83a7 BootID:9b4ebaf6-c092-498f-95e0-8af60e0cb9cd Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:dd:3c:9d:13 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:dd:3c:9d:13 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:16:b9:6b:07:af:b0 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 17:56:47.214986 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.214973    2553 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 17:56:47.215130 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.215075    2553 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 17:56:47.216925 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.216895    2553 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 17:56:47.217095 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.216927    2553 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-95.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 17:56:47.217638 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.217614    2553 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 17:56:47.217674 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.217641    2553 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 17:56:47.217674 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.217648    2553 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 17:56:47.217674 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.217660    2553 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 17:56:47.218939 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.218929    2553 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 17:56:47.221603 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.221589    2553 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 17:56:47.221746 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.221736    2553 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 17:56:47.221783 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.221774    2553 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 17:56:47.221814 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.221786    2553 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 17:56:47.221814 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.221799    2553 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 17:56:47.221814 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.221808    2553 kubelet.go:398] "Adding apiserver pod source"
Sep 17 17:56:47.221920 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.221818    2553 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 17:56:47.222203 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.222182    2553 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 17:56:47.222203 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.222203    2553 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 17:56:47.229996 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.229972    2553 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 17:56:47.230120 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.230100    2553 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 17:56:47.232390 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.232376    2553 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 17:56:47.232433 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.232407    2553 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 17:56:47.232560 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.232550    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 17:56:47.232612 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.232565    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 17:56:47.232612 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.232572    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 17:56:47.232612 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.232577    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 17:56:47.232612 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.232586    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 17:56:47.232612 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.232591    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 17:56:47.233181 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.233172    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 17:56:47.233181 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.233181    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 17:56:47.233245 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.233188    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 17:56:47.233245 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.233194    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 17:56:47.233245 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.233203    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 17:56:47.233245 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.233212    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 17:56:47.233245 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.233237    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 17:56:47.233245 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.233242    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 17:56:47.240422 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.240402    2553 server.go:1268] "Started kubelet"
Sep 17 17:56:47.240534 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.240483    2553 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 17:56:47.240588 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.240538    2553 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 17:56:47.240645 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.240612    2553 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 17:56:47.240950 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.240922    2553 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 17:56:47.241455 ip-10-0-137-95 systemd[1]: Started Kubernetes Kubelet.
Sep 17 17:56:47.242260 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.242233    2553 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 17:56:47.244043 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.244024    2553 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-95.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 17:56:47.244341 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:47.244316    2553 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-95.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 17:56:47.244410 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:47.244318    2553 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 17:56:47.245473 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:47.245454    2553 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 17:56:47.248498 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.248480    2553 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 17:56:47.248587 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.248487    2553 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 17:56:47.248587 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:47.248576    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-95.ec2.internal\" not found"
Sep 17 17:56:47.249098 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.249082    2553 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 17:56:47.249207 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.249082    2553 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 17:56:47.249207 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.249209    2553 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 17:56:47.249402 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.249274    2553 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 17:56:47.249402 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.249283    2553 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 17:56:47.249402 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.249306    2553 factory.go:55] Registering systemd factory
Sep 17 17:56:47.249402 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.249320    2553 factory.go:223] Registration of the systemd container factory successfully
Sep 17 17:56:47.249556 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.249488    2553 factory.go:153] Registering CRI-O factory
Sep 17 17:56:47.249556 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.249496    2553 factory.go:223] Registration of the crio container factory successfully
Sep 17 17:56:47.249653 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.249567    2553 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 17:56:47.249653 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.249589    2553 factory.go:103] Registering Raw factory
Sep 17 17:56:47.249653 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.249606    2553 manager.go:1196] Started watching for new ooms in manager
Sep 17 17:56:47.250033 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.250021    2553 manager.go:349] Starting recovery of all containers
Sep 17 17:56:47.271186 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:47.271139    2553 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-95.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 17:56:47.271186 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:47.271148    2553 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 17:56:47.279809 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.279665    2553 manager.go:354] Recovery completed
Sep 17 17:56:47.284153 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.284137    2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:56:47.286049 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.286029    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-95.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:56:47.286148 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.286063    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-95.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:56:47.286148 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.286073    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-95.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:56:47.286705 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.286691    2553 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 17:56:47.286705 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.286699    2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:56:47.286705 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.286704    2553 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 17:56:47.286817 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.286720    2553 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 17:56:47.287006 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:47.285958    2553 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-95.ec2.internal.18d62d637c83a570  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-95.ec2.internal,UID:ip-10-0-137-95.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-95.ec2.internal,},FirstTimestamp:2026-09-17 17:56:47.240365424 +0000 UTC m=+1.936456099,LastTimestamp:2026-09-17 17:56:47.240365424 +0000 UTC m=+1.936456099,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-95.ec2.internal,}"
Sep 17 17:56:47.287456 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.287441    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-95.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:56:47.287527 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.287471    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-95.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:56:47.287527 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.287485    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-95.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:56:47.287980 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.287967    2553 policy_none.go:49] "None policy: Start"
Sep 17 17:56:47.288033 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.287983    2553 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 17:56:47.288033 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.287994    2553 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 17:56:47.289089 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.289077    2553 policy_none.go:47] "Start"
Sep 17 17:56:47.295408 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:47.295316    2553 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-95.ec2.internal.18d62d637f3cb690  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-95.ec2.internal,UID:ip-10-0-137-95.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-95.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-95.ec2.internal,},FirstTimestamp:2026-09-17 17:56:47.2860484 +0000 UTC m=+1.982139075,LastTimestamp:2026-09-17 17:56:47.2860484 +0000 UTC m=+1.982139075,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-95.ec2.internal,}"
Sep 17 17:56:47.298095 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:47.298012    2553 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-95.ec2.internal.18d62d637f3d0299  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-95.ec2.internal,UID:ip-10-0-137-95.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-137-95.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-137-95.ec2.internal,},FirstTimestamp:2026-09-17 17:56:47.286067865 +0000 UTC m=+1.982158539,LastTimestamp:2026-09-17 17:56:47.286067865 +0000 UTC m=+1.982158539,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-95.ec2.internal,}"
Sep 17 17:56:47.304990 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:47.304919    2553 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-95.ec2.internal.18d62d637f3d28b3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-95.ec2.internal,UID:ip-10-0-137-95.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-137-95.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-137-95.ec2.internal,},FirstTimestamp:2026-09-17 17:56:47.286077619 +0000 UTC m=+1.982168294,LastTimestamp:2026-09-17 17:56:47.286077619 +0000 UTC m=+1.982168294,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-95.ec2.internal,}"
Sep 17 17:56:47.316567 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:47.316490    2553 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-137-95.ec2.internal.18d62d637f3cb690\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-95.ec2.internal.18d62d637f3cb690  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-95.ec2.internal,UID:ip-10-0-137-95.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-95.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-95.ec2.internal,},FirstTimestamp:2026-09-17 17:56:47.2860484 +0000 UTC m=+1.982139075,LastTimestamp:2026-09-17 17:56:47.287455665 +0000 UTC m=+1.983546341,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-95.ec2.internal,}"
Sep 17 17:56:47.327879 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.327859    2553 manager.go:337] "Starting Device Plugin manager"
Sep 17 17:56:47.338311 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:47.327909    2553 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 17:56:47.338311 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.327920    2553 server.go:87] "Starting device plugin registration server"
Sep 17 17:56:47.338311 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.328212    2553 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 17:56:47.338311 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.328239    2553 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 17:56:47.338311 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.328327    2553 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 17:56:47.338311 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.328529    2553 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 17:56:47.338311 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.328539    2553 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 17:56:47.338311 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:47.329074    2553 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 17:56:47.338311 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:47.329152    2553 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-95.ec2.internal\" not found"
Sep 17 17:56:47.338311 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:47.329229    2553 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-137-95.ec2.internal.18d62d637f3d0299\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-95.ec2.internal.18d62d637f3d0299  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-95.ec2.internal,UID:ip-10-0-137-95.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-137-95.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-137-95.ec2.internal,},FirstTimestamp:2026-09-17 17:56:47.286067865 +0000 UTC m=+1.982158539,LastTimestamp:2026-09-17 17:56:47.287477808 +0000 UTC m=+1.983568486,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-95.ec2.internal,}"
Sep 17 17:56:47.340324 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:47.340250    2553 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-137-95.ec2.internal.18d62d637f3d28b3\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-95.ec2.internal.18d62d637f3d28b3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-95.ec2.internal,UID:ip-10-0-137-95.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-137-95.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-137-95.ec2.internal,},FirstTimestamp:2026-09-17 17:56:47.286077619 +0000 UTC m=+1.982168294,LastTimestamp:2026-09-17 17:56:47.287491128 +0000 UTC m=+1.983581803,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-95.ec2.internal,}"
Sep 17 17:56:47.349741 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:47.349663    2553 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-95.ec2.internal.18d62d6381e48c1d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-95.ec2.internal,UID:ip-10-0-137-95.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-137-95.ec2.internal,},FirstTimestamp:2026-09-17 17:56:47.330602013 +0000 UTC m=+2.026692674,LastTimestamp:2026-09-17 17:56:47.330602013 +0000 UTC m=+2.026692674,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-95.ec2.internal,}"
Sep 17 17:56:47.368778 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.368729    2553 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 17:56:47.370054 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.370033    2553 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 17:56:47.370119 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.370101    2553 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 17:56:47.370193 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.370176    2553 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 17:56:47.370246 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:47.370232    2553 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 17:56:47.383494 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:47.383464    2553 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass"
Sep 17 17:56:47.428742 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.428649    2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:56:47.429871 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.429855    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-95.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:56:47.429972 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.429888    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-95.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:56:47.429972 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.429902    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-95.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:56:47.429972 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.429939    2553 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-95.ec2.internal"
Sep 17 17:56:47.440096 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:47.439960    2553 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-137-95.ec2.internal.18d62d637f3cb690\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-95.ec2.internal.18d62d637f3cb690  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-95.ec2.internal,UID:ip-10-0-137-95.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-95.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-95.ec2.internal,},FirstTimestamp:2026-09-17 17:56:47.2860484 +0000 UTC m=+1.982139075,LastTimestamp:2026-09-17 17:56:47.429871584 +0000 UTC m=+2.125962259,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-95.ec2.internal,}"
Sep 17 17:56:47.449138 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:47.449110    2553 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-137-95.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-137-95.ec2.internal"
Sep 17 17:56:47.449273 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:47.449180    2553 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-137-95.ec2.internal.18d62d637f3d0299\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-95.ec2.internal.18d62d637f3d0299  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-95.ec2.internal,UID:ip-10-0-137-95.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-137-95.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-137-95.ec2.internal,},FirstTimestamp:2026-09-17 17:56:47.286067865 +0000 UTC m=+1.982158539,LastTimestamp:2026-09-17 17:56:47.429895256 +0000 UTC m=+2.125985935,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-95.ec2.internal,}"
Sep 17 17:56:47.458307 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:47.458204    2553 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-137-95.ec2.internal.18d62d637f3d28b3\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-95.ec2.internal.18d62d637f3d28b3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-95.ec2.internal,UID:ip-10-0-137-95.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-137-95.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-137-95.ec2.internal,},FirstTimestamp:2026-09-17 17:56:47.286077619 +0000 UTC m=+1.982168294,LastTimestamp:2026-09-17 17:56:47.429907547 +0000 UTC m=+2.125998223,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-95.ec2.internal,}"
Sep 17 17:56:47.471387 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.471345    2553 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-137-95.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-95.ec2.internal"]
Sep 17 17:56:47.471500 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.471448    2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:56:47.473754 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.473398    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-95.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:56:47.473754 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.473432    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-95.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:56:47.473754 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.473463    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-95.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:56:47.474612 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.474586    2553 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dm7ts"
Sep 17 17:56:47.475061 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.475044    2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:56:47.475173 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.475157    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-95.ec2.internal"
Sep 17 17:56:47.475210 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.475191    2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:56:47.475825 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.475808    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-95.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:56:47.475899 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.475859    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-95.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:56:47.475899 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.475869    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-95.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:56:47.475899 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.475885    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-95.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:56:47.475899 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.475894    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-95.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:56:47.476042 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.475905    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-95.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:56:47.477046 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.477031    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-95.ec2.internal"
Sep 17 17:56:47.477132 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.477055    2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:56:47.477829 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.477807    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-95.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:56:47.477829 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.477835    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-95.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:56:47.478000 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.477852    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-95.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:56:47.479217 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:47.479200    2553 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-95.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="400ms"
Sep 17 17:56:47.479298 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:47.479168    2553 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-137-95.ec2.internal.18d62d637f3cb690\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-95.ec2.internal.18d62d637f3cb690  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-95.ec2.internal,UID:ip-10-0-137-95.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-95.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-95.ec2.internal,},FirstTimestamp:2026-09-17 17:56:47.2860484 +0000 UTC m=+1.982139075,LastTimestamp:2026-09-17 17:56:47.473420111 +0000 UTC m=+2.169510787,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-95.ec2.internal,}"
Sep 17 17:56:47.482586 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.482569    2553 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dm7ts"
Sep 17 17:56:47.483940 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:47.483877    2553 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-137-95.ec2.internal.18d62d637f3d0299\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-95.ec2.internal.18d62d637f3d0299  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-95.ec2.internal,UID:ip-10-0-137-95.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-137-95.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-137-95.ec2.internal,},FirstTimestamp:2026-09-17 17:56:47.286067865 +0000 UTC m=+1.982158539,LastTimestamp:2026-09-17 17:56:47.473454474 +0000 UTC m=+2.169545150,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-95.ec2.internal,}"
Sep 17 17:56:47.528380 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:47.511692    2553 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-95.ec2.internal\" not found" node="ip-10-0-137-95.ec2.internal"
Sep 17 17:56:47.533892 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:47.533874    2553 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-95.ec2.internal\" not found" node="ip-10-0-137-95.ec2.internal"
Sep 17 17:56:47.649188 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.649158    2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:56:47.650239 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.650223    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-95.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:56:47.650311 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.650254    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-95.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:56:47.650311 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.650264    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-95.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:56:47.650311 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.650288    2553 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-95.ec2.internal"
Sep 17 17:56:47.650408 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.650347    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/01e1e8be5fa1c8721ccc54011f518b1a-config\") pod \"kube-apiserver-proxy-ip-10-0-137-95.ec2.internal\" (UID: \"01e1e8be5fa1c8721ccc54011f518b1a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-95.ec2.internal"
Sep 17 17:56:47.650408 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.650389    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7c35fbfd6aae0a9c81a0ec1cdc209b72-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-95.ec2.internal\" (UID: \"7c35fbfd6aae0a9c81a0ec1cdc209b72\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-95.ec2.internal"
Sep 17 17:56:47.650476 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.650421    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c35fbfd6aae0a9c81a0ec1cdc209b72-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-95.ec2.internal\" (UID: \"7c35fbfd6aae0a9c81a0ec1cdc209b72\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-95.ec2.internal"
Sep 17 17:56:47.660734 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.660714    2553 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-95.ec2.internal"
Sep 17 17:56:47.660812 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:47.660742    2553 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-95.ec2.internal\": node \"ip-10-0-137-95.ec2.internal\" not found"
Sep 17 17:56:47.672690 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:47.672661    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-95.ec2.internal\" not found"
Sep 17 17:56:47.751506 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.751442    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/01e1e8be5fa1c8721ccc54011f518b1a-config\") pod \"kube-apiserver-proxy-ip-10-0-137-95.ec2.internal\" (UID: \"01e1e8be5fa1c8721ccc54011f518b1a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-95.ec2.internal"
Sep 17 17:56:47.751506 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.751474    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7c35fbfd6aae0a9c81a0ec1cdc209b72-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-95.ec2.internal\" (UID: \"7c35fbfd6aae0a9c81a0ec1cdc209b72\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-95.ec2.internal"
Sep 17 17:56:47.751506 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.751493    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c35fbfd6aae0a9c81a0ec1cdc209b72-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-95.ec2.internal\" (UID: \"7c35fbfd6aae0a9c81a0ec1cdc209b72\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-95.ec2.internal"
Sep 17 17:56:47.751686 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.751558    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/01e1e8be5fa1c8721ccc54011f518b1a-config\") pod \"kube-apiserver-proxy-ip-10-0-137-95.ec2.internal\" (UID: \"01e1e8be5fa1c8721ccc54011f518b1a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-95.ec2.internal"
Sep 17 17:56:47.751686 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.751607    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c35fbfd6aae0a9c81a0ec1cdc209b72-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-95.ec2.internal\" (UID: \"7c35fbfd6aae0a9c81a0ec1cdc209b72\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-95.ec2.internal"
Sep 17 17:56:47.751686 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.751663    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7c35fbfd6aae0a9c81a0ec1cdc209b72-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-95.ec2.internal\" (UID: \"7c35fbfd6aae0a9c81a0ec1cdc209b72\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-95.ec2.internal"
Sep 17 17:56:47.773600 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:47.773552    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-95.ec2.internal\" not found"
Sep 17 17:56:47.831775 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.831751    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-95.ec2.internal"
Sep 17 17:56:47.838316 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:47.838297    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-95.ec2.internal"
Sep 17 17:56:47.874336 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:47.874301    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-95.ec2.internal\" not found"
Sep 17 17:56:47.975422 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:47.975378    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-95.ec2.internal\" not found"
Sep 17 17:56:48.075989 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:48.075910    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-95.ec2.internal\" not found"
Sep 17 17:56:48.077385 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.077367    2553 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:56:48.149426 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.149389    2553 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-95.ec2.internal"
Sep 17 17:56:48.162880 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.162851    2553 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 17:56:48.164570 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.164556    2553 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-95.ec2.internal"
Sep 17 17:56:48.181236 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.181214    2553 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 17:56:48.194396 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.194371    2553 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 17:56:48.221997 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.221964    2553 apiserver.go:52] "Watching apiserver"
Sep 17 17:56:48.240337 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.240308    2553 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 17:56:48.240704 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.240683    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-4fdj7","openshift-image-registry/node-ca-6pxfw","openshift-multus/multus-additional-cni-plugins-bmjrp","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-95.ec2.internal","openshift-multus/multus-tjrts","openshift-multus/network-metrics-daemon-5gdn2","openshift-network-diagnostics/network-check-target-lkn4v","openshift-network-operator/iptables-alerter-p6z75","openshift-ovn-kubernetes/ovnkube-node-btmqz","kube-system/global-pull-secret-syncer-xr8hx","kube-system/konnectivity-agent-jz66f","kube-system/kube-apiserver-proxy-ip-10-0-137-95.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gg9tp","openshift-cluster-node-tuning-operator/tuned-t4pkh"]
Sep 17 17:56:48.242147 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.242131    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xr8hx"
Sep 17 17:56:48.242216 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:48.242198    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xr8hx" podUID="45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb"
Sep 17 17:56:48.243420 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.243404    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gg9tp"
Sep 17 17:56:48.244668 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.244647    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bmjrp"
Sep 17 17:56:48.245803 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.245783    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-t4pkh"
Sep 17 17:56:48.246841 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.246827    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tjrts"
Sep 17 17:56:48.247880 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.247866    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gdn2"
Sep 17 17:56:48.247946 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:48.247929    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5gdn2" podUID="dfc1cb73-d5de-4902-b16c-5744271855c5"
Sep 17 17:56:48.248183 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.248168    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 17:56:48.248229 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.248190    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 17:56:48.248229 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.248225    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 17:56:48.248455 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.248441    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-tkw8p\""
Sep 17 17:56:48.248604 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.248588    2553 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 17:56:48.249115 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.249099    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-btmqz"
Sep 17 17:56:48.250397 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.250378    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 17:56:48.250502 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.250404    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 17:56:48.250502 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.250488    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lkn4v"
Sep 17 17:56:48.250575 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:48.250555    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lkn4v" podUID="cb953471-4308-48e7-9316-6b0bdf2b8738"
Sep 17 17:56:48.251602 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.251587    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6pxfw"
Sep 17 17:56:48.252691 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.252676    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4fdj7"
Sep 17 17:56:48.254127 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.254107    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b7c581ed-7ab3-4a26-930c-34fdf021affd-etc-kubernetes\") pod \"tuned-t4pkh\" (UID: \"b7c581ed-7ab3-4a26-930c-34fdf021affd\") " pod="openshift-cluster-node-tuning-operator/tuned-t4pkh"
Sep 17 17:56:48.254201 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.254137    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/622ee5be-fb50-42b6-ae23-42900b7c2d98-host-cni-bin\") pod \"ovnkube-node-btmqz\" (UID: \"622ee5be-fb50-42b6-ae23-42900b7c2d98\") " pod="openshift-ovn-kubernetes/ovnkube-node-btmqz"
Sep 17 17:56:48.254201 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.254153    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/622ee5be-fb50-42b6-ae23-42900b7c2d98-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-btmqz\" (UID: \"622ee5be-fb50-42b6-ae23-42900b7c2d98\") " pod="openshift-ovn-kubernetes/ovnkube-node-btmqz"
Sep 17 17:56:48.254201 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.254169    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/622ee5be-fb50-42b6-ae23-42900b7c2d98-env-overrides\") pod \"ovnkube-node-btmqz\" (UID: \"622ee5be-fb50-42b6-ae23-42900b7c2d98\") " pod="openshift-ovn-kubernetes/ovnkube-node-btmqz"
Sep 17 17:56:48.254201 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.254184    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/855a7f82-274d-4b6d-bcde-0c6f4c19016b-etc-kubernetes\") pod \"multus-tjrts\" (UID: \"855a7f82-274d-4b6d-bcde-0c6f4c19016b\") " pod="openshift-multus/multus-tjrts"
Sep 17 17:56:48.254339 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.254206    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b7c581ed-7ab3-4a26-930c-34fdf021affd-etc-tuned\") pod \"tuned-t4pkh\" (UID: \"b7c581ed-7ab3-4a26-930c-34fdf021affd\") " pod="openshift-cluster-node-tuning-operator/tuned-t4pkh"
Sep 17 17:56:48.254339 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.254228    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/622ee5be-fb50-42b6-ae23-42900b7c2d98-host-run-netns\") pod \"ovnkube-node-btmqz\" (UID: \"622ee5be-fb50-42b6-ae23-42900b7c2d98\") " pod="openshift-ovn-kubernetes/ovnkube-node-btmqz"
Sep 17 17:56:48.254339 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.254242    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb-original-pull-secret\") pod \"global-pull-secret-syncer-xr8hx\" (UID: \"45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb\") " pod="kube-system/global-pull-secret-syncer-xr8hx"
Sep 17 17:56:48.254339 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.254257    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e137835-2bdc-4b32-aa6b-64635bdc26d7-system-cni-dir\") pod \"multus-additional-cni-plugins-bmjrp\" (UID: \"4e137835-2bdc-4b32-aa6b-64635bdc26d7\") " pod="openshift-multus/multus-additional-cni-plugins-bmjrp"
Sep 17 17:56:48.254339 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.254277    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4e137835-2bdc-4b32-aa6b-64635bdc26d7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bmjrp\" (UID: \"4e137835-2bdc-4b32-aa6b-64635bdc26d7\") " pod="openshift-multus/multus-additional-cni-plugins-bmjrp"
Sep 17 17:56:48.254339 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.254311    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6skh2\" (UniqueName: \"kubernetes.io/projected/b7c581ed-7ab3-4a26-930c-34fdf021affd-kube-api-access-6skh2\") pod \"tuned-t4pkh\" (UID: \"b7c581ed-7ab3-4a26-930c-34fdf021affd\") " pod="openshift-cluster-node-tuning-operator/tuned-t4pkh"
Sep 17 17:56:48.254339 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.254337    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/622ee5be-fb50-42b6-ae23-42900b7c2d98-host-cni-netd\") pod \"ovnkube-node-btmqz\" (UID: \"622ee5be-fb50-42b6-ae23-42900b7c2d98\") " pod="openshift-ovn-kubernetes/ovnkube-node-btmqz"
Sep 17 17:56:48.254673 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.254355    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmv25\" (UniqueName: \"kubernetes.io/projected/dfc1cb73-d5de-4902-b16c-5744271855c5-kube-api-access-pmv25\") pod \"network-metrics-daemon-5gdn2\" (UID: \"dfc1cb73-d5de-4902-b16c-5744271855c5\") " pod="openshift-multus/network-metrics-daemon-5gdn2"
Sep 17 17:56:48.254673 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.254372    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3f19101a-6f40-4457-8f18-9c7dc294323b-socket-dir\") pod \"aws-ebs-csi-driver-node-gg9tp\" (UID: \"3f19101a-6f40-4457-8f18-9c7dc294323b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gg9tp"
Sep 17 17:56:48.254673 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.254393    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b7c581ed-7ab3-4a26-930c-34fdf021affd-lib-modules\") pod \"tuned-t4pkh\" (UID: \"b7c581ed-7ab3-4a26-930c-34fdf021affd\") " pod="openshift-cluster-node-tuning-operator/tuned-t4pkh"
Sep 17 17:56:48.254673 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.254406    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b7c581ed-7ab3-4a26-930c-34fdf021affd-tmp\") pod \"tuned-t4pkh\" (UID: \"b7c581ed-7ab3-4a26-930c-34fdf021affd\") " pod="openshift-cluster-node-tuning-operator/tuned-t4pkh"
Sep 17 17:56:48.254673 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.254433    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/622ee5be-fb50-42b6-ae23-42900b7c2d98-log-socket\") pod \"ovnkube-node-btmqz\" (UID: \"622ee5be-fb50-42b6-ae23-42900b7c2d98\") " pod="openshift-ovn-kubernetes/ovnkube-node-btmqz"
Sep 17 17:56:48.254673 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.254469    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4e137835-2bdc-4b32-aa6b-64635bdc26d7-cni-binary-copy\") pod \"multus-additional-cni-plugins-bmjrp\" (UID: \"4e137835-2bdc-4b32-aa6b-64635bdc26d7\") " pod="openshift-multus/multus-additional-cni-plugins-bmjrp"
Sep 17 17:56:48.254673 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.254494    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4e137835-2bdc-4b32-aa6b-64635bdc26d7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bmjrp\" (UID: \"4e137835-2bdc-4b32-aa6b-64635bdc26d7\") " pod="openshift-multus/multus-additional-cni-plugins-bmjrp"
Sep 17 17:56:48.254673 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.254514    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jz66f"
Sep 17 17:56:48.254673 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.254525    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/855a7f82-274d-4b6d-bcde-0c6f4c19016b-system-cni-dir\") pod \"multus-tjrts\" (UID: \"855a7f82-274d-4b6d-bcde-0c6f4c19016b\") " pod="openshift-multus/multus-tjrts"
Sep 17 17:56:48.254673 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.254554    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/855a7f82-274d-4b6d-bcde-0c6f4c19016b-multus-socket-dir-parent\") pod \"multus-tjrts\" (UID: \"855a7f82-274d-4b6d-bcde-0c6f4c19016b\") " pod="openshift-multus/multus-tjrts"
Sep 17 17:56:48.254673 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.254575    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/855a7f82-274d-4b6d-bcde-0c6f4c19016b-host-var-lib-kubelet\") pod \"multus-tjrts\" (UID: \"855a7f82-274d-4b6d-bcde-0c6f4c19016b\") " pod="openshift-multus/multus-tjrts"
Sep 17 17:56:48.254673 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.254598    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g74cc\" (UniqueName: \"kubernetes.io/projected/3f19101a-6f40-4457-8f18-9c7dc294323b-kube-api-access-g74cc\") pod \"aws-ebs-csi-driver-node-gg9tp\" (UID: \"3f19101a-6f40-4457-8f18-9c7dc294323b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gg9tp"
Sep 17 17:56:48.254673 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.254636    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b7c581ed-7ab3-4a26-930c-34fdf021affd-etc-systemd\") pod \"tuned-t4pkh\" (UID: \"b7c581ed-7ab3-4a26-930c-34fdf021affd\") " pod="openshift-cluster-node-tuning-operator/tuned-t4pkh"
Sep 17 17:56:48.254673 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.254660    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/622ee5be-fb50-42b6-ae23-42900b7c2d98-ovnkube-script-lib\") pod \"ovnkube-node-btmqz\" (UID: \"622ee5be-fb50-42b6-ae23-42900b7c2d98\") " pod="openshift-ovn-kubernetes/ovnkube-node-btmqz"
Sep 17 17:56:48.255287 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.254695    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb-kubelet-config\") pod \"global-pull-secret-syncer-xr8hx\" (UID: \"45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb\") " pod="kube-system/global-pull-secret-syncer-xr8hx"
Sep 17 17:56:48.255287 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.254753    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfc1cb73-d5de-4902-b16c-5744271855c5-metrics-certs\") pod \"network-metrics-daemon-5gdn2\" (UID: \"dfc1cb73-d5de-4902-b16c-5744271855c5\") " pod="openshift-multus/network-metrics-daemon-5gdn2"
Sep 17 17:56:48.255287 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.254776    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b7c581ed-7ab3-4a26-930c-34fdf021affd-run\") pod \"tuned-t4pkh\" (UID: \"b7c581ed-7ab3-4a26-930c-34fdf021affd\") " pod="openshift-cluster-node-tuning-operator/tuned-t4pkh"
Sep 17 17:56:48.255287 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.254792    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/622ee5be-fb50-42b6-ae23-42900b7c2d98-ovn-node-metrics-cert\") pod \"ovnkube-node-btmqz\" (UID: \"622ee5be-fb50-42b6-ae23-42900b7c2d98\") " pod="openshift-ovn-kubernetes/ovnkube-node-btmqz"
Sep 17 17:56:48.255287 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.254825    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b7c581ed-7ab3-4a26-930c-34fdf021affd-etc-sysconfig\") pod \"tuned-t4pkh\" (UID: \"b7c581ed-7ab3-4a26-930c-34fdf021affd\") " pod="openshift-cluster-node-tuning-operator/tuned-t4pkh"
Sep 17 17:56:48.255287 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.254843    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/622ee5be-fb50-42b6-ae23-42900b7c2d98-run-openvswitch\") pod \"ovnkube-node-btmqz\" (UID: \"622ee5be-fb50-42b6-ae23-42900b7c2d98\") " pod="openshift-ovn-kubernetes/ovnkube-node-btmqz"
Sep 17 17:56:48.255287 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.254858    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/622ee5be-fb50-42b6-ae23-42900b7c2d98-run-ovn\") pod \"ovnkube-node-btmqz\" (UID: \"622ee5be-fb50-42b6-ae23-42900b7c2d98\") " pod="openshift-ovn-kubernetes/ovnkube-node-btmqz"
Sep 17 17:56:48.255287 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.254872    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/855a7f82-274d-4b6d-bcde-0c6f4c19016b-multus-cni-dir\") pod \"multus-tjrts\" (UID: \"855a7f82-274d-4b6d-bcde-0c6f4c19016b\") " pod="openshift-multus/multus-tjrts"
Sep 17 17:56:48.255287 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.254885    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b7c581ed-7ab3-4a26-930c-34fdf021affd-etc-modprobe-d\") pod \"tuned-t4pkh\" (UID: \"b7c581ed-7ab3-4a26-930c-34fdf021affd\") " pod="openshift-cluster-node-tuning-operator/tuned-t4pkh"
Sep 17 17:56:48.255287 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.254899    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b7c581ed-7ab3-4a26-930c-34fdf021affd-etc-sysctl-d\") pod \"tuned-t4pkh\" (UID: \"b7c581ed-7ab3-4a26-930c-34fdf021affd\") " pod="openshift-cluster-node-tuning-operator/tuned-t4pkh"
Sep 17 17:56:48.255287 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.254912    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b7c581ed-7ab3-4a26-930c-34fdf021affd-host\") pod \"tuned-t4pkh\" (UID: \"b7c581ed-7ab3-4a26-930c-34fdf021affd\") " pod="openshift-cluster-node-tuning-operator/tuned-t4pkh"
Sep 17 17:56:48.255287 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.254944    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4e137835-2bdc-4b32-aa6b-64635bdc26d7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bmjrp\" (UID: \"4e137835-2bdc-4b32-aa6b-64635bdc26d7\") " pod="openshift-multus/multus-additional-cni-plugins-bmjrp"
Sep 17 17:56:48.255287 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.254966    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3f19101a-6f40-4457-8f18-9c7dc294323b-etc-selinux\") pod \"aws-ebs-csi-driver-node-gg9tp\" (UID: \"3f19101a-6f40-4457-8f18-9c7dc294323b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gg9tp"
Sep 17 17:56:48.255287 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.254980    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/622ee5be-fb50-42b6-ae23-42900b7c2d98-var-lib-openvswitch\") pod \"ovnkube-node-btmqz\" (UID: \"622ee5be-fb50-42b6-ae23-42900b7c2d98\") " pod="openshift-ovn-kubernetes/ovnkube-node-btmqz"
Sep 17 17:56:48.255287 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.254996    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/855a7f82-274d-4b6d-bcde-0c6f4c19016b-hostroot\") pod \"multus-tjrts\" (UID: \"855a7f82-274d-4b6d-bcde-0c6f4c19016b\") " pod="openshift-multus/multus-tjrts"
Sep 17 17:56:48.255287 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.255015    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3f19101a-6f40-4457-8f18-9c7dc294323b-device-dir\") pod \"aws-ebs-csi-driver-node-gg9tp\" (UID: \"3f19101a-6f40-4457-8f18-9c7dc294323b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gg9tp"
Sep 17 17:56:48.255915 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.255052    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7c581ed-7ab3-4a26-930c-34fdf021affd-var-lib-kubelet\") pod \"tuned-t4pkh\" (UID: \"b7c581ed-7ab3-4a26-930c-34fdf021affd\") " pod="openshift-cluster-node-tuning-operator/tuned-t4pkh"
Sep 17 17:56:48.255915 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.255089    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/622ee5be-fb50-42b6-ae23-42900b7c2d98-systemd-units\") pod \"ovnkube-node-btmqz\" (UID: \"622ee5be-fb50-42b6-ae23-42900b7c2d98\") " pod="openshift-ovn-kubernetes/ovnkube-node-btmqz"
Sep 17 17:56:48.255915 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.255117    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6s8w\" (UniqueName: \"kubernetes.io/projected/622ee5be-fb50-42b6-ae23-42900b7c2d98-kube-api-access-c6s8w\") pod \"ovnkube-node-btmqz\" (UID: \"622ee5be-fb50-42b6-ae23-42900b7c2d98\") " pod="openshift-ovn-kubernetes/ovnkube-node-btmqz"
Sep 17 17:56:48.255915 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.255136    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/855a7f82-274d-4b6d-bcde-0c6f4c19016b-cnibin\") pod \"multus-tjrts\" (UID: \"855a7f82-274d-4b6d-bcde-0c6f4c19016b\") " pod="openshift-multus/multus-tjrts"
Sep 17 17:56:48.255915 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.255153    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/855a7f82-274d-4b6d-bcde-0c6f4c19016b-host-run-netns\") pod \"multus-tjrts\" (UID: \"855a7f82-274d-4b6d-bcde-0c6f4c19016b\") " pod="openshift-multus/multus-tjrts"
Sep 17 17:56:48.255915 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.255176    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd8gk\" (UniqueName: \"kubernetes.io/projected/855a7f82-274d-4b6d-bcde-0c6f4c19016b-kube-api-access-jd8gk\") pod \"multus-tjrts\" (UID: \"855a7f82-274d-4b6d-bcde-0c6f4c19016b\") " pod="openshift-multus/multus-tjrts"
Sep 17 17:56:48.255915 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.255200    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/855a7f82-274d-4b6d-bcde-0c6f4c19016b-cni-binary-copy\") pod \"multus-tjrts\" (UID: \"855a7f82-274d-4b6d-bcde-0c6f4c19016b\") " pod="openshift-multus/multus-tjrts"
Sep 17 17:56:48.255915 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.255521    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/855a7f82-274d-4b6d-bcde-0c6f4c19016b-host-run-k8s-cni-cncf-io\") pod \"multus-tjrts\" (UID: \"855a7f82-274d-4b6d-bcde-0c6f4c19016b\") " pod="openshift-multus/multus-tjrts"
Sep 17 17:56:48.255915 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.255556    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3f19101a-6f40-4457-8f18-9c7dc294323b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gg9tp\" (UID: \"3f19101a-6f40-4457-8f18-9c7dc294323b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gg9tp"
Sep 17 17:56:48.255915 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.255577    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3f19101a-6f40-4457-8f18-9c7dc294323b-registration-dir\") pod \"aws-ebs-csi-driver-node-gg9tp\" (UID: \"3f19101a-6f40-4457-8f18-9c7dc294323b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gg9tp"
Sep 17 17:56:48.255915 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.255614    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b7c581ed-7ab3-4a26-930c-34fdf021affd-etc-sysctl-conf\") pod \"tuned-t4pkh\" (UID: \"b7c581ed-7ab3-4a26-930c-34fdf021affd\") " pod="openshift-cluster-node-tuning-operator/tuned-t4pkh"
Sep 17 17:56:48.256437 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.255972    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/855a7f82-274d-4b6d-bcde-0c6f4c19016b-host-var-lib-cni-bin\") pod \"multus-tjrts\" (UID: \"855a7f82-274d-4b6d-bcde-0c6f4c19016b\") " pod="openshift-multus/multus-tjrts"
Sep 17 17:56:48.256437 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.255997    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/855a7f82-274d-4b6d-bcde-0c6f4c19016b-multus-daemon-config\") pod \"multus-tjrts\" (UID: \"855a7f82-274d-4b6d-bcde-0c6f4c19016b\") " pod="openshift-multus/multus-tjrts"
Sep 17 17:56:48.256437 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.256023    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/855a7f82-274d-4b6d-bcde-0c6f4c19016b-host-run-multus-certs\") pod \"multus-tjrts\" (UID: \"855a7f82-274d-4b6d-bcde-0c6f4c19016b\") " pod="openshift-multus/multus-tjrts"
Sep 17 17:56:48.256437 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.256048    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3f19101a-6f40-4457-8f18-9c7dc294323b-sys-fs\") pod \"aws-ebs-csi-driver-node-gg9tp\" (UID: \"3f19101a-6f40-4457-8f18-9c7dc294323b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gg9tp"
Sep 17 17:56:48.256437 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.256066    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b7c581ed-7ab3-4a26-930c-34fdf021affd-sys\") pod \"tuned-t4pkh\" (UID: \"b7c581ed-7ab3-4a26-930c-34fdf021affd\") " pod="openshift-cluster-node-tuning-operator/tuned-t4pkh"
Sep 17 17:56:48.256437 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.256097    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/622ee5be-fb50-42b6-ae23-42900b7c2d98-host-kubelet\") pod \"ovnkube-node-btmqz\" (UID: \"622ee5be-fb50-42b6-ae23-42900b7c2d98\") " pod="openshift-ovn-kubernetes/ovnkube-node-btmqz"
Sep 17 17:56:48.256437 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.256326    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-p6z75"
Sep 17 17:56:48.256763 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.256429    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/622ee5be-fb50-42b6-ae23-42900b7c2d98-host-slash\") pod \"ovnkube-node-btmqz\" (UID: \"622ee5be-fb50-42b6-ae23-42900b7c2d98\") " pod="openshift-ovn-kubernetes/ovnkube-node-btmqz"
Sep 17 17:56:48.256763 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.256495    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4e137835-2bdc-4b32-aa6b-64635bdc26d7-cnibin\") pod \"multus-additional-cni-plugins-bmjrp\" (UID: \"4e137835-2bdc-4b32-aa6b-64635bdc26d7\") " pod="openshift-multus/multus-additional-cni-plugins-bmjrp"
Sep 17 17:56:48.256763 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.256543    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/622ee5be-fb50-42b6-ae23-42900b7c2d98-run-systemd\") pod \"ovnkube-node-btmqz\" (UID: \"622ee5be-fb50-42b6-ae23-42900b7c2d98\") " pod="openshift-ovn-kubernetes/ovnkube-node-btmqz"
Sep 17 17:56:48.256763 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.256570    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/622ee5be-fb50-42b6-ae23-42900b7c2d98-etc-openvswitch\") pod \"ovnkube-node-btmqz\" (UID: \"622ee5be-fb50-42b6-ae23-42900b7c2d98\") " pod="openshift-ovn-kubernetes/ovnkube-node-btmqz"
Sep 17 17:56:48.257027 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.257004    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/622ee5be-fb50-42b6-ae23-42900b7c2d98-node-log\") pod \"ovnkube-node-btmqz\" (UID: \"622ee5be-fb50-42b6-ae23-42900b7c2d98\") " pod="openshift-ovn-kubernetes/ovnkube-node-btmqz"
Sep 17 17:56:48.257116 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.257095    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/622ee5be-fb50-42b6-ae23-42900b7c2d98-host-run-ovn-kubernetes\") pod \"ovnkube-node-btmqz\" (UID: \"622ee5be-fb50-42b6-ae23-42900b7c2d98\") " pod="openshift-ovn-kubernetes/ovnkube-node-btmqz"
Sep 17 17:56:48.257170 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.257144    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/622ee5be-fb50-42b6-ae23-42900b7c2d98-ovnkube-config\") pod \"ovnkube-node-btmqz\" (UID: \"622ee5be-fb50-42b6-ae23-42900b7c2d98\") " pod="openshift-ovn-kubernetes/ovnkube-node-btmqz"
Sep 17 17:56:48.257407 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.257233    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb-dbus\") pod \"global-pull-secret-syncer-xr8hx\" (UID: \"45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb\") " pod="kube-system/global-pull-secret-syncer-xr8hx"
Sep 17 17:56:48.257470 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.257424    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9msrf\" (UniqueName: \"kubernetes.io/projected/4e137835-2bdc-4b32-aa6b-64635bdc26d7-kube-api-access-9msrf\") pod \"multus-additional-cni-plugins-bmjrp\" (UID: \"4e137835-2bdc-4b32-aa6b-64635bdc26d7\") " pod="openshift-multus/multus-additional-cni-plugins-bmjrp"
Sep 17 17:56:48.257470 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.257456    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/855a7f82-274d-4b6d-bcde-0c6f4c19016b-host-var-lib-cni-multus\") pod \"multus-tjrts\" (UID: \"855a7f82-274d-4b6d-bcde-0c6f4c19016b\") " pod="openshift-multus/multus-tjrts"
Sep 17 17:56:48.257557 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.257486    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/855a7f82-274d-4b6d-bcde-0c6f4c19016b-multus-conf-dir\") pod \"multus-tjrts\" (UID: \"855a7f82-274d-4b6d-bcde-0c6f4c19016b\") " pod="openshift-multus/multus-tjrts"
Sep 17 17:56:48.257557 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.257516    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4p88\" (UniqueName: \"kubernetes.io/projected/cb953471-4308-48e7-9316-6b0bdf2b8738-kube-api-access-x4p88\") pod \"network-check-target-lkn4v\" (UID: \"cb953471-4308-48e7-9316-6b0bdf2b8738\") " pod="openshift-network-diagnostics/network-check-target-lkn4v"
Sep 17 17:56:48.258926 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.258852    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 17:56:48.258926 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.258886    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 17:56:48.259512 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.259497    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 17:56:48.259591 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.259575    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 17:56:48.264268 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.264247    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 17:56:48.264268 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.264266    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 17:56:48.264610 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.264593    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:56:48.264674 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.264593    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 17:56:48.265648 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.265634    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 17:56:48.265716 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.265635    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 17:56:48.265716 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.265636    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 17:56:48.274353 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.274327    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 17:56:48.274474 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.274459    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 17:56:48.274698 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.274684    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:56:48.274746 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.274723    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-kvrll\""
Sep 17 17:56:48.274999 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.274964    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 17:56:48.274999 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.274994    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 17:56:48.275073 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.275061    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bsz4n\""
Sep 17 17:56:48.275229 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.275210    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sg25s\""
Sep 17 17:56:48.275834 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.275812    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 17:56:48.275925 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.275885    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 17:56:48.276377 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.276359    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-x47xl\""
Sep 17 17:56:48.276467 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.276377    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-wlctl\""
Sep 17 17:56:48.276467 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.276419    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 17:56:48.276827 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.276813    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 17:56:48.276827 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.276823    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hl8j9\""
Sep 17 17:56:48.277081 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.277066    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6cxcg\""
Sep 17 17:56:48.277449 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.277434    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 17:56:48.277754 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.277738    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 17:56:48.278785 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.278760    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-nd77d\""
Sep 17 17:56:48.290727 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.290693    2553 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 17:56:48.322267 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.322241    2553 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-dl269"
Sep 17 17:56:48.341962 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.341938    2553 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-dl269"
Sep 17 17:56:48.349994 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.349965    2553 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 17:56:48.358499 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.358467    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b7c581ed-7ab3-4a26-930c-34fdf021affd-lib-modules\") pod \"tuned-t4pkh\" (UID: \"b7c581ed-7ab3-4a26-930c-34fdf021affd\") " pod="openshift-cluster-node-tuning-operator/tuned-t4pkh"
Sep 17 17:56:48.358499 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.358497    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b7c581ed-7ab3-4a26-930c-34fdf021affd-tmp\") pod \"tuned-t4pkh\" (UID: \"b7c581ed-7ab3-4a26-930c-34fdf021affd\") " pod="openshift-cluster-node-tuning-operator/tuned-t4pkh"
Sep 17 17:56:48.358753 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.358514    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/622ee5be-fb50-42b6-ae23-42900b7c2d98-log-socket\") pod \"ovnkube-node-btmqz\" (UID: \"622ee5be-fb50-42b6-ae23-42900b7c2d98\") " pod="openshift-ovn-kubernetes/ovnkube-node-btmqz"
Sep 17 17:56:48.358753 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.358531    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4e137835-2bdc-4b32-aa6b-64635bdc26d7-cni-binary-copy\") pod \"multus-additional-cni-plugins-bmjrp\" (UID: \"4e137835-2bdc-4b32-aa6b-64635bdc26d7\") " pod="openshift-multus/multus-additional-cni-plugins-bmjrp"
Sep 17 17:56:48.358753 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.358598    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/622ee5be-fb50-42b6-ae23-42900b7c2d98-log-socket\") pod \"ovnkube-node-btmqz\" (UID: \"622ee5be-fb50-42b6-ae23-42900b7c2d98\") " pod="openshift-ovn-kubernetes/ovnkube-node-btmqz"
Sep 17 17:56:48.358753 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.358654    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4e137835-2bdc-4b32-aa6b-64635bdc26d7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bmjrp\" (UID: \"4e137835-2bdc-4b32-aa6b-64635bdc26d7\") " pod="openshift-multus/multus-additional-cni-plugins-bmjrp"
Sep 17 17:56:48.358753 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.358660    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b7c581ed-7ab3-4a26-930c-34fdf021affd-lib-modules\") pod \"tuned-t4pkh\" (UID: \"b7c581ed-7ab3-4a26-930c-34fdf021affd\") " pod="openshift-cluster-node-tuning-operator/tuned-t4pkh"
Sep 17 17:56:48.358753 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.358684    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/855a7f82-274d-4b6d-bcde-0c6f4c19016b-system-cni-dir\") pod \"multus-tjrts\" (UID: \"855a7f82-274d-4b6d-bcde-0c6f4c19016b\") " pod="openshift-multus/multus-tjrts"
Sep 17 17:56:48.358753 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.358711    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/855a7f82-274d-4b6d-bcde-0c6f4c19016b-multus-socket-dir-parent\") pod \"multus-tjrts\" (UID: \"855a7f82-274d-4b6d-bcde-0c6f4c19016b\") " pod="openshift-multus/multus-tjrts"
Sep 17 17:56:48.358753 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.358735    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/855a7f82-274d-4b6d-bcde-0c6f4c19016b-host-var-lib-kubelet\") pod \"multus-tjrts\" (UID: \"855a7f82-274d-4b6d-bcde-0c6f4c19016b\") " pod="openshift-multus/multus-tjrts"
Sep 17 17:56:48.358753 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.358751    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/855a7f82-274d-4b6d-bcde-0c6f4c19016b-system-cni-dir\") pod \"multus-tjrts\" (UID: \"855a7f82-274d-4b6d-bcde-0c6f4c19016b\") " pod="openshift-multus/multus-tjrts"
Sep 17 17:56:48.359205 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.358764    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g74cc\" (UniqueName: \"kubernetes.io/projected/3f19101a-6f40-4457-8f18-9c7dc294323b-kube-api-access-g74cc\") pod \"aws-ebs-csi-driver-node-gg9tp\" (UID: \"3f19101a-6f40-4457-8f18-9c7dc294323b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gg9tp"
Sep 17 17:56:48.359205 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.358787    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b7c581ed-7ab3-4a26-930c-34fdf021affd-etc-systemd\") pod \"tuned-t4pkh\" (UID: \"b7c581ed-7ab3-4a26-930c-34fdf021affd\") " pod="openshift-cluster-node-tuning-operator/tuned-t4pkh"
Sep 17 17:56:48.359205 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.358804    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4e137835-2bdc-4b32-aa6b-64635bdc26d7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bmjrp\" (UID: \"4e137835-2bdc-4b32-aa6b-64635bdc26d7\") " pod="openshift-multus/multus-additional-cni-plugins-bmjrp"
Sep 17 17:56:48.359205 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.358805    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/855a7f82-274d-4b6d-bcde-0c6f4c19016b-multus-socket-dir-parent\") pod \"multus-tjrts\" (UID: \"855a7f82-274d-4b6d-bcde-0c6f4c19016b\") " pod="openshift-multus/multus-tjrts"
Sep 17 17:56:48.359205 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.358822    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/622ee5be-fb50-42b6-ae23-42900b7c2d98-ovnkube-script-lib\") pod \"ovnkube-node-btmqz\" (UID: \"622ee5be-fb50-42b6-ae23-42900b7c2d98\") " pod="openshift-ovn-kubernetes/ovnkube-node-btmqz"
Sep 17 17:56:48.359205 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.358836    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/855a7f82-274d-4b6d-bcde-0c6f4c19016b-host-var-lib-kubelet\") pod \"multus-tjrts\" (UID: \"855a7f82-274d-4b6d-bcde-0c6f4c19016b\") " pod="openshift-multus/multus-tjrts"
Sep 17 17:56:48.359205 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.358861    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b7c581ed-7ab3-4a26-930c-34fdf021affd-etc-systemd\") pod \"tuned-t4pkh\" (UID: \"b7c581ed-7ab3-4a26-930c-34fdf021affd\") " pod="openshift-cluster-node-tuning-operator/tuned-t4pkh"
Sep 17 17:56:48.359205 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.358890    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb-kubelet-config\") pod \"global-pull-secret-syncer-xr8hx\" (UID: \"45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb\") " pod="kube-system/global-pull-secret-syncer-xr8hx"
Sep 17 17:56:48.359205 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.358910    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfc1cb73-d5de-4902-b16c-5744271855c5-metrics-certs\") pod \"network-metrics-daemon-5gdn2\" (UID: \"dfc1cb73-d5de-4902-b16c-5744271855c5\") " pod="openshift-multus/network-metrics-daemon-5gdn2"
Sep 17 17:56:48.359205 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.358925    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b7c581ed-7ab3-4a26-930c-34fdf021affd-run\") pod \"tuned-t4pkh\" (UID: \"b7c581ed-7ab3-4a26-930c-34fdf021affd\") " pod="openshift-cluster-node-tuning-operator/tuned-t4pkh"
Sep 17 17:56:48.359205 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.358962    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/622ee5be-fb50-42b6-ae23-42900b7c2d98-ovn-node-metrics-cert\") pod \"ovnkube-node-btmqz\" (UID: \"622ee5be-fb50-42b6-ae23-42900b7c2d98\") " pod="openshift-ovn-kubernetes/ovnkube-node-btmqz"
Sep 17 17:56:48.359205 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.358988    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b7c581ed-7ab3-4a26-930c-34fdf021affd-etc-sysconfig\") pod \"tuned-t4pkh\" (UID: \"b7c581ed-7ab3-4a26-930c-34fdf021affd\") " pod="openshift-cluster-node-tuning-operator/tuned-t4pkh"
Sep 17 17:56:48.359205 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.358979    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb-kubelet-config\") pod \"global-pull-secret-syncer-xr8hx\" (UID: \"45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb\") " pod="kube-system/global-pull-secret-syncer-xr8hx"
Sep 17 17:56:48.359205 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:48.359009    2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:56:48.359205 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.359021    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/622ee5be-fb50-42b6-ae23-42900b7c2d98-run-openvswitch\") pod \"ovnkube-node-btmqz\" (UID: \"622ee5be-fb50-42b6-ae23-42900b7c2d98\") " pod="openshift-ovn-kubernetes/ovnkube-node-btmqz"
Sep 17 17:56:48.359205 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.359045    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/622ee5be-fb50-42b6-ae23-42900b7c2d98-run-ovn\") pod \"ovnkube-node-btmqz\" (UID: \"622ee5be-fb50-42b6-ae23-42900b7c2d98\") " pod="openshift-ovn-kubernetes/ovnkube-node-btmqz"
Sep 17 17:56:48.359205 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.359045    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b7c581ed-7ab3-4a26-930c-34fdf021affd-run\") pod \"tuned-t4pkh\" (UID: \"b7c581ed-7ab3-4a26-930c-34fdf021affd\") " pod="openshift-cluster-node-tuning-operator/tuned-t4pkh"
Sep 17 17:56:48.359205 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.359083    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b7c581ed-7ab3-4a26-930c-34fdf021affd-etc-sysconfig\") pod \"tuned-t4pkh\" (UID: \"b7c581ed-7ab3-4a26-930c-34fdf021affd\") " pod="openshift-cluster-node-tuning-operator/tuned-t4pkh"
Sep 17 17:56:48.360205 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:48.359095    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfc1cb73-d5de-4902-b16c-5744271855c5-metrics-certs podName:dfc1cb73-d5de-4902-b16c-5744271855c5 nodeName:}" failed. No retries permitted until 2026-09-17 17:56:48.859053864 +0000 UTC m=+3.555144541 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dfc1cb73-d5de-4902-b16c-5744271855c5-metrics-certs") pod "network-metrics-daemon-5gdn2" (UID: "dfc1cb73-d5de-4902-b16c-5744271855c5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:56:48.360205 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.359131    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/622ee5be-fb50-42b6-ae23-42900b7c2d98-run-ovn\") pod \"ovnkube-node-btmqz\" (UID: \"622ee5be-fb50-42b6-ae23-42900b7c2d98\") " pod="openshift-ovn-kubernetes/ovnkube-node-btmqz"
Sep 17 17:56:48.360205 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.359092    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/622ee5be-fb50-42b6-ae23-42900b7c2d98-run-openvswitch\") pod \"ovnkube-node-btmqz\" (UID: \"622ee5be-fb50-42b6-ae23-42900b7c2d98\") " pod="openshift-ovn-kubernetes/ovnkube-node-btmqz"
Sep 17 17:56:48.360205 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.359178    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92plc\" (UniqueName: \"kubernetes.io/projected/56b9ff80-4b68-4449-bd38-e2e4d2de6aa2-kube-api-access-92plc\") pod \"node-resolver-4fdj7\" (UID: \"56b9ff80-4b68-4449-bd38-e2e4d2de6aa2\") " pod="openshift-dns/node-resolver-4fdj7"
Sep 17 17:56:48.360205 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.359175    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4e137835-2bdc-4b32-aa6b-64635bdc26d7-cni-binary-copy\") pod \"multus-additional-cni-plugins-bmjrp\" (UID: \"4e137835-2bdc-4b32-aa6b-64635bdc26d7\") " pod="openshift-multus/multus-additional-cni-plugins-bmjrp"
Sep 17 17:56:48.360205 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.359211    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4468ff09-5965-493f-ab71-8ccfb33fbb61-host-slash\") pod \"iptables-alerter-p6z75\" (UID: \"4468ff09-5965-493f-ab71-8ccfb33fbb61\") " pod="openshift-network-operator/iptables-alerter-p6z75"
Sep 17 17:56:48.360205 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.359256    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/855a7f82-274d-4b6d-bcde-0c6f4c19016b-multus-cni-dir\") pod \"multus-tjrts\" (UID: \"855a7f82-274d-4b6d-bcde-0c6f4c19016b\") " pod="openshift-multus/multus-tjrts"
Sep 17 17:56:48.360205 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.359285    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b7c581ed-7ab3-4a26-930c-34fdf021affd-etc-modprobe-d\") pod \"tuned-t4pkh\" (UID: \"b7c581ed-7ab3-4a26-930c-34fdf021affd\") " pod="openshift-cluster-node-tuning-operator/tuned-t4pkh"
Sep 17 17:56:48.360205 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.359308    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b7c581ed-7ab3-4a26-930c-34fdf021affd-etc-sysctl-d\") pod \"tuned-t4pkh\" (UID: \"b7c581ed-7ab3-4a26-930c-34fdf021affd\") " pod="openshift-cluster-node-tuning-operator/tuned-t4pkh"
Sep 17 17:56:48.360205 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.359330    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b7c581ed-7ab3-4a26-930c-34fdf021affd-host\") pod \"tuned-t4pkh\" (UID: \"b7c581ed-7ab3-4a26-930c-34fdf021affd\") " pod="openshift-cluster-node-tuning-operator/tuned-t4pkh"
Sep 17 17:56:48.360205 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.359350    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/855a7f82-274d-4b6d-bcde-0c6f4c19016b-multus-cni-dir\") pod \"multus-tjrts\" (UID: \"855a7f82-274d-4b6d-bcde-0c6f4c19016b\") " pod="openshift-multus/multus-tjrts"
Sep 17 17:56:48.360205 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.359382    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b7c581ed-7ab3-4a26-930c-34fdf021affd-host\") pod \"tuned-t4pkh\" (UID: \"b7c581ed-7ab3-4a26-930c-34fdf021affd\") " pod="openshift-cluster-node-tuning-operator/tuned-t4pkh"
Sep 17 17:56:48.360205 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.359403    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b7c581ed-7ab3-4a26-930c-34fdf021affd-etc-modprobe-d\") pod \"tuned-t4pkh\" (UID: \"b7c581ed-7ab3-4a26-930c-34fdf021affd\") " pod="openshift-cluster-node-tuning-operator/tuned-t4pkh"
Sep 17 17:56:48.360205 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.359393    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4468ff09-5965-493f-ab71-8ccfb33fbb61-iptables-alerter-script\") pod \"iptables-alerter-p6z75\" (UID: \"4468ff09-5965-493f-ab71-8ccfb33fbb61\") " pod="openshift-network-operator/iptables-alerter-p6z75"
Sep 17 17:56:48.360205 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.359441    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b7c581ed-7ab3-4a26-930c-34fdf021affd-etc-sysctl-d\") pod \"tuned-t4pkh\" (UID: \"b7c581ed-7ab3-4a26-930c-34fdf021affd\") " pod="openshift-cluster-node-tuning-operator/tuned-t4pkh"
Sep 17 17:56:48.360205 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.359452    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4e137835-2bdc-4b32-aa6b-64635bdc26d7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bmjrp\" (UID: \"4e137835-2bdc-4b32-aa6b-64635bdc26d7\") " pod="openshift-multus/multus-additional-cni-plugins-bmjrp"
Sep 17 17:56:48.360205 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.359461    2553 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 17:56:48.361082 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.359483    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3f19101a-6f40-4457-8f18-9c7dc294323b-etc-selinux\") pod \"aws-ebs-csi-driver-node-gg9tp\" (UID: \"3f19101a-6f40-4457-8f18-9c7dc294323b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gg9tp"
Sep 17 17:56:48.361082 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.359490    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/622ee5be-fb50-42b6-ae23-42900b7c2d98-ovnkube-script-lib\") pod \"ovnkube-node-btmqz\" (UID: \"622ee5be-fb50-42b6-ae23-42900b7c2d98\") " pod="openshift-ovn-kubernetes/ovnkube-node-btmqz"
Sep 17 17:56:48.361082 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.359506    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/622ee5be-fb50-42b6-ae23-42900b7c2d98-var-lib-openvswitch\") pod \"ovnkube-node-btmqz\" (UID: \"622ee5be-fb50-42b6-ae23-42900b7c2d98\") " pod="openshift-ovn-kubernetes/ovnkube-node-btmqz"
Sep 17 17:56:48.361082 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.359529    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/855a7f82-274d-4b6d-bcde-0c6f4c19016b-hostroot\") pod \"multus-tjrts\" (UID: \"855a7f82-274d-4b6d-bcde-0c6f4c19016b\") " pod="openshift-multus/multus-tjrts"
Sep 17 17:56:48.361082 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.359554    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3f19101a-6f40-4457-8f18-9c7dc294323b-device-dir\") pod \"aws-ebs-csi-driver-node-gg9tp\" (UID: \"3f19101a-6f40-4457-8f18-9c7dc294323b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gg9tp"
Sep 17 17:56:48.361082 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.359570    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/622ee5be-fb50-42b6-ae23-42900b7c2d98-var-lib-openvswitch\") pod \"ovnkube-node-btmqz\" (UID: \"622ee5be-fb50-42b6-ae23-42900b7c2d98\") " pod="openshift-ovn-kubernetes/ovnkube-node-btmqz"
Sep 17 17:56:48.361082 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.359577    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7c581ed-7ab3-4a26-930c-34fdf021affd-var-lib-kubelet\") pod \"tuned-t4pkh\" (UID: \"b7c581ed-7ab3-4a26-930c-34fdf021affd\") " pod="openshift-cluster-node-tuning-operator/tuned-t4pkh"
Sep 17 17:56:48.361082 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.359594    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/855a7f82-274d-4b6d-bcde-0c6f4c19016b-hostroot\") pod \"multus-tjrts\" (UID: \"855a7f82-274d-4b6d-bcde-0c6f4c19016b\") " pod="openshift-multus/multus-tjrts"
Sep 17 17:56:48.361082 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.359586    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3f19101a-6f40-4457-8f18-9c7dc294323b-etc-selinux\") pod \"aws-ebs-csi-driver-node-gg9tp\" (UID: \"3f19101a-6f40-4457-8f18-9c7dc294323b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gg9tp"
Sep 17 17:56:48.361082 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.359644    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3f19101a-6f40-4457-8f18-9c7dc294323b-device-dir\") pod \"aws-ebs-csi-driver-node-gg9tp\" (UID: \"3f19101a-6f40-4457-8f18-9c7dc294323b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gg9tp"
Sep 17 17:56:48.361082 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.359655    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/622ee5be-fb50-42b6-ae23-42900b7c2d98-systemd-units\") pod \"ovnkube-node-btmqz\" (UID: \"622ee5be-fb50-42b6-ae23-42900b7c2d98\") " pod="openshift-ovn-kubernetes/ovnkube-node-btmqz"
Sep 17 17:56:48.361082 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.359639    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7c581ed-7ab3-4a26-930c-34fdf021affd-var-lib-kubelet\") pod \"tuned-t4pkh\" (UID: \"b7c581ed-7ab3-4a26-930c-34fdf021affd\") " pod="openshift-cluster-node-tuning-operator/tuned-t4pkh"
Sep 17 17:56:48.361082 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.359682    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6s8w\" (UniqueName: \"kubernetes.io/projected/622ee5be-fb50-42b6-ae23-42900b7c2d98-kube-api-access-c6s8w\") pod \"ovnkube-node-btmqz\" (UID: \"622ee5be-fb50-42b6-ae23-42900b7c2d98\") " pod="openshift-ovn-kubernetes/ovnkube-node-btmqz"
Sep 17 17:56:48.361082 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.359710    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/622ee5be-fb50-42b6-ae23-42900b7c2d98-systemd-units\") pod \"ovnkube-node-btmqz\" (UID: \"622ee5be-fb50-42b6-ae23-42900b7c2d98\") " pod="openshift-ovn-kubernetes/ovnkube-node-btmqz"
Sep 17 17:56:48.361082 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.359758    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/56b9ff80-4b68-4449-bd38-e2e4d2de6aa2-hosts-file\") pod \"node-resolver-4fdj7\" (UID: \"56b9ff80-4b68-4449-bd38-e2e4d2de6aa2\") " pod="openshift-dns/node-resolver-4fdj7"
Sep 17 17:56:48.361082 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.359788    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/855a7f82-274d-4b6d-bcde-0c6f4c19016b-cnibin\") pod \"multus-tjrts\" (UID: \"855a7f82-274d-4b6d-bcde-0c6f4c19016b\") " pod="openshift-multus/multus-tjrts"
Sep 17 17:56:48.361082 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.359812    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/855a7f82-274d-4b6d-bcde-0c6f4c19016b-host-run-netns\") pod \"multus-tjrts\" (UID: \"855a7f82-274d-4b6d-bcde-0c6f4c19016b\") " pod="openshift-multus/multus-tjrts"
Sep 17 17:56:48.362483 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.359837    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd8gk\" (UniqueName: \"kubernetes.io/projected/855a7f82-274d-4b6d-bcde-0c6f4c19016b-kube-api-access-jd8gk\") pod \"multus-tjrts\" (UID: \"855a7f82-274d-4b6d-bcde-0c6f4c19016b\") " pod="openshift-multus/multus-tjrts"
Sep 17 17:56:48.362483 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.359861    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/855a7f82-274d-4b6d-bcde-0c6f4c19016b-cni-binary-copy\") pod \"multus-tjrts\" (UID: \"855a7f82-274d-4b6d-bcde-0c6f4c19016b\") " pod="openshift-multus/multus-tjrts"
Sep 17 17:56:48.362483 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.359887    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/855a7f82-274d-4b6d-bcde-0c6f4c19016b-host-run-netns\") pod \"multus-tjrts\" (UID: \"855a7f82-274d-4b6d-bcde-0c6f4c19016b\") " pod="openshift-multus/multus-tjrts"
Sep 17 17:56:48.362483 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.359893    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/855a7f82-274d-4b6d-bcde-0c6f4c19016b-cnibin\") pod \"multus-tjrts\" (UID: \"855a7f82-274d-4b6d-bcde-0c6f4c19016b\") " pod="openshift-multus/multus-tjrts"
Sep 17 17:56:48.362483 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.359892    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/855a7f82-274d-4b6d-bcde-0c6f4c19016b-host-run-k8s-cni-cncf-io\") pod \"multus-tjrts\" (UID: \"855a7f82-274d-4b6d-bcde-0c6f4c19016b\") " pod="openshift-multus/multus-tjrts"
Sep 17 17:56:48.362483 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.359936    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3f19101a-6f40-4457-8f18-9c7dc294323b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gg9tp\" (UID: \"3f19101a-6f40-4457-8f18-9c7dc294323b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gg9tp"
Sep 17 17:56:48.362483 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.359960    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4e137835-2bdc-4b32-aa6b-64635bdc26d7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bmjrp\" (UID: \"4e137835-2bdc-4b32-aa6b-64635bdc26d7\") " pod="openshift-multus/multus-additional-cni-plugins-bmjrp"
Sep 17 17:56:48.362483 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.359965    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/855a7f82-274d-4b6d-bcde-0c6f4c19016b-host-run-k8s-cni-cncf-io\") pod \"multus-tjrts\" (UID: \"855a7f82-274d-4b6d-bcde-0c6f4c19016b\") " pod="openshift-multus/multus-tjrts"
Sep 17 17:56:48.362483 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.359963    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3f19101a-6f40-4457-8f18-9c7dc294323b-registration-dir\") pod \"aws-ebs-csi-driver-node-gg9tp\" (UID: \"3f19101a-6f40-4457-8f18-9c7dc294323b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gg9tp"
Sep 17 17:56:48.362483 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.360009    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b7c581ed-7ab3-4a26-930c-34fdf021affd-etc-sysctl-conf\") pod \"tuned-t4pkh\" (UID: \"b7c581ed-7ab3-4a26-930c-34fdf021affd\") " pod="openshift-cluster-node-tuning-operator/tuned-t4pkh"
Sep 17 17:56:48.362483 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.360012    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3f19101a-6f40-4457-8f18-9c7dc294323b-registration-dir\") pod \"aws-ebs-csi-driver-node-gg9tp\" (UID: \"3f19101a-6f40-4457-8f18-9c7dc294323b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gg9tp"
Sep 17 17:56:48.362483 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.360019    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3f19101a-6f40-4457-8f18-9c7dc294323b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gg9tp\" (UID: \"3f19101a-6f40-4457-8f18-9c7dc294323b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gg9tp"
Sep 17 17:56:48.362483 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.360035    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b281f20e-0314-4794-98c1-50e5cba38212-konnectivity-ca\") pod \"konnectivity-agent-jz66f\" (UID: \"b281f20e-0314-4794-98c1-50e5cba38212\") " pod="kube-system/konnectivity-agent-jz66f"
Sep 17 17:56:48.362483 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.360068    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/855a7f82-274d-4b6d-bcde-0c6f4c19016b-host-var-lib-cni-bin\") pod \"multus-tjrts\" (UID: \"855a7f82-274d-4b6d-bcde-0c6f4c19016b\") " pod="openshift-multus/multus-tjrts"
Sep 17 17:56:48.362483 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.360090    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/855a7f82-274d-4b6d-bcde-0c6f4c19016b-multus-daemon-config\") pod \"multus-tjrts\" (UID: \"855a7f82-274d-4b6d-bcde-0c6f4c19016b\") " pod="openshift-multus/multus-tjrts"
Sep 17 17:56:48.362483 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.360113    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/855a7f82-274d-4b6d-bcde-0c6f4c19016b-host-run-multus-certs\") pod \"multus-tjrts\" (UID: \"855a7f82-274d-4b6d-bcde-0c6f4c19016b\") " pod="openshift-multus/multus-tjrts"
Sep 17 17:56:48.362483 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.360136    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3f19101a-6f40-4457-8f18-9c7dc294323b-sys-fs\") pod \"aws-ebs-csi-driver-node-gg9tp\" (UID: \"3f19101a-6f40-4457-8f18-9c7dc294323b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gg9tp"
Sep 17 17:56:48.363332 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.360144    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/855a7f82-274d-4b6d-bcde-0c6f4c19016b-host-var-lib-cni-bin\") pod \"multus-tjrts\" (UID: \"855a7f82-274d-4b6d-bcde-0c6f4c19016b\") " pod="openshift-multus/multus-tjrts"
Sep 17 17:56:48.363332 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.360159    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b7c581ed-7ab3-4a26-930c-34fdf021affd-sys\") pod \"tuned-t4pkh\" (UID: \"b7c581ed-7ab3-4a26-930c-34fdf021affd\") " pod="openshift-cluster-node-tuning-operator/tuned-t4pkh"
Sep 17 17:56:48.363332 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.360167    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b7c581ed-7ab3-4a26-930c-34fdf021affd-etc-sysctl-conf\") pod \"tuned-t4pkh\" (UID: \"b7c581ed-7ab3-4a26-930c-34fdf021affd\") " pod="openshift-cluster-node-tuning-operator/tuned-t4pkh"
Sep 17 17:56:48.363332 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.360182    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/622ee5be-fb50-42b6-ae23-42900b7c2d98-host-kubelet\") pod \"ovnkube-node-btmqz\" (UID: \"622ee5be-fb50-42b6-ae23-42900b7c2d98\") " pod="openshift-ovn-kubernetes/ovnkube-node-btmqz"
Sep 17 17:56:48.363332 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.360200    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/855a7f82-274d-4b6d-bcde-0c6f4c19016b-host-run-multus-certs\") pod \"multus-tjrts\" (UID: \"855a7f82-274d-4b6d-bcde-0c6f4c19016b\") " pod="openshift-multus/multus-tjrts"
Sep 17 17:56:48.363332 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.360237    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3f19101a-6f40-4457-8f18-9c7dc294323b-sys-fs\") pod \"aws-ebs-csi-driver-node-gg9tp\" (UID: \"3f19101a-6f40-4457-8f18-9c7dc294323b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gg9tp"
Sep 17 17:56:48.363332 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.360269    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/622ee5be-fb50-42b6-ae23-42900b7c2d98-host-slash\") pod \"ovnkube-node-btmqz\" (UID: \"622ee5be-fb50-42b6-ae23-42900b7c2d98\") " pod="openshift-ovn-kubernetes/ovnkube-node-btmqz"
Sep 17 17:56:48.363332 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.360295    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c5297750-e3a6-4690-9f70-4be7eac03a96-serviceca\") pod \"node-ca-6pxfw\" (UID: \"c5297750-e3a6-4690-9f70-4be7eac03a96\") " pod="openshift-image-registry/node-ca-6pxfw"
Sep 17 17:56:48.363332 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.360303    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/622ee5be-fb50-42b6-ae23-42900b7c2d98-host-kubelet\") pod \"ovnkube-node-btmqz\" (UID: \"622ee5be-fb50-42b6-ae23-42900b7c2d98\") " pod="openshift-ovn-kubernetes/ovnkube-node-btmqz"
Sep 17 17:56:48.363332 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.360248    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b7c581ed-7ab3-4a26-930c-34fdf021affd-sys\") pod \"tuned-t4pkh\" (UID: \"b7c581ed-7ab3-4a26-930c-34fdf021affd\") " pod="openshift-cluster-node-tuning-operator/tuned-t4pkh"
Sep 17 17:56:48.363332 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.360322    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4e137835-2bdc-4b32-aa6b-64635bdc26d7-cnibin\") pod \"multus-additional-cni-plugins-bmjrp\" (UID: \"4e137835-2bdc-4b32-aa6b-64635bdc26d7\") " pod="openshift-multus/multus-additional-cni-plugins-bmjrp"
Sep 17 17:56:48.363332 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.360348    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/622ee5be-fb50-42b6-ae23-42900b7c2d98-run-systemd\") pod \"ovnkube-node-btmqz\" (UID: \"622ee5be-fb50-42b6-ae23-42900b7c2d98\") " pod="openshift-ovn-kubernetes/ovnkube-node-btmqz"
Sep 17 17:56:48.363332 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.360372    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/622ee5be-fb50-42b6-ae23-42900b7c2d98-etc-openvswitch\") pod \"ovnkube-node-btmqz\" (UID: \"622ee5be-fb50-42b6-ae23-42900b7c2d98\") " pod="openshift-ovn-kubernetes/ovnkube-node-btmqz"
Sep 17 17:56:48.363332 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.360376    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/855a7f82-274d-4b6d-bcde-0c6f4c19016b-cni-binary-copy\") pod \"multus-tjrts\" (UID: \"855a7f82-274d-4b6d-bcde-0c6f4c19016b\") " pod="openshift-multus/multus-tjrts"
Sep 17 17:56:48.363332 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.360393    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4e137835-2bdc-4b32-aa6b-64635bdc26d7-cnibin\") pod \"multus-additional-cni-plugins-bmjrp\" (UID: \"4e137835-2bdc-4b32-aa6b-64635bdc26d7\") " pod="openshift-multus/multus-additional-cni-plugins-bmjrp"
Sep 17 17:56:48.363332 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.360397    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/622ee5be-fb50-42b6-ae23-42900b7c2d98-node-log\") pod \"ovnkube-node-btmqz\" (UID: \"622ee5be-fb50-42b6-ae23-42900b7c2d98\") " pod="openshift-ovn-kubernetes/ovnkube-node-btmqz"
Sep 17 17:56:48.363332 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.360348    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/622ee5be-fb50-42b6-ae23-42900b7c2d98-host-slash\") pod \"ovnkube-node-btmqz\" (UID: \"622ee5be-fb50-42b6-ae23-42900b7c2d98\") " pod="openshift-ovn-kubernetes/ovnkube-node-btmqz"
Sep 17 17:56:48.363332 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.360428    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/622ee5be-fb50-42b6-ae23-42900b7c2d98-host-run-ovn-kubernetes\") pod \"ovnkube-node-btmqz\" (UID: \"622ee5be-fb50-42b6-ae23-42900b7c2d98\") " pod="openshift-ovn-kubernetes/ovnkube-node-btmqz"
Sep 17 17:56:48.363900 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.360435    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/622ee5be-fb50-42b6-ae23-42900b7c2d98-node-log\") pod \"ovnkube-node-btmqz\" (UID: \"622ee5be-fb50-42b6-ae23-42900b7c2d98\") " pod="openshift-ovn-kubernetes/ovnkube-node-btmqz"
Sep 17 17:56:48.363900 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.360448    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/622ee5be-fb50-42b6-ae23-42900b7c2d98-run-systemd\") pod \"ovnkube-node-btmqz\" (UID: \"622ee5be-fb50-42b6-ae23-42900b7c2d98\") " pod="openshift-ovn-kubernetes/ovnkube-node-btmqz"
Sep 17 17:56:48.363900 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.360453    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/622ee5be-fb50-42b6-ae23-42900b7c2d98-ovnkube-config\") pod \"ovnkube-node-btmqz\" (UID: \"622ee5be-fb50-42b6-ae23-42900b7c2d98\") " pod="openshift-ovn-kubernetes/ovnkube-node-btmqz"
Sep 17 17:56:48.363900 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.360468    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/622ee5be-fb50-42b6-ae23-42900b7c2d98-etc-openvswitch\") pod \"ovnkube-node-btmqz\" (UID: \"622ee5be-fb50-42b6-ae23-42900b7c2d98\") " pod="openshift-ovn-kubernetes/ovnkube-node-btmqz"
Sep 17 17:56:48.363900 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.360487    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/622ee5be-fb50-42b6-ae23-42900b7c2d98-host-run-ovn-kubernetes\") pod \"ovnkube-node-btmqz\" (UID: \"622ee5be-fb50-42b6-ae23-42900b7c2d98\") " pod="openshift-ovn-kubernetes/ovnkube-node-btmqz"
Sep 17 17:56:48.363900 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.360487    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjvjg\" (UniqueName: \"kubernetes.io/projected/c5297750-e3a6-4690-9f70-4be7eac03a96-kube-api-access-fjvjg\") pod \"node-ca-6pxfw\" (UID: \"c5297750-e3a6-4690-9f70-4be7eac03a96\") " pod="openshift-image-registry/node-ca-6pxfw"
Sep 17 17:56:48.363900 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.360544    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r49v\" (UniqueName: \"kubernetes.io/projected/4468ff09-5965-493f-ab71-8ccfb33fbb61-kube-api-access-4r49v\") pod \"iptables-alerter-p6z75\" (UID: \"4468ff09-5965-493f-ab71-8ccfb33fbb61\") " pod="openshift-network-operator/iptables-alerter-p6z75"
Sep 17 17:56:48.363900 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.360579    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb-dbus\") pod \"global-pull-secret-syncer-xr8hx\" (UID: \"45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb\") " pod="kube-system/global-pull-secret-syncer-xr8hx"
Sep 17 17:56:48.363900 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.360611    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9msrf\" (UniqueName: \"kubernetes.io/projected/4e137835-2bdc-4b32-aa6b-64635bdc26d7-kube-api-access-9msrf\") pod \"multus-additional-cni-plugins-bmjrp\" (UID: \"4e137835-2bdc-4b32-aa6b-64635bdc26d7\") " pod="openshift-multus/multus-additional-cni-plugins-bmjrp"
Sep 17 17:56:48.363900 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.360652    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/855a7f82-274d-4b6d-bcde-0c6f4c19016b-host-var-lib-cni-multus\") pod \"multus-tjrts\" (UID: \"855a7f82-274d-4b6d-bcde-0c6f4c19016b\") " pod="openshift-multus/multus-tjrts"
Sep 17 17:56:48.363900 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.360682    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/855a7f82-274d-4b6d-bcde-0c6f4c19016b-multus-daemon-config\") pod \"multus-tjrts\" (UID: \"855a7f82-274d-4b6d-bcde-0c6f4c19016b\") " pod="openshift-multus/multus-tjrts"
Sep 17 17:56:48.363900 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.360693    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/855a7f82-274d-4b6d-bcde-0c6f4c19016b-multus-conf-dir\") pod \"multus-tjrts\" (UID: \"855a7f82-274d-4b6d-bcde-0c6f4c19016b\") " pod="openshift-multus/multus-tjrts"
Sep 17 17:56:48.363900 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.360717    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4p88\" (UniqueName: \"kubernetes.io/projected/cb953471-4308-48e7-9316-6b0bdf2b8738-kube-api-access-x4p88\") pod \"network-check-target-lkn4v\" (UID: \"cb953471-4308-48e7-9316-6b0bdf2b8738\") " pod="openshift-network-diagnostics/network-check-target-lkn4v"
Sep 17 17:56:48.363900 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.360728    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb-dbus\") pod \"global-pull-secret-syncer-xr8hx\" (UID: \"45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb\") " pod="kube-system/global-pull-secret-syncer-xr8hx"
Sep 17 17:56:48.363900 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.360739    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b7c581ed-7ab3-4a26-930c-34fdf021affd-etc-kubernetes\") pod \"tuned-t4pkh\" (UID: \"b7c581ed-7ab3-4a26-930c-34fdf021affd\") " pod="openshift-cluster-node-tuning-operator/tuned-t4pkh"
Sep 17 17:56:48.363900 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.360747    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/855a7f82-274d-4b6d-bcde-0c6f4c19016b-multus-conf-dir\") pod \"multus-tjrts\" (UID: \"855a7f82-274d-4b6d-bcde-0c6f4c19016b\") " pod="openshift-multus/multus-tjrts"
Sep 17 17:56:48.363900 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.360764    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/855a7f82-274d-4b6d-bcde-0c6f4c19016b-host-var-lib-cni-multus\") pod \"multus-tjrts\" (UID: \"855a7f82-274d-4b6d-bcde-0c6f4c19016b\") " pod="openshift-multus/multus-tjrts"
Sep 17 17:56:48.364464 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.360780    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/622ee5be-fb50-42b6-ae23-42900b7c2d98-host-cni-bin\") pod \"ovnkube-node-btmqz\" (UID: \"622ee5be-fb50-42b6-ae23-42900b7c2d98\") " pod="openshift-ovn-kubernetes/ovnkube-node-btmqz"
Sep 17 17:56:48.364464 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.360807    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/622ee5be-fb50-42b6-ae23-42900b7c2d98-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-btmqz\" (UID: \"622ee5be-fb50-42b6-ae23-42900b7c2d98\") " pod="openshift-ovn-kubernetes/ovnkube-node-btmqz"
Sep 17 17:56:48.364464 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.360856    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/622ee5be-fb50-42b6-ae23-42900b7c2d98-env-overrides\") pod \"ovnkube-node-btmqz\" (UID: \"622ee5be-fb50-42b6-ae23-42900b7c2d98\") " pod="openshift-ovn-kubernetes/ovnkube-node-btmqz"
Sep 17 17:56:48.364464 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.360882    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56b9ff80-4b68-4449-bd38-e2e4d2de6aa2-tmp-dir\") pod \"node-resolver-4fdj7\" (UID: \"56b9ff80-4b68-4449-bd38-e2e4d2de6aa2\") " pod="openshift-dns/node-resolver-4fdj7"
Sep 17 17:56:48.364464 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.360907    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b281f20e-0314-4794-98c1-50e5cba38212-agent-certs\") pod \"konnectivity-agent-jz66f\" (UID: \"b281f20e-0314-4794-98c1-50e5cba38212\") " pod="kube-system/konnectivity-agent-jz66f"
Sep 17 17:56:48.364464 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.360911    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b7c581ed-7ab3-4a26-930c-34fdf021affd-etc-kubernetes\") pod \"tuned-t4pkh\" (UID: \"b7c581ed-7ab3-4a26-930c-34fdf021affd\") " pod="openshift-cluster-node-tuning-operator/tuned-t4pkh"
Sep 17 17:56:48.364464 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.360951    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/622ee5be-fb50-42b6-ae23-42900b7c2d98-ovnkube-config\") pod \"ovnkube-node-btmqz\" (UID: \"622ee5be-fb50-42b6-ae23-42900b7c2d98\") " pod="openshift-ovn-kubernetes/ovnkube-node-btmqz"
Sep 17 17:56:48.364464 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.360958    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/855a7f82-274d-4b6d-bcde-0c6f4c19016b-etc-kubernetes\") pod \"multus-tjrts\" (UID: \"855a7f82-274d-4b6d-bcde-0c6f4c19016b\") " pod="openshift-multus/multus-tjrts"
Sep 17 17:56:48.364464 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.360986    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b7c581ed-7ab3-4a26-930c-34fdf021affd-etc-tuned\") pod \"tuned-t4pkh\" (UID: \"b7c581ed-7ab3-4a26-930c-34fdf021affd\") " pod="openshift-cluster-node-tuning-operator/tuned-t4pkh"
Sep 17 17:56:48.364464 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.361019    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/622ee5be-fb50-42b6-ae23-42900b7c2d98-host-cni-bin\") pod \"ovnkube-node-btmqz\" (UID: \"622ee5be-fb50-42b6-ae23-42900b7c2d98\") " pod="openshift-ovn-kubernetes/ovnkube-node-btmqz"
Sep 17 17:56:48.364464 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.361036    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/622ee5be-fb50-42b6-ae23-42900b7c2d98-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-btmqz\" (UID: \"622ee5be-fb50-42b6-ae23-42900b7c2d98\") " pod="openshift-ovn-kubernetes/ovnkube-node-btmqz"
Sep 17 17:56:48.364464 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.361059    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/622ee5be-fb50-42b6-ae23-42900b7c2d98-host-run-netns\") pod \"ovnkube-node-btmqz\" (UID: \"622ee5be-fb50-42b6-ae23-42900b7c2d98\") " pod="openshift-ovn-kubernetes/ovnkube-node-btmqz"
Sep 17 17:56:48.364464 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.361065    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/855a7f82-274d-4b6d-bcde-0c6f4c19016b-etc-kubernetes\") pod \"multus-tjrts\" (UID: \"855a7f82-274d-4b6d-bcde-0c6f4c19016b\") " pod="openshift-multus/multus-tjrts"
Sep 17 17:56:48.364464 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.361108    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb-original-pull-secret\") pod \"global-pull-secret-syncer-xr8hx\" (UID: \"45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb\") " pod="kube-system/global-pull-secret-syncer-xr8hx"
Sep 17 17:56:48.364464 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.361134    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e137835-2bdc-4b32-aa6b-64635bdc26d7-system-cni-dir\") pod \"multus-additional-cni-plugins-bmjrp\" (UID: \"4e137835-2bdc-4b32-aa6b-64635bdc26d7\") " pod="openshift-multus/multus-additional-cni-plugins-bmjrp"
Sep 17 17:56:48.364464 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.361155    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/622ee5be-fb50-42b6-ae23-42900b7c2d98-host-run-netns\") pod \"ovnkube-node-btmqz\" (UID: \"622ee5be-fb50-42b6-ae23-42900b7c2d98\") " pod="openshift-ovn-kubernetes/ovnkube-node-btmqz"
Sep 17 17:56:48.364464 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.361162    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4e137835-2bdc-4b32-aa6b-64635bdc26d7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bmjrp\" (UID: \"4e137835-2bdc-4b32-aa6b-64635bdc26d7\") " pod="openshift-multus/multus-additional-cni-plugins-bmjrp"
Sep 17 17:56:48.364971 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.361191    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6skh2\" (UniqueName: \"kubernetes.io/projected/b7c581ed-7ab3-4a26-930c-34fdf021affd-kube-api-access-6skh2\") pod \"tuned-t4pkh\" (UID: \"b7c581ed-7ab3-4a26-930c-34fdf021affd\") " pod="openshift-cluster-node-tuning-operator/tuned-t4pkh"
Sep 17 17:56:48.364971 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.361225    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e137835-2bdc-4b32-aa6b-64635bdc26d7-system-cni-dir\") pod \"multus-additional-cni-plugins-bmjrp\" (UID: \"4e137835-2bdc-4b32-aa6b-64635bdc26d7\") " pod="openshift-multus/multus-additional-cni-plugins-bmjrp"
Sep 17 17:56:48.364971 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:48.361273    2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:56:48.364971 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.361293    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/622ee5be-fb50-42b6-ae23-42900b7c2d98-env-overrides\") pod \"ovnkube-node-btmqz\" (UID: \"622ee5be-fb50-42b6-ae23-42900b7c2d98\") " pod="openshift-ovn-kubernetes/ovnkube-node-btmqz"
Sep 17 17:56:48.364971 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:48.361326    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb-original-pull-secret podName:45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb nodeName:}" failed. No retries permitted until 2026-09-17 17:56:48.86130967 +0000 UTC m=+3.557400350 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb-original-pull-secret") pod "global-pull-secret-syncer-xr8hx" (UID: "45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:56:48.364971 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.361354    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/622ee5be-fb50-42b6-ae23-42900b7c2d98-host-cni-netd\") pod \"ovnkube-node-btmqz\" (UID: \"622ee5be-fb50-42b6-ae23-42900b7c2d98\") " pod="openshift-ovn-kubernetes/ovnkube-node-btmqz"
Sep 17 17:56:48.364971 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.361383    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c5297750-e3a6-4690-9f70-4be7eac03a96-host\") pod \"node-ca-6pxfw\" (UID: \"c5297750-e3a6-4690-9f70-4be7eac03a96\") " pod="openshift-image-registry/node-ca-6pxfw"
Sep 17 17:56:48.364971 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.361409    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmv25\" (UniqueName: \"kubernetes.io/projected/dfc1cb73-d5de-4902-b16c-5744271855c5-kube-api-access-pmv25\") pod \"network-metrics-daemon-5gdn2\" (UID: \"dfc1cb73-d5de-4902-b16c-5744271855c5\") " pod="openshift-multus/network-metrics-daemon-5gdn2"
Sep 17 17:56:48.364971 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.361435    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3f19101a-6f40-4457-8f18-9c7dc294323b-socket-dir\") pod \"aws-ebs-csi-driver-node-gg9tp\" (UID: \"3f19101a-6f40-4457-8f18-9c7dc294323b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gg9tp"
Sep 17 17:56:48.364971 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.361473    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/622ee5be-fb50-42b6-ae23-42900b7c2d98-host-cni-netd\") pod \"ovnkube-node-btmqz\" (UID: \"622ee5be-fb50-42b6-ae23-42900b7c2d98\") " pod="openshift-ovn-kubernetes/ovnkube-node-btmqz"
Sep 17 17:56:48.364971 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.361569    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3f19101a-6f40-4457-8f18-9c7dc294323b-socket-dir\") pod \"aws-ebs-csi-driver-node-gg9tp\" (UID: \"3f19101a-6f40-4457-8f18-9c7dc294323b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gg9tp"
Sep 17 17:56:48.364971 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.361700    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4e137835-2bdc-4b32-aa6b-64635bdc26d7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bmjrp\" (UID: \"4e137835-2bdc-4b32-aa6b-64635bdc26d7\") " pod="openshift-multus/multus-additional-cni-plugins-bmjrp"
Sep 17 17:56:48.364971 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.363078    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b7c581ed-7ab3-4a26-930c-34fdf021affd-etc-tuned\") pod \"tuned-t4pkh\" (UID: \"b7c581ed-7ab3-4a26-930c-34fdf021affd\") " pod="openshift-cluster-node-tuning-operator/tuned-t4pkh"
Sep 17 17:56:48.364971 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.363255    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b7c581ed-7ab3-4a26-930c-34fdf021affd-tmp\") pod \"tuned-t4pkh\" (UID: \"b7c581ed-7ab3-4a26-930c-34fdf021affd\") " pod="openshift-cluster-node-tuning-operator/tuned-t4pkh"
Sep 17 17:56:48.364971 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.363279    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/622ee5be-fb50-42b6-ae23-42900b7c2d98-ovn-node-metrics-cert\") pod \"ovnkube-node-btmqz\" (UID: \"622ee5be-fb50-42b6-ae23-42900b7c2d98\") " pod="openshift-ovn-kubernetes/ovnkube-node-btmqz"
Sep 17 17:56:48.383594 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.383401    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd8gk\" (UniqueName: \"kubernetes.io/projected/855a7f82-274d-4b6d-bcde-0c6f4c19016b-kube-api-access-jd8gk\") pod \"multus-tjrts\" (UID: \"855a7f82-274d-4b6d-bcde-0c6f4c19016b\") " pod="openshift-multus/multus-tjrts"
Sep 17 17:56:48.386944 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.386919    2553 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:56:48.393568 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:48.393538    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:56:48.393568 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:48.393569    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:56:48.393896 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:48.393583    2553 projected.go:196] Error preparing data for projected volume kube-api-access-x4p88 for pod openshift-network-diagnostics/network-check-target-lkn4v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:56:48.393896 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:48.393694    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb953471-4308-48e7-9316-6b0bdf2b8738-kube-api-access-x4p88 podName:cb953471-4308-48e7-9316-6b0bdf2b8738 nodeName:}" failed. No retries permitted until 2026-09-17 17:56:48.893671888 +0000 UTC m=+3.589762555 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-x4p88" (UniqueName: "kubernetes.io/projected/cb953471-4308-48e7-9316-6b0bdf2b8738-kube-api-access-x4p88") pod "network-check-target-lkn4v" (UID: "cb953471-4308-48e7-9316-6b0bdf2b8738") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:56:48.394064 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.394028    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g74cc\" (UniqueName: \"kubernetes.io/projected/3f19101a-6f40-4457-8f18-9c7dc294323b-kube-api-access-g74cc\") pod \"aws-ebs-csi-driver-node-gg9tp\" (UID: \"3f19101a-6f40-4457-8f18-9c7dc294323b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gg9tp"
Sep 17 17:56:48.395880 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.395854    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6s8w\" (UniqueName: \"kubernetes.io/projected/622ee5be-fb50-42b6-ae23-42900b7c2d98-kube-api-access-c6s8w\") pod \"ovnkube-node-btmqz\" (UID: \"622ee5be-fb50-42b6-ae23-42900b7c2d98\") " pod="openshift-ovn-kubernetes/ovnkube-node-btmqz"
Sep 17 17:56:48.396925 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.396898    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmv25\" (UniqueName: \"kubernetes.io/projected/dfc1cb73-d5de-4902-b16c-5744271855c5-kube-api-access-pmv25\") pod \"network-metrics-daemon-5gdn2\" (UID: \"dfc1cb73-d5de-4902-b16c-5744271855c5\") " pod="openshift-multus/network-metrics-daemon-5gdn2"
Sep 17 17:56:48.398179 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.398158    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6skh2\" (UniqueName: \"kubernetes.io/projected/b7c581ed-7ab3-4a26-930c-34fdf021affd-kube-api-access-6skh2\") pod \"tuned-t4pkh\" (UID: \"b7c581ed-7ab3-4a26-930c-34fdf021affd\") " pod="openshift-cluster-node-tuning-operator/tuned-t4pkh"
Sep 17 17:56:48.398403 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.398383    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9msrf\" (UniqueName: \"kubernetes.io/projected/4e137835-2bdc-4b32-aa6b-64635bdc26d7-kube-api-access-9msrf\") pod \"multus-additional-cni-plugins-bmjrp\" (UID: \"4e137835-2bdc-4b32-aa6b-64635bdc26d7\") " pod="openshift-multus/multus-additional-cni-plugins-bmjrp"
Sep 17 17:56:48.462678 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.462646    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4468ff09-5965-493f-ab71-8ccfb33fbb61-iptables-alerter-script\") pod \"iptables-alerter-p6z75\" (UID: \"4468ff09-5965-493f-ab71-8ccfb33fbb61\") " pod="openshift-network-operator/iptables-alerter-p6z75"
Sep 17 17:56:48.462678 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.462685    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/56b9ff80-4b68-4449-bd38-e2e4d2de6aa2-hosts-file\") pod \"node-resolver-4fdj7\" (UID: \"56b9ff80-4b68-4449-bd38-e2e4d2de6aa2\") " pod="openshift-dns/node-resolver-4fdj7"
Sep 17 17:56:48.462905 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.462705    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b281f20e-0314-4794-98c1-50e5cba38212-konnectivity-ca\") pod \"konnectivity-agent-jz66f\" (UID: \"b281f20e-0314-4794-98c1-50e5cba38212\") " pod="kube-system/konnectivity-agent-jz66f"
Sep 17 17:56:48.462905 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.462722    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c5297750-e3a6-4690-9f70-4be7eac03a96-serviceca\") pod \"node-ca-6pxfw\" (UID: \"c5297750-e3a6-4690-9f70-4be7eac03a96\") " pod="openshift-image-registry/node-ca-6pxfw"
Sep 17 17:56:48.462905 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.462799    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/56b9ff80-4b68-4449-bd38-e2e4d2de6aa2-hosts-file\") pod \"node-resolver-4fdj7\" (UID: \"56b9ff80-4b68-4449-bd38-e2e4d2de6aa2\") " pod="openshift-dns/node-resolver-4fdj7"
Sep 17 17:56:48.462905 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.462807    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjvjg\" (UniqueName: \"kubernetes.io/projected/c5297750-e3a6-4690-9f70-4be7eac03a96-kube-api-access-fjvjg\") pod \"node-ca-6pxfw\" (UID: \"c5297750-e3a6-4690-9f70-4be7eac03a96\") " pod="openshift-image-registry/node-ca-6pxfw"
Sep 17 17:56:48.462905 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.462864    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r49v\" (UniqueName: \"kubernetes.io/projected/4468ff09-5965-493f-ab71-8ccfb33fbb61-kube-api-access-4r49v\") pod \"iptables-alerter-p6z75\" (UID: \"4468ff09-5965-493f-ab71-8ccfb33fbb61\") " pod="openshift-network-operator/iptables-alerter-p6z75"
Sep 17 17:56:48.463156 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.462908    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56b9ff80-4b68-4449-bd38-e2e4d2de6aa2-tmp-dir\") pod \"node-resolver-4fdj7\" (UID: \"56b9ff80-4b68-4449-bd38-e2e4d2de6aa2\") " pod="openshift-dns/node-resolver-4fdj7"
Sep 17 17:56:48.463156 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.462932    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b281f20e-0314-4794-98c1-50e5cba38212-agent-certs\") pod \"konnectivity-agent-jz66f\" (UID: \"b281f20e-0314-4794-98c1-50e5cba38212\") " pod="kube-system/konnectivity-agent-jz66f"
Sep 17 17:56:48.463156 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.462971    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c5297750-e3a6-4690-9f70-4be7eac03a96-host\") pod \"node-ca-6pxfw\" (UID: \"c5297750-e3a6-4690-9f70-4be7eac03a96\") " pod="openshift-image-registry/node-ca-6pxfw"
Sep 17 17:56:48.463156 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.463020    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92plc\" (UniqueName: \"kubernetes.io/projected/56b9ff80-4b68-4449-bd38-e2e4d2de6aa2-kube-api-access-92plc\") pod \"node-resolver-4fdj7\" (UID: \"56b9ff80-4b68-4449-bd38-e2e4d2de6aa2\") " pod="openshift-dns/node-resolver-4fdj7"
Sep 17 17:56:48.463156 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.463043    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4468ff09-5965-493f-ab71-8ccfb33fbb61-host-slash\") pod \"iptables-alerter-p6z75\" (UID: \"4468ff09-5965-493f-ab71-8ccfb33fbb61\") " pod="openshift-network-operator/iptables-alerter-p6z75"
Sep 17 17:56:48.463156 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.463108    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c5297750-e3a6-4690-9f70-4be7eac03a96-host\") pod \"node-ca-6pxfw\" (UID: \"c5297750-e3a6-4690-9f70-4be7eac03a96\") " pod="openshift-image-registry/node-ca-6pxfw"
Sep 17 17:56:48.463425 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.463198    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c5297750-e3a6-4690-9f70-4be7eac03a96-serviceca\") pod \"node-ca-6pxfw\" (UID: \"c5297750-e3a6-4690-9f70-4be7eac03a96\") " pod="openshift-image-registry/node-ca-6pxfw"
Sep 17 17:56:48.463425 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.463263    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4468ff09-5965-493f-ab71-8ccfb33fbb61-host-slash\") pod \"iptables-alerter-p6z75\" (UID: \"4468ff09-5965-493f-ab71-8ccfb33fbb61\") " pod="openshift-network-operator/iptables-alerter-p6z75"
Sep 17 17:56:48.463425 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.463340    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4468ff09-5965-493f-ab71-8ccfb33fbb61-iptables-alerter-script\") pod \"iptables-alerter-p6z75\" (UID: \"4468ff09-5965-493f-ab71-8ccfb33fbb61\") " pod="openshift-network-operator/iptables-alerter-p6z75"
Sep 17 17:56:48.463425 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.463372    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b281f20e-0314-4794-98c1-50e5cba38212-konnectivity-ca\") pod \"konnectivity-agent-jz66f\" (UID: \"b281f20e-0314-4794-98c1-50e5cba38212\") " pod="kube-system/konnectivity-agent-jz66f"
Sep 17 17:56:48.463558 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.463429    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56b9ff80-4b68-4449-bd38-e2e4d2de6aa2-tmp-dir\") pod \"node-resolver-4fdj7\" (UID: \"56b9ff80-4b68-4449-bd38-e2e4d2de6aa2\") " pod="openshift-dns/node-resolver-4fdj7"
Sep 17 17:56:48.465350 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.465331    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b281f20e-0314-4794-98c1-50e5cba38212-agent-certs\") pod \"konnectivity-agent-jz66f\" (UID: \"b281f20e-0314-4794-98c1-50e5cba38212\") " pod="kube-system/konnectivity-agent-jz66f"
Sep 17 17:56:48.476653 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.476607    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjvjg\" (UniqueName: \"kubernetes.io/projected/c5297750-e3a6-4690-9f70-4be7eac03a96-kube-api-access-fjvjg\") pod \"node-ca-6pxfw\" (UID: \"c5297750-e3a6-4690-9f70-4be7eac03a96\") " pod="openshift-image-registry/node-ca-6pxfw"
Sep 17 17:56:48.476941 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.476921    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r49v\" (UniqueName: \"kubernetes.io/projected/4468ff09-5965-493f-ab71-8ccfb33fbb61-kube-api-access-4r49v\") pod \"iptables-alerter-p6z75\" (UID: \"4468ff09-5965-493f-ab71-8ccfb33fbb61\") " pod="openshift-network-operator/iptables-alerter-p6z75"
Sep 17 17:56:48.476941 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.476934    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92plc\" (UniqueName: \"kubernetes.io/projected/56b9ff80-4b68-4449-bd38-e2e4d2de6aa2-kube-api-access-92plc\") pod \"node-resolver-4fdj7\" (UID: \"56b9ff80-4b68-4449-bd38-e2e4d2de6aa2\") " pod="openshift-dns/node-resolver-4fdj7"
Sep 17 17:56:48.485965 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.485917    2553 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 17:51:47 +0000 UTC" deadline="2028-05-21 12:23:35.191963793 +0000 UTC"
Sep 17 17:56:48.485965 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.485960    2553 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14682h26m46.706006134s"
Sep 17 17:56:48.520457 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:48.520415    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01e1e8be5fa1c8721ccc54011f518b1a.slice/crio-628e8ca2f7b5781bbd498f577421993a260ae6b94541cf2e4f50dc4c8f73151e WatchSource:0}: Error finding container 628e8ca2f7b5781bbd498f577421993a260ae6b94541cf2e4f50dc4c8f73151e: Status 404 returned error can't find the container with id 628e8ca2f7b5781bbd498f577421993a260ae6b94541cf2e4f50dc4c8f73151e
Sep 17 17:56:48.522222 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:48.522193    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c35fbfd6aae0a9c81a0ec1cdc209b72.slice/crio-e604a6c79e93596b354abfa3731c3e8d5525dbb058b57c72c7785541099dcabd WatchSource:0}: Error finding container e604a6c79e93596b354abfa3731c3e8d5525dbb058b57c72c7785541099dcabd: Status 404 returned error can't find the container with id e604a6c79e93596b354abfa3731c3e8d5525dbb058b57c72c7785541099dcabd
Sep 17 17:56:48.527680 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.527660    2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:56:48.537728 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.537702    2553 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:56:48.578823 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.578792    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gg9tp"
Sep 17 17:56:48.587160 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:48.587132    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f19101a_6f40_4457_8f18_9c7dc294323b.slice/crio-b30228a49fa462020bad3242072809491aa498a0a3c2080a3abbaba57c25d901 WatchSource:0}: Error finding container b30228a49fa462020bad3242072809491aa498a0a3c2080a3abbaba57c25d901: Status 404 returned error can't find the container with id b30228a49fa462020bad3242072809491aa498a0a3c2080a3abbaba57c25d901
Sep 17 17:56:48.594374 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.594353    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bmjrp"
Sep 17 17:56:48.602233 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:48.602198    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e137835_2bdc_4b32_aa6b_64635bdc26d7.slice/crio-5edcb98b4b5c5c5a76bf95b430b78f973a77a6dd5e5099e53f5f6e0565ea1957 WatchSource:0}: Error finding container 5edcb98b4b5c5c5a76bf95b430b78f973a77a6dd5e5099e53f5f6e0565ea1957: Status 404 returned error can't find the container with id 5edcb98b4b5c5c5a76bf95b430b78f973a77a6dd5e5099e53f5f6e0565ea1957
Sep 17 17:56:48.610207 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.610182    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-t4pkh"
Sep 17 17:56:48.617545 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:48.617517    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7c581ed_7ab3_4a26_930c_34fdf021affd.slice/crio-217fd0ebeaa8a29fbd81aee356c7631c11ee0577e73f07e01c221c89ada42ed8 WatchSource:0}: Error finding container 217fd0ebeaa8a29fbd81aee356c7631c11ee0577e73f07e01c221c89ada42ed8: Status 404 returned error can't find the container with id 217fd0ebeaa8a29fbd81aee356c7631c11ee0577e73f07e01c221c89ada42ed8
Sep 17 17:56:48.625666 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.625640    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tjrts"
Sep 17 17:56:48.634174 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:48.634147    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod855a7f82_274d_4b6d_bcde_0c6f4c19016b.slice/crio-c4c264bc5bec59dd1a6a8020bc660c06bfe490946fe2986e187d41fc3ff82402 WatchSource:0}: Error finding container c4c264bc5bec59dd1a6a8020bc660c06bfe490946fe2986e187d41fc3ff82402: Status 404 returned error can't find the container with id c4c264bc5bec59dd1a6a8020bc660c06bfe490946fe2986e187d41fc3ff82402
Sep 17 17:56:48.640593 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.640569    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-btmqz"
Sep 17 17:56:48.647852 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:48.647823    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod622ee5be_fb50_42b6_ae23_42900b7c2d98.slice/crio-81da29683fcee5fdc47f4bc7a1b2c57fc19ca84a7a5e4f94e6ab2c99acfcd522 WatchSource:0}: Error finding container 81da29683fcee5fdc47f4bc7a1b2c57fc19ca84a7a5e4f94e6ab2c99acfcd522: Status 404 returned error can't find the container with id 81da29683fcee5fdc47f4bc7a1b2c57fc19ca84a7a5e4f94e6ab2c99acfcd522
Sep 17 17:56:48.661183 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.661152    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6pxfw"
Sep 17 17:56:48.669899 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:48.669863    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5297750_e3a6_4690_9f70_4be7eac03a96.slice/crio-9dc59f0d19bebeb188bf6d03b18d5d6e8a9653578ca438b3eaef2a69ad95e002 WatchSource:0}: Error finding container 9dc59f0d19bebeb188bf6d03b18d5d6e8a9653578ca438b3eaef2a69ad95e002: Status 404 returned error can't find the container with id 9dc59f0d19bebeb188bf6d03b18d5d6e8a9653578ca438b3eaef2a69ad95e002
Sep 17 17:56:48.689444 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.689416    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4fdj7"
Sep 17 17:56:48.698593 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:48.698550    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56b9ff80_4b68_4449_bd38_e2e4d2de6aa2.slice/crio-2bae7e6d8cb95548e3062377505a8fbc9579f293458484d266f5cf6f689920e7 WatchSource:0}: Error finding container 2bae7e6d8cb95548e3062377505a8fbc9579f293458484d266f5cf6f689920e7: Status 404 returned error can't find the container with id 2bae7e6d8cb95548e3062377505a8fbc9579f293458484d266f5cf6f689920e7
Sep 17 17:56:48.707946 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.707924    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jz66f"
Sep 17 17:56:48.714315 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.714293    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-p6z75"
Sep 17 17:56:48.716396 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:48.716369    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb281f20e_0314_4794_98c1_50e5cba38212.slice/crio-bb1b33b1df8d06ac3faf2e11d7e5d5e326acbb302c4795baaab8c464c06995aa WatchSource:0}: Error finding container bb1b33b1df8d06ac3faf2e11d7e5d5e326acbb302c4795baaab8c464c06995aa: Status 404 returned error can't find the container with id bb1b33b1df8d06ac3faf2e11d7e5d5e326acbb302c4795baaab8c464c06995aa
Sep 17 17:56:48.721889 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:56:48.721866    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4468ff09_5965_493f_ab71_8ccfb33fbb61.slice/crio-fc048c9b17013e0699a2414f80b1ad81719bcb482dc93116da2e342d7d5142c8 WatchSource:0}: Error finding container fc048c9b17013e0699a2414f80b1ad81719bcb482dc93116da2e342d7d5142c8: Status 404 returned error can't find the container with id fc048c9b17013e0699a2414f80b1ad81719bcb482dc93116da2e342d7d5142c8
Sep 17 17:56:48.723916 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:48.723896    2553 kuberuntime_manager.go:1459] "Unhandled Error" err="container iptables-alerter start failed in pod iptables-alerter-p6z75_openshift-network-operator(4468ff09-5965-493f-ab71-8ccfb33fbb61): ErrImagePull: pull QPS exceeded" logger="UnhandledError"
Sep 17 17:56:48.723963 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:48.723934    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"iptables-alerter\" with ErrImagePull: \"pull QPS exceeded\"" pod="openshift-network-operator/iptables-alerter-p6z75" podUID="4468ff09-5965-493f-ab71-8ccfb33fbb61"
Sep 17 17:56:48.865659 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.865547    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb-original-pull-secret\") pod \"global-pull-secret-syncer-xr8hx\" (UID: \"45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb\") " pod="kube-system/global-pull-secret-syncer-xr8hx"
Sep 17 17:56:48.865659 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.865611    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfc1cb73-d5de-4902-b16c-5744271855c5-metrics-certs\") pod \"network-metrics-daemon-5gdn2\" (UID: \"dfc1cb73-d5de-4902-b16c-5744271855c5\") " pod="openshift-multus/network-metrics-daemon-5gdn2"
Sep 17 17:56:48.865883 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:48.865776    2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:56:48.865883 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:48.865837    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfc1cb73-d5de-4902-b16c-5744271855c5-metrics-certs podName:dfc1cb73-d5de-4902-b16c-5744271855c5 nodeName:}" failed. No retries permitted until 2026-09-17 17:56:49.865821228 +0000 UTC m=+4.561911889 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dfc1cb73-d5de-4902-b16c-5744271855c5-metrics-certs") pod "network-metrics-daemon-5gdn2" (UID: "dfc1cb73-d5de-4902-b16c-5744271855c5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:56:48.866246 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:48.866225    2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:56:48.866336 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:48.866277    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb-original-pull-secret podName:45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb nodeName:}" failed. No retries permitted until 2026-09-17 17:56:49.866263199 +0000 UTC m=+4.562353875 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb-original-pull-secret") pod "global-pull-secret-syncer-xr8hx" (UID: "45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:56:48.964030 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.963995    2553 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:56:48.966545 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:48.966508    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4p88\" (UniqueName: \"kubernetes.io/projected/cb953471-4308-48e7-9316-6b0bdf2b8738-kube-api-access-x4p88\") pod \"network-check-target-lkn4v\" (UID: \"cb953471-4308-48e7-9316-6b0bdf2b8738\") " pod="openshift-network-diagnostics/network-check-target-lkn4v"
Sep 17 17:56:48.966762 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:48.966745    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:56:48.966851 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:48.966771    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:56:48.966851 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:48.966786    2553 projected.go:196] Error preparing data for projected volume kube-api-access-x4p88 for pod openshift-network-diagnostics/network-check-target-lkn4v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:56:48.966851 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:48.966844    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb953471-4308-48e7-9316-6b0bdf2b8738-kube-api-access-x4p88 podName:cb953471-4308-48e7-9316-6b0bdf2b8738 nodeName:}" failed. No retries permitted until 2026-09-17 17:56:49.966825127 +0000 UTC m=+4.662915806 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-x4p88" (UniqueName: "kubernetes.io/projected/cb953471-4308-48e7-9316-6b0bdf2b8738-kube-api-access-x4p88") pod "network-check-target-lkn4v" (UID: "cb953471-4308-48e7-9316-6b0bdf2b8738") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:56:49.344303 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:49.344238    2553 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 17:51:48 +0000 UTC" deadline="2028-05-10 01:30:13.894635227 +0000 UTC"
Sep 17 17:56:49.344303 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:49.344291    2553 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14407h33m24.550348068s"
Sep 17 17:56:49.373040 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:49.373009    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gdn2"
Sep 17 17:56:49.373193 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:49.373162    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5gdn2" podUID="dfc1cb73-d5de-4902-b16c-5744271855c5"
Sep 17 17:56:49.400179 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:49.400118    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4fdj7" event={"ID":"56b9ff80-4b68-4449-bd38-e2e4d2de6aa2","Type":"ContainerStarted","Data":"2bae7e6d8cb95548e3062377505a8fbc9579f293458484d266f5cf6f689920e7"}
Sep 17 17:56:49.408247 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:49.408207    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6pxfw" event={"ID":"c5297750-e3a6-4690-9f70-4be7eac03a96","Type":"ContainerStarted","Data":"9dc59f0d19bebeb188bf6d03b18d5d6e8a9653578ca438b3eaef2a69ad95e002"}
Sep 17 17:56:49.419945 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:49.419893    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-btmqz" event={"ID":"622ee5be-fb50-42b6-ae23-42900b7c2d98","Type":"ContainerStarted","Data":"81da29683fcee5fdc47f4bc7a1b2c57fc19ca84a7a5e4f94e6ab2c99acfcd522"}
Sep 17 17:56:49.438053 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:49.437957    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tjrts" event={"ID":"855a7f82-274d-4b6d-bcde-0c6f4c19016b","Type":"ContainerStarted","Data":"c4c264bc5bec59dd1a6a8020bc660c06bfe490946fe2986e187d41fc3ff82402"}
Sep 17 17:56:49.445312 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:49.445200    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bmjrp" event={"ID":"4e137835-2bdc-4b32-aa6b-64635bdc26d7","Type":"ContainerStarted","Data":"5edcb98b4b5c5c5a76bf95b430b78f973a77a6dd5e5099e53f5f6e0565ea1957"}
Sep 17 17:56:49.450574 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:49.450452    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-95.ec2.internal" event={"ID":"01e1e8be5fa1c8721ccc54011f518b1a","Type":"ContainerStarted","Data":"628e8ca2f7b5781bbd498f577421993a260ae6b94541cf2e4f50dc4c8f73151e"}
Sep 17 17:56:49.463227 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:49.463183    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-p6z75" event={"ID":"4468ff09-5965-493f-ab71-8ccfb33fbb61","Type":"ContainerStarted","Data":"fc048c9b17013e0699a2414f80b1ad81719bcb482dc93116da2e342d7d5142c8"}
Sep 17 17:56:49.474351 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:49.474309    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-t4pkh" event={"ID":"b7c581ed-7ab3-4a26-930c-34fdf021affd","Type":"ContainerStarted","Data":"217fd0ebeaa8a29fbd81aee356c7631c11ee0577e73f07e01c221c89ada42ed8"}
Sep 17 17:56:49.476536 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:49.476498    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"iptables-alerter\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8946e875b8c70aba812220ab6f6607097da88ae08ef6bebdabadea24dd62613d\\\": ErrImagePull: pull QPS exceeded\"" pod="openshift-network-operator/iptables-alerter-p6z75" podUID="4468ff09-5965-493f-ab71-8ccfb33fbb61"
Sep 17 17:56:49.494018 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:49.493977    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gg9tp" event={"ID":"3f19101a-6f40-4457-8f18-9c7dc294323b","Type":"ContainerStarted","Data":"b30228a49fa462020bad3242072809491aa498a0a3c2080a3abbaba57c25d901"}
Sep 17 17:56:49.510677 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:49.510633    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-95.ec2.internal" event={"ID":"7c35fbfd6aae0a9c81a0ec1cdc209b72","Type":"ContainerStarted","Data":"e604a6c79e93596b354abfa3731c3e8d5525dbb058b57c72c7785541099dcabd"}
Sep 17 17:56:49.528336 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:49.528240    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jz66f" event={"ID":"b281f20e-0314-4794-98c1-50e5cba38212","Type":"ContainerStarted","Data":"bb1b33b1df8d06ac3faf2e11d7e5d5e326acbb302c4795baaab8c464c06995aa"}
Sep 17 17:56:49.872851 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:49.872759    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb-original-pull-secret\") pod \"global-pull-secret-syncer-xr8hx\" (UID: \"45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb\") " pod="kube-system/global-pull-secret-syncer-xr8hx"
Sep 17 17:56:49.872851 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:49.872817    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfc1cb73-d5de-4902-b16c-5744271855c5-metrics-certs\") pod \"network-metrics-daemon-5gdn2\" (UID: \"dfc1cb73-d5de-4902-b16c-5744271855c5\") " pod="openshift-multus/network-metrics-daemon-5gdn2"
Sep 17 17:56:49.873075 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:49.872969    2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:56:49.873075 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:49.873016    2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:56:49.873075 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:49.873034    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb-original-pull-secret podName:45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb nodeName:}" failed. No retries permitted until 2026-09-17 17:56:51.873015179 +0000 UTC m=+6.569105849 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb-original-pull-secret") pod "global-pull-secret-syncer-xr8hx" (UID: "45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:56:49.873245 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:49.873079    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfc1cb73-d5de-4902-b16c-5744271855c5-metrics-certs podName:dfc1cb73-d5de-4902-b16c-5744271855c5 nodeName:}" failed. No retries permitted until 2026-09-17 17:56:51.873061313 +0000 UTC m=+6.569151981 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dfc1cb73-d5de-4902-b16c-5744271855c5-metrics-certs") pod "network-metrics-daemon-5gdn2" (UID: "dfc1cb73-d5de-4902-b16c-5744271855c5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:56:49.973769 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:49.973728    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4p88\" (UniqueName: \"kubernetes.io/projected/cb953471-4308-48e7-9316-6b0bdf2b8738-kube-api-access-x4p88\") pod \"network-check-target-lkn4v\" (UID: \"cb953471-4308-48e7-9316-6b0bdf2b8738\") " pod="openshift-network-diagnostics/network-check-target-lkn4v"
Sep 17 17:56:49.973957 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:49.973922    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:56:49.973957 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:49.973949    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:56:49.974101 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:49.973963    2553 projected.go:196] Error preparing data for projected volume kube-api-access-x4p88 for pod openshift-network-diagnostics/network-check-target-lkn4v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:56:49.974101 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:49.974025    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb953471-4308-48e7-9316-6b0bdf2b8738-kube-api-access-x4p88 podName:cb953471-4308-48e7-9316-6b0bdf2b8738 nodeName:}" failed. No retries permitted until 2026-09-17 17:56:51.974006701 +0000 UTC m=+6.670097376 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-x4p88" (UniqueName: "kubernetes.io/projected/cb953471-4308-48e7-9316-6b0bdf2b8738-kube-api-access-x4p88") pod "network-check-target-lkn4v" (UID: "cb953471-4308-48e7-9316-6b0bdf2b8738") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:56:50.344750 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:50.344694    2553 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 17:51:48 +0000 UTC" deadline="2028-04-09 13:31:47.629176652 +0000 UTC"
Sep 17 17:56:50.344750 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:50.344743    2553 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13675h34m57.284438395s"
Sep 17 17:56:50.371358 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:50.371320    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lkn4v"
Sep 17 17:56:50.371550 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:50.371458    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lkn4v" podUID="cb953471-4308-48e7-9316-6b0bdf2b8738"
Sep 17 17:56:50.371955 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:50.371931    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xr8hx"
Sep 17 17:56:50.372082 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:50.372041    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xr8hx" podUID="45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb"
Sep 17 17:56:51.371161 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:51.371123    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gdn2"
Sep 17 17:56:51.371630 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:51.371289    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5gdn2" podUID="dfc1cb73-d5de-4902-b16c-5744271855c5"
Sep 17 17:56:51.891976 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:51.891933    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb-original-pull-secret\") pod \"global-pull-secret-syncer-xr8hx\" (UID: \"45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb\") " pod="kube-system/global-pull-secret-syncer-xr8hx"
Sep 17 17:56:51.892176 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:51.891998    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfc1cb73-d5de-4902-b16c-5744271855c5-metrics-certs\") pod \"network-metrics-daemon-5gdn2\" (UID: \"dfc1cb73-d5de-4902-b16c-5744271855c5\") " pod="openshift-multus/network-metrics-daemon-5gdn2"
Sep 17 17:56:51.892176 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:51.892118    2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:56:51.892285 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:51.892182    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfc1cb73-d5de-4902-b16c-5744271855c5-metrics-certs podName:dfc1cb73-d5de-4902-b16c-5744271855c5 nodeName:}" failed. No retries permitted until 2026-09-17 17:56:55.892161891 +0000 UTC m=+10.588252568 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dfc1cb73-d5de-4902-b16c-5744271855c5-metrics-certs") pod "network-metrics-daemon-5gdn2" (UID: "dfc1cb73-d5de-4902-b16c-5744271855c5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:56:51.892702 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:51.892680    2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:56:51.892826 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:51.892751    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb-original-pull-secret podName:45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb nodeName:}" failed. No retries permitted until 2026-09-17 17:56:55.892727213 +0000 UTC m=+10.588817889 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb-original-pull-secret") pod "global-pull-secret-syncer-xr8hx" (UID: "45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:56:51.993426 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:51.993379    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4p88\" (UniqueName: \"kubernetes.io/projected/cb953471-4308-48e7-9316-6b0bdf2b8738-kube-api-access-x4p88\") pod \"network-check-target-lkn4v\" (UID: \"cb953471-4308-48e7-9316-6b0bdf2b8738\") " pod="openshift-network-diagnostics/network-check-target-lkn4v"
Sep 17 17:56:51.993603 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:51.993575    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:56:51.993682 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:51.993651    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:56:51.993737 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:51.993691    2553 projected.go:196] Error preparing data for projected volume kube-api-access-x4p88 for pod openshift-network-diagnostics/network-check-target-lkn4v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:56:51.993841 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:51.993764    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb953471-4308-48e7-9316-6b0bdf2b8738-kube-api-access-x4p88 podName:cb953471-4308-48e7-9316-6b0bdf2b8738 nodeName:}" failed. No retries permitted until 2026-09-17 17:56:55.993743566 +0000 UTC m=+10.689834244 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-x4p88" (UniqueName: "kubernetes.io/projected/cb953471-4308-48e7-9316-6b0bdf2b8738-kube-api-access-x4p88") pod "network-check-target-lkn4v" (UID: "cb953471-4308-48e7-9316-6b0bdf2b8738") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:56:52.371456 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:52.370743    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lkn4v"
Sep 17 17:56:52.371456 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:52.370877    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lkn4v" podUID="cb953471-4308-48e7-9316-6b0bdf2b8738"
Sep 17 17:56:52.371456 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:52.371253    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xr8hx"
Sep 17 17:56:52.371456 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:52.371342    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xr8hx" podUID="45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb"
Sep 17 17:56:53.370697 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:53.370630    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gdn2"
Sep 17 17:56:53.371076 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:53.371043    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5gdn2" podUID="dfc1cb73-d5de-4902-b16c-5744271855c5"
Sep 17 17:56:54.371262 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:54.371224    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lkn4v"
Sep 17 17:56:54.371750 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:54.371394    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lkn4v" podUID="cb953471-4308-48e7-9316-6b0bdf2b8738"
Sep 17 17:56:54.371750 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:54.371438    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xr8hx"
Sep 17 17:56:54.371750 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:54.371526    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xr8hx" podUID="45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb"
Sep 17 17:56:55.374174 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:55.372844    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gdn2"
Sep 17 17:56:55.374174 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:55.373041    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5gdn2" podUID="dfc1cb73-d5de-4902-b16c-5744271855c5"
Sep 17 17:56:55.930786 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:55.930747    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb-original-pull-secret\") pod \"global-pull-secret-syncer-xr8hx\" (UID: \"45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb\") " pod="kube-system/global-pull-secret-syncer-xr8hx"
Sep 17 17:56:55.930974 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:55.930800    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfc1cb73-d5de-4902-b16c-5744271855c5-metrics-certs\") pod \"network-metrics-daemon-5gdn2\" (UID: \"dfc1cb73-d5de-4902-b16c-5744271855c5\") " pod="openshift-multus/network-metrics-daemon-5gdn2"
Sep 17 17:56:55.930974 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:55.930924    2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:56:55.930974 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:55.930925    2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:56:55.931136 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:55.930979    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfc1cb73-d5de-4902-b16c-5744271855c5-metrics-certs podName:dfc1cb73-d5de-4902-b16c-5744271855c5 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:03.930962913 +0000 UTC m=+18.627053576 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dfc1cb73-d5de-4902-b16c-5744271855c5-metrics-certs") pod "network-metrics-daemon-5gdn2" (UID: "dfc1cb73-d5de-4902-b16c-5744271855c5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:56:55.931136 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:55.931038    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb-original-pull-secret podName:45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb nodeName:}" failed. No retries permitted until 2026-09-17 17:57:03.931020555 +0000 UTC m=+18.627111221 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb-original-pull-secret") pod "global-pull-secret-syncer-xr8hx" (UID: "45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:56:56.031938 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:56.031277    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4p88\" (UniqueName: \"kubernetes.io/projected/cb953471-4308-48e7-9316-6b0bdf2b8738-kube-api-access-x4p88\") pod \"network-check-target-lkn4v\" (UID: \"cb953471-4308-48e7-9316-6b0bdf2b8738\") " pod="openshift-network-diagnostics/network-check-target-lkn4v"
Sep 17 17:56:56.031938 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:56.031479    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:56:56.031938 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:56.031502    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:56:56.031938 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:56.031519    2553 projected.go:196] Error preparing data for projected volume kube-api-access-x4p88 for pod openshift-network-diagnostics/network-check-target-lkn4v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:56:56.031938 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:56.031579    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb953471-4308-48e7-9316-6b0bdf2b8738-kube-api-access-x4p88 podName:cb953471-4308-48e7-9316-6b0bdf2b8738 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:04.031559884 +0000 UTC m=+18.727650569 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-x4p88" (UniqueName: "kubernetes.io/projected/cb953471-4308-48e7-9316-6b0bdf2b8738-kube-api-access-x4p88") pod "network-check-target-lkn4v" (UID: "cb953471-4308-48e7-9316-6b0bdf2b8738") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:56:56.371466 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:56.371379    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lkn4v"
Sep 17 17:56:56.371466 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:56.371426    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xr8hx"
Sep 17 17:56:56.371691 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:56.371512    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lkn4v" podUID="cb953471-4308-48e7-9316-6b0bdf2b8738"
Sep 17 17:56:56.371691 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:56.371651    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xr8hx" podUID="45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb"
Sep 17 17:56:57.372175 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:57.371926    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gdn2"
Sep 17 17:56:57.372734 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:57.372281    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5gdn2" podUID="dfc1cb73-d5de-4902-b16c-5744271855c5"
Sep 17 17:56:58.370972 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:58.370932    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lkn4v"
Sep 17 17:56:58.370972 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:58.370963    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xr8hx"
Sep 17 17:56:58.371242 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:58.371064    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lkn4v" podUID="cb953471-4308-48e7-9316-6b0bdf2b8738"
Sep 17 17:56:58.371242 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:58.371180    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xr8hx" podUID="45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb"
Sep 17 17:56:59.370944 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:56:59.370911    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gdn2"
Sep 17 17:56:59.371338 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:56:59.371030    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5gdn2" podUID="dfc1cb73-d5de-4902-b16c-5744271855c5"
Sep 17 17:57:00.370402 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:00.370368    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xr8hx"
Sep 17 17:57:00.370402 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:00.370395    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lkn4v"
Sep 17 17:57:00.370785 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:57:00.370470    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xr8hx" podUID="45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb"
Sep 17 17:57:00.370785 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:57:00.370612    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lkn4v" podUID="cb953471-4308-48e7-9316-6b0bdf2b8738"
Sep 17 17:57:01.371391 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:01.371356    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gdn2"
Sep 17 17:57:01.371830 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:57:01.371472    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5gdn2" podUID="dfc1cb73-d5de-4902-b16c-5744271855c5"
Sep 17 17:57:02.371291 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:02.371256    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xr8hx"
Sep 17 17:57:02.371497 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:02.371375    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lkn4v"
Sep 17 17:57:02.371497 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:57:02.371384    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xr8hx" podUID="45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb"
Sep 17 17:57:02.371943 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:57:02.371490    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lkn4v" podUID="cb953471-4308-48e7-9316-6b0bdf2b8738"
Sep 17 17:57:03.371406 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:03.371309    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gdn2"
Sep 17 17:57:03.371554 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:57:03.371479    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5gdn2" podUID="dfc1cb73-d5de-4902-b16c-5744271855c5"
Sep 17 17:57:03.987681 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:03.987634    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb-original-pull-secret\") pod \"global-pull-secret-syncer-xr8hx\" (UID: \"45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb\") " pod="kube-system/global-pull-secret-syncer-xr8hx"
Sep 17 17:57:03.987876 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:03.987705    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfc1cb73-d5de-4902-b16c-5744271855c5-metrics-certs\") pod \"network-metrics-daemon-5gdn2\" (UID: \"dfc1cb73-d5de-4902-b16c-5744271855c5\") " pod="openshift-multus/network-metrics-daemon-5gdn2"
Sep 17 17:57:03.987876 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:57:03.987797    2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:57:03.987876 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:57:03.987826    2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:57:03.988006 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:57:03.987889    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb-original-pull-secret podName:45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb nodeName:}" failed. No retries permitted until 2026-09-17 17:57:19.987871768 +0000 UTC m=+34.683962449 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb-original-pull-secret") pod "global-pull-secret-syncer-xr8hx" (UID: "45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:57:03.988006 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:57:03.987905    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfc1cb73-d5de-4902-b16c-5744271855c5-metrics-certs podName:dfc1cb73-d5de-4902-b16c-5744271855c5 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:19.987897554 +0000 UTC m=+34.683988216 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dfc1cb73-d5de-4902-b16c-5744271855c5-metrics-certs") pod "network-metrics-daemon-5gdn2" (UID: "dfc1cb73-d5de-4902-b16c-5744271855c5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:57:04.089004 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:04.088967    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4p88\" (UniqueName: \"kubernetes.io/projected/cb953471-4308-48e7-9316-6b0bdf2b8738-kube-api-access-x4p88\") pod \"network-check-target-lkn4v\" (UID: \"cb953471-4308-48e7-9316-6b0bdf2b8738\") " pod="openshift-network-diagnostics/network-check-target-lkn4v"
Sep 17 17:57:04.089204 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:57:04.089115    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:57:04.089204 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:57:04.089134    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:57:04.089204 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:57:04.089147    2553 projected.go:196] Error preparing data for projected volume kube-api-access-x4p88 for pod openshift-network-diagnostics/network-check-target-lkn4v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:57:04.089204 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:57:04.089200    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb953471-4308-48e7-9316-6b0bdf2b8738-kube-api-access-x4p88 podName:cb953471-4308-48e7-9316-6b0bdf2b8738 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:20.089186714 +0000 UTC m=+34.785277375 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-x4p88" (UniqueName: "kubernetes.io/projected/cb953471-4308-48e7-9316-6b0bdf2b8738-kube-api-access-x4p88") pod "network-check-target-lkn4v" (UID: "cb953471-4308-48e7-9316-6b0bdf2b8738") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:57:04.371199 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:04.371117    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lkn4v"
Sep 17 17:57:04.371199 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:04.371144    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xr8hx"
Sep 17 17:57:04.371422 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:57:04.371238    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lkn4v" podUID="cb953471-4308-48e7-9316-6b0bdf2b8738"
Sep 17 17:57:04.371422 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:57:04.371356    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xr8hx" podUID="45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb"
Sep 17 17:57:05.370768 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:05.370731    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gdn2"
Sep 17 17:57:05.371341 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:57:05.370857    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5gdn2" podUID="dfc1cb73-d5de-4902-b16c-5744271855c5"
Sep 17 17:57:06.370442 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:06.370369    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lkn4v"
Sep 17 17:57:06.370652 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:57:06.370499    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lkn4v" podUID="cb953471-4308-48e7-9316-6b0bdf2b8738"
Sep 17 17:57:06.370652 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:06.370503    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xr8hx"
Sep 17 17:57:06.370652 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:57:06.370594    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xr8hx" podUID="45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb"
Sep 17 17:57:07.374281 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:07.373852    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gdn2"
Sep 17 17:57:07.374281 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:57:07.374004    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5gdn2" podUID="dfc1cb73-d5de-4902-b16c-5744271855c5"
Sep 17 17:57:07.585751 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:07.585495    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-btmqz_622ee5be-fb50-42b6-ae23-42900b7c2d98/ovn-acl-logging/0.log"
Sep 17 17:57:07.586552 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:07.586473    2553 generic.go:356] "Generic (PLEG): container finished" podID="622ee5be-fb50-42b6-ae23-42900b7c2d98" containerID="7d5ae2e5d3895224ac051c13607e380f50758d5d08b14877d497a10d37d50878" exitCode=1
Sep 17 17:57:07.586672 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:07.586574    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-btmqz" event={"ID":"622ee5be-fb50-42b6-ae23-42900b7c2d98","Type":"ContainerDied","Data":"7d5ae2e5d3895224ac051c13607e380f50758d5d08b14877d497a10d37d50878"}
Sep 17 17:57:07.586672 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:07.586668    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-btmqz" event={"ID":"622ee5be-fb50-42b6-ae23-42900b7c2d98","Type":"ContainerStarted","Data":"1ff91ef19c9041a868f31cdc49d19c6cd91eabc6efe340cae0b49a7ba09d9e3d"}
Sep 17 17:57:07.593161 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:07.593128    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-95.ec2.internal" event={"ID":"01e1e8be5fa1c8721ccc54011f518b1a","Type":"ContainerStarted","Data":"458e731995415ab4dbc49de4df02c85432886bf1792750e995cc5fcdd544ef34"}
Sep 17 17:57:07.596697 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:07.596584    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-t4pkh" event={"ID":"b7c581ed-7ab3-4a26-930c-34fdf021affd","Type":"ContainerStarted","Data":"8450d0461e658392357f9b17c574eaab910fba26a4b5a0d78357159c7bf7101b"}
Sep 17 17:57:07.640720 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:07.640673    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-95.ec2.internal" podStartSLOduration=19.640655945 podStartE2EDuration="19.640655945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:56:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:07.61544326 +0000 UTC m=+22.311533946" watchObservedRunningTime="2026-09-17 17:57:07.640655945 +0000 UTC m=+22.336746628"
Sep 17 17:57:07.640831 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:07.640796    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-t4pkh" podStartSLOduration=2.230676719 podStartE2EDuration="20.640790062s" totalImagesPullingTime="18.410113343s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:56:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:07.640192209 +0000 UTC m=+22.336282893" watchObservedRunningTime="2026-09-17 17:57:07.640790062 +0000 UTC m=+22.336880794"
Sep 17 17:57:08.370677 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:08.370637    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lkn4v"
Sep 17 17:57:08.370677 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:08.370666    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xr8hx"
Sep 17 17:57:08.370934 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:57:08.370751    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lkn4v" podUID="cb953471-4308-48e7-9316-6b0bdf2b8738"
Sep 17 17:57:08.370934 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:57:08.370894    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xr8hx" podUID="45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb"
Sep 17 17:57:08.600102 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:08.600073    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jz66f" event={"ID":"b281f20e-0314-4794-98c1-50e5cba38212","Type":"ContainerStarted","Data":"8b55c0be5749ffdc16ffec992f0a8dd20ec07309ab48a5b18ca0ffe9cdefee45"}
Sep 17 17:57:08.601733 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:08.601704    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4fdj7" event={"ID":"56b9ff80-4b68-4449-bd38-e2e4d2de6aa2","Type":"ContainerStarted","Data":"f12d9daa0261585766d2ea27c659f78e1def0b6e34047ba37786feeaeac87c6a"}
Sep 17 17:57:08.603886 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:08.603499    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6pxfw" event={"ID":"c5297750-e3a6-4690-9f70-4be7eac03a96","Type":"ContainerStarted","Data":"e17d69a40bf91cc48a876571dd09fd7ecea2fe861bf0d8658539a7e989b4f7e9"}
Sep 17 17:57:08.608227 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:08.608205    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-btmqz_622ee5be-fb50-42b6-ae23-42900b7c2d98/ovn-acl-logging/0.log"
Sep 17 17:57:08.608962 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:08.608938    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-btmqz" event={"ID":"622ee5be-fb50-42b6-ae23-42900b7c2d98","Type":"ContainerStarted","Data":"a78e57709a7998a7cb8bc607bc4f2c31357c350ca54ddc45f2497481240351b3"}
Sep 17 17:57:08.609047 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:08.609015    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-btmqz" event={"ID":"622ee5be-fb50-42b6-ae23-42900b7c2d98","Type":"ContainerStarted","Data":"053c7634dbdc19d02e806849007ff381c2d3ce496014537549c6749866700405"}
Sep 17 17:57:08.609115 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:08.609082    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-btmqz" event={"ID":"622ee5be-fb50-42b6-ae23-42900b7c2d98","Type":"ContainerStarted","Data":"71fc5f32cfaf6bb507fa137bcc8302ca6d7b3702abc4126fe2818dc3d59a8b7a"}
Sep 17 17:57:08.609169 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:08.609132    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-btmqz" event={"ID":"622ee5be-fb50-42b6-ae23-42900b7c2d98","Type":"ContainerStarted","Data":"8f553f6dedb688de88af5b439b29caa38ab81b2a8e6d3adb50a2174954835733"}
Sep 17 17:57:08.610647 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:08.610579    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tjrts" event={"ID":"855a7f82-274d-4b6d-bcde-0c6f4c19016b","Type":"ContainerStarted","Data":"9d95b519622f8899e82eaa733fb37518b90ae8c715fd91557b5d0ff6592e284d"}
Sep 17 17:57:08.612434 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:08.612408    2553 generic.go:356] "Generic (PLEG): container finished" podID="4e137835-2bdc-4b32-aa6b-64635bdc26d7" containerID="0674ce9859f0dd76ae484c349319dd59d0d51be4c3544aa675a3107a99021397" exitCode=0
Sep 17 17:57:08.612544 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:08.612493    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bmjrp" event={"ID":"4e137835-2bdc-4b32-aa6b-64635bdc26d7","Type":"ContainerDied","Data":"0674ce9859f0dd76ae484c349319dd59d0d51be4c3544aa675a3107a99021397"}
Sep 17 17:57:08.614862 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:08.614838    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gg9tp" event={"ID":"3f19101a-6f40-4457-8f18-9c7dc294323b","Type":"ContainerStarted","Data":"cdb7a9b06c59f3c7b492d9a46bc0c764a5fd161e018f457f548831918dc2de12"}
Sep 17 17:57:08.617379 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:08.617349    2553 generic.go:356] "Generic (PLEG): container finished" podID="7c35fbfd6aae0a9c81a0ec1cdc209b72" containerID="cc2e78fbe9a1192a2c900226e0070fabe05bab398281e4cec134d5493e06157b" exitCode=0
Sep 17 17:57:08.617650 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:08.617404    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-95.ec2.internal" event={"ID":"7c35fbfd6aae0a9c81a0ec1cdc209b72","Type":"ContainerDied","Data":"cc2e78fbe9a1192a2c900226e0070fabe05bab398281e4cec134d5493e06157b"}
Sep 17 17:57:08.627342 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:08.627293    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-jz66f" podStartSLOduration=3.215031005 podStartE2EDuration="21.627276432s" totalImagesPullingTime="18.412245427s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:56:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:08.627259888 +0000 UTC m=+23.323350584" watchObservedRunningTime="2026-09-17 17:57:08.627276432 +0000 UTC m=+23.323367118"
Sep 17 17:57:08.710210 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:08.710166    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-4fdj7" podStartSLOduration=3.259133672 podStartE2EDuration="21.710150063s" totalImagesPullingTime="18.451016391s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:56:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:08.710149953 +0000 UTC m=+23.406240636" watchObservedRunningTime="2026-09-17 17:57:08.710150063 +0000 UTC m=+23.406240747"
Sep 17 17:57:08.717314 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:08.717287    2553 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 17:57:08.746341 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:08.746296    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-tjrts" podStartSLOduration=2.851037343 podStartE2EDuration="21.746280843s" totalImagesPullingTime="18.8952435s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:56:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:08.746170927 +0000 UTC m=+23.442261610" watchObservedRunningTime="2026-09-17 17:57:08.746280843 +0000 UTC m=+23.442371527"
Sep 17 17:57:09.339152 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:09.339029    2553 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T17:57:08.717310082Z","UUID":"81949952-55ca-4403-acef-f834f3c940ad","Handler":null,"Name":"","Endpoint":""}
Sep 17 17:57:09.340934 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:09.340901    2553 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 17:57:09.341051 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:09.340943    2553 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 17:57:09.370526 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:09.370492    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gdn2"
Sep 17 17:57:09.370772 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:57:09.370644    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5gdn2" podUID="dfc1cb73-d5de-4902-b16c-5744271855c5"
Sep 17 17:57:09.622036 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:09.621997    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-p6z75" event={"ID":"4468ff09-5965-493f-ab71-8ccfb33fbb61","Type":"ContainerStarted","Data":"a58a9f87bc9bc8b30b626ede05239af5ca0c68b543decd55444430ffadb6432e"}
Sep 17 17:57:09.624579 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:09.624532    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gg9tp" event={"ID":"3f19101a-6f40-4457-8f18-9c7dc294323b","Type":"ContainerStarted","Data":"0dcd5d3fe3ed12ca0f01086ac72fd6dfd2b007c7b6b1dc1fce723cd902108274"}
Sep 17 17:57:09.624706 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:09.624596    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gg9tp" event={"ID":"3f19101a-6f40-4457-8f18-9c7dc294323b","Type":"ContainerStarted","Data":"013a043362e4eabe98be464839df38f3be8b63d77d0ef367a379a11684a20e6e"}
Sep 17 17:57:09.626889 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:09.626836    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-95.ec2.internal" event={"ID":"7c35fbfd6aae0a9c81a0ec1cdc209b72","Type":"ContainerStarted","Data":"4684cdcbfcd51370455e6287ca2133448f55a08c44ac61edcbfb071e82add57c"}
Sep 17 17:57:09.643215 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:09.643160    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-6pxfw" podStartSLOduration=4.183902874 podStartE2EDuration="22.643139114s" totalImagesPullingTime="18.45923624s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:56:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:08.78106476 +0000 UTC m=+23.477155443" watchObservedRunningTime="2026-09-17 17:57:09.643139114 +0000 UTC m=+24.339229790"
Sep 17 17:57:09.643689 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:09.643611    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-p6z75" podStartSLOduration=4.147080954 podStartE2EDuration="22.643599637s" totalImagesPullingTime="18.496518683s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:56:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=1 observedRunningTime="2026-09-17 17:57:09.64324001 +0000 UTC m=+24.339330695" watchObservedRunningTime="2026-09-17 17:57:09.643599637 +0000 UTC m=+24.339690322"
Sep 17 17:57:09.703807 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:09.703697    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gg9tp" podStartSLOduration=2.718985016 podStartE2EDuration="22.703678224s" totalImagesPullingTime="19.984693208s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:56:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:09.673776426 +0000 UTC m=+24.369867110" watchObservedRunningTime="2026-09-17 17:57:09.703678224 +0000 UTC m=+24.399768907"
Sep 17 17:57:09.704151 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:09.704120    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-95.ec2.internal" podStartSLOduration=21.704108142 podStartE2EDuration="21.704108142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:56:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:09.704068588 +0000 UTC m=+24.400159272" watchObservedRunningTime="2026-09-17 17:57:09.704108142 +0000 UTC m=+24.400198826"
Sep 17 17:57:10.370832 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:10.370739    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xr8hx"
Sep 17 17:57:10.371006 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:57:10.370871    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xr8hx" podUID="45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb"
Sep 17 17:57:10.371006 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:10.370881    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lkn4v"
Sep 17 17:57:10.371006 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:57:10.370967    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lkn4v" podUID="cb953471-4308-48e7-9316-6b0bdf2b8738"
Sep 17 17:57:10.632848 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:10.632779    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-btmqz_622ee5be-fb50-42b6-ae23-42900b7c2d98/ovn-acl-logging/0.log"
Sep 17 17:57:10.633286 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:10.633242    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-btmqz" event={"ID":"622ee5be-fb50-42b6-ae23-42900b7c2d98","Type":"ContainerStarted","Data":"78b5dab47a24fc1e1fb4d98e0d97b317fa0c4f9f8fa9feab03ce682e6a0daee2"}
Sep 17 17:57:11.371706 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:11.371435    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gdn2"
Sep 17 17:57:11.371864 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:57:11.371822    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5gdn2" podUID="dfc1cb73-d5de-4902-b16c-5744271855c5"
Sep 17 17:57:12.371127 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:12.371055    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xr8hx"
Sep 17 17:57:12.371534 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:57:12.371164    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xr8hx" podUID="45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb"
Sep 17 17:57:12.371534 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:12.371229    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lkn4v"
Sep 17 17:57:12.371534 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:57:12.371347    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lkn4v" podUID="cb953471-4308-48e7-9316-6b0bdf2b8738"
Sep 17 17:57:12.642870 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:12.642848    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-btmqz_622ee5be-fb50-42b6-ae23-42900b7c2d98/ovn-acl-logging/0.log"
Sep 17 17:57:12.643282 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:12.643255    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-btmqz" event={"ID":"622ee5be-fb50-42b6-ae23-42900b7c2d98","Type":"ContainerStarted","Data":"2ac3c65c03974d405e28f9fba5e41ae5c47714da16280a95112b64e03ed6f80c"}
Sep 17 17:57:12.643597 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:12.643523    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-btmqz"
Sep 17 17:57:12.643752 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:12.643736    2553 scope.go:117] "RemoveContainer" containerID="7d5ae2e5d3895224ac051c13607e380f50758d5d08b14877d497a10d37d50878"
Sep 17 17:57:12.661582 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:12.661556    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-btmqz"
Sep 17 17:57:13.370603 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:13.370566    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gdn2"
Sep 17 17:57:13.370850 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:57:13.370707    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5gdn2" podUID="dfc1cb73-d5de-4902-b16c-5744271855c5"
Sep 17 17:57:13.386269 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:13.386239    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-jz66f"
Sep 17 17:57:13.387181 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:13.386852    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-jz66f"
Sep 17 17:57:13.646760 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:13.646724    2553 generic.go:356] "Generic (PLEG): container finished" podID="4e137835-2bdc-4b32-aa6b-64635bdc26d7" containerID="daeacb3a06f89e85ff157aa371bb0433c5f03e473fc9a1dbfa710aaeee130020" exitCode=0
Sep 17 17:57:13.646958 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:13.646807    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bmjrp" event={"ID":"4e137835-2bdc-4b32-aa6b-64635bdc26d7","Type":"ContainerDied","Data":"daeacb3a06f89e85ff157aa371bb0433c5f03e473fc9a1dbfa710aaeee130020"}
Sep 17 17:57:13.650888 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:13.650871    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-btmqz_622ee5be-fb50-42b6-ae23-42900b7c2d98/ovn-acl-logging/0.log"
Sep 17 17:57:13.651238 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:13.651215    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-btmqz" event={"ID":"622ee5be-fb50-42b6-ae23-42900b7c2d98","Type":"ContainerStarted","Data":"3d765fe340838af9bb33ecc22143538915959cedf4a23bc7297c8f1714d957f5"}
Sep 17 17:57:13.651334 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:13.651320    2553 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 17:57:13.651372 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:13.651337    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-jz66f"
Sep 17 17:57:13.651955 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:13.651558    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-btmqz"
Sep 17 17:57:13.651955 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:13.651899    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jz66f"
Sep 17 17:57:13.666387 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:13.666364    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-btmqz"
Sep 17 17:57:13.729047 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:13.728999    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-btmqz" podStartSLOduration=8.156481864 podStartE2EDuration="26.728982826s" totalImagesPullingTime="18.572500962s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:56:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:13.710391858 +0000 UTC m=+28.406482550" watchObservedRunningTime="2026-09-17 17:57:13.728982826 +0000 UTC m=+28.425073509"
Sep 17 17:57:14.105346 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:14.105264    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-btmqz"
Sep 17 17:57:14.370451 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:14.370365    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xr8hx"
Sep 17 17:57:14.370586 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:57:14.370470    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xr8hx" podUID="45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb"
Sep 17 17:57:14.370586 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:14.370522    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lkn4v"
Sep 17 17:57:14.370684 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:57:14.370634    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lkn4v" podUID="cb953471-4308-48e7-9316-6b0bdf2b8738"
Sep 17 17:57:14.459258 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:14.459222    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5gdn2"]
Sep 17 17:57:14.459682 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:14.459344    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gdn2"
Sep 17 17:57:14.459682 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:57:14.459433    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5gdn2" podUID="dfc1cb73-d5de-4902-b16c-5744271855c5"
Sep 17 17:57:14.460795 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:14.460772    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xr8hx"]
Sep 17 17:57:14.477496 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:14.477464    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lkn4v"]
Sep 17 17:57:14.656239 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:14.656209    2553 generic.go:356] "Generic (PLEG): container finished" podID="4e137835-2bdc-4b32-aa6b-64635bdc26d7" containerID="2436bd9ae3219ae901c647843167743f9e7d27d706542d6883704090281c26ce" exitCode=0
Sep 17 17:57:14.656412 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:14.656305    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lkn4v"
Sep 17 17:57:14.656412 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:14.656312    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bmjrp" event={"ID":"4e137835-2bdc-4b32-aa6b-64635bdc26d7","Type":"ContainerDied","Data":"2436bd9ae3219ae901c647843167743f9e7d27d706542d6883704090281c26ce"}
Sep 17 17:57:14.656412 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:14.656395    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xr8hx"
Sep 17 17:57:14.656412 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:57:14.656395    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lkn4v" podUID="cb953471-4308-48e7-9316-6b0bdf2b8738"
Sep 17 17:57:14.656586 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:57:14.656558    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xr8hx" podUID="45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb"
Sep 17 17:57:15.660697 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:15.660479    2553 generic.go:356] "Generic (PLEG): container finished" podID="4e137835-2bdc-4b32-aa6b-64635bdc26d7" containerID="47387fcbe658a0a6caa777d1553900cf3464eb7e9aca44f8e1d80d48bc1e93c2" exitCode=0
Sep 17 17:57:15.661088 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:15.660565    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bmjrp" event={"ID":"4e137835-2bdc-4b32-aa6b-64635bdc26d7","Type":"ContainerDied","Data":"47387fcbe658a0a6caa777d1553900cf3464eb7e9aca44f8e1d80d48bc1e93c2"}
Sep 17 17:57:16.370696 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:16.370601    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lkn4v"
Sep 17 17:57:16.370696 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:16.370653    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gdn2"
Sep 17 17:57:16.370876 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:16.370601    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xr8hx"
Sep 17 17:57:16.370876 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:57:16.370751    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lkn4v" podUID="cb953471-4308-48e7-9316-6b0bdf2b8738"
Sep 17 17:57:16.370876 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:57:16.370827    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xr8hx" podUID="45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb"
Sep 17 17:57:16.370974 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:57:16.370908    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5gdn2" podUID="dfc1cb73-d5de-4902-b16c-5744271855c5"
Sep 17 17:57:18.370758 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:18.370697    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xr8hx"
Sep 17 17:57:18.371388 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:18.370823    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lkn4v"
Sep 17 17:57:18.371388 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:57:18.370834    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xr8hx" podUID="45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb"
Sep 17 17:57:18.371388 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:18.370857    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gdn2"
Sep 17 17:57:18.371388 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:57:18.370941    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lkn4v" podUID="cb953471-4308-48e7-9316-6b0bdf2b8738"
Sep 17 17:57:18.371388 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:57:18.371028    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5gdn2" podUID="dfc1cb73-d5de-4902-b16c-5744271855c5"
Sep 17 17:57:20.012770 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:20.012744    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb-original-pull-secret\") pod \"global-pull-secret-syncer-xr8hx\" (UID: \"45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb\") " pod="kube-system/global-pull-secret-syncer-xr8hx"
Sep 17 17:57:20.013251 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:20.012786    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfc1cb73-d5de-4902-b16c-5744271855c5-metrics-certs\") pod \"network-metrics-daemon-5gdn2\" (UID: \"dfc1cb73-d5de-4902-b16c-5744271855c5\") " pod="openshift-multus/network-metrics-daemon-5gdn2"
Sep 17 17:57:20.013251 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:57:20.012886    2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:57:20.013251 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:57:20.012889    2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:57:20.013251 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:57:20.012941    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfc1cb73-d5de-4902-b16c-5744271855c5-metrics-certs podName:dfc1cb73-d5de-4902-b16c-5744271855c5 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:52.012925276 +0000 UTC m=+66.709015940 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dfc1cb73-d5de-4902-b16c-5744271855c5-metrics-certs") pod "network-metrics-daemon-5gdn2" (UID: "dfc1cb73-d5de-4902-b16c-5744271855c5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:57:20.013251 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:57:20.012954    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb-original-pull-secret podName:45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb nodeName:}" failed. No retries permitted until 2026-09-17 17:57:52.012947952 +0000 UTC m=+66.709038614 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb-original-pull-secret") pod "global-pull-secret-syncer-xr8hx" (UID: "45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:57:20.113820 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:20.113795    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4p88\" (UniqueName: \"kubernetes.io/projected/cb953471-4308-48e7-9316-6b0bdf2b8738-kube-api-access-x4p88\") pod \"network-check-target-lkn4v\" (UID: \"cb953471-4308-48e7-9316-6b0bdf2b8738\") " pod="openshift-network-diagnostics/network-check-target-lkn4v"
Sep 17 17:57:20.113924 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:57:20.113910    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:57:20.113963 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:57:20.113930    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:57:20.113963 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:57:20.113942    2553 projected.go:196] Error preparing data for projected volume kube-api-access-x4p88 for pod openshift-network-diagnostics/network-check-target-lkn4v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:57:20.114036 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:57:20.113988    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb953471-4308-48e7-9316-6b0bdf2b8738-kube-api-access-x4p88 podName:cb953471-4308-48e7-9316-6b0bdf2b8738 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:52.113975082 +0000 UTC m=+66.810065744 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-x4p88" (UniqueName: "kubernetes.io/projected/cb953471-4308-48e7-9316-6b0bdf2b8738-kube-api-access-x4p88") pod "network-check-target-lkn4v" (UID: "cb953471-4308-48e7-9316-6b0bdf2b8738") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:57:20.197255 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:20.197233    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-95.ec2.internal" event="NodeReady"
Sep 17 17:57:20.197382 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:20.197357    2553 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 17:57:20.271080 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:20.271011    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-7pqps"]
Sep 17 17:57:20.288895 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:20.288868    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-dm5l8"]
Sep 17 17:57:20.289043 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:20.289008    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7pqps"
Sep 17 17:57:20.294339 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:20.294314    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-n54wg\""
Sep 17 17:57:20.294457 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:20.294313    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 17:57:20.294457 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:20.294450    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 17:57:20.295372 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:20.295351    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 17:57:20.310633 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:20.310602    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7pqps"]
Sep 17 17:57:20.310744 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:20.310650    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dm5l8"]
Sep 17 17:57:20.310744 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:20.310717    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dm5l8"
Sep 17 17:57:20.315011 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:20.314990    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 17:57:20.315011 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:20.314998    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5wl9d\""
Sep 17 17:57:20.315397 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:20.315381    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 17:57:20.370542 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:20.370505    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xr8hx"
Sep 17 17:57:20.370732 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:20.370607    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lkn4v"
Sep 17 17:57:20.370732 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:20.370656    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gdn2"
Sep 17 17:57:20.373771 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:20.373752    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 17:57:20.374643 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:20.374611    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-74sx5\""
Sep 17 17:57:20.374789 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:20.374771    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 17:57:20.374852 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:20.374838    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 17:57:20.375465 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:20.375452    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-c2jtj\""
Sep 17 17:57:20.376813 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:20.376790    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 17:57:20.415091 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:20.415061    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkx8h\" (UniqueName: \"kubernetes.io/projected/94bf369d-1ce4-4ed5-bf52-c45425223e50-kube-api-access-jkx8h\") pod \"ingress-canary-7pqps\" (UID: \"94bf369d-1ce4-4ed5-bf52-c45425223e50\") " pod="openshift-ingress-canary/ingress-canary-7pqps"
Sep 17 17:57:20.415091 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:20.415092    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/05707133-87f8-4036-89b3-f783c8f5e563-config-volume\") pod \"dns-default-dm5l8\" (UID: \"05707133-87f8-4036-89b3-f783c8f5e563\") " pod="openshift-dns/dns-default-dm5l8"
Sep 17 17:57:20.415256 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:20.415111    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05707133-87f8-4036-89b3-f783c8f5e563-metrics-tls\") pod \"dns-default-dm5l8\" (UID: \"05707133-87f8-4036-89b3-f783c8f5e563\") " pod="openshift-dns/dns-default-dm5l8"
Sep 17 17:57:20.415256 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:20.415186    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pdpq\" (UniqueName: \"kubernetes.io/projected/05707133-87f8-4036-89b3-f783c8f5e563-kube-api-access-8pdpq\") pod \"dns-default-dm5l8\" (UID: \"05707133-87f8-4036-89b3-f783c8f5e563\") " pod="openshift-dns/dns-default-dm5l8"
Sep 17 17:57:20.415319 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:20.415251    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05707133-87f8-4036-89b3-f783c8f5e563-tmp-dir\") pod \"dns-default-dm5l8\" (UID: \"05707133-87f8-4036-89b3-f783c8f5e563\") " pod="openshift-dns/dns-default-dm5l8"
Sep 17 17:57:20.415319 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:20.415275    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/94bf369d-1ce4-4ed5-bf52-c45425223e50-cert\") pod \"ingress-canary-7pqps\" (UID: \"94bf369d-1ce4-4ed5-bf52-c45425223e50\") " pod="openshift-ingress-canary/ingress-canary-7pqps"
Sep 17 17:57:20.516439 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:20.516406    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkx8h\" (UniqueName: \"kubernetes.io/projected/94bf369d-1ce4-4ed5-bf52-c45425223e50-kube-api-access-jkx8h\") pod \"ingress-canary-7pqps\" (UID: \"94bf369d-1ce4-4ed5-bf52-c45425223e50\") " pod="openshift-ingress-canary/ingress-canary-7pqps"
Sep 17 17:57:20.516439 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:20.516441    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/05707133-87f8-4036-89b3-f783c8f5e563-config-volume\") pod \"dns-default-dm5l8\" (UID: \"05707133-87f8-4036-89b3-f783c8f5e563\") " pod="openshift-dns/dns-default-dm5l8"
Sep 17 17:57:20.516610 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:20.516461    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05707133-87f8-4036-89b3-f783c8f5e563-metrics-tls\") pod \"dns-default-dm5l8\" (UID: \"05707133-87f8-4036-89b3-f783c8f5e563\") " pod="openshift-dns/dns-default-dm5l8"
Sep 17 17:57:20.516610 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:20.516498    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pdpq\" (UniqueName: \"kubernetes.io/projected/05707133-87f8-4036-89b3-f783c8f5e563-kube-api-access-8pdpq\") pod \"dns-default-dm5l8\" (UID: \"05707133-87f8-4036-89b3-f783c8f5e563\") " pod="openshift-dns/dns-default-dm5l8"
Sep 17 17:57:20.516610 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:20.516530    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05707133-87f8-4036-89b3-f783c8f5e563-tmp-dir\") pod \"dns-default-dm5l8\" (UID: \"05707133-87f8-4036-89b3-f783c8f5e563\") " pod="openshift-dns/dns-default-dm5l8"
Sep 17 17:57:20.516610 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:20.516556    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/94bf369d-1ce4-4ed5-bf52-c45425223e50-cert\") pod \"ingress-canary-7pqps\" (UID: \"94bf369d-1ce4-4ed5-bf52-c45425223e50\") " pod="openshift-ingress-canary/ingress-canary-7pqps"
Sep 17 17:57:20.516610 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:57:20.516579    2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:57:20.516858 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:57:20.516680    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05707133-87f8-4036-89b3-f783c8f5e563-metrics-tls podName:05707133-87f8-4036-89b3-f783c8f5e563 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:21.016659544 +0000 UTC m=+35.712750210 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/05707133-87f8-4036-89b3-f783c8f5e563-metrics-tls") pod "dns-default-dm5l8" (UID: "05707133-87f8-4036-89b3-f783c8f5e563") : secret "dns-default-metrics-tls" not found
Sep 17 17:57:20.516974 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:20.516936    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05707133-87f8-4036-89b3-f783c8f5e563-tmp-dir\") pod \"dns-default-dm5l8\" (UID: \"05707133-87f8-4036-89b3-f783c8f5e563\") " pod="openshift-dns/dns-default-dm5l8"
Sep 17 17:57:20.517247 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:20.517225    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/05707133-87f8-4036-89b3-f783c8f5e563-config-volume\") pod \"dns-default-dm5l8\" (UID: \"05707133-87f8-4036-89b3-f783c8f5e563\") " pod="openshift-dns/dns-default-dm5l8"
Sep 17 17:57:20.520553 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:20.520534    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/94bf369d-1ce4-4ed5-bf52-c45425223e50-cert\") pod \"ingress-canary-7pqps\" (UID: \"94bf369d-1ce4-4ed5-bf52-c45425223e50\") " pod="openshift-ingress-canary/ingress-canary-7pqps"
Sep 17 17:57:20.531662 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:20.531594    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pdpq\" (UniqueName: \"kubernetes.io/projected/05707133-87f8-4036-89b3-f783c8f5e563-kube-api-access-8pdpq\") pod \"dns-default-dm5l8\" (UID: \"05707133-87f8-4036-89b3-f783c8f5e563\") " pod="openshift-dns/dns-default-dm5l8"
Sep 17 17:57:20.531781 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:20.531765    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkx8h\" (UniqueName: \"kubernetes.io/projected/94bf369d-1ce4-4ed5-bf52-c45425223e50-kube-api-access-jkx8h\") pod \"ingress-canary-7pqps\" (UID: \"94bf369d-1ce4-4ed5-bf52-c45425223e50\") " pod="openshift-ingress-canary/ingress-canary-7pqps"
Sep 17 17:57:20.598660 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:20.598614    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7pqps"
Sep 17 17:57:20.676452 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:20.676421    2553 generic.go:356] "Generic (PLEG): container finished" podID="4e137835-2bdc-4b32-aa6b-64635bdc26d7" containerID="2ced7cbc869c54dbd6d925ea2ac6237e44c2460e57d5e768844e2eef549e0f33" exitCode=0
Sep 17 17:57:20.676648 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:20.676476    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bmjrp" event={"ID":"4e137835-2bdc-4b32-aa6b-64635bdc26d7","Type":"ContainerDied","Data":"2ced7cbc869c54dbd6d925ea2ac6237e44c2460e57d5e768844e2eef549e0f33"}
Sep 17 17:57:20.930553 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:20.930524    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7pqps"]
Sep 17 17:57:20.934408 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:57:20.934380    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94bf369d_1ce4_4ed5_bf52_c45425223e50.slice/crio-ae6d970f3e2db6578c7a7a7b8642f7935e6b422afe6a987dfc75a96abb7aa645 WatchSource:0}: Error finding container ae6d970f3e2db6578c7a7a7b8642f7935e6b422afe6a987dfc75a96abb7aa645: Status 404 returned error can't find the container with id ae6d970f3e2db6578c7a7a7b8642f7935e6b422afe6a987dfc75a96abb7aa645
Sep 17 17:57:21.019914 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:21.019874    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05707133-87f8-4036-89b3-f783c8f5e563-metrics-tls\") pod \"dns-default-dm5l8\" (UID: \"05707133-87f8-4036-89b3-f783c8f5e563\") " pod="openshift-dns/dns-default-dm5l8"
Sep 17 17:57:21.020355 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:57:21.019991    2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:57:21.020355 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:57:21.020040    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05707133-87f8-4036-89b3-f783c8f5e563-metrics-tls podName:05707133-87f8-4036-89b3-f783c8f5e563 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:22.020027064 +0000 UTC m=+36.716117730 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/05707133-87f8-4036-89b3-f783c8f5e563-metrics-tls") pod "dns-default-dm5l8" (UID: "05707133-87f8-4036-89b3-f783c8f5e563") : secret "dns-default-metrics-tls" not found
Sep 17 17:57:21.682111 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:21.682075    2553 generic.go:356] "Generic (PLEG): container finished" podID="4e137835-2bdc-4b32-aa6b-64635bdc26d7" containerID="88a6ff409f57374264bc0e03f73b32dffab773bce8c7249e2424ebc52f10b63f" exitCode=0
Sep 17 17:57:21.682406 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:21.682159    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bmjrp" event={"ID":"4e137835-2bdc-4b32-aa6b-64635bdc26d7","Type":"ContainerDied","Data":"88a6ff409f57374264bc0e03f73b32dffab773bce8c7249e2424ebc52f10b63f"}
Sep 17 17:57:21.683492 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:21.683331    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7pqps" event={"ID":"94bf369d-1ce4-4ed5-bf52-c45425223e50","Type":"ContainerStarted","Data":"ae6d970f3e2db6578c7a7a7b8642f7935e6b422afe6a987dfc75a96abb7aa645"}
Sep 17 17:57:22.029255 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:22.029160    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05707133-87f8-4036-89b3-f783c8f5e563-metrics-tls\") pod \"dns-default-dm5l8\" (UID: \"05707133-87f8-4036-89b3-f783c8f5e563\") " pod="openshift-dns/dns-default-dm5l8"
Sep 17 17:57:22.029708 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:57:22.029342    2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:57:22.029708 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:57:22.029408    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05707133-87f8-4036-89b3-f783c8f5e563-metrics-tls podName:05707133-87f8-4036-89b3-f783c8f5e563 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:24.02939289 +0000 UTC m=+38.725483557 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/05707133-87f8-4036-89b3-f783c8f5e563-metrics-tls") pod "dns-default-dm5l8" (UID: "05707133-87f8-4036-89b3-f783c8f5e563") : secret "dns-default-metrics-tls" not found
Sep 17 17:57:22.689777 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:22.689747    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bmjrp" event={"ID":"4e137835-2bdc-4b32-aa6b-64635bdc26d7","Type":"ContainerStarted","Data":"2956e867abab5f7c5415bc33d571b017fbcffaefa7abb4b32113c4a757e57b69"}
Sep 17 17:57:24.046691 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:24.046461    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05707133-87f8-4036-89b3-f783c8f5e563-metrics-tls\") pod \"dns-default-dm5l8\" (UID: \"05707133-87f8-4036-89b3-f783c8f5e563\") " pod="openshift-dns/dns-default-dm5l8"
Sep 17 17:57:24.046691 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:57:24.046611    2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:57:24.047075 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:57:24.046698    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05707133-87f8-4036-89b3-f783c8f5e563-metrics-tls podName:05707133-87f8-4036-89b3-f783c8f5e563 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:28.046682365 +0000 UTC m=+42.742773032 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/05707133-87f8-4036-89b3-f783c8f5e563-metrics-tls") pod "dns-default-dm5l8" (UID: "05707133-87f8-4036-89b3-f783c8f5e563") : secret "dns-default-metrics-tls" not found
Sep 17 17:57:24.696033 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:24.695994    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7pqps" event={"ID":"94bf369d-1ce4-4ed5-bf52-c45425223e50","Type":"ContainerStarted","Data":"1216cc74d4f21b6c9f9fd961688efca94cc7da3513e60db923ce9f1c3a55a32a"}
Sep 17 17:57:24.716766 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:24.716692    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-bmjrp" podStartSLOduration=9.838012116 podStartE2EDuration="37.716675842s" totalImagesPullingTime="27.878663726s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:56:47 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:22.723192051 +0000 UTC m=+37.419282735" watchObservedRunningTime="2026-09-17 17:57:24.716675842 +0000 UTC m=+39.412766523"
Sep 17 17:57:24.717204 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:24.717179    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-7pqps" podStartSLOduration=1.905836656 podStartE2EDuration="4.717162324s" totalImagesPullingTime="2.811325668s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:57:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:24.716647093 +0000 UTC m=+39.412737771" watchObservedRunningTime="2026-09-17 17:57:24.717162324 +0000 UTC m=+39.413253008"
Sep 17 17:57:28.075289 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:28.075252    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05707133-87f8-4036-89b3-f783c8f5e563-metrics-tls\") pod \"dns-default-dm5l8\" (UID: \"05707133-87f8-4036-89b3-f783c8f5e563\") " pod="openshift-dns/dns-default-dm5l8"
Sep 17 17:57:28.075761 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:57:28.075416    2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:57:28.075761 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:57:28.075484    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05707133-87f8-4036-89b3-f783c8f5e563-metrics-tls podName:05707133-87f8-4036-89b3-f783c8f5e563 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:36.075466342 +0000 UTC m=+50.771557011 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/05707133-87f8-4036-89b3-f783c8f5e563-metrics-tls") pod "dns-default-dm5l8" (UID: "05707133-87f8-4036-89b3-f783c8f5e563") : secret "dns-default-metrics-tls" not found
Sep 17 17:57:36.130862 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:36.130821    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05707133-87f8-4036-89b3-f783c8f5e563-metrics-tls\") pod \"dns-default-dm5l8\" (UID: \"05707133-87f8-4036-89b3-f783c8f5e563\") " pod="openshift-dns/dns-default-dm5l8"
Sep 17 17:57:36.131359 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:57:36.130997    2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:57:36.131359 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:57:36.131094    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05707133-87f8-4036-89b3-f783c8f5e563-metrics-tls podName:05707133-87f8-4036-89b3-f783c8f5e563 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:52.131072365 +0000 UTC m=+66.827163046 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/05707133-87f8-4036-89b3-f783c8f5e563-metrics-tls") pod "dns-default-dm5l8" (UID: "05707133-87f8-4036-89b3-f783c8f5e563") : secret "dns-default-metrics-tls" not found
Sep 17 17:57:45.671889 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:45.671862    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-btmqz"
Sep 17 17:57:52.037024 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:52.036985    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb-original-pull-secret\") pod \"global-pull-secret-syncer-xr8hx\" (UID: \"45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb\") " pod="kube-system/global-pull-secret-syncer-xr8hx"
Sep 17 17:57:52.037024 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:52.037030    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfc1cb73-d5de-4902-b16c-5744271855c5-metrics-certs\") pod \"network-metrics-daemon-5gdn2\" (UID: \"dfc1cb73-d5de-4902-b16c-5744271855c5\") " pod="openshift-multus/network-metrics-daemon-5gdn2"
Sep 17 17:57:52.040046 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:52.040023    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 17:57:52.040162 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:52.040146    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 17:57:52.047811 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:57:52.047786    2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 17:57:52.047898 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:57:52.047853    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfc1cb73-d5de-4902-b16c-5744271855c5-metrics-certs podName:dfc1cb73-d5de-4902-b16c-5744271855c5 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:56.047834795 +0000 UTC m=+130.743925461 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dfc1cb73-d5de-4902-b16c-5744271855c5-metrics-certs") pod "network-metrics-daemon-5gdn2" (UID: "dfc1cb73-d5de-4902-b16c-5744271855c5") : secret "metrics-daemon-secret" not found
Sep 17 17:57:52.051491 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:52.051470    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb-original-pull-secret\") pod \"global-pull-secret-syncer-xr8hx\" (UID: \"45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb\") " pod="kube-system/global-pull-secret-syncer-xr8hx"
Sep 17 17:57:52.137285 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:52.137235    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4p88\" (UniqueName: \"kubernetes.io/projected/cb953471-4308-48e7-9316-6b0bdf2b8738-kube-api-access-x4p88\") pod \"network-check-target-lkn4v\" (UID: \"cb953471-4308-48e7-9316-6b0bdf2b8738\") " pod="openshift-network-diagnostics/network-check-target-lkn4v"
Sep 17 17:57:52.137285 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:52.137291    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05707133-87f8-4036-89b3-f783c8f5e563-metrics-tls\") pod \"dns-default-dm5l8\" (UID: \"05707133-87f8-4036-89b3-f783c8f5e563\") " pod="openshift-dns/dns-default-dm5l8"
Sep 17 17:57:52.137506 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:57:52.137402    2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:57:52.137506 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:57:52.137448    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05707133-87f8-4036-89b3-f783c8f5e563-metrics-tls podName:05707133-87f8-4036-89b3-f783c8f5e563 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:24.137435544 +0000 UTC m=+98.833526211 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/05707133-87f8-4036-89b3-f783c8f5e563-metrics-tls") pod "dns-default-dm5l8" (UID: "05707133-87f8-4036-89b3-f783c8f5e563") : secret "dns-default-metrics-tls" not found
Sep 17 17:57:52.141000 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:52.140967    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 17:57:52.150132 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:52.150109    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 17:57:52.160057 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:52.160023    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4p88\" (UniqueName: \"kubernetes.io/projected/cb953471-4308-48e7-9316-6b0bdf2b8738-kube-api-access-x4p88\") pod \"network-check-target-lkn4v\" (UID: \"cb953471-4308-48e7-9316-6b0bdf2b8738\") " pod="openshift-network-diagnostics/network-check-target-lkn4v"
Sep 17 17:57:52.180939 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:52.180915    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xr8hx"
Sep 17 17:57:52.189890 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:52.189863    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-c2jtj\""
Sep 17 17:57:52.197254 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:52.197233    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lkn4v"
Sep 17 17:57:52.464632 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:52.464589    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xr8hx"]
Sep 17 17:57:52.468198 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:57:52.468163    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45254d28_1b05_4ddd_8ea9_22f6fb0eb3eb.slice/crio-b4683ab484a9a12e1897c44252a6525e36628c90e1635ccbfdaf2c58d227358c WatchSource:0}: Error finding container b4683ab484a9a12e1897c44252a6525e36628c90e1635ccbfdaf2c58d227358c: Status 404 returned error can't find the container with id b4683ab484a9a12e1897c44252a6525e36628c90e1635ccbfdaf2c58d227358c
Sep 17 17:57:52.482824 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:52.482794    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lkn4v"]
Sep 17 17:57:52.483996 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:57:52.483970    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb953471_4308_48e7_9316_6b0bdf2b8738.slice/crio-b82aa63da24d8f48d626734e9df365c910431000b48b68dfed5775f0fd53bb85 WatchSource:0}: Error finding container b82aa63da24d8f48d626734e9df365c910431000b48b68dfed5775f0fd53bb85: Status 404 returned error can't find the container with id b82aa63da24d8f48d626734e9df365c910431000b48b68dfed5775f0fd53bb85
Sep 17 17:57:52.763977 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:52.763890    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xr8hx" event={"ID":"45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb","Type":"ContainerStarted","Data":"b4683ab484a9a12e1897c44252a6525e36628c90e1635ccbfdaf2c58d227358c"}
Sep 17 17:57:52.764919 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:52.764897    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lkn4v" event={"ID":"cb953471-4308-48e7-9316-6b0bdf2b8738","Type":"ContainerStarted","Data":"b82aa63da24d8f48d626734e9df365c910431000b48b68dfed5775f0fd53bb85"}
Sep 17 17:57:57.781887 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:57.781841    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lkn4v" event={"ID":"cb953471-4308-48e7-9316-6b0bdf2b8738","Type":"ContainerStarted","Data":"b20c615cdaf219f882295822f4b62c8e201cde137c798ec488e1b61a1e9d4d16"}
Sep 17 17:57:57.782287 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:57.781938    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-lkn4v"
Sep 17 17:57:57.783377 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:57.783350    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xr8hx" event={"ID":"45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb","Type":"ContainerStarted","Data":"a4fc6e7926cf104c475b67bf98f4f66b4473cd5a1f537b359335f056a11ee12a"}
Sep 17 17:57:57.826240 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:57.826174    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-xr8hx" podStartSLOduration=66.407947871 podStartE2EDuration="1m10.826156693s" totalImagesPullingTime="4.418208822s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:56:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:57.825893827 +0000 UTC m=+72.521984523" watchObservedRunningTime="2026-09-17 17:57:57.826156693 +0000 UTC m=+72.522247378"
Sep 17 17:57:57.826765 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:57:57.826733    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-lkn4v" podStartSLOduration=66.427470993 podStartE2EDuration="1m10.826724466s" totalImagesPullingTime="4.399253473s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:56:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:57.802346098 +0000 UTC m=+72.498436782" watchObservedRunningTime="2026-09-17 17:57:57.826724466 +0000 UTC m=+72.522815141"
Sep 17 17:58:03.374920 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.374788    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-sxqrz"]
Sep 17 17:58:03.380211 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.380187    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-9vjmf"]
Sep 17 17:58:03.380348 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.380294    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-sxqrz"
Sep 17 17:58:03.382169 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.382152    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-wrv4g"]
Sep 17 17:58:03.382278 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.382264    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-9vjmf"
Sep 17 17:58:03.384355 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.384336    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-wrv4g"
Sep 17 17:58:03.385454 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.385413    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-hd862\""
Sep 17 17:58:03.385666 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.385651    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 17 17:58:03.385866 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.385846    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 17 17:58:03.385917 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.385866    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:58:03.386076 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.386064    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:58:03.386476 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.386456    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-l452v\""
Sep 17 17:58:03.386557 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.386509    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 17 17:58:03.386614 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.386566    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 17 17:58:03.386950 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.386768    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 17 17:58:03.387768 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.387751    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 17 17:58:03.389248 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.389229    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-94wbt\""
Sep 17 17:58:03.389248 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.389240    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 17 17:58:03.390295 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.390279    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 17:58:03.407476 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.407453    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 17:58:03.415901 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.415874    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-wrv4g"]
Sep 17 17:58:03.416433 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.416407    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-9vjmf"]
Sep 17 17:58:03.417381 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.417361    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-sxqrz"]
Sep 17 17:58:03.419613 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.419593    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f677181c-d82b-4a44-9ca3-6f46ac50b28d-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-9vjmf\" (UID: \"f677181c-d82b-4a44-9ca3-6f46ac50b28d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-9vjmf"
Sep 17 17:58:03.419718 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.419638    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4zr7\" (UniqueName: \"kubernetes.io/projected/f677181c-d82b-4a44-9ca3-6f46ac50b28d-kube-api-access-s4zr7\") pod \"cluster-samples-operator-685d7d68b5-9vjmf\" (UID: \"f677181c-d82b-4a44-9ca3-6f46ac50b28d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-9vjmf"
Sep 17 17:58:03.419718 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.419690    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb30f947-1c4d-478f-95b1-939d82415f95-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-wrv4g\" (UID: \"bb30f947-1c4d-478f-95b1-939d82415f95\") " pod="openshift-insights/insights-operator-6f9c644dd9-wrv4g"
Sep 17 17:58:03.419786 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.419727    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb30f947-1c4d-478f-95b1-939d82415f95-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-wrv4g\" (UID: \"bb30f947-1c4d-478f-95b1-939d82415f95\") " pod="openshift-insights/insights-operator-6f9c644dd9-wrv4g"
Sep 17 17:58:03.419786 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.419754    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-847mw\" (UniqueName: \"kubernetes.io/projected/bb30f947-1c4d-478f-95b1-939d82415f95-kube-api-access-847mw\") pod \"insights-operator-6f9c644dd9-wrv4g\" (UID: \"bb30f947-1c4d-478f-95b1-939d82415f95\") " pod="openshift-insights/insights-operator-6f9c644dd9-wrv4g"
Sep 17 17:58:03.419853 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.419804    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1ec26660-d938-4072-b5da-eaa607ea5ba5-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-sxqrz\" (UID: \"1ec26660-d938-4072-b5da-eaa607ea5ba5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-sxqrz"
Sep 17 17:58:03.419853 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.419821    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb30f947-1c4d-478f-95b1-939d82415f95-tmp\") pod \"insights-operator-6f9c644dd9-wrv4g\" (UID: \"bb30f947-1c4d-478f-95b1-939d82415f95\") " pod="openshift-insights/insights-operator-6f9c644dd9-wrv4g"
Sep 17 17:58:03.419853 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.419842    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pttx5\" (UniqueName: \"kubernetes.io/projected/1ec26660-d938-4072-b5da-eaa607ea5ba5-kube-api-access-pttx5\") pod \"kube-storage-version-migrator-operator-55889f7b47-sxqrz\" (UID: \"1ec26660-d938-4072-b5da-eaa607ea5ba5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-sxqrz"
Sep 17 17:58:03.419997 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.419867    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/bb30f947-1c4d-478f-95b1-939d82415f95-snapshots\") pod \"insights-operator-6f9c644dd9-wrv4g\" (UID: \"bb30f947-1c4d-478f-95b1-939d82415f95\") " pod="openshift-insights/insights-operator-6f9c644dd9-wrv4g"
Sep 17 17:58:03.419997 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.419939    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1ec26660-d938-4072-b5da-eaa607ea5ba5-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-sxqrz\" (UID: \"1ec26660-d938-4072-b5da-eaa607ea5ba5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-sxqrz"
Sep 17 17:58:03.419997 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.419972    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bb30f947-1c4d-478f-95b1-939d82415f95-serving-cert\") pod \"insights-operator-6f9c644dd9-wrv4g\" (UID: \"bb30f947-1c4d-478f-95b1-939d82415f95\") " pod="openshift-insights/insights-operator-6f9c644dd9-wrv4g"
Sep 17 17:58:03.426385 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.426358    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 17 17:58:03.460601 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.460568    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-c9mxg"]
Sep 17 17:58:03.462994 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.462978    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-c9mxg"
Sep 17 17:58:03.475708 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.475686    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 17 17:58:03.475829 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.475755    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-pztqd\""
Sep 17 17:58:03.477252 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.477236    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 17:58:03.477424 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.477412    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 17:58:03.479551 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.479534    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 17 17:58:03.492197 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.492171    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-c9mxg"]
Sep 17 17:58:03.520470 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.520439    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f677181c-d82b-4a44-9ca3-6f46ac50b28d-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-9vjmf\" (UID: \"f677181c-d82b-4a44-9ca3-6f46ac50b28d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-9vjmf"
Sep 17 17:58:03.520470 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.520472    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4zr7\" (UniqueName: \"kubernetes.io/projected/f677181c-d82b-4a44-9ca3-6f46ac50b28d-kube-api-access-s4zr7\") pod \"cluster-samples-operator-685d7d68b5-9vjmf\" (UID: \"f677181c-d82b-4a44-9ca3-6f46ac50b28d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-9vjmf"
Sep 17 17:58:03.520713 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.520491    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb30f947-1c4d-478f-95b1-939d82415f95-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-wrv4g\" (UID: \"bb30f947-1c4d-478f-95b1-939d82415f95\") " pod="openshift-insights/insights-operator-6f9c644dd9-wrv4g"
Sep 17 17:58:03.520713 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.520515    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb30f947-1c4d-478f-95b1-939d82415f95-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-wrv4g\" (UID: \"bb30f947-1c4d-478f-95b1-939d82415f95\") " pod="openshift-insights/insights-operator-6f9c644dd9-wrv4g"
Sep 17 17:58:03.520713 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:58:03.520642    2553 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 17:58:03.520824 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:58:03.520724    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f677181c-d82b-4a44-9ca3-6f46ac50b28d-samples-operator-tls podName:f677181c-d82b-4a44-9ca3-6f46ac50b28d nodeName:}" failed. No retries permitted until 2026-09-17 17:58:04.020698566 +0000 UTC m=+78.716789231 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f677181c-d82b-4a44-9ca3-6f46ac50b28d-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-9vjmf" (UID: "f677181c-d82b-4a44-9ca3-6f46ac50b28d") : secret "samples-operator-tls" not found
Sep 17 17:58:03.520824 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.520750    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-847mw\" (UniqueName: \"kubernetes.io/projected/bb30f947-1c4d-478f-95b1-939d82415f95-kube-api-access-847mw\") pod \"insights-operator-6f9c644dd9-wrv4g\" (UID: \"bb30f947-1c4d-478f-95b1-939d82415f95\") " pod="openshift-insights/insights-operator-6f9c644dd9-wrv4g"
Sep 17 17:58:03.520925 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.520835    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1ec26660-d938-4072-b5da-eaa607ea5ba5-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-sxqrz\" (UID: \"1ec26660-d938-4072-b5da-eaa607ea5ba5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-sxqrz"
Sep 17 17:58:03.520925 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.520867    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb30f947-1c4d-478f-95b1-939d82415f95-tmp\") pod \"insights-operator-6f9c644dd9-wrv4g\" (UID: \"bb30f947-1c4d-478f-95b1-939d82415f95\") " pod="openshift-insights/insights-operator-6f9c644dd9-wrv4g"
Sep 17 17:58:03.520925 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.520899    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtmd4\" (UniqueName: \"kubernetes.io/projected/8433dbe1-addc-4e5e-acdb-381db8334300-kube-api-access-vtmd4\") pod \"cluster-monitoring-operator-847b8bd8d7-c9mxg\" (UID: \"8433dbe1-addc-4e5e-acdb-381db8334300\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-c9mxg"
Sep 17 17:58:03.521072 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.520936    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pttx5\" (UniqueName: \"kubernetes.io/projected/1ec26660-d938-4072-b5da-eaa607ea5ba5-kube-api-access-pttx5\") pod \"kube-storage-version-migrator-operator-55889f7b47-sxqrz\" (UID: \"1ec26660-d938-4072-b5da-eaa607ea5ba5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-sxqrz"
Sep 17 17:58:03.521072 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.520961    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/bb30f947-1c4d-478f-95b1-939d82415f95-snapshots\") pod \"insights-operator-6f9c644dd9-wrv4g\" (UID: \"bb30f947-1c4d-478f-95b1-939d82415f95\") " pod="openshift-insights/insights-operator-6f9c644dd9-wrv4g"
Sep 17 17:58:03.521072 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.521012    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1ec26660-d938-4072-b5da-eaa607ea5ba5-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-sxqrz\" (UID: \"1ec26660-d938-4072-b5da-eaa607ea5ba5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-sxqrz"
Sep 17 17:58:03.521072 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.521046    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bb30f947-1c4d-478f-95b1-939d82415f95-serving-cert\") pod \"insights-operator-6f9c644dd9-wrv4g\" (UID: \"bb30f947-1c4d-478f-95b1-939d82415f95\") " pod="openshift-insights/insights-operator-6f9c644dd9-wrv4g"
Sep 17 17:58:03.521287 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.521078    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8433dbe1-addc-4e5e-acdb-381db8334300-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-c9mxg\" (UID: \"8433dbe1-addc-4e5e-acdb-381db8334300\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-c9mxg"
Sep 17 17:58:03.521287 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.521115    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8433dbe1-addc-4e5e-acdb-381db8334300-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-c9mxg\" (UID: \"8433dbe1-addc-4e5e-acdb-381db8334300\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-c9mxg"
Sep 17 17:58:03.521287 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.521169    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb30f947-1c4d-478f-95b1-939d82415f95-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-wrv4g\" (UID: \"bb30f947-1c4d-478f-95b1-939d82415f95\") " pod="openshift-insights/insights-operator-6f9c644dd9-wrv4g"
Sep 17 17:58:03.521433 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.521323    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb30f947-1c4d-478f-95b1-939d82415f95-tmp\") pod \"insights-operator-6f9c644dd9-wrv4g\" (UID: \"bb30f947-1c4d-478f-95b1-939d82415f95\") " pod="openshift-insights/insights-operator-6f9c644dd9-wrv4g"
Sep 17 17:58:03.521657 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.521637    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1ec26660-d938-4072-b5da-eaa607ea5ba5-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-sxqrz\" (UID: \"1ec26660-d938-4072-b5da-eaa607ea5ba5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-sxqrz"
Sep 17 17:58:03.521741 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.521667    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb30f947-1c4d-478f-95b1-939d82415f95-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-wrv4g\" (UID: \"bb30f947-1c4d-478f-95b1-939d82415f95\") " pod="openshift-insights/insights-operator-6f9c644dd9-wrv4g"
Sep 17 17:58:03.521775 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.521759    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/bb30f947-1c4d-478f-95b1-939d82415f95-snapshots\") pod \"insights-operator-6f9c644dd9-wrv4g\" (UID: \"bb30f947-1c4d-478f-95b1-939d82415f95\") " pod="openshift-insights/insights-operator-6f9c644dd9-wrv4g"
Sep 17 17:58:03.523187 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.523157    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1ec26660-d938-4072-b5da-eaa607ea5ba5-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-sxqrz\" (UID: \"1ec26660-d938-4072-b5da-eaa607ea5ba5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-sxqrz"
Sep 17 17:58:03.523288 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.523244    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bb30f947-1c4d-478f-95b1-939d82415f95-serving-cert\") pod \"insights-operator-6f9c644dd9-wrv4g\" (UID: \"bb30f947-1c4d-478f-95b1-939d82415f95\") " pod="openshift-insights/insights-operator-6f9c644dd9-wrv4g"
Sep 17 17:58:03.568535 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.568498    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4zr7\" (UniqueName: \"kubernetes.io/projected/f677181c-d82b-4a44-9ca3-6f46ac50b28d-kube-api-access-s4zr7\") pod \"cluster-samples-operator-685d7d68b5-9vjmf\" (UID: \"f677181c-d82b-4a44-9ca3-6f46ac50b28d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-9vjmf"
Sep 17 17:58:03.592165 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.592134    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pttx5\" (UniqueName: \"kubernetes.io/projected/1ec26660-d938-4072-b5da-eaa607ea5ba5-kube-api-access-pttx5\") pod \"kube-storage-version-migrator-operator-55889f7b47-sxqrz\" (UID: \"1ec26660-d938-4072-b5da-eaa607ea5ba5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-sxqrz"
Sep 17 17:58:03.609982 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.609948    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-847mw\" (UniqueName: \"kubernetes.io/projected/bb30f947-1c4d-478f-95b1-939d82415f95-kube-api-access-847mw\") pod \"insights-operator-6f9c644dd9-wrv4g\" (UID: \"bb30f947-1c4d-478f-95b1-939d82415f95\") " pod="openshift-insights/insights-operator-6f9c644dd9-wrv4g"
Sep 17 17:58:03.621957 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.621926    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8433dbe1-addc-4e5e-acdb-381db8334300-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-c9mxg\" (UID: \"8433dbe1-addc-4e5e-acdb-381db8334300\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-c9mxg"
Sep 17 17:58:03.622106 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.621964    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8433dbe1-addc-4e5e-acdb-381db8334300-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-c9mxg\" (UID: \"8433dbe1-addc-4e5e-acdb-381db8334300\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-c9mxg"
Sep 17 17:58:03.622106 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:58:03.622076    2553 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 17:58:03.622223 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.622112    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtmd4\" (UniqueName: \"kubernetes.io/projected/8433dbe1-addc-4e5e-acdb-381db8334300-kube-api-access-vtmd4\") pod \"cluster-monitoring-operator-847b8bd8d7-c9mxg\" (UID: \"8433dbe1-addc-4e5e-acdb-381db8334300\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-c9mxg"
Sep 17 17:58:03.622223 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:58:03.622125    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8433dbe1-addc-4e5e-acdb-381db8334300-cluster-monitoring-operator-tls podName:8433dbe1-addc-4e5e-acdb-381db8334300 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:04.122111239 +0000 UTC m=+78.818201900 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8433dbe1-addc-4e5e-acdb-381db8334300-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-c9mxg" (UID: "8433dbe1-addc-4e5e-acdb-381db8334300") : secret "cluster-monitoring-operator-tls" not found
Sep 17 17:58:03.622737 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.622718    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8433dbe1-addc-4e5e-acdb-381db8334300-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-c9mxg\" (UID: \"8433dbe1-addc-4e5e-acdb-381db8334300\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-c9mxg"
Sep 17 17:58:03.640613 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.640585    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtmd4\" (UniqueName: \"kubernetes.io/projected/8433dbe1-addc-4e5e-acdb-381db8334300-kube-api-access-vtmd4\") pod \"cluster-monitoring-operator-847b8bd8d7-c9mxg\" (UID: \"8433dbe1-addc-4e5e-acdb-381db8334300\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-c9mxg"
Sep 17 17:58:03.681040 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.681007    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-c8f6d6959-bqc4q"]
Sep 17 17:58:03.684793 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.684770    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c8f6d6959-bqc4q"
Sep 17 17:58:03.689895 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.689869    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-sxqrz"
Sep 17 17:58:03.693602 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.693583    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-ljtnz\""
Sep 17 17:58:03.695956 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.695937    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 17:58:03.699524 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.699507    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 17:58:03.700545 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.700525    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-wrv4g"
Sep 17 17:58:03.715819 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.715791    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 17:58:03.723334 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.723299    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cb42b3bd-4639-4ab7-b044-b9cef036ae24-trusted-ca\") pod \"image-registry-c8f6d6959-bqc4q\" (UID: \"cb42b3bd-4639-4ab7-b044-b9cef036ae24\") " pod="openshift-image-registry/image-registry-c8f6d6959-bqc4q"
Sep 17 17:58:03.723467 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.723378    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cb42b3bd-4639-4ab7-b044-b9cef036ae24-image-registry-private-configuration\") pod \"image-registry-c8f6d6959-bqc4q\" (UID: \"cb42b3bd-4639-4ab7-b044-b9cef036ae24\") " pod="openshift-image-registry/image-registry-c8f6d6959-bqc4q"
Sep 17 17:58:03.723467 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.723440    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cb42b3bd-4639-4ab7-b044-b9cef036ae24-ca-trust-extracted\") pod \"image-registry-c8f6d6959-bqc4q\" (UID: \"cb42b3bd-4639-4ab7-b044-b9cef036ae24\") " pod="openshift-image-registry/image-registry-c8f6d6959-bqc4q"
Sep 17 17:58:03.723575 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.723485    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-259km\" (UniqueName: \"kubernetes.io/projected/cb42b3bd-4639-4ab7-b044-b9cef036ae24-kube-api-access-259km\") pod \"image-registry-c8f6d6959-bqc4q\" (UID: \"cb42b3bd-4639-4ab7-b044-b9cef036ae24\") " pod="openshift-image-registry/image-registry-c8f6d6959-bqc4q"
Sep 17 17:58:03.723575 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.723536    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cb42b3bd-4639-4ab7-b044-b9cef036ae24-registry-tls\") pod \"image-registry-c8f6d6959-bqc4q\" (UID: \"cb42b3bd-4639-4ab7-b044-b9cef036ae24\") " pod="openshift-image-registry/image-registry-c8f6d6959-bqc4q"
Sep 17 17:58:03.723575 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.723567    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cb42b3bd-4639-4ab7-b044-b9cef036ae24-registry-certificates\") pod \"image-registry-c8f6d6959-bqc4q\" (UID: \"cb42b3bd-4639-4ab7-b044-b9cef036ae24\") " pod="openshift-image-registry/image-registry-c8f6d6959-bqc4q"
Sep 17 17:58:03.723733 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.723610    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cb42b3bd-4639-4ab7-b044-b9cef036ae24-bound-sa-token\") pod \"image-registry-c8f6d6959-bqc4q\" (UID: \"cb42b3bd-4639-4ab7-b044-b9cef036ae24\") " pod="openshift-image-registry/image-registry-c8f6d6959-bqc4q"
Sep 17 17:58:03.723733 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.723665    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cb42b3bd-4639-4ab7-b044-b9cef036ae24-installation-pull-secrets\") pod \"image-registry-c8f6d6959-bqc4q\" (UID: \"cb42b3bd-4639-4ab7-b044-b9cef036ae24\") " pod="openshift-image-registry/image-registry-c8f6d6959-bqc4q"
Sep 17 17:58:03.727513 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.727305    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 17:58:03.824313 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.824276    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cb42b3bd-4639-4ab7-b044-b9cef036ae24-trusted-ca\") pod \"image-registry-c8f6d6959-bqc4q\" (UID: \"cb42b3bd-4639-4ab7-b044-b9cef036ae24\") " pod="openshift-image-registry/image-registry-c8f6d6959-bqc4q"
Sep 17 17:58:03.824313 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.824328    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cb42b3bd-4639-4ab7-b044-b9cef036ae24-image-registry-private-configuration\") pod \"image-registry-c8f6d6959-bqc4q\" (UID: \"cb42b3bd-4639-4ab7-b044-b9cef036ae24\") " pod="openshift-image-registry/image-registry-c8f6d6959-bqc4q"
Sep 17 17:58:03.824582 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.824463    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cb42b3bd-4639-4ab7-b044-b9cef036ae24-ca-trust-extracted\") pod \"image-registry-c8f6d6959-bqc4q\" (UID: \"cb42b3bd-4639-4ab7-b044-b9cef036ae24\") " pod="openshift-image-registry/image-registry-c8f6d6959-bqc4q"
Sep 17 17:58:03.824582 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.824517    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-259km\" (UniqueName: \"kubernetes.io/projected/cb42b3bd-4639-4ab7-b044-b9cef036ae24-kube-api-access-259km\") pod \"image-registry-c8f6d6959-bqc4q\" (UID: \"cb42b3bd-4639-4ab7-b044-b9cef036ae24\") " pod="openshift-image-registry/image-registry-c8f6d6959-bqc4q"
Sep 17 17:58:03.824582 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.824563    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cb42b3bd-4639-4ab7-b044-b9cef036ae24-registry-tls\") pod \"image-registry-c8f6d6959-bqc4q\" (UID: \"cb42b3bd-4639-4ab7-b044-b9cef036ae24\") " pod="openshift-image-registry/image-registry-c8f6d6959-bqc4q"
Sep 17 17:58:03.824765 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.824590    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cb42b3bd-4639-4ab7-b044-b9cef036ae24-registry-certificates\") pod \"image-registry-c8f6d6959-bqc4q\" (UID: \"cb42b3bd-4639-4ab7-b044-b9cef036ae24\") " pod="openshift-image-registry/image-registry-c8f6d6959-bqc4q"
Sep 17 17:58:03.824765 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.824647    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cb42b3bd-4639-4ab7-b044-b9cef036ae24-bound-sa-token\") pod \"image-registry-c8f6d6959-bqc4q\" (UID: \"cb42b3bd-4639-4ab7-b044-b9cef036ae24\") " pod="openshift-image-registry/image-registry-c8f6d6959-bqc4q"
Sep 17 17:58:03.824765 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.824679    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cb42b3bd-4639-4ab7-b044-b9cef036ae24-installation-pull-secrets\") pod \"image-registry-c8f6d6959-bqc4q\" (UID: \"cb42b3bd-4639-4ab7-b044-b9cef036ae24\") " pod="openshift-image-registry/image-registry-c8f6d6959-bqc4q"
Sep 17 17:58:03.824765 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:58:03.824747    2553 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:58:03.824955 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:58:03.824768    2553 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c8f6d6959-bqc4q: secret "image-registry-tls" not found
Sep 17 17:58:03.824955 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:58:03.824842    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb42b3bd-4639-4ab7-b044-b9cef036ae24-registry-tls podName:cb42b3bd-4639-4ab7-b044-b9cef036ae24 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:04.324820425 +0000 UTC m=+79.020911103 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cb42b3bd-4639-4ab7-b044-b9cef036ae24-registry-tls") pod "image-registry-c8f6d6959-bqc4q" (UID: "cb42b3bd-4639-4ab7-b044-b9cef036ae24") : secret "image-registry-tls" not found
Sep 17 17:58:03.824955 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.824887    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cb42b3bd-4639-4ab7-b044-b9cef036ae24-ca-trust-extracted\") pod \"image-registry-c8f6d6959-bqc4q\" (UID: \"cb42b3bd-4639-4ab7-b044-b9cef036ae24\") " pod="openshift-image-registry/image-registry-c8f6d6959-bqc4q"
Sep 17 17:58:03.825202 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.825183    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cb42b3bd-4639-4ab7-b044-b9cef036ae24-registry-certificates\") pod \"image-registry-c8f6d6959-bqc4q\" (UID: \"cb42b3bd-4639-4ab7-b044-b9cef036ae24\") " pod="openshift-image-registry/image-registry-c8f6d6959-bqc4q"
Sep 17 17:58:03.825554 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.825534    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cb42b3bd-4639-4ab7-b044-b9cef036ae24-trusted-ca\") pod \"image-registry-c8f6d6959-bqc4q\" (UID: \"cb42b3bd-4639-4ab7-b044-b9cef036ae24\") " pod="openshift-image-registry/image-registry-c8f6d6959-bqc4q"
Sep 17 17:58:03.827317 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.827293    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cb42b3bd-4639-4ab7-b044-b9cef036ae24-installation-pull-secrets\") pod \"image-registry-c8f6d6959-bqc4q\" (UID: \"cb42b3bd-4639-4ab7-b044-b9cef036ae24\") " pod="openshift-image-registry/image-registry-c8f6d6959-bqc4q"
Sep 17 17:58:03.827317 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.827307    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cb42b3bd-4639-4ab7-b044-b9cef036ae24-image-registry-private-configuration\") pod \"image-registry-c8f6d6959-bqc4q\" (UID: \"cb42b3bd-4639-4ab7-b044-b9cef036ae24\") " pod="openshift-image-registry/image-registry-c8f6d6959-bqc4q"
Sep 17 17:58:03.865485 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.865452    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cb42b3bd-4639-4ab7-b044-b9cef036ae24-bound-sa-token\") pod \"image-registry-c8f6d6959-bqc4q\" (UID: \"cb42b3bd-4639-4ab7-b044-b9cef036ae24\") " pod="openshift-image-registry/image-registry-c8f6d6959-bqc4q"
Sep 17 17:58:03.865590 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.865573    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-259km\" (UniqueName: \"kubernetes.io/projected/cb42b3bd-4639-4ab7-b044-b9cef036ae24-kube-api-access-259km\") pod \"image-registry-c8f6d6959-bqc4q\" (UID: \"cb42b3bd-4639-4ab7-b044-b9cef036ae24\") " pod="openshift-image-registry/image-registry-c8f6d6959-bqc4q"
Sep 17 17:58:03.879415 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.879385    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-c8f6d6959-bqc4q"]
Sep 17 17:58:03.998271 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:03.998183    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-sxqrz"]
Sep 17 17:58:04.001657 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:04.001630    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-wrv4g"]
Sep 17 17:58:04.002768 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:58:04.002733    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ec26660_d938_4072_b5da_eaa607ea5ba5.slice/crio-03b521d87fe503174c710564dcfc8ef720d9d37860d1198a73c0f05cd8ef33e6 WatchSource:0}: Error finding container 03b521d87fe503174c710564dcfc8ef720d9d37860d1198a73c0f05cd8ef33e6: Status 404 returned error can't find the container with id 03b521d87fe503174c710564dcfc8ef720d9d37860d1198a73c0f05cd8ef33e6
Sep 17 17:58:04.005314 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:58:04.005294    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb30f947_1c4d_478f_95b1_939d82415f95.slice/crio-9694464dc5c4af05751c461fcd2e37c86fd962549868e2376fb7535f7dca2374 WatchSource:0}: Error finding container 9694464dc5c4af05751c461fcd2e37c86fd962549868e2376fb7535f7dca2374: Status 404 returned error can't find the container with id 9694464dc5c4af05751c461fcd2e37c86fd962549868e2376fb7535f7dca2374
Sep 17 17:58:04.026534 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:04.026509    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f677181c-d82b-4a44-9ca3-6f46ac50b28d-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-9vjmf\" (UID: \"f677181c-d82b-4a44-9ca3-6f46ac50b28d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-9vjmf"
Sep 17 17:58:04.026690 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:58:04.026673    2553 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 17:58:04.026745 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:58:04.026729    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f677181c-d82b-4a44-9ca3-6f46ac50b28d-samples-operator-tls podName:f677181c-d82b-4a44-9ca3-6f46ac50b28d nodeName:}" failed. No retries permitted until 2026-09-17 17:58:05.026714968 +0000 UTC m=+79.722805634 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f677181c-d82b-4a44-9ca3-6f46ac50b28d-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-9vjmf" (UID: "f677181c-d82b-4a44-9ca3-6f46ac50b28d") : secret "samples-operator-tls" not found
Sep 17 17:58:04.126873 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:04.126841    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8433dbe1-addc-4e5e-acdb-381db8334300-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-c9mxg\" (UID: \"8433dbe1-addc-4e5e-acdb-381db8334300\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-c9mxg"
Sep 17 17:58:04.127035 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:58:04.126985    2553 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 17:58:04.127079 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:58:04.127048    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8433dbe1-addc-4e5e-acdb-381db8334300-cluster-monitoring-operator-tls podName:8433dbe1-addc-4e5e-acdb-381db8334300 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:05.127031042 +0000 UTC m=+79.823121722 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8433dbe1-addc-4e5e-acdb-381db8334300-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-c9mxg" (UID: "8433dbe1-addc-4e5e-acdb-381db8334300") : secret "cluster-monitoring-operator-tls" not found
Sep 17 17:58:04.328433 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:04.328338    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cb42b3bd-4639-4ab7-b044-b9cef036ae24-registry-tls\") pod \"image-registry-c8f6d6959-bqc4q\" (UID: \"cb42b3bd-4639-4ab7-b044-b9cef036ae24\") " pod="openshift-image-registry/image-registry-c8f6d6959-bqc4q"
Sep 17 17:58:04.328578 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:58:04.328487    2553 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:58:04.328578 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:58:04.328499    2553 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c8f6d6959-bqc4q: secret "image-registry-tls" not found
Sep 17 17:58:04.328578 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:58:04.328547    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb42b3bd-4639-4ab7-b044-b9cef036ae24-registry-tls podName:cb42b3bd-4639-4ab7-b044-b9cef036ae24 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:05.328533163 +0000 UTC m=+80.024623829 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cb42b3bd-4639-4ab7-b044-b9cef036ae24-registry-tls") pod "image-registry-c8f6d6959-bqc4q" (UID: "cb42b3bd-4639-4ab7-b044-b9cef036ae24") : secret "image-registry-tls" not found
Sep 17 17:58:04.804256 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:04.804213    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-sxqrz" event={"ID":"1ec26660-d938-4072-b5da-eaa607ea5ba5","Type":"ContainerStarted","Data":"03b521d87fe503174c710564dcfc8ef720d9d37860d1198a73c0f05cd8ef33e6"}
Sep 17 17:58:04.805748 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:04.805712    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-wrv4g" event={"ID":"bb30f947-1c4d-478f-95b1-939d82415f95","Type":"ContainerStarted","Data":"9694464dc5c4af05751c461fcd2e37c86fd962549868e2376fb7535f7dca2374"}
Sep 17 17:58:05.034994 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:05.034937    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f677181c-d82b-4a44-9ca3-6f46ac50b28d-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-9vjmf\" (UID: \"f677181c-d82b-4a44-9ca3-6f46ac50b28d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-9vjmf"
Sep 17 17:58:05.035192 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:58:05.035100    2553 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 17:58:05.035192 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:58:05.035177    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f677181c-d82b-4a44-9ca3-6f46ac50b28d-samples-operator-tls podName:f677181c-d82b-4a44-9ca3-6f46ac50b28d nodeName:}" failed. No retries permitted until 2026-09-17 17:58:07.03515504 +0000 UTC m=+81.731245702 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f677181c-d82b-4a44-9ca3-6f46ac50b28d-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-9vjmf" (UID: "f677181c-d82b-4a44-9ca3-6f46ac50b28d") : secret "samples-operator-tls" not found
Sep 17 17:58:05.135444 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:05.135409    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8433dbe1-addc-4e5e-acdb-381db8334300-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-c9mxg\" (UID: \"8433dbe1-addc-4e5e-acdb-381db8334300\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-c9mxg"
Sep 17 17:58:05.135659 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:58:05.135582    2553 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 17:58:05.135732 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:58:05.135678    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8433dbe1-addc-4e5e-acdb-381db8334300-cluster-monitoring-operator-tls podName:8433dbe1-addc-4e5e-acdb-381db8334300 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:07.13565603 +0000 UTC m=+81.831746694 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8433dbe1-addc-4e5e-acdb-381db8334300-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-c9mxg" (UID: "8433dbe1-addc-4e5e-acdb-381db8334300") : secret "cluster-monitoring-operator-tls" not found
Sep 17 17:58:05.338001 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:05.337956    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cb42b3bd-4639-4ab7-b044-b9cef036ae24-registry-tls\") pod \"image-registry-c8f6d6959-bqc4q\" (UID: \"cb42b3bd-4639-4ab7-b044-b9cef036ae24\") " pod="openshift-image-registry/image-registry-c8f6d6959-bqc4q"
Sep 17 17:58:05.338182 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:58:05.338096    2553 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:58:05.338182 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:58:05.338120    2553 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c8f6d6959-bqc4q: secret "image-registry-tls" not found
Sep 17 17:58:05.338296 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:58:05.338184    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb42b3bd-4639-4ab7-b044-b9cef036ae24-registry-tls podName:cb42b3bd-4639-4ab7-b044-b9cef036ae24 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:07.338168358 +0000 UTC m=+82.034259026 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cb42b3bd-4639-4ab7-b044-b9cef036ae24-registry-tls") pod "image-registry-c8f6d6959-bqc4q" (UID: "cb42b3bd-4639-4ab7-b044-b9cef036ae24") : secret "image-registry-tls" not found
Sep 17 17:58:06.813995 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:06.813951    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-sxqrz" event={"ID":"1ec26660-d938-4072-b5da-eaa607ea5ba5","Type":"ContainerStarted","Data":"c603b90c7cb6a0356947c6a102499a7b1b9d054f636ff860f8158feb02b58ec9"}
Sep 17 17:58:06.815576 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:06.815545    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-wrv4g" event={"ID":"bb30f947-1c4d-478f-95b1-939d82415f95","Type":"ContainerStarted","Data":"bd70b83a95d11f5ad3b5542ed20b1bd7afb02951c9af9a23e549bba86969e566"}
Sep 17 17:58:06.833073 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:06.833022    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-sxqrz" podStartSLOduration=1.769441065 podStartE2EDuration="3.833003957s" totalImagesPullingTime="2.063562892s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:06.83237186 +0000 UTC m=+81.528462545" watchObservedRunningTime="2026-09-17 17:58:06.833003957 +0000 UTC m=+81.529094641"
Sep 17 17:58:06.864078 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:06.860498    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-6f9c644dd9-wrv4g" podStartSLOduration=1.8024024349999999 podStartE2EDuration="3.86047756s" totalImagesPullingTime="2.058075125s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:06.859079072 +0000 UTC m=+81.555169757" watchObservedRunningTime="2026-09-17 17:58:06.86047756 +0000 UTC m=+81.556568240"
Sep 17 17:58:07.049890 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:07.049846    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f677181c-d82b-4a44-9ca3-6f46ac50b28d-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-9vjmf\" (UID: \"f677181c-d82b-4a44-9ca3-6f46ac50b28d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-9vjmf"
Sep 17 17:58:07.050077 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:58:07.050051    2553 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 17:58:07.050155 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:58:07.050144    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f677181c-d82b-4a44-9ca3-6f46ac50b28d-samples-operator-tls podName:f677181c-d82b-4a44-9ca3-6f46ac50b28d nodeName:}" failed. No retries permitted until 2026-09-17 17:58:11.050121547 +0000 UTC m=+85.746212226 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f677181c-d82b-4a44-9ca3-6f46ac50b28d-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-9vjmf" (UID: "f677181c-d82b-4a44-9ca3-6f46ac50b28d") : secret "samples-operator-tls" not found
Sep 17 17:58:07.151218 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:07.151184    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8433dbe1-addc-4e5e-acdb-381db8334300-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-c9mxg\" (UID: \"8433dbe1-addc-4e5e-acdb-381db8334300\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-c9mxg"
Sep 17 17:58:07.151408 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:58:07.151385    2553 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 17:58:07.151498 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:58:07.151485    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8433dbe1-addc-4e5e-acdb-381db8334300-cluster-monitoring-operator-tls podName:8433dbe1-addc-4e5e-acdb-381db8334300 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:11.151466862 +0000 UTC m=+85.847557524 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8433dbe1-addc-4e5e-acdb-381db8334300-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-c9mxg" (UID: "8433dbe1-addc-4e5e-acdb-381db8334300") : secret "cluster-monitoring-operator-tls" not found
Sep 17 17:58:07.224813 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:07.224780    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-s782r"]
Sep 17 17:58:07.227976 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:07.227950    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-s782r"
Sep 17 17:58:07.232832 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:07.232803    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 17:58:07.232832 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:07.232825    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-d5pr9\""
Sep 17 17:58:07.233077 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:07.233063    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 17:58:07.246701 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:07.246673    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-s782r"]
Sep 17 17:58:07.252034 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:07.252009    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8c882b94-d6bf-422b-b04f-13232d9828f3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s782r\" (UID: \"8c882b94-d6bf-422b-b04f-13232d9828f3\") " pod="openshift-insights/insights-runtime-extractor-s782r"
Sep 17 17:58:07.252137 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:07.252039    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5v6z\" (UniqueName: \"kubernetes.io/projected/8c882b94-d6bf-422b-b04f-13232d9828f3-kube-api-access-j5v6z\") pod \"insights-runtime-extractor-s782r\" (UID: \"8c882b94-d6bf-422b-b04f-13232d9828f3\") " pod="openshift-insights/insights-runtime-extractor-s782r"
Sep 17 17:58:07.252137 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:07.252061    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8c882b94-d6bf-422b-b04f-13232d9828f3-data-volume\") pod \"insights-runtime-extractor-s782r\" (UID: \"8c882b94-d6bf-422b-b04f-13232d9828f3\") " pod="openshift-insights/insights-runtime-extractor-s782r"
Sep 17 17:58:07.252218 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:07.252155    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8c882b94-d6bf-422b-b04f-13232d9828f3-crio-socket\") pod \"insights-runtime-extractor-s782r\" (UID: \"8c882b94-d6bf-422b-b04f-13232d9828f3\") " pod="openshift-insights/insights-runtime-extractor-s782r"
Sep 17 17:58:07.252251 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:07.252230    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8c882b94-d6bf-422b-b04f-13232d9828f3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s782r\" (UID: \"8c882b94-d6bf-422b-b04f-13232d9828f3\") " pod="openshift-insights/insights-runtime-extractor-s782r"
Sep 17 17:58:07.352555 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:07.352521    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8c882b94-d6bf-422b-b04f-13232d9828f3-crio-socket\") pod \"insights-runtime-extractor-s782r\" (UID: \"8c882b94-d6bf-422b-b04f-13232d9828f3\") " pod="openshift-insights/insights-runtime-extractor-s782r"
Sep 17 17:58:07.352744 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:07.352610    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8c882b94-d6bf-422b-b04f-13232d9828f3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s782r\" (UID: \"8c882b94-d6bf-422b-b04f-13232d9828f3\") " pod="openshift-insights/insights-runtime-extractor-s782r"
Sep 17 17:58:07.352744 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:07.352650    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8c882b94-d6bf-422b-b04f-13232d9828f3-crio-socket\") pod \"insights-runtime-extractor-s782r\" (UID: \"8c882b94-d6bf-422b-b04f-13232d9828f3\") " pod="openshift-insights/insights-runtime-extractor-s782r"
Sep 17 17:58:07.352744 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:07.352678    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cb42b3bd-4639-4ab7-b044-b9cef036ae24-registry-tls\") pod \"image-registry-c8f6d6959-bqc4q\" (UID: \"cb42b3bd-4639-4ab7-b044-b9cef036ae24\") " pod="openshift-image-registry/image-registry-c8f6d6959-bqc4q"
Sep 17 17:58:07.352744 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:07.352714    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8c882b94-d6bf-422b-b04f-13232d9828f3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s782r\" (UID: \"8c882b94-d6bf-422b-b04f-13232d9828f3\") " pod="openshift-insights/insights-runtime-extractor-s782r"
Sep 17 17:58:07.352744 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:58:07.352727    2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:58:07.352744 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:07.352741    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5v6z\" (UniqueName: \"kubernetes.io/projected/8c882b94-d6bf-422b-b04f-13232d9828f3-kube-api-access-j5v6z\") pod \"insights-runtime-extractor-s782r\" (UID: \"8c882b94-d6bf-422b-b04f-13232d9828f3\") " pod="openshift-insights/insights-runtime-extractor-s782r"
Sep 17 17:58:07.353022 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:07.352767    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8c882b94-d6bf-422b-b04f-13232d9828f3-data-volume\") pod \"insights-runtime-extractor-s782r\" (UID: \"8c882b94-d6bf-422b-b04f-13232d9828f3\") " pod="openshift-insights/insights-runtime-extractor-s782r"
Sep 17 17:58:07.353022 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:58:07.352826    2553 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:58:07.353022 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:58:07.352839    2553 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:58:07.353022 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:58:07.352845    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c882b94-d6bf-422b-b04f-13232d9828f3-insights-runtime-extractor-tls podName:8c882b94-d6bf-422b-b04f-13232d9828f3 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:07.852824186 +0000 UTC m=+82.548914862 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8c882b94-d6bf-422b-b04f-13232d9828f3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s782r" (UID: "8c882b94-d6bf-422b-b04f-13232d9828f3") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:58:07.353022 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:58:07.352988    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8c882b94-d6bf-422b-b04f-13232d9828f3-kube-rbac-proxy-cm podName:8c882b94-d6bf-422b-b04f-13232d9828f3 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:07.852966309 +0000 UTC m=+82.549056982 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8c882b94-d6bf-422b-b04f-13232d9828f3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-s782r" (UID: "8c882b94-d6bf-422b-b04f-13232d9828f3") : configmap "kube-rbac-proxy" not found
Sep 17 17:58:07.353022 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:58:07.353020    2553 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c8f6d6959-bqc4q: secret "image-registry-tls" not found
Sep 17 17:58:07.353295 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:58:07.353096    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb42b3bd-4639-4ab7-b044-b9cef036ae24-registry-tls podName:cb42b3bd-4639-4ab7-b044-b9cef036ae24 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:11.35308043 +0000 UTC m=+86.049171111 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cb42b3bd-4639-4ab7-b044-b9cef036ae24-registry-tls") pod "image-registry-c8f6d6959-bqc4q" (UID: "cb42b3bd-4639-4ab7-b044-b9cef036ae24") : secret "image-registry-tls" not found
Sep 17 17:58:07.353701 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:07.353683    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8c882b94-d6bf-422b-b04f-13232d9828f3-data-volume\") pod \"insights-runtime-extractor-s782r\" (UID: \"8c882b94-d6bf-422b-b04f-13232d9828f3\") " pod="openshift-insights/insights-runtime-extractor-s782r"
Sep 17 17:58:07.375786 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:07.375761    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5v6z\" (UniqueName: \"kubernetes.io/projected/8c882b94-d6bf-422b-b04f-13232d9828f3-kube-api-access-j5v6z\") pod \"insights-runtime-extractor-s782r\" (UID: \"8c882b94-d6bf-422b-b04f-13232d9828f3\") " pod="openshift-insights/insights-runtime-extractor-s782r"
Sep 17 17:58:07.857286 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:07.857245    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8c882b94-d6bf-422b-b04f-13232d9828f3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s782r\" (UID: \"8c882b94-d6bf-422b-b04f-13232d9828f3\") " pod="openshift-insights/insights-runtime-extractor-s782r"
Sep 17 17:58:07.857780 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:58:07.857336    2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:58:07.857780 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:07.857359    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8c882b94-d6bf-422b-b04f-13232d9828f3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s782r\" (UID: \"8c882b94-d6bf-422b-b04f-13232d9828f3\") " pod="openshift-insights/insights-runtime-extractor-s782r"
Sep 17 17:58:07.857780 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:58:07.857382    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c882b94-d6bf-422b-b04f-13232d9828f3-insights-runtime-extractor-tls podName:8c882b94-d6bf-422b-b04f-13232d9828f3 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:08.857368288 +0000 UTC m=+83.553458950 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8c882b94-d6bf-422b-b04f-13232d9828f3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s782r" (UID: "8c882b94-d6bf-422b-b04f-13232d9828f3") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:58:07.857780 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:58:07.857406    2553 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:58:07.857780 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:58:07.857543    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8c882b94-d6bf-422b-b04f-13232d9828f3-kube-rbac-proxy-cm podName:8c882b94-d6bf-422b-b04f-13232d9828f3 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:08.85752636 +0000 UTC m=+83.553617048 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8c882b94-d6bf-422b-b04f-13232d9828f3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-s782r" (UID: "8c882b94-d6bf-422b-b04f-13232d9828f3") : configmap "kube-rbac-proxy" not found
Sep 17 17:58:08.864942 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:08.864894    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8c882b94-d6bf-422b-b04f-13232d9828f3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s782r\" (UID: \"8c882b94-d6bf-422b-b04f-13232d9828f3\") " pod="openshift-insights/insights-runtime-extractor-s782r"
Sep 17 17:58:08.865390 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:08.864980    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8c882b94-d6bf-422b-b04f-13232d9828f3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s782r\" (UID: \"8c882b94-d6bf-422b-b04f-13232d9828f3\") " pod="openshift-insights/insights-runtime-extractor-s782r"
Sep 17 17:58:08.865390 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:58:08.865050    2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:58:08.865390 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:58:08.865072    2553 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:58:08.865390 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:58:08.865116    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c882b94-d6bf-422b-b04f-13232d9828f3-insights-runtime-extractor-tls podName:8c882b94-d6bf-422b-b04f-13232d9828f3 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:10.865099347 +0000 UTC m=+85.561190014 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8c882b94-d6bf-422b-b04f-13232d9828f3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s782r" (UID: "8c882b94-d6bf-422b-b04f-13232d9828f3") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:58:08.865390 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:58:08.865132    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8c882b94-d6bf-422b-b04f-13232d9828f3-kube-rbac-proxy-cm podName:8c882b94-d6bf-422b-b04f-13232d9828f3 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:10.865124043 +0000 UTC m=+85.561214705 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8c882b94-d6bf-422b-b04f-13232d9828f3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-s782r" (UID: "8c882b94-d6bf-422b-b04f-13232d9828f3") : configmap "kube-rbac-proxy" not found
Sep 17 17:58:08.995206 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:08.995181    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4fdj7_56b9ff80-4b68-4449-bd38-e2e4d2de6aa2/dns-node-resolver/0.log"
Sep 17 17:58:09.990428 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:09.990398    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6pxfw_c5297750-e3a6-4690-9f70-4be7eac03a96/node-ca/0.log"
Sep 17 17:58:10.880524 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:10.880486    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8c882b94-d6bf-422b-b04f-13232d9828f3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s782r\" (UID: \"8c882b94-d6bf-422b-b04f-13232d9828f3\") " pod="openshift-insights/insights-runtime-extractor-s782r"
Sep 17 17:58:10.880733 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:10.880593    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8c882b94-d6bf-422b-b04f-13232d9828f3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s782r\" (UID: \"8c882b94-d6bf-422b-b04f-13232d9828f3\") " pod="openshift-insights/insights-runtime-extractor-s782r"
Sep 17 17:58:10.880733 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:58:10.880637    2553 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:58:10.880733 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:58:10.880703    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8c882b94-d6bf-422b-b04f-13232d9828f3-kube-rbac-proxy-cm podName:8c882b94-d6bf-422b-b04f-13232d9828f3 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:14.880687772 +0000 UTC m=+89.576778438 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8c882b94-d6bf-422b-b04f-13232d9828f3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-s782r" (UID: "8c882b94-d6bf-422b-b04f-13232d9828f3") : configmap "kube-rbac-proxy" not found
Sep 17 17:58:10.880845 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:58:10.880743    2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:58:10.880845 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:58:10.880799    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c882b94-d6bf-422b-b04f-13232d9828f3-insights-runtime-extractor-tls podName:8c882b94-d6bf-422b-b04f-13232d9828f3 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:14.880783975 +0000 UTC m=+89.576874642 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8c882b94-d6bf-422b-b04f-13232d9828f3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s782r" (UID: "8c882b94-d6bf-422b-b04f-13232d9828f3") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:58:10.992371 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:10.992340    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7pqps_94bf369d-1ce4-4ed5-bf52-c45425223e50/serve-healthcheck-canary/0.log"
Sep 17 17:58:11.081963 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:11.081905    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f677181c-d82b-4a44-9ca3-6f46ac50b28d-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-9vjmf\" (UID: \"f677181c-d82b-4a44-9ca3-6f46ac50b28d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-9vjmf"
Sep 17 17:58:11.082108 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:58:11.082058    2553 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 17:58:11.082145 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:58:11.082119    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f677181c-d82b-4a44-9ca3-6f46ac50b28d-samples-operator-tls podName:f677181c-d82b-4a44-9ca3-6f46ac50b28d nodeName:}" failed. No retries permitted until 2026-09-17 17:58:19.082103881 +0000 UTC m=+93.778194544 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f677181c-d82b-4a44-9ca3-6f46ac50b28d-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-9vjmf" (UID: "f677181c-d82b-4a44-9ca3-6f46ac50b28d") : secret "samples-operator-tls" not found
Sep 17 17:58:11.182883 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:11.182844    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8433dbe1-addc-4e5e-acdb-381db8334300-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-c9mxg\" (UID: \"8433dbe1-addc-4e5e-acdb-381db8334300\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-c9mxg"
Sep 17 17:58:11.183056 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:58:11.182991    2553 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 17:58:11.183098 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:58:11.183055    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8433dbe1-addc-4e5e-acdb-381db8334300-cluster-monitoring-operator-tls podName:8433dbe1-addc-4e5e-acdb-381db8334300 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:19.183040402 +0000 UTC m=+93.879131068 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8433dbe1-addc-4e5e-acdb-381db8334300-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-c9mxg" (UID: "8433dbe1-addc-4e5e-acdb-381db8334300") : secret "cluster-monitoring-operator-tls" not found
Sep 17 17:58:11.384713 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:11.384686    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cb42b3bd-4639-4ab7-b044-b9cef036ae24-registry-tls\") pod \"image-registry-c8f6d6959-bqc4q\" (UID: \"cb42b3bd-4639-4ab7-b044-b9cef036ae24\") " pod="openshift-image-registry/image-registry-c8f6d6959-bqc4q"
Sep 17 17:58:11.384854 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:58:11.384835    2553 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:58:11.384854 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:58:11.384851    2553 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c8f6d6959-bqc4q: secret "image-registry-tls" not found
Sep 17 17:58:11.384942 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:58:11.384905    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb42b3bd-4639-4ab7-b044-b9cef036ae24-registry-tls podName:cb42b3bd-4639-4ab7-b044-b9cef036ae24 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:19.384888485 +0000 UTC m=+94.080979152 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cb42b3bd-4639-4ab7-b044-b9cef036ae24-registry-tls") pod "image-registry-c8f6d6959-bqc4q" (UID: "cb42b3bd-4639-4ab7-b044-b9cef036ae24") : secret "image-registry-tls" not found
Sep 17 17:58:14.913432 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:14.913387    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8c882b94-d6bf-422b-b04f-13232d9828f3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s782r\" (UID: \"8c882b94-d6bf-422b-b04f-13232d9828f3\") " pod="openshift-insights/insights-runtime-extractor-s782r"
Sep 17 17:58:14.913896 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:14.913478    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8c882b94-d6bf-422b-b04f-13232d9828f3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s782r\" (UID: \"8c882b94-d6bf-422b-b04f-13232d9828f3\") " pod="openshift-insights/insights-runtime-extractor-s782r"
Sep 17 17:58:14.913896 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:58:14.913566    2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:58:14.913896 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:58:14.913595    2553 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:58:14.913896 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:58:14.913654    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c882b94-d6bf-422b-b04f-13232d9828f3-insights-runtime-extractor-tls podName:8c882b94-d6bf-422b-b04f-13232d9828f3 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:22.91363696 +0000 UTC m=+97.609727628 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8c882b94-d6bf-422b-b04f-13232d9828f3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s782r" (UID: "8c882b94-d6bf-422b-b04f-13232d9828f3") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:58:14.913896 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:58:14.913669    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8c882b94-d6bf-422b-b04f-13232d9828f3-kube-rbac-proxy-cm podName:8c882b94-d6bf-422b-b04f-13232d9828f3 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:22.913663067 +0000 UTC m=+97.609753730 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8c882b94-d6bf-422b-b04f-13232d9828f3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-s782r" (UID: "8c882b94-d6bf-422b-b04f-13232d9828f3") : configmap "kube-rbac-proxy" not found
Sep 17 17:58:19.149006 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:19.148968    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f677181c-d82b-4a44-9ca3-6f46ac50b28d-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-9vjmf\" (UID: \"f677181c-d82b-4a44-9ca3-6f46ac50b28d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-9vjmf"
Sep 17 17:58:19.151411 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:19.151391    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f677181c-d82b-4a44-9ca3-6f46ac50b28d-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-9vjmf\" (UID: \"f677181c-d82b-4a44-9ca3-6f46ac50b28d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-9vjmf"
Sep 17 17:58:19.249984 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:19.249946    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8433dbe1-addc-4e5e-acdb-381db8334300-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-c9mxg\" (UID: \"8433dbe1-addc-4e5e-acdb-381db8334300\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-c9mxg"
Sep 17 17:58:19.252356 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:19.252336    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8433dbe1-addc-4e5e-acdb-381db8334300-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-c9mxg\" (UID: \"8433dbe1-addc-4e5e-acdb-381db8334300\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-c9mxg"
Sep 17 17:58:19.296252 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:19.296208    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-9vjmf"
Sep 17 17:58:19.371789 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:19.371759    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-c9mxg"
Sep 17 17:58:19.452121 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:19.452086    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cb42b3bd-4639-4ab7-b044-b9cef036ae24-registry-tls\") pod \"image-registry-c8f6d6959-bqc4q\" (UID: \"cb42b3bd-4639-4ab7-b044-b9cef036ae24\") " pod="openshift-image-registry/image-registry-c8f6d6959-bqc4q"
Sep 17 17:58:19.454360 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:19.454338    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cb42b3bd-4639-4ab7-b044-b9cef036ae24-registry-tls\") pod \"image-registry-c8f6d6959-bqc4q\" (UID: \"cb42b3bd-4639-4ab7-b044-b9cef036ae24\") " pod="openshift-image-registry/image-registry-c8f6d6959-bqc4q"
Sep 17 17:58:19.581206 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:19.581172    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-9vjmf"]
Sep 17 17:58:19.593330 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:19.593299    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c8f6d6959-bqc4q"
Sep 17 17:58:19.652525 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:19.652449    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-c9mxg"]
Sep 17 17:58:19.657420 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:58:19.657393    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8433dbe1_addc_4e5e_acdb_381db8334300.slice/crio-389225969ccf420f0c3a3f1c2eac894fac47b4723a10c98487915add81fc0ea1 WatchSource:0}: Error finding container 389225969ccf420f0c3a3f1c2eac894fac47b4723a10c98487915add81fc0ea1: Status 404 returned error can't find the container with id 389225969ccf420f0c3a3f1c2eac894fac47b4723a10c98487915add81fc0ea1
Sep 17 17:58:19.859922 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:19.859838    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-c9mxg" event={"ID":"8433dbe1-addc-4e5e-acdb-381db8334300","Type":"ContainerStarted","Data":"389225969ccf420f0c3a3f1c2eac894fac47b4723a10c98487915add81fc0ea1"}
Sep 17 17:58:19.860939 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:19.860918    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-9vjmf" event={"ID":"f677181c-d82b-4a44-9ca3-6f46ac50b28d","Type":"ContainerStarted","Data":"a34322beaf73a6898bd5a6e5492820128e4e147cbff58735f6907b9240c33f03"}
Sep 17 17:58:19.874676 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:19.874651    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-c8f6d6959-bqc4q"]
Sep 17 17:58:19.877776 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:58:19.877755    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb42b3bd_4639_4ab7_b044_b9cef036ae24.slice/crio-a5322fc57d375d85c6c04c8289babfabc2aa1575cad13594a9e16cb095afbed5 WatchSource:0}: Error finding container a5322fc57d375d85c6c04c8289babfabc2aa1575cad13594a9e16cb095afbed5: Status 404 returned error can't find the container with id a5322fc57d375d85c6c04c8289babfabc2aa1575cad13594a9e16cb095afbed5
Sep 17 17:58:20.867649 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:20.867224    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c8f6d6959-bqc4q" event={"ID":"cb42b3bd-4639-4ab7-b044-b9cef036ae24","Type":"ContainerStarted","Data":"b5c7de08185d4926cee699b03c72605f299644589e9e61fd69e7c549e46094e6"}
Sep 17 17:58:20.867649 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:20.867287    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c8f6d6959-bqc4q" event={"ID":"cb42b3bd-4639-4ab7-b044-b9cef036ae24","Type":"ContainerStarted","Data":"a5322fc57d375d85c6c04c8289babfabc2aa1575cad13594a9e16cb095afbed5"}
Sep 17 17:58:20.867649 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:20.867560    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-c8f6d6959-bqc4q"
Sep 17 17:58:20.900740 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:20.900687    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-c8f6d6959-bqc4q" podStartSLOduration=17.900669873 podStartE2EDuration="17.900669873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:20.899789574 +0000 UTC m=+95.595880259" watchObservedRunningTime="2026-09-17 17:58:20.900669873 +0000 UTC m=+95.596760568"
Sep 17 17:58:22.878704 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:22.878673    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-c9mxg" event={"ID":"8433dbe1-addc-4e5e-acdb-381db8334300","Type":"ContainerStarted","Data":"c7f55a0cf1ae2de56d73c50d17b383875b3525eb08b610c08695145b8f7244c6"}
Sep 17 17:58:22.884383 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:22.884353    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-9vjmf" event={"ID":"f677181c-d82b-4a44-9ca3-6f46ac50b28d","Type":"ContainerStarted","Data":"859e10c6373e5671bf69800fdefd61fbb6c639903f02a0d3a73a63de058fd16e"}
Sep 17 17:58:22.884526 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:22.884423    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-9vjmf" event={"ID":"f677181c-d82b-4a44-9ca3-6f46ac50b28d","Type":"ContainerStarted","Data":"2748e46cad17ecc5bbd53e4bfb815ce1592062b5b759956f40917f1a6f9fee5a"}
Sep 17 17:58:22.908545 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:22.908501    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-c9mxg" podStartSLOduration=17.755688853 podStartE2EDuration="19.90848654s" totalImagesPullingTime="2.152797687s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:22.907842062 +0000 UTC m=+97.603932745" watchObservedRunningTime="2026-09-17 17:58:22.90848654 +0000 UTC m=+97.604577224"
Sep 17 17:58:22.930202 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:22.930150    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-9vjmf" podStartSLOduration=17.744715829 podStartE2EDuration="19.930131843s" totalImagesPullingTime="2.185416014s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:22.928679412 +0000 UTC m=+97.624770096" watchObservedRunningTime="2026-09-17 17:58:22.930131843 +0000 UTC m=+97.626222526"
Sep 17 17:58:22.981707 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:22.981665    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8c882b94-d6bf-422b-b04f-13232d9828f3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s782r\" (UID: \"8c882b94-d6bf-422b-b04f-13232d9828f3\") " pod="openshift-insights/insights-runtime-extractor-s782r"
Sep 17 17:58:22.981886 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:22.981754    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8c882b94-d6bf-422b-b04f-13232d9828f3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s782r\" (UID: \"8c882b94-d6bf-422b-b04f-13232d9828f3\") " pod="openshift-insights/insights-runtime-extractor-s782r"
Sep 17 17:58:22.982338 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:22.982309    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8c882b94-d6bf-422b-b04f-13232d9828f3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s782r\" (UID: \"8c882b94-d6bf-422b-b04f-13232d9828f3\") " pod="openshift-insights/insights-runtime-extractor-s782r"
Sep 17 17:58:22.984195 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:22.984167    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8c882b94-d6bf-422b-b04f-13232d9828f3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s782r\" (UID: \"8c882b94-d6bf-422b-b04f-13232d9828f3\") " pod="openshift-insights/insights-runtime-extractor-s782r"
Sep 17 17:58:23.137032 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:23.136991    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-s782r"
Sep 17 17:58:23.461453 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:23.461371    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-s782r"]
Sep 17 17:58:23.464502 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:58:23.464474    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c882b94_d6bf_422b_b04f_13232d9828f3.slice/crio-c0e375cee6596d24eb08b9ee742d136ca299f35dfad019e83dee801723b04037 WatchSource:0}: Error finding container c0e375cee6596d24eb08b9ee742d136ca299f35dfad019e83dee801723b04037: Status 404 returned error can't find the container with id c0e375cee6596d24eb08b9ee742d136ca299f35dfad019e83dee801723b04037
Sep 17 17:58:23.892628 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:23.892580    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s782r" event={"ID":"8c882b94-d6bf-422b-b04f-13232d9828f3","Type":"ContainerStarted","Data":"695adfd252f4399fe9f337e98f2db0cc1d1d75b7a5bd8f7c13ea0275739f4b7c"}
Sep 17 17:58:23.892984 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:23.892666    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s782r" event={"ID":"8c882b94-d6bf-422b-b04f-13232d9828f3","Type":"ContainerStarted","Data":"c0e375cee6596d24eb08b9ee742d136ca299f35dfad019e83dee801723b04037"}
Sep 17 17:58:24.192601 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:24.192570    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05707133-87f8-4036-89b3-f783c8f5e563-metrics-tls\") pod \"dns-default-dm5l8\" (UID: \"05707133-87f8-4036-89b3-f783c8f5e563\") " pod="openshift-dns/dns-default-dm5l8"
Sep 17 17:58:24.195273 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:24.195252    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05707133-87f8-4036-89b3-f783c8f5e563-metrics-tls\") pod \"dns-default-dm5l8\" (UID: \"05707133-87f8-4036-89b3-f783c8f5e563\") " pod="openshift-dns/dns-default-dm5l8"
Sep 17 17:58:24.221728 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:24.221706    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5wl9d\""
Sep 17 17:58:24.229593 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:24.229572    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dm5l8"
Sep 17 17:58:24.504328 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:24.504255    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dm5l8"]
Sep 17 17:58:24.506942 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:58:24.506921    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05707133_87f8_4036_89b3_f783c8f5e563.slice/crio-8ed87de407d01916826bf75a50064679119962317ba26ec0be65031e8fe2a817 WatchSource:0}: Error finding container 8ed87de407d01916826bf75a50064679119962317ba26ec0be65031e8fe2a817: Status 404 returned error can't find the container with id 8ed87de407d01916826bf75a50064679119962317ba26ec0be65031e8fe2a817
Sep 17 17:58:24.899958 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:24.899920    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dm5l8" event={"ID":"05707133-87f8-4036-89b3-f783c8f5e563","Type":"ContainerStarted","Data":"8ed87de407d01916826bf75a50064679119962317ba26ec0be65031e8fe2a817"}
Sep 17 17:58:24.901767 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:24.901735    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s782r" event={"ID":"8c882b94-d6bf-422b-b04f-13232d9828f3","Type":"ContainerStarted","Data":"c2e7678e04f8641c3a5764c65c330e9e037fa65d59cd430212c4728c00f0bfa0"}
Sep 17 17:58:25.907763 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:25.907725    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s782r" event={"ID":"8c882b94-d6bf-422b-b04f-13232d9828f3","Type":"ContainerStarted","Data":"0ca676d4b9a2791edbb5a7844958f8406f92f9964683627cf13822380058a275"}
Sep 17 17:58:26.917102 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:26.917062    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dm5l8" event={"ID":"05707133-87f8-4036-89b3-f783c8f5e563","Type":"ContainerStarted","Data":"2f52c5a835a08493b9453fb3dac3e7312793523ce9fe444121296807e6ad56c2"}
Sep 17 17:58:26.917511 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:26.917147    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dm5l8" event={"ID":"05707133-87f8-4036-89b3-f783c8f5e563","Type":"ContainerStarted","Data":"e4bcc595f23bf546ef8349d86389b8ccc633257336246e1de32f131afd32aea9"}
Sep 17 17:58:26.948506 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:26.948449    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-s782r" podStartSLOduration=18.093566585 podStartE2EDuration="19.948428177s" totalImagesPullingTime="1.854861592s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:58:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:25.932037242 +0000 UTC m=+100.628127925" watchObservedRunningTime="2026-09-17 17:58:26.948428177 +0000 UTC m=+101.644518863"
Sep 17 17:58:27.920709 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:27.920674    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-dm5l8"
Sep 17 17:58:28.788741 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:28.788710    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-lkn4v"
Sep 17 17:58:28.816109 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:28.816055    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-dm5l8" podStartSLOduration=66.79400964 podStartE2EDuration="1m8.816034229s" totalImagesPullingTime="2.022024589s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:57:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:26.948836131 +0000 UTC m=+101.644926817" watchObservedRunningTime="2026-09-17 17:58:28.816034229 +0000 UTC m=+103.512124914"
Sep 17 17:58:37.178505 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:37.178469    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-f8ff7f478-9xdbz"]
Sep 17 17:58:37.181611 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:37.181591    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f8ff7f478-9xdbz"
Sep 17 17:58:37.185350 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:37.185329    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 17 17:58:37.186135 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:37.186117    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 17 17:58:37.186530 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:37.186512    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 17 17:58:37.186594 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:37.186555    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 17 17:58:37.199386 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:37.199360    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-c8f6d6959-bqc4q"]
Sep 17 17:58:37.200427 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:37.200402    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-f8ff7f478-9xdbz"]
Sep 17 17:58:37.203858 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:37.203819    2553 patch_prober.go:28] interesting pod/image-registry-c8f6d6959-bqc4q container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 17 17:58:37.203957 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:37.203881    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-c8f6d6959-bqc4q" podUID="cb42b3bd-4639-4ab7-b044-b9cef036ae24" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:58:37.289805 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:37.289768    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5b5767bf9b-cckrb"]
Sep 17 17:58:37.291851 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:37.291835    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b5767bf9b-cckrb"
Sep 17 17:58:37.292875 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:37.292851    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/301a28ba-e70e-4112-a0ca-d7dad9bb0880-tmp\") pod \"klusterlet-addon-workmgr-f8ff7f478-9xdbz\" (UID: \"301a28ba-e70e-4112-a0ca-d7dad9bb0880\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f8ff7f478-9xdbz"
Sep 17 17:58:37.292967 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:37.292947    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/301a28ba-e70e-4112-a0ca-d7dad9bb0880-klusterlet-config\") pod \"klusterlet-addon-workmgr-f8ff7f478-9xdbz\" (UID: \"301a28ba-e70e-4112-a0ca-d7dad9bb0880\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f8ff7f478-9xdbz"
Sep 17 17:58:37.293022 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:37.292966    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp4l9\" (UniqueName: \"kubernetes.io/projected/301a28ba-e70e-4112-a0ca-d7dad9bb0880-kube-api-access-dp4l9\") pod \"klusterlet-addon-workmgr-f8ff7f478-9xdbz\" (UID: \"301a28ba-e70e-4112-a0ca-d7dad9bb0880\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f8ff7f478-9xdbz"
Sep 17 17:58:37.333435 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:37.333402    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5b5767bf9b-cckrb"]
Sep 17 17:58:37.393657 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:37.393604    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f467fe4f-0b64-4a65-9cf1-d1aebdb1baa1-ca-trust-extracted\") pod \"image-registry-5b5767bf9b-cckrb\" (UID: \"f467fe4f-0b64-4a65-9cf1-d1aebdb1baa1\") " pod="openshift-image-registry/image-registry-5b5767bf9b-cckrb"
Sep 17 17:58:37.393889 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:37.393666    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f467fe4f-0b64-4a65-9cf1-d1aebdb1baa1-registry-tls\") pod \"image-registry-5b5767bf9b-cckrb\" (UID: \"f467fe4f-0b64-4a65-9cf1-d1aebdb1baa1\") " pod="openshift-image-registry/image-registry-5b5767bf9b-cckrb"
Sep 17 17:58:37.393889 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:37.393713    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f467fe4f-0b64-4a65-9cf1-d1aebdb1baa1-installation-pull-secrets\") pod \"image-registry-5b5767bf9b-cckrb\" (UID: \"f467fe4f-0b64-4a65-9cf1-d1aebdb1baa1\") " pod="openshift-image-registry/image-registry-5b5767bf9b-cckrb"
Sep 17 17:58:37.393889 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:37.393738    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f467fe4f-0b64-4a65-9cf1-d1aebdb1baa1-registry-certificates\") pod \"image-registry-5b5767bf9b-cckrb\" (UID: \"f467fe4f-0b64-4a65-9cf1-d1aebdb1baa1\") " pod="openshift-image-registry/image-registry-5b5767bf9b-cckrb"
Sep 17 17:58:37.393889 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:37.393770    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/301a28ba-e70e-4112-a0ca-d7dad9bb0880-tmp\") pod \"klusterlet-addon-workmgr-f8ff7f478-9xdbz\" (UID: \"301a28ba-e70e-4112-a0ca-d7dad9bb0880\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f8ff7f478-9xdbz"
Sep 17 17:58:37.394068 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:37.393893    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42q2c\" (UniqueName: \"kubernetes.io/projected/f467fe4f-0b64-4a65-9cf1-d1aebdb1baa1-kube-api-access-42q2c\") pod \"image-registry-5b5767bf9b-cckrb\" (UID: \"f467fe4f-0b64-4a65-9cf1-d1aebdb1baa1\") " pod="openshift-image-registry/image-registry-5b5767bf9b-cckrb"
Sep 17 17:58:37.394068 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:37.393944    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f467fe4f-0b64-4a65-9cf1-d1aebdb1baa1-trusted-ca\") pod \"image-registry-5b5767bf9b-cckrb\" (UID: \"f467fe4f-0b64-4a65-9cf1-d1aebdb1baa1\") " pod="openshift-image-registry/image-registry-5b5767bf9b-cckrb"
Sep 17 17:58:37.394068 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:37.394011    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f467fe4f-0b64-4a65-9cf1-d1aebdb1baa1-bound-sa-token\") pod \"image-registry-5b5767bf9b-cckrb\" (UID: \"f467fe4f-0b64-4a65-9cf1-d1aebdb1baa1\") " pod="openshift-image-registry/image-registry-5b5767bf9b-cckrb"
Sep 17 17:58:37.394168 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:37.394069    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f467fe4f-0b64-4a65-9cf1-d1aebdb1baa1-image-registry-private-configuration\") pod \"image-registry-5b5767bf9b-cckrb\" (UID: \"f467fe4f-0b64-4a65-9cf1-d1aebdb1baa1\") " pod="openshift-image-registry/image-registry-5b5767bf9b-cckrb"
Sep 17 17:58:37.394168 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:37.394082    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/301a28ba-e70e-4112-a0ca-d7dad9bb0880-tmp\") pod \"klusterlet-addon-workmgr-f8ff7f478-9xdbz\" (UID: \"301a28ba-e70e-4112-a0ca-d7dad9bb0880\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f8ff7f478-9xdbz"
Sep 17 17:58:37.394168 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:37.394106    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/301a28ba-e70e-4112-a0ca-d7dad9bb0880-klusterlet-config\") pod \"klusterlet-addon-workmgr-f8ff7f478-9xdbz\" (UID: \"301a28ba-e70e-4112-a0ca-d7dad9bb0880\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f8ff7f478-9xdbz"
Sep 17 17:58:37.394168 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:37.394129    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp4l9\" (UniqueName: \"kubernetes.io/projected/301a28ba-e70e-4112-a0ca-d7dad9bb0880-kube-api-access-dp4l9\") pod \"klusterlet-addon-workmgr-f8ff7f478-9xdbz\" (UID: \"301a28ba-e70e-4112-a0ca-d7dad9bb0880\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f8ff7f478-9xdbz"
Sep 17 17:58:37.396471 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:37.396449    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/301a28ba-e70e-4112-a0ca-d7dad9bb0880-klusterlet-config\") pod \"klusterlet-addon-workmgr-f8ff7f478-9xdbz\" (UID: \"301a28ba-e70e-4112-a0ca-d7dad9bb0880\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f8ff7f478-9xdbz"
Sep 17 17:58:37.407554 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:37.407530    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp4l9\" (UniqueName: \"kubernetes.io/projected/301a28ba-e70e-4112-a0ca-d7dad9bb0880-kube-api-access-dp4l9\") pod \"klusterlet-addon-workmgr-f8ff7f478-9xdbz\" (UID: \"301a28ba-e70e-4112-a0ca-d7dad9bb0880\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f8ff7f478-9xdbz"
Sep 17 17:58:37.491346 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:37.491257    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f8ff7f478-9xdbz"
Sep 17 17:58:37.495381 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:37.495363    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f467fe4f-0b64-4a65-9cf1-d1aebdb1baa1-installation-pull-secrets\") pod \"image-registry-5b5767bf9b-cckrb\" (UID: \"f467fe4f-0b64-4a65-9cf1-d1aebdb1baa1\") " pod="openshift-image-registry/image-registry-5b5767bf9b-cckrb"
Sep 17 17:58:37.495442 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:37.495398    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f467fe4f-0b64-4a65-9cf1-d1aebdb1baa1-registry-certificates\") pod \"image-registry-5b5767bf9b-cckrb\" (UID: \"f467fe4f-0b64-4a65-9cf1-d1aebdb1baa1\") " pod="openshift-image-registry/image-registry-5b5767bf9b-cckrb"
Sep 17 17:58:37.495442 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:37.495431    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42q2c\" (UniqueName: \"kubernetes.io/projected/f467fe4f-0b64-4a65-9cf1-d1aebdb1baa1-kube-api-access-42q2c\") pod \"image-registry-5b5767bf9b-cckrb\" (UID: \"f467fe4f-0b64-4a65-9cf1-d1aebdb1baa1\") " pod="openshift-image-registry/image-registry-5b5767bf9b-cckrb"
Sep 17 17:58:37.495530 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:37.495490    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f467fe4f-0b64-4a65-9cf1-d1aebdb1baa1-trusted-ca\") pod \"image-registry-5b5767bf9b-cckrb\" (UID: \"f467fe4f-0b64-4a65-9cf1-d1aebdb1baa1\") " pod="openshift-image-registry/image-registry-5b5767bf9b-cckrb"
Sep 17 17:58:37.495580 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:37.495534    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f467fe4f-0b64-4a65-9cf1-d1aebdb1baa1-bound-sa-token\") pod \"image-registry-5b5767bf9b-cckrb\" (UID: \"f467fe4f-0b64-4a65-9cf1-d1aebdb1baa1\") " pod="openshift-image-registry/image-registry-5b5767bf9b-cckrb"
Sep 17 17:58:37.495662 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:37.495581    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f467fe4f-0b64-4a65-9cf1-d1aebdb1baa1-image-registry-private-configuration\") pod \"image-registry-5b5767bf9b-cckrb\" (UID: \"f467fe4f-0b64-4a65-9cf1-d1aebdb1baa1\") " pod="openshift-image-registry/image-registry-5b5767bf9b-cckrb"
Sep 17 17:58:37.495728 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:37.495679    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f467fe4f-0b64-4a65-9cf1-d1aebdb1baa1-ca-trust-extracted\") pod \"image-registry-5b5767bf9b-cckrb\" (UID: \"f467fe4f-0b64-4a65-9cf1-d1aebdb1baa1\") " pod="openshift-image-registry/image-registry-5b5767bf9b-cckrb"
Sep 17 17:58:37.495728 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:37.495710    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f467fe4f-0b64-4a65-9cf1-d1aebdb1baa1-registry-tls\") pod \"image-registry-5b5767bf9b-cckrb\" (UID: \"f467fe4f-0b64-4a65-9cf1-d1aebdb1baa1\") " pod="openshift-image-registry/image-registry-5b5767bf9b-cckrb"
Sep 17 17:58:37.496263 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:37.496225    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f467fe4f-0b64-4a65-9cf1-d1aebdb1baa1-ca-trust-extracted\") pod \"image-registry-5b5767bf9b-cckrb\" (UID: \"f467fe4f-0b64-4a65-9cf1-d1aebdb1baa1\") " pod="openshift-image-registry/image-registry-5b5767bf9b-cckrb"
Sep 17 17:58:37.496774 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:37.496747    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f467fe4f-0b64-4a65-9cf1-d1aebdb1baa1-registry-certificates\") pod \"image-registry-5b5767bf9b-cckrb\" (UID: \"f467fe4f-0b64-4a65-9cf1-d1aebdb1baa1\") " pod="openshift-image-registry/image-registry-5b5767bf9b-cckrb"
Sep 17 17:58:37.497187 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:37.497161    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f467fe4f-0b64-4a65-9cf1-d1aebdb1baa1-trusted-ca\") pod \"image-registry-5b5767bf9b-cckrb\" (UID: \"f467fe4f-0b64-4a65-9cf1-d1aebdb1baa1\") " pod="openshift-image-registry/image-registry-5b5767bf9b-cckrb"
Sep 17 17:58:37.498046 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:37.498022    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f467fe4f-0b64-4a65-9cf1-d1aebdb1baa1-installation-pull-secrets\") pod \"image-registry-5b5767bf9b-cckrb\" (UID: \"f467fe4f-0b64-4a65-9cf1-d1aebdb1baa1\") " pod="openshift-image-registry/image-registry-5b5767bf9b-cckrb"
Sep 17 17:58:37.498371 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:37.498355    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f467fe4f-0b64-4a65-9cf1-d1aebdb1baa1-registry-tls\") pod \"image-registry-5b5767bf9b-cckrb\" (UID: \"f467fe4f-0b64-4a65-9cf1-d1aebdb1baa1\") " pod="openshift-image-registry/image-registry-5b5767bf9b-cckrb"
Sep 17 17:58:37.498524 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:37.498500    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f467fe4f-0b64-4a65-9cf1-d1aebdb1baa1-image-registry-private-configuration\") pod \"image-registry-5b5767bf9b-cckrb\" (UID: \"f467fe4f-0b64-4a65-9cf1-d1aebdb1baa1\") " pod="openshift-image-registry/image-registry-5b5767bf9b-cckrb"
Sep 17 17:58:37.505298 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:37.505265    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42q2c\" (UniqueName: \"kubernetes.io/projected/f467fe4f-0b64-4a65-9cf1-d1aebdb1baa1-kube-api-access-42q2c\") pod \"image-registry-5b5767bf9b-cckrb\" (UID: \"f467fe4f-0b64-4a65-9cf1-d1aebdb1baa1\") " pod="openshift-image-registry/image-registry-5b5767bf9b-cckrb"
Sep 17 17:58:37.505427 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:37.505316    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f467fe4f-0b64-4a65-9cf1-d1aebdb1baa1-bound-sa-token\") pod \"image-registry-5b5767bf9b-cckrb\" (UID: \"f467fe4f-0b64-4a65-9cf1-d1aebdb1baa1\") " pod="openshift-image-registry/image-registry-5b5767bf9b-cckrb"
Sep 17 17:58:37.600399 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:37.600352    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b5767bf9b-cckrb"
Sep 17 17:58:37.775687 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:37.775579    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-f8ff7f478-9xdbz"]
Sep 17 17:58:37.778450 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:58:37.778423    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod301a28ba_e70e_4112_a0ca_d7dad9bb0880.slice/crio-2aaf81f1a03d97b8a6fdb86d1ec5c8fb92bccc719f9d13c13ecb74303167857f WatchSource:0}: Error finding container 2aaf81f1a03d97b8a6fdb86d1ec5c8fb92bccc719f9d13c13ecb74303167857f: Status 404 returned error can't find the container with id 2aaf81f1a03d97b8a6fdb86d1ec5c8fb92bccc719f9d13c13ecb74303167857f
Sep 17 17:58:37.900524 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:37.900492    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5b5767bf9b-cckrb"]
Sep 17 17:58:37.903779 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:58:37.903752    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf467fe4f_0b64_4a65_9cf1_d1aebdb1baa1.slice/crio-a5402980df2dbed3b0759b0b7db754540b45e3d73570ade9bd356e0782416adb WatchSource:0}: Error finding container a5402980df2dbed3b0759b0b7db754540b45e3d73570ade9bd356e0782416adb: Status 404 returned error can't find the container with id a5402980df2dbed3b0759b0b7db754540b45e3d73570ade9bd356e0782416adb
Sep 17 17:58:37.927377 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:37.927351    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-dm5l8"
Sep 17 17:58:37.955570 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:37.955542    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b5767bf9b-cckrb" event={"ID":"f467fe4f-0b64-4a65-9cf1-d1aebdb1baa1","Type":"ContainerStarted","Data":"a5402980df2dbed3b0759b0b7db754540b45e3d73570ade9bd356e0782416adb"}
Sep 17 17:58:37.956723 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:37.956700    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f8ff7f478-9xdbz" event={"ID":"301a28ba-e70e-4112-a0ca-d7dad9bb0880","Type":"ContainerStarted","Data":"2aaf81f1a03d97b8a6fdb86d1ec5c8fb92bccc719f9d13c13ecb74303167857f"}
Sep 17 17:58:38.962890 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:38.962856    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b5767bf9b-cckrb" event={"ID":"f467fe4f-0b64-4a65-9cf1-d1aebdb1baa1","Type":"ContainerStarted","Data":"94bb9d41cb7f664d39dc0aa29318924a4ea62f4bad78fc4d965700206ff58b53"}
Sep 17 17:58:38.963459 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:38.962991    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5b5767bf9b-cckrb"
Sep 17 17:58:39.001935 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:39.001874    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5b5767bf9b-cckrb" podStartSLOduration=2.001853708 podStartE2EDuration="2.001853708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:39.000696268 +0000 UTC m=+113.696786972" watchObservedRunningTime="2026-09-17 17:58:39.001853708 +0000 UTC m=+113.697944393"
Sep 17 17:58:40.721227 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:40.721180    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-gbkmm"]
Sep 17 17:58:40.724721 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:40.724697    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-gbkmm"
Sep 17 17:58:40.733041 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:40.733010    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 17 17:58:40.733167 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:40.733010    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 17 17:58:40.733167 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:40.733091    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-jtmxf\""
Sep 17 17:58:40.733713 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:40.733527    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 17:58:40.739532 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:40.739503    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-gbkmm"]
Sep 17 17:58:40.826420 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:40.826383    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df8d85da-2c74-414a-9aa9-98270323c887-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-gbkmm\" (UID: \"df8d85da-2c74-414a-9aa9-98270323c887\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-gbkmm"
Sep 17 17:58:40.826583 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:40.826491    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/df8d85da-2c74-414a-9aa9-98270323c887-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-gbkmm\" (UID: \"df8d85da-2c74-414a-9aa9-98270323c887\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-gbkmm"
Sep 17 17:58:40.826583 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:40.826543    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d45f\" (UniqueName: \"kubernetes.io/projected/df8d85da-2c74-414a-9aa9-98270323c887-kube-api-access-5d45f\") pod \"prometheus-operator-7bd5d56d6-gbkmm\" (UID: \"df8d85da-2c74-414a-9aa9-98270323c887\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-gbkmm"
Sep 17 17:58:40.826583 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:40.826574    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/df8d85da-2c74-414a-9aa9-98270323c887-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-gbkmm\" (UID: \"df8d85da-2c74-414a-9aa9-98270323c887\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-gbkmm"
Sep 17 17:58:40.927293 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:40.927244    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df8d85da-2c74-414a-9aa9-98270323c887-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-gbkmm\" (UID: \"df8d85da-2c74-414a-9aa9-98270323c887\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-gbkmm"
Sep 17 17:58:40.927470 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:40.927337    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/df8d85da-2c74-414a-9aa9-98270323c887-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-gbkmm\" (UID: \"df8d85da-2c74-414a-9aa9-98270323c887\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-gbkmm"
Sep 17 17:58:40.927470 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:40.927388    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d45f\" (UniqueName: \"kubernetes.io/projected/df8d85da-2c74-414a-9aa9-98270323c887-kube-api-access-5d45f\") pod \"prometheus-operator-7bd5d56d6-gbkmm\" (UID: \"df8d85da-2c74-414a-9aa9-98270323c887\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-gbkmm"
Sep 17 17:58:40.927470 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:40.927412    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/df8d85da-2c74-414a-9aa9-98270323c887-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-gbkmm\" (UID: \"df8d85da-2c74-414a-9aa9-98270323c887\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-gbkmm"
Sep 17 17:58:40.928059 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:40.928031    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df8d85da-2c74-414a-9aa9-98270323c887-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-gbkmm\" (UID: \"df8d85da-2c74-414a-9aa9-98270323c887\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-gbkmm"
Sep 17 17:58:40.930144 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:40.930122    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/df8d85da-2c74-414a-9aa9-98270323c887-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-gbkmm\" (UID: \"df8d85da-2c74-414a-9aa9-98270323c887\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-gbkmm"
Sep 17 17:58:40.930304 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:40.930281    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/df8d85da-2c74-414a-9aa9-98270323c887-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-gbkmm\" (UID: \"df8d85da-2c74-414a-9aa9-98270323c887\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-gbkmm"
Sep 17 17:58:40.940196 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:40.940171    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d45f\" (UniqueName: \"kubernetes.io/projected/df8d85da-2c74-414a-9aa9-98270323c887-kube-api-access-5d45f\") pod \"prometheus-operator-7bd5d56d6-gbkmm\" (UID: \"df8d85da-2c74-414a-9aa9-98270323c887\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-gbkmm"
Sep 17 17:58:41.035358 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:41.035265    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-gbkmm"
Sep 17 17:58:41.672728 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:41.672697    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-gbkmm"]
Sep 17 17:58:41.672842 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:58:41.672769    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf8d85da_2c74_414a_9aa9_98270323c887.slice/crio-a7acfa73dc59a8921951bbdcb3be030aa17db3c79d86f5c8199b52c1209174d9 WatchSource:0}: Error finding container a7acfa73dc59a8921951bbdcb3be030aa17db3c79d86f5c8199b52c1209174d9: Status 404 returned error can't find the container with id a7acfa73dc59a8921951bbdcb3be030aa17db3c79d86f5c8199b52c1209174d9
Sep 17 17:58:41.974662 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:41.974557    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f8ff7f478-9xdbz" event={"ID":"301a28ba-e70e-4112-a0ca-d7dad9bb0880","Type":"ContainerStarted","Data":"4745cb36ebb96e0ed633b72dcd432389e80620cfd0e2bb5ffc991132eb942ab6"}
Sep 17 17:58:41.975085 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:41.974776    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f8ff7f478-9xdbz"
Sep 17 17:58:41.976022 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:41.975995    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-gbkmm" event={"ID":"df8d85da-2c74-414a-9aa9-98270323c887","Type":"ContainerStarted","Data":"a7acfa73dc59a8921951bbdcb3be030aa17db3c79d86f5c8199b52c1209174d9"}
Sep 17 17:58:41.976557 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:41.976540    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f8ff7f478-9xdbz"
Sep 17 17:58:42.002242 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:42.002186    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f8ff7f478-9xdbz" podStartSLOduration=1.328288561 podStartE2EDuration="5.002169877s" totalImagesPullingTime="3.673881316s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:42.00060997 +0000 UTC m=+116.696700674" watchObservedRunningTime="2026-09-17 17:58:42.002169877 +0000 UTC m=+116.698260564"
Sep 17 17:58:43.983517 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:43.983479    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-gbkmm" event={"ID":"df8d85da-2c74-414a-9aa9-98270323c887","Type":"ContainerStarted","Data":"da442dcb498b7b94b797d1c57ded25004ed330801392968515415627f1436f18"}
Sep 17 17:58:43.983922 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:43.983542    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-gbkmm" event={"ID":"df8d85da-2c74-414a-9aa9-98270323c887","Type":"ContainerStarted","Data":"89510ba59c9cd75bde4f80d70e5b8351b0ac3b2afa969e708d6361e6ec0d2b42"}
Sep 17 17:58:44.004358 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:44.004289    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-gbkmm" podStartSLOduration=2.74602981 podStartE2EDuration="4.004268063s" totalImagesPullingTime="1.258238253s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:44.002786142 +0000 UTC m=+118.698876825" watchObservedRunningTime="2026-09-17 17:58:44.004268063 +0000 UTC m=+118.700358749"
Sep 17 17:58:46.613775 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:46.613741    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-fbfzt"]
Sep 17 17:58:46.618437 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:46.618417    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-fbfzt"
Sep 17 17:58:46.621843 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:46.621824    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-jqnms\""
Sep 17 17:58:46.623286 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:46.623267    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 17 17:58:46.623345 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:46.623323    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 17 17:58:46.624166 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:46.624142    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 17 17:58:46.636604 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:46.636583    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-fbfzt"]
Sep 17 17:58:46.681486 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:46.681452    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-6w2lq"]
Sep 17 17:58:46.684932 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:46.684909    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6w2lq"
Sep 17 17:58:46.700532 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:46.700515    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 17:58:46.702049 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:46.702033    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 17:58:46.702121 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:46.702049    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-kp7tj\""
Sep 17 17:58:46.702179 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:46.702138    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 17:58:46.779408 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:46.779373    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9eb07b72-ea20-4e7b-b92f-7bbf3a882223-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-fbfzt\" (UID: \"9eb07b72-ea20-4e7b-b92f-7bbf3a882223\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-fbfzt"
Sep 17 17:58:46.779582 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:46.779449    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9eb07b72-ea20-4e7b-b92f-7bbf3a882223-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-fbfzt\" (UID: \"9eb07b72-ea20-4e7b-b92f-7bbf3a882223\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-fbfzt"
Sep 17 17:58:46.779582 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:46.779486    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxxs9\" (UniqueName: \"kubernetes.io/projected/9eb07b72-ea20-4e7b-b92f-7bbf3a882223-kube-api-access-qxxs9\") pod \"kube-state-metrics-84c885658c-fbfzt\" (UID: \"9eb07b72-ea20-4e7b-b92f-7bbf3a882223\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-fbfzt"
Sep 17 17:58:46.779582 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:46.779509    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9eb07b72-ea20-4e7b-b92f-7bbf3a882223-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-fbfzt\" (UID: \"9eb07b72-ea20-4e7b-b92f-7bbf3a882223\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-fbfzt"
Sep 17 17:58:46.779582 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:46.779537    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9eb07b72-ea20-4e7b-b92f-7bbf3a882223-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-fbfzt\" (UID: \"9eb07b72-ea20-4e7b-b92f-7bbf3a882223\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-fbfzt"
Sep 17 17:58:46.779582 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:46.779555    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9eb07b72-ea20-4e7b-b92f-7bbf3a882223-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-fbfzt\" (UID: \"9eb07b72-ea20-4e7b-b92f-7bbf3a882223\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-fbfzt"
Sep 17 17:58:46.880382 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:46.880299    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9592822d-5079-442d-94ec-242a4f10ee82-node-exporter-tls\") pod \"node-exporter-6w2lq\" (UID: \"9592822d-5079-442d-94ec-242a4f10ee82\") " pod="openshift-monitoring/node-exporter-6w2lq"
Sep 17 17:58:46.880382 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:46.880337    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9592822d-5079-442d-94ec-242a4f10ee82-node-exporter-accelerators-collector-config\") pod \"node-exporter-6w2lq\" (UID: \"9592822d-5079-442d-94ec-242a4f10ee82\") " pod="openshift-monitoring/node-exporter-6w2lq"
Sep 17 17:58:46.880593 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:46.880392    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9eb07b72-ea20-4e7b-b92f-7bbf3a882223-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-fbfzt\" (UID: \"9eb07b72-ea20-4e7b-b92f-7bbf3a882223\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-fbfzt"
Sep 17 17:58:46.880593 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:46.880421    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9592822d-5079-442d-94ec-242a4f10ee82-sys\") pod \"node-exporter-6w2lq\" (UID: \"9592822d-5079-442d-94ec-242a4f10ee82\") " pod="openshift-monitoring/node-exporter-6w2lq"
Sep 17 17:58:46.880593 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:46.880446    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9592822d-5079-442d-94ec-242a4f10ee82-node-exporter-textfile\") pod \"node-exporter-6w2lq\" (UID: \"9592822d-5079-442d-94ec-242a4f10ee82\") " pod="openshift-monitoring/node-exporter-6w2lq"
Sep 17 17:58:46.880593 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:46.880490    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9592822d-5079-442d-94ec-242a4f10ee82-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6w2lq\" (UID: \"9592822d-5079-442d-94ec-242a4f10ee82\") " pod="openshift-monitoring/node-exporter-6w2lq"
Sep 17 17:58:46.880593 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:46.880530    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9eb07b72-ea20-4e7b-b92f-7bbf3a882223-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-fbfzt\" (UID: \"9eb07b72-ea20-4e7b-b92f-7bbf3a882223\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-fbfzt"
Sep 17 17:58:46.880882 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:46.880640    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9592822d-5079-442d-94ec-242a4f10ee82-root\") pod \"node-exporter-6w2lq\" (UID: \"9592822d-5079-442d-94ec-242a4f10ee82\") " pod="openshift-monitoring/node-exporter-6w2lq"
Sep 17 17:58:46.880882 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:46.880678    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxxs9\" (UniqueName: \"kubernetes.io/projected/9eb07b72-ea20-4e7b-b92f-7bbf3a882223-kube-api-access-qxxs9\") pod \"kube-state-metrics-84c885658c-fbfzt\" (UID: \"9eb07b72-ea20-4e7b-b92f-7bbf3a882223\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-fbfzt"
Sep 17 17:58:46.880882 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:46.880700    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9592822d-5079-442d-94ec-242a4f10ee82-node-exporter-wtmp\") pod \"node-exporter-6w2lq\" (UID: \"9592822d-5079-442d-94ec-242a4f10ee82\") " pod="openshift-monitoring/node-exporter-6w2lq"
Sep 17 17:58:46.880882 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:46.880815    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9eb07b72-ea20-4e7b-b92f-7bbf3a882223-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-fbfzt\" (UID: \"9eb07b72-ea20-4e7b-b92f-7bbf3a882223\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-fbfzt"
Sep 17 17:58:46.881096 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:46.880893    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdv86\" (UniqueName: \"kubernetes.io/projected/9592822d-5079-442d-94ec-242a4f10ee82-kube-api-access-wdv86\") pod \"node-exporter-6w2lq\" (UID: \"9592822d-5079-442d-94ec-242a4f10ee82\") " pod="openshift-monitoring/node-exporter-6w2lq"
Sep 17 17:58:46.881096 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:46.880917    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9eb07b72-ea20-4e7b-b92f-7bbf3a882223-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-fbfzt\" (UID: \"9eb07b72-ea20-4e7b-b92f-7bbf3a882223\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-fbfzt"
Sep 17 17:58:46.881096 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:46.880935    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9eb07b72-ea20-4e7b-b92f-7bbf3a882223-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-fbfzt\" (UID: \"9eb07b72-ea20-4e7b-b92f-7bbf3a882223\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-fbfzt"
Sep 17 17:58:46.881096 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:46.880963    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9592822d-5079-442d-94ec-242a4f10ee82-metrics-client-ca\") pod \"node-exporter-6w2lq\" (UID: \"9592822d-5079-442d-94ec-242a4f10ee82\") " pod="openshift-monitoring/node-exporter-6w2lq"
Sep 17 17:58:46.881096 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:58:46.881054    2553 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 17 17:58:46.881319 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:58:46.881112    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9eb07b72-ea20-4e7b-b92f-7bbf3a882223-kube-state-metrics-tls podName:9eb07b72-ea20-4e7b-b92f-7bbf3a882223 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:47.381097015 +0000 UTC m=+122.077187681 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/9eb07b72-ea20-4e7b-b92f-7bbf3a882223-kube-state-metrics-tls") pod "kube-state-metrics-84c885658c-fbfzt" (UID: "9eb07b72-ea20-4e7b-b92f-7bbf3a882223") : secret "kube-state-metrics-tls" not found
Sep 17 17:58:46.881319 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:46.881133    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9eb07b72-ea20-4e7b-b92f-7bbf3a882223-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-fbfzt\" (UID: \"9eb07b72-ea20-4e7b-b92f-7bbf3a882223\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-fbfzt"
Sep 17 17:58:46.881319 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:46.881200    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9eb07b72-ea20-4e7b-b92f-7bbf3a882223-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-fbfzt\" (UID: \"9eb07b72-ea20-4e7b-b92f-7bbf3a882223\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-fbfzt"
Sep 17 17:58:46.881319 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:46.881246    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9eb07b72-ea20-4e7b-b92f-7bbf3a882223-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-fbfzt\" (UID: \"9eb07b72-ea20-4e7b-b92f-7bbf3a882223\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-fbfzt"
Sep 17 17:58:46.883155 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:46.883132    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9eb07b72-ea20-4e7b-b92f-7bbf3a882223-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-fbfzt\" (UID: \"9eb07b72-ea20-4e7b-b92f-7bbf3a882223\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-fbfzt"
Sep 17 17:58:46.892099 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:46.892077    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxxs9\" (UniqueName: \"kubernetes.io/projected/9eb07b72-ea20-4e7b-b92f-7bbf3a882223-kube-api-access-qxxs9\") pod \"kube-state-metrics-84c885658c-fbfzt\" (UID: \"9eb07b72-ea20-4e7b-b92f-7bbf3a882223\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-fbfzt"
Sep 17 17:58:46.981852 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:46.981818    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9592822d-5079-442d-94ec-242a4f10ee82-metrics-client-ca\") pod \"node-exporter-6w2lq\" (UID: \"9592822d-5079-442d-94ec-242a4f10ee82\") " pod="openshift-monitoring/node-exporter-6w2lq"
Sep 17 17:58:46.981852 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:46.981852    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9592822d-5079-442d-94ec-242a4f10ee82-node-exporter-tls\") pod \"node-exporter-6w2lq\" (UID: \"9592822d-5079-442d-94ec-242a4f10ee82\") " pod="openshift-monitoring/node-exporter-6w2lq"
Sep 17 17:58:46.982089 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:46.981870    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9592822d-5079-442d-94ec-242a4f10ee82-node-exporter-accelerators-collector-config\") pod \"node-exporter-6w2lq\" (UID: \"9592822d-5079-442d-94ec-242a4f10ee82\") " pod="openshift-monitoring/node-exporter-6w2lq"
Sep 17 17:58:46.982089 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:46.981914    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9592822d-5079-442d-94ec-242a4f10ee82-sys\") pod \"node-exporter-6w2lq\" (UID: \"9592822d-5079-442d-94ec-242a4f10ee82\") " pod="openshift-monitoring/node-exporter-6w2lq"
Sep 17 17:58:46.982089 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:46.981937    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9592822d-5079-442d-94ec-242a4f10ee82-node-exporter-textfile\") pod \"node-exporter-6w2lq\" (UID: \"9592822d-5079-442d-94ec-242a4f10ee82\") " pod="openshift-monitoring/node-exporter-6w2lq"
Sep 17 17:58:46.982089 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:46.981983    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9592822d-5079-442d-94ec-242a4f10ee82-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6w2lq\" (UID: \"9592822d-5079-442d-94ec-242a4f10ee82\") " pod="openshift-monitoring/node-exporter-6w2lq"
Sep 17 17:58:46.982089 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:58:46.982025    2553 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 17 17:58:46.982089 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:46.982036    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9592822d-5079-442d-94ec-242a4f10ee82-sys\") pod \"node-exporter-6w2lq\" (UID: \"9592822d-5079-442d-94ec-242a4f10ee82\") " pod="openshift-monitoring/node-exporter-6w2lq"
Sep 17 17:58:46.982089 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:46.982066    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9592822d-5079-442d-94ec-242a4f10ee82-root\") pod \"node-exporter-6w2lq\" (UID: \"9592822d-5079-442d-94ec-242a4f10ee82\") " pod="openshift-monitoring/node-exporter-6w2lq"
Sep 17 17:58:46.982089 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:46.982026    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9592822d-5079-442d-94ec-242a4f10ee82-root\") pod \"node-exporter-6w2lq\" (UID: \"9592822d-5079-442d-94ec-242a4f10ee82\") " pod="openshift-monitoring/node-exporter-6w2lq"
Sep 17 17:58:46.982479 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:58:46.982101    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9592822d-5079-442d-94ec-242a4f10ee82-node-exporter-tls podName:9592822d-5079-442d-94ec-242a4f10ee82 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:47.482081157 +0000 UTC m=+122.178171848 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/9592822d-5079-442d-94ec-242a4f10ee82-node-exporter-tls") pod "node-exporter-6w2lq" (UID: "9592822d-5079-442d-94ec-242a4f10ee82") : secret "node-exporter-tls" not found
Sep 17 17:58:46.982479 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:46.982178    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9592822d-5079-442d-94ec-242a4f10ee82-node-exporter-wtmp\") pod \"node-exporter-6w2lq\" (UID: \"9592822d-5079-442d-94ec-242a4f10ee82\") " pod="openshift-monitoring/node-exporter-6w2lq"
Sep 17 17:58:46.982479 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:46.982212    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdv86\" (UniqueName: \"kubernetes.io/projected/9592822d-5079-442d-94ec-242a4f10ee82-kube-api-access-wdv86\") pod \"node-exporter-6w2lq\" (UID: \"9592822d-5079-442d-94ec-242a4f10ee82\") " pod="openshift-monitoring/node-exporter-6w2lq"
Sep 17 17:58:46.982479 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:46.982314    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9592822d-5079-442d-94ec-242a4f10ee82-node-exporter-textfile\") pod \"node-exporter-6w2lq\" (UID: \"9592822d-5079-442d-94ec-242a4f10ee82\") " pod="openshift-monitoring/node-exporter-6w2lq"
Sep 17 17:58:46.982479 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:46.982362    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9592822d-5079-442d-94ec-242a4f10ee82-node-exporter-wtmp\") pod \"node-exporter-6w2lq\" (UID: \"9592822d-5079-442d-94ec-242a4f10ee82\") " pod="openshift-monitoring/node-exporter-6w2lq"
Sep 17 17:58:46.982705 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:46.982543    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9592822d-5079-442d-94ec-242a4f10ee82-metrics-client-ca\") pod \"node-exporter-6w2lq\" (UID: \"9592822d-5079-442d-94ec-242a4f10ee82\") " pod="openshift-monitoring/node-exporter-6w2lq"
Sep 17 17:58:46.982705 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:46.982602    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9592822d-5079-442d-94ec-242a4f10ee82-node-exporter-accelerators-collector-config\") pod \"node-exporter-6w2lq\" (UID: \"9592822d-5079-442d-94ec-242a4f10ee82\") " pod="openshift-monitoring/node-exporter-6w2lq"
Sep 17 17:58:46.984365 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:46.984347    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9592822d-5079-442d-94ec-242a4f10ee82-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6w2lq\" (UID: \"9592822d-5079-442d-94ec-242a4f10ee82\") " pod="openshift-monitoring/node-exporter-6w2lq"
Sep 17 17:58:47.003696 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:47.003675    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdv86\" (UniqueName: \"kubernetes.io/projected/9592822d-5079-442d-94ec-242a4f10ee82-kube-api-access-wdv86\") pod \"node-exporter-6w2lq\" (UID: \"9592822d-5079-442d-94ec-242a4f10ee82\") " pod="openshift-monitoring/node-exporter-6w2lq"
Sep 17 17:58:47.203962 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:47.203928    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-c8f6d6959-bqc4q"
Sep 17 17:58:47.385999 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:47.385968    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9eb07b72-ea20-4e7b-b92f-7bbf3a882223-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-fbfzt\" (UID: \"9eb07b72-ea20-4e7b-b92f-7bbf3a882223\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-fbfzt"
Sep 17 17:58:47.388256 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:47.388231    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9eb07b72-ea20-4e7b-b92f-7bbf3a882223-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-fbfzt\" (UID: \"9eb07b72-ea20-4e7b-b92f-7bbf3a882223\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-fbfzt"
Sep 17 17:58:47.486464 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:47.486389    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9592822d-5079-442d-94ec-242a4f10ee82-node-exporter-tls\") pod \"node-exporter-6w2lq\" (UID: \"9592822d-5079-442d-94ec-242a4f10ee82\") " pod="openshift-monitoring/node-exporter-6w2lq"
Sep 17 17:58:47.488685 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:47.488664    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9592822d-5079-442d-94ec-242a4f10ee82-node-exporter-tls\") pod \"node-exporter-6w2lq\" (UID: \"9592822d-5079-442d-94ec-242a4f10ee82\") " pod="openshift-monitoring/node-exporter-6w2lq"
Sep 17 17:58:47.530242 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:47.530213    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-jqnms\""
Sep 17 17:58:47.537973 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:47.537956    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-fbfzt"
Sep 17 17:58:47.603644 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:47.603600    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-kp7tj\""
Sep 17 17:58:47.613258 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:47.613229    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6w2lq"
Sep 17 17:58:47.625013 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:58:47.624990    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9592822d_5079_442d_94ec_242a4f10ee82.slice/crio-dcf4083c0d4566e754a4133fee8fab32e7710839dd9b535099d2210b9673f168 WatchSource:0}: Error finding container dcf4083c0d4566e754a4133fee8fab32e7710839dd9b535099d2210b9673f168: Status 404 returned error can't find the container with id dcf4083c0d4566e754a4133fee8fab32e7710839dd9b535099d2210b9673f168
Sep 17 17:58:47.828882 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:47.828855    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-fbfzt"]
Sep 17 17:58:47.831041 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:58:47.831012    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eb07b72_ea20_4e7b_b92f_7bbf3a882223.slice/crio-522aae167976d28596d0133a576c903b9039f326db140c0339db2221c1f5ae8e WatchSource:0}: Error finding container 522aae167976d28596d0133a576c903b9039f326db140c0339db2221c1f5ae8e: Status 404 returned error can't find the container with id 522aae167976d28596d0133a576c903b9039f326db140c0339db2221c1f5ae8e
Sep 17 17:58:47.998685 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:47.998649    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6w2lq" event={"ID":"9592822d-5079-442d-94ec-242a4f10ee82","Type":"ContainerStarted","Data":"dcf4083c0d4566e754a4133fee8fab32e7710839dd9b535099d2210b9673f168"}
Sep 17 17:58:47.999707 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:47.999679    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-fbfzt" event={"ID":"9eb07b72-ea20-4e7b-b92f-7bbf3a882223","Type":"ContainerStarted","Data":"522aae167976d28596d0133a576c903b9039f326db140c0339db2221c1f5ae8e"}
Sep 17 17:58:50.016158 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:50.016123    2553 generic.go:356] "Generic (PLEG): container finished" podID="9592822d-5079-442d-94ec-242a4f10ee82" containerID="5d47cfa822d2798fed30eecc23ef6776377478b06829c52784bde64d3b0ec67d" exitCode=0
Sep 17 17:58:50.016648 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:50.016209    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6w2lq" event={"ID":"9592822d-5079-442d-94ec-242a4f10ee82","Type":"ContainerDied","Data":"5d47cfa822d2798fed30eecc23ef6776377478b06829c52784bde64d3b0ec67d"}
Sep 17 17:58:50.018995 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:50.018965    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-fbfzt" event={"ID":"9eb07b72-ea20-4e7b-b92f-7bbf3a882223","Type":"ContainerStarted","Data":"c2109a9c058d56596902c0131471216fe59f170c4503d5d8a929c57e1f99c745"}
Sep 17 17:58:50.019091 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:50.019025    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-fbfzt" event={"ID":"9eb07b72-ea20-4e7b-b92f-7bbf3a882223","Type":"ContainerStarted","Data":"2261e9ef2bf237fb55eebb3b9790039f774592d135d5ca191c5f930c7f3b2365"}
Sep 17 17:58:50.019091 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:50.019055    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-fbfzt" event={"ID":"9eb07b72-ea20-4e7b-b92f-7bbf3a882223","Type":"ContainerStarted","Data":"880694f0809265680fc3ac23f7967e91ff2967a565c4b5530a80578e25ddec2d"}
Sep 17 17:58:50.168798 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:50.168745    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-84c885658c-fbfzt" podStartSLOduration=2.86055229 podStartE2EDuration="4.168726449s" totalImagesPullingTime="1.308174159s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:50.168350023 +0000 UTC m=+124.864440707" watchObservedRunningTime="2026-09-17 17:58:50.168726449 +0000 UTC m=+124.864817135"
Sep 17 17:58:51.025993 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:51.025946    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6w2lq" event={"ID":"9592822d-5079-442d-94ec-242a4f10ee82","Type":"ContainerStarted","Data":"fcfd32257dc5e711f339a2e3a5530c708dc6b90eb7c7d033b9da1fe6ff8b49fe"}
Sep 17 17:58:51.026390 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:51.026022    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6w2lq" event={"ID":"9592822d-5079-442d-94ec-242a4f10ee82","Type":"ContainerStarted","Data":"c8a638533a75dc08a2d99629970af150309c6ae24931d6358ad06d5b5477e168"}
Sep 17 17:58:51.078105 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:51.078047    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-6w2lq" podStartSLOduration=3.4895435949999998 podStartE2EDuration="5.078024498s" totalImagesPullingTime="1.511971056s" totalInitContainerRuntime="76.509847ms" isStatefulPod=true podCreationTimestamp="2026-09-17 17:58:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:51.076726581 +0000 UTC m=+125.772817265" watchObservedRunningTime="2026-09-17 17:58:51.078024498 +0000 UTC m=+125.774115184"
Sep 17 17:58:51.137502 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:51.137470    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-b5zt5"]
Sep 17 17:58:51.140534 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:51.140513    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-b5zt5"
Sep 17 17:58:51.148531 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:51.148504    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-jd776\""
Sep 17 17:58:51.149090 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:51.149071    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 17 17:58:51.181551 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:51.181515    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-b5zt5"]
Sep 17 17:58:51.215588 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:51.215554    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ghsl\" (UniqueName: \"kubernetes.io/projected/e1a66bb3-53a8-4e8d-be66-ecf547963837-kube-api-access-6ghsl\") pod \"monitoring-plugin-5984459ff8-b5zt5\" (UID: \"e1a66bb3-53a8-4e8d-be66-ecf547963837\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-b5zt5"
Sep 17 17:58:51.215742 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:51.215641    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e1a66bb3-53a8-4e8d-be66-ecf547963837-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-b5zt5\" (UID: \"e1a66bb3-53a8-4e8d-be66-ecf547963837\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-b5zt5"
Sep 17 17:58:51.316977 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:51.316895    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ghsl\" (UniqueName: \"kubernetes.io/projected/e1a66bb3-53a8-4e8d-be66-ecf547963837-kube-api-access-6ghsl\") pod \"monitoring-plugin-5984459ff8-b5zt5\" (UID: \"e1a66bb3-53a8-4e8d-be66-ecf547963837\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-b5zt5"
Sep 17 17:58:51.317120 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:51.317034    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e1a66bb3-53a8-4e8d-be66-ecf547963837-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-b5zt5\" (UID: \"e1a66bb3-53a8-4e8d-be66-ecf547963837\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-b5zt5"
Sep 17 17:58:51.319500 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:51.319478    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e1a66bb3-53a8-4e8d-be66-ecf547963837-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-b5zt5\" (UID: \"e1a66bb3-53a8-4e8d-be66-ecf547963837\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-b5zt5"
Sep 17 17:58:51.331994 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:51.331966    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ghsl\" (UniqueName: \"kubernetes.io/projected/e1a66bb3-53a8-4e8d-be66-ecf547963837-kube-api-access-6ghsl\") pod \"monitoring-plugin-5984459ff8-b5zt5\" (UID: \"e1a66bb3-53a8-4e8d-be66-ecf547963837\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-b5zt5"
Sep 17 17:58:51.449928 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:51.449890    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-b5zt5"
Sep 17 17:58:51.730931 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:51.730897    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-b5zt5"]
Sep 17 17:58:51.734915 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:58:51.734882    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1a66bb3_53a8_4e8d_be66_ecf547963837.slice/crio-6c680bec33a7759876162ad778a08f9cd6781e7c9f4e164de02c08f11263c10e WatchSource:0}: Error finding container 6c680bec33a7759876162ad778a08f9cd6781e7c9f4e164de02c08f11263c10e: Status 404 returned error can't find the container with id 6c680bec33a7759876162ad778a08f9cd6781e7c9f4e164de02c08f11263c10e
Sep 17 17:58:52.031273 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:52.031185    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-b5zt5" event={"ID":"e1a66bb3-53a8-4e8d-be66-ecf547963837","Type":"ContainerStarted","Data":"6c680bec33a7759876162ad778a08f9cd6781e7c9f4e164de02c08f11263c10e"}
Sep 17 17:58:53.037096 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:53.037054    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-b5zt5" event={"ID":"e1a66bb3-53a8-4e8d-be66-ecf547963837","Type":"ContainerStarted","Data":"d4f19e6c3b35bebc3fbff5f56fa1b2a45850b24b91284f6b87303cfb8a2994aa"}
Sep 17 17:58:53.037502 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:53.037219    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-b5zt5"
Sep 17 17:58:53.042200 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:53.042174    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-b5zt5"
Sep 17 17:58:53.082231 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:53.082166    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5984459ff8-b5zt5" podStartSLOduration=0.879413176 podStartE2EDuration="2.082147798s" totalImagesPullingTime="1.202734622s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:53.078084166 +0000 UTC m=+127.774174852" watchObservedRunningTime="2026-09-17 17:58:53.082147798 +0000 UTC m=+127.778238482"
Sep 17 17:58:56.062724 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:56.062686    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfc1cb73-d5de-4902-b16c-5744271855c5-metrics-certs\") pod \"network-metrics-daemon-5gdn2\" (UID: \"dfc1cb73-d5de-4902-b16c-5744271855c5\") " pod="openshift-multus/network-metrics-daemon-5gdn2"
Sep 17 17:58:56.065042 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:56.065016    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfc1cb73-d5de-4902-b16c-5744271855c5-metrics-certs\") pod \"network-metrics-daemon-5gdn2\" (UID: \"dfc1cb73-d5de-4902-b16c-5744271855c5\") " pod="openshift-multus/network-metrics-daemon-5gdn2"
Sep 17 17:58:56.093445 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:56.093415    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-74sx5\""
Sep 17 17:58:56.101886 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:56.101852    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5gdn2"
Sep 17 17:58:56.394803 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:56.394778    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5gdn2"]
Sep 17 17:58:56.397003 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:58:56.396976    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfc1cb73_d5de_4902_b16c_5744271855c5.slice/crio-45728908d5e253432e1d8cdcfcb385a48a7a5648a0dcf8470801174ceb8faf6a WatchSource:0}: Error finding container 45728908d5e253432e1d8cdcfcb385a48a7a5648a0dcf8470801174ceb8faf6a: Status 404 returned error can't find the container with id 45728908d5e253432e1d8cdcfcb385a48a7a5648a0dcf8470801174ceb8faf6a
Sep 17 17:58:57.053885 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:57.053844    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5gdn2" event={"ID":"dfc1cb73-d5de-4902-b16c-5744271855c5","Type":"ContainerStarted","Data":"45728908d5e253432e1d8cdcfcb385a48a7a5648a0dcf8470801174ceb8faf6a"}
Sep 17 17:58:58.060419 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:58.060372    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5gdn2" event={"ID":"dfc1cb73-d5de-4902-b16c-5744271855c5","Type":"ContainerStarted","Data":"938719a4375f2d575a32cf3ea4a5a905cad7cda43bff1cc953c73a6b31b2afa2"}
Sep 17 17:58:58.060930 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:58.060448    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5gdn2" event={"ID":"dfc1cb73-d5de-4902-b16c-5744271855c5","Type":"ContainerStarted","Data":"867ccba0b66305acd9f58e7510563f3dbbad48b5b551ccea9f329c0182dec6af"}
Sep 17 17:58:58.098174 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:58.098118    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-5gdn2" podStartSLOduration=130.232699708 podStartE2EDuration="2m11.098101565s" totalImagesPullingTime="865.401857ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:56:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:58.095602915 +0000 UTC m=+132.791693598" watchObservedRunningTime="2026-09-17 17:58:58.098101565 +0000 UTC m=+132.794192249"
Sep 17 17:58:59.971079 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:58:59.971049    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5b5767bf9b-cckrb"
Sep 17 17:59:02.219562 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:02.219518    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-c8f6d6959-bqc4q" podUID="cb42b3bd-4639-4ab7-b044-b9cef036ae24" containerName="registry" containerID="cri-o://b5c7de08185d4926cee699b03c72605f299644589e9e61fd69e7c549e46094e6" gracePeriod=30
Sep 17 17:59:02.465176 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:02.465150    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c8f6d6959-bqc4q"
Sep 17 17:59:02.516341 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:02.516260    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cb42b3bd-4639-4ab7-b044-b9cef036ae24-image-registry-private-configuration\") pod \"cb42b3bd-4639-4ab7-b044-b9cef036ae24\" (UID: \"cb42b3bd-4639-4ab7-b044-b9cef036ae24\") "
Sep 17 17:59:02.516341 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:02.516298    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-259km\" (UniqueName: \"kubernetes.io/projected/cb42b3bd-4639-4ab7-b044-b9cef036ae24-kube-api-access-259km\") pod \"cb42b3bd-4639-4ab7-b044-b9cef036ae24\" (UID: \"cb42b3bd-4639-4ab7-b044-b9cef036ae24\") "
Sep 17 17:59:02.516341 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:02.516343    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cb42b3bd-4639-4ab7-b044-b9cef036ae24-ca-trust-extracted\") pod \"cb42b3bd-4639-4ab7-b044-b9cef036ae24\" (UID: \"cb42b3bd-4639-4ab7-b044-b9cef036ae24\") "
Sep 17 17:59:02.516606 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:02.516372    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cb42b3bd-4639-4ab7-b044-b9cef036ae24-installation-pull-secrets\") pod \"cb42b3bd-4639-4ab7-b044-b9cef036ae24\" (UID: \"cb42b3bd-4639-4ab7-b044-b9cef036ae24\") "
Sep 17 17:59:02.516606 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:02.516401    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cb42b3bd-4639-4ab7-b044-b9cef036ae24-registry-certificates\") pod \"cb42b3bd-4639-4ab7-b044-b9cef036ae24\" (UID: \"cb42b3bd-4639-4ab7-b044-b9cef036ae24\") "
Sep 17 17:59:02.516606 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:02.516432    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cb42b3bd-4639-4ab7-b044-b9cef036ae24-trusted-ca\") pod \"cb42b3bd-4639-4ab7-b044-b9cef036ae24\" (UID: \"cb42b3bd-4639-4ab7-b044-b9cef036ae24\") "
Sep 17 17:59:02.516606 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:02.516458    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cb42b3bd-4639-4ab7-b044-b9cef036ae24-registry-tls\") pod \"cb42b3bd-4639-4ab7-b044-b9cef036ae24\" (UID: \"cb42b3bd-4639-4ab7-b044-b9cef036ae24\") "
Sep 17 17:59:02.516606 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:02.516522    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cb42b3bd-4639-4ab7-b044-b9cef036ae24-bound-sa-token\") pod \"cb42b3bd-4639-4ab7-b044-b9cef036ae24\" (UID: \"cb42b3bd-4639-4ab7-b044-b9cef036ae24\") "
Sep 17 17:59:02.516874 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:02.516847    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb42b3bd-4639-4ab7-b044-b9cef036ae24-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "cb42b3bd-4639-4ab7-b044-b9cef036ae24" (UID: "cb42b3bd-4639-4ab7-b044-b9cef036ae24"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:59:02.517225 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:02.517190    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb42b3bd-4639-4ab7-b044-b9cef036ae24-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "cb42b3bd-4639-4ab7-b044-b9cef036ae24" (UID: "cb42b3bd-4639-4ab7-b044-b9cef036ae24"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:59:02.518918 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:02.518866    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb42b3bd-4639-4ab7-b044-b9cef036ae24-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "cb42b3bd-4639-4ab7-b044-b9cef036ae24" (UID: "cb42b3bd-4639-4ab7-b044-b9cef036ae24"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:59:02.518918 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:02.518888    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb42b3bd-4639-4ab7-b044-b9cef036ae24-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "cb42b3bd-4639-4ab7-b044-b9cef036ae24" (UID: "cb42b3bd-4639-4ab7-b044-b9cef036ae24"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:59:02.519066 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:02.518996    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb42b3bd-4639-4ab7-b044-b9cef036ae24-kube-api-access-259km" (OuterVolumeSpecName: "kube-api-access-259km") pod "cb42b3bd-4639-4ab7-b044-b9cef036ae24" (UID: "cb42b3bd-4639-4ab7-b044-b9cef036ae24"). InnerVolumeSpecName "kube-api-access-259km". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:59:02.519122 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:02.519104    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb42b3bd-4639-4ab7-b044-b9cef036ae24-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "cb42b3bd-4639-4ab7-b044-b9cef036ae24" (UID: "cb42b3bd-4639-4ab7-b044-b9cef036ae24"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:59:02.519334 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:02.519317    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb42b3bd-4639-4ab7-b044-b9cef036ae24-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "cb42b3bd-4639-4ab7-b044-b9cef036ae24" (UID: "cb42b3bd-4639-4ab7-b044-b9cef036ae24"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:59:02.525132 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:02.525106    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb42b3bd-4639-4ab7-b044-b9cef036ae24-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "cb42b3bd-4639-4ab7-b044-b9cef036ae24" (UID: "cb42b3bd-4639-4ab7-b044-b9cef036ae24"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:59:02.617953 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:02.617922    2553 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cb42b3bd-4639-4ab7-b044-b9cef036ae24-image-registry-private-configuration\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 17:59:02.617953 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:02.617950    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-259km\" (UniqueName: \"kubernetes.io/projected/cb42b3bd-4639-4ab7-b044-b9cef036ae24-kube-api-access-259km\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 17:59:02.617953 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:02.617961    2553 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cb42b3bd-4639-4ab7-b044-b9cef036ae24-ca-trust-extracted\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 17:59:02.618148 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:02.617970    2553 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cb42b3bd-4639-4ab7-b044-b9cef036ae24-installation-pull-secrets\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 17:59:02.618148 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:02.617979    2553 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cb42b3bd-4639-4ab7-b044-b9cef036ae24-registry-certificates\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 17:59:02.618148 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:02.617988    2553 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cb42b3bd-4639-4ab7-b044-b9cef036ae24-trusted-ca\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 17:59:02.618148 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:02.617996    2553 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cb42b3bd-4639-4ab7-b044-b9cef036ae24-registry-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 17:59:02.618148 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:02.618004    2553 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cb42b3bd-4639-4ab7-b044-b9cef036ae24-bound-sa-token\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 17:59:03.082328 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:03.082290    2553 generic.go:356] "Generic (PLEG): container finished" podID="cb42b3bd-4639-4ab7-b044-b9cef036ae24" containerID="b5c7de08185d4926cee699b03c72605f299644589e9e61fd69e7c549e46094e6" exitCode=0
Sep 17 17:59:03.082492 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:03.082332    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c8f6d6959-bqc4q"
Sep 17 17:59:03.082492 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:03.082341    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c8f6d6959-bqc4q" event={"ID":"cb42b3bd-4639-4ab7-b044-b9cef036ae24","Type":"ContainerDied","Data":"b5c7de08185d4926cee699b03c72605f299644589e9e61fd69e7c549e46094e6"}
Sep 17 17:59:03.082492 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:03.082413    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c8f6d6959-bqc4q" event={"ID":"cb42b3bd-4639-4ab7-b044-b9cef036ae24","Type":"ContainerDied","Data":"a5322fc57d375d85c6c04c8289babfabc2aa1575cad13594a9e16cb095afbed5"}
Sep 17 17:59:03.082492 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:03.082434    2553 scope.go:117] "RemoveContainer" containerID="b5c7de08185d4926cee699b03c72605f299644589e9e61fd69e7c549e46094e6"
Sep 17 17:59:03.091126 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:03.091109    2553 scope.go:117] "RemoveContainer" containerID="b5c7de08185d4926cee699b03c72605f299644589e9e61fd69e7c549e46094e6"
Sep 17 17:59:03.091403 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:59:03.091383    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5c7de08185d4926cee699b03c72605f299644589e9e61fd69e7c549e46094e6\": container with ID starting with b5c7de08185d4926cee699b03c72605f299644589e9e61fd69e7c549e46094e6 not found: ID does not exist" containerID="b5c7de08185d4926cee699b03c72605f299644589e9e61fd69e7c549e46094e6"
Sep 17 17:59:03.091471 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:03.091416    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5c7de08185d4926cee699b03c72605f299644589e9e61fd69e7c549e46094e6"} err="failed to get container status \"b5c7de08185d4926cee699b03c72605f299644589e9e61fd69e7c549e46094e6\": rpc error: code = NotFound desc = could not find container \"b5c7de08185d4926cee699b03c72605f299644589e9e61fd69e7c549e46094e6\": container with ID starting with b5c7de08185d4926cee699b03c72605f299644589e9e61fd69e7c549e46094e6 not found: ID does not exist"
Sep 17 17:59:03.108137 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:03.108114    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-c8f6d6959-bqc4q"]
Sep 17 17:59:03.116241 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:03.116214    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-c8f6d6959-bqc4q"]
Sep 17 17:59:03.375801 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:03.375713    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb42b3bd-4639-4ab7-b044-b9cef036ae24" path="/var/lib/kubelet/pods/cb42b3bd-4639-4ab7-b044-b9cef036ae24/volumes"
Sep 17 17:59:09.923027 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:09.922994    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7bf79c54bd-nr8jr"]
Sep 17 17:59:09.923403 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:09.923279    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb42b3bd-4639-4ab7-b044-b9cef036ae24" containerName="registry"
Sep 17 17:59:09.923403 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:09.923291    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb42b3bd-4639-4ab7-b044-b9cef036ae24" containerName="registry"
Sep 17 17:59:09.923403 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:09.923342    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb42b3bd-4639-4ab7-b044-b9cef036ae24" containerName="registry"
Sep 17 17:59:09.928095 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:09.928077    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bf79c54bd-nr8jr"
Sep 17 17:59:09.932104 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:09.932082    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 17:59:09.932249 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:09.932147    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 17:59:09.933635 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:09.933597    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 17:59:09.943723 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:09.943693    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-9m74q\""
Sep 17 17:59:09.943723 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:09.943708    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 17:59:09.945084 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:09.945056    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bf79c54bd-nr8jr"]
Sep 17 17:59:09.947200 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:09.947183    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 17:59:09.949401 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:09.949387    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 17:59:09.951832 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:09.951816    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 17:59:09.976162 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:09.976128    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/35595709-c4a6-473a-aff6-154a3f5fe81f-console-serving-cert\") pod \"console-7bf79c54bd-nr8jr\" (UID: \"35595709-c4a6-473a-aff6-154a3f5fe81f\") " pod="openshift-console/console-7bf79c54bd-nr8jr"
Sep 17 17:59:09.976334 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:09.976169    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/35595709-c4a6-473a-aff6-154a3f5fe81f-console-oauth-config\") pod \"console-7bf79c54bd-nr8jr\" (UID: \"35595709-c4a6-473a-aff6-154a3f5fe81f\") " pod="openshift-console/console-7bf79c54bd-nr8jr"
Sep 17 17:59:09.976334 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:09.976194    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/35595709-c4a6-473a-aff6-154a3f5fe81f-service-ca\") pod \"console-7bf79c54bd-nr8jr\" (UID: \"35595709-c4a6-473a-aff6-154a3f5fe81f\") " pod="openshift-console/console-7bf79c54bd-nr8jr"
Sep 17 17:59:09.976334 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:09.976229    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/35595709-c4a6-473a-aff6-154a3f5fe81f-oauth-serving-cert\") pod \"console-7bf79c54bd-nr8jr\" (UID: \"35595709-c4a6-473a-aff6-154a3f5fe81f\") " pod="openshift-console/console-7bf79c54bd-nr8jr"
Sep 17 17:59:09.976334 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:09.976270    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxks5\" (UniqueName: \"kubernetes.io/projected/35595709-c4a6-473a-aff6-154a3f5fe81f-kube-api-access-xxks5\") pod \"console-7bf79c54bd-nr8jr\" (UID: \"35595709-c4a6-473a-aff6-154a3f5fe81f\") " pod="openshift-console/console-7bf79c54bd-nr8jr"
Sep 17 17:59:09.976334 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:09.976316    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/35595709-c4a6-473a-aff6-154a3f5fe81f-console-config\") pod \"console-7bf79c54bd-nr8jr\" (UID: \"35595709-c4a6-473a-aff6-154a3f5fe81f\") " pod="openshift-console/console-7bf79c54bd-nr8jr"
Sep 17 17:59:10.077138 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:10.077103    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/35595709-c4a6-473a-aff6-154a3f5fe81f-console-oauth-config\") pod \"console-7bf79c54bd-nr8jr\" (UID: \"35595709-c4a6-473a-aff6-154a3f5fe81f\") " pod="openshift-console/console-7bf79c54bd-nr8jr"
Sep 17 17:59:10.077138 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:10.077137    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/35595709-c4a6-473a-aff6-154a3f5fe81f-service-ca\") pod \"console-7bf79c54bd-nr8jr\" (UID: \"35595709-c4a6-473a-aff6-154a3f5fe81f\") " pod="openshift-console/console-7bf79c54bd-nr8jr"
Sep 17 17:59:10.077388 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:10.077173    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/35595709-c4a6-473a-aff6-154a3f5fe81f-oauth-serving-cert\") pod \"console-7bf79c54bd-nr8jr\" (UID: \"35595709-c4a6-473a-aff6-154a3f5fe81f\") " pod="openshift-console/console-7bf79c54bd-nr8jr"
Sep 17 17:59:10.077388 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:10.077205    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxks5\" (UniqueName: \"kubernetes.io/projected/35595709-c4a6-473a-aff6-154a3f5fe81f-kube-api-access-xxks5\") pod \"console-7bf79c54bd-nr8jr\" (UID: \"35595709-c4a6-473a-aff6-154a3f5fe81f\") " pod="openshift-console/console-7bf79c54bd-nr8jr"
Sep 17 17:59:10.077388 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:10.077252    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/35595709-c4a6-473a-aff6-154a3f5fe81f-console-config\") pod \"console-7bf79c54bd-nr8jr\" (UID: \"35595709-c4a6-473a-aff6-154a3f5fe81f\") " pod="openshift-console/console-7bf79c54bd-nr8jr"
Sep 17 17:59:10.077388 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:10.077315    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/35595709-c4a6-473a-aff6-154a3f5fe81f-console-serving-cert\") pod \"console-7bf79c54bd-nr8jr\" (UID: \"35595709-c4a6-473a-aff6-154a3f5fe81f\") " pod="openshift-console/console-7bf79c54bd-nr8jr"
Sep 17 17:59:10.077993 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:10.077962    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/35595709-c4a6-473a-aff6-154a3f5fe81f-service-ca\") pod \"console-7bf79c54bd-nr8jr\" (UID: \"35595709-c4a6-473a-aff6-154a3f5fe81f\") " pod="openshift-console/console-7bf79c54bd-nr8jr"
Sep 17 17:59:10.078107 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:10.077964    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/35595709-c4a6-473a-aff6-154a3f5fe81f-oauth-serving-cert\") pod \"console-7bf79c54bd-nr8jr\" (UID: \"35595709-c4a6-473a-aff6-154a3f5fe81f\") " pod="openshift-console/console-7bf79c54bd-nr8jr"
Sep 17 17:59:10.078107 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:10.078042    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/35595709-c4a6-473a-aff6-154a3f5fe81f-console-config\") pod \"console-7bf79c54bd-nr8jr\" (UID: \"35595709-c4a6-473a-aff6-154a3f5fe81f\") " pod="openshift-console/console-7bf79c54bd-nr8jr"
Sep 17 17:59:10.079766 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:10.079748    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/35595709-c4a6-473a-aff6-154a3f5fe81f-console-serving-cert\") pod \"console-7bf79c54bd-nr8jr\" (UID: \"35595709-c4a6-473a-aff6-154a3f5fe81f\") " pod="openshift-console/console-7bf79c54bd-nr8jr"
Sep 17 17:59:10.079866 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:10.079845    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/35595709-c4a6-473a-aff6-154a3f5fe81f-console-oauth-config\") pod \"console-7bf79c54bd-nr8jr\" (UID: \"35595709-c4a6-473a-aff6-154a3f5fe81f\") " pod="openshift-console/console-7bf79c54bd-nr8jr"
Sep 17 17:59:10.088395 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:10.088374    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxks5\" (UniqueName: \"kubernetes.io/projected/35595709-c4a6-473a-aff6-154a3f5fe81f-kube-api-access-xxks5\") pod \"console-7bf79c54bd-nr8jr\" (UID: \"35595709-c4a6-473a-aff6-154a3f5fe81f\") " pod="openshift-console/console-7bf79c54bd-nr8jr"
Sep 17 17:59:10.237278 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:10.237190    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bf79c54bd-nr8jr"
Sep 17 17:59:10.523636 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:10.523546    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bf79c54bd-nr8jr"]
Sep 17 17:59:10.526384 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:59:10.526343    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35595709_c4a6_473a_aff6_154a3f5fe81f.slice/crio-c0220e7177b5df6de7da4bf5c790659d0bd25b2fae83e7ba33816ab994b667cb WatchSource:0}: Error finding container c0220e7177b5df6de7da4bf5c790659d0bd25b2fae83e7ba33816ab994b667cb: Status 404 returned error can't find the container with id c0220e7177b5df6de7da4bf5c790659d0bd25b2fae83e7ba33816ab994b667cb
Sep 17 17:59:11.114999 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:11.114958    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bf79c54bd-nr8jr" event={"ID":"35595709-c4a6-473a-aff6-154a3f5fe81f","Type":"ContainerStarted","Data":"c0220e7177b5df6de7da4bf5c790659d0bd25b2fae83e7ba33816ab994b667cb"}
Sep 17 17:59:12.122091 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:12.122013    2553 generic.go:356] "Generic (PLEG): container finished" podID="bb30f947-1c4d-478f-95b1-939d82415f95" containerID="bd70b83a95d11f5ad3b5542ed20b1bd7afb02951c9af9a23e549bba86969e566" exitCode=0
Sep 17 17:59:12.122540 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:12.122096    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-wrv4g" event={"ID":"bb30f947-1c4d-478f-95b1-939d82415f95","Type":"ContainerDied","Data":"bd70b83a95d11f5ad3b5542ed20b1bd7afb02951c9af9a23e549bba86969e566"}
Sep 17 17:59:12.122540 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:12.122492    2553 scope.go:117] "RemoveContainer" containerID="bd70b83a95d11f5ad3b5542ed20b1bd7afb02951c9af9a23e549bba86969e566"
Sep 17 17:59:15.137759 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:15.137693    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-wrv4g" event={"ID":"bb30f947-1c4d-478f-95b1-939d82415f95","Type":"ContainerStarted","Data":"dccb3cff5a441c0c7bbacb0bb97f62db51c595416d2f73623ab1959c8ec2982b"}
Sep 17 17:59:15.140191 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:15.140144    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bf79c54bd-nr8jr" event={"ID":"35595709-c4a6-473a-aff6-154a3f5fe81f","Type":"ContainerStarted","Data":"2087724a4f7aede7b1b2eecec19da0046e3dcb3ece3f90f00dc19705b8b38936"}
Sep 17 17:59:15.223924 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:15.223867    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7bf79c54bd-nr8jr" podStartSLOduration=2.09582022 podStartE2EDuration="6.22385015s" totalImagesPullingTime="4.12802993s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:59:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:59:15.222434886 +0000 UTC m=+149.918525609" watchObservedRunningTime="2026-09-17 17:59:15.22385015 +0000 UTC m=+149.919940834"
Sep 17 17:59:15.349641 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:15.349593    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dm5l8_05707133-87f8-4036-89b3-f783c8f5e563/dns/0.log"
Sep 17 17:59:15.374922 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:15.374883    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dm5l8_05707133-87f8-4036-89b3-f783c8f5e563/kube-rbac-proxy/0.log"
Sep 17 17:59:15.490277 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:15.490193    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4fdj7_56b9ff80-4b68-4449-bd38-e2e4d2de6aa2/dns-node-resolver/0.log"
Sep 17 17:59:19.749401 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:19.749368    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6545f6b45-r5zn8"]
Sep 17 17:59:19.754704 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:19.754686    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6545f6b45-r5zn8"
Sep 17 17:59:19.768215 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:19.768191    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 17 17:59:19.788812 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:19.788781    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6545f6b45-r5zn8"]
Sep 17 17:59:19.863807 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:19.863767    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9273459d-ae23-40b9-9a3b-716ddff44c06-service-ca\") pod \"console-6545f6b45-r5zn8\" (UID: \"9273459d-ae23-40b9-9a3b-716ddff44c06\") " pod="openshift-console/console-6545f6b45-r5zn8"
Sep 17 17:59:19.863971 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:19.863823    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9273459d-ae23-40b9-9a3b-716ddff44c06-console-config\") pod \"console-6545f6b45-r5zn8\" (UID: \"9273459d-ae23-40b9-9a3b-716ddff44c06\") " pod="openshift-console/console-6545f6b45-r5zn8"
Sep 17 17:59:19.863971 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:19.863891    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9273459d-ae23-40b9-9a3b-716ddff44c06-oauth-serving-cert\") pod \"console-6545f6b45-r5zn8\" (UID: \"9273459d-ae23-40b9-9a3b-716ddff44c06\") " pod="openshift-console/console-6545f6b45-r5zn8"
Sep 17 17:59:19.863971 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:19.863945    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9273459d-ae23-40b9-9a3b-716ddff44c06-console-serving-cert\") pod \"console-6545f6b45-r5zn8\" (UID: \"9273459d-ae23-40b9-9a3b-716ddff44c06\") " pod="openshift-console/console-6545f6b45-r5zn8"
Sep 17 17:59:19.863971 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:19.863968    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9273459d-ae23-40b9-9a3b-716ddff44c06-trusted-ca-bundle\") pod \"console-6545f6b45-r5zn8\" (UID: \"9273459d-ae23-40b9-9a3b-716ddff44c06\") " pod="openshift-console/console-6545f6b45-r5zn8"
Sep 17 17:59:19.864123 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:19.863988    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9273459d-ae23-40b9-9a3b-716ddff44c06-console-oauth-config\") pod \"console-6545f6b45-r5zn8\" (UID: \"9273459d-ae23-40b9-9a3b-716ddff44c06\") " pod="openshift-console/console-6545f6b45-r5zn8"
Sep 17 17:59:19.864123 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:19.864016    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wb6f\" (UniqueName: \"kubernetes.io/projected/9273459d-ae23-40b9-9a3b-716ddff44c06-kube-api-access-5wb6f\") pod \"console-6545f6b45-r5zn8\" (UID: \"9273459d-ae23-40b9-9a3b-716ddff44c06\") " pod="openshift-console/console-6545f6b45-r5zn8"
Sep 17 17:59:19.964734 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:19.964689    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9273459d-ae23-40b9-9a3b-716ddff44c06-service-ca\") pod \"console-6545f6b45-r5zn8\" (UID: \"9273459d-ae23-40b9-9a3b-716ddff44c06\") " pod="openshift-console/console-6545f6b45-r5zn8"
Sep 17 17:59:19.964926 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:19.964751    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9273459d-ae23-40b9-9a3b-716ddff44c06-console-config\") pod \"console-6545f6b45-r5zn8\" (UID: \"9273459d-ae23-40b9-9a3b-716ddff44c06\") " pod="openshift-console/console-6545f6b45-r5zn8"
Sep 17 17:59:19.964926 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:19.964784    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9273459d-ae23-40b9-9a3b-716ddff44c06-oauth-serving-cert\") pod \"console-6545f6b45-r5zn8\" (UID: \"9273459d-ae23-40b9-9a3b-716ddff44c06\") " pod="openshift-console/console-6545f6b45-r5zn8"
Sep 17 17:59:19.964926 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:19.964816    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9273459d-ae23-40b9-9a3b-716ddff44c06-console-serving-cert\") pod \"console-6545f6b45-r5zn8\" (UID: \"9273459d-ae23-40b9-9a3b-716ddff44c06\") " pod="openshift-console/console-6545f6b45-r5zn8"
Sep 17 17:59:19.964926 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:19.964844    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9273459d-ae23-40b9-9a3b-716ddff44c06-trusted-ca-bundle\") pod \"console-6545f6b45-r5zn8\" (UID: \"9273459d-ae23-40b9-9a3b-716ddff44c06\") " pod="openshift-console/console-6545f6b45-r5zn8"
Sep 17 17:59:19.964926 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:19.964871    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9273459d-ae23-40b9-9a3b-716ddff44c06-console-oauth-config\") pod \"console-6545f6b45-r5zn8\" (UID: \"9273459d-ae23-40b9-9a3b-716ddff44c06\") " pod="openshift-console/console-6545f6b45-r5zn8"
Sep 17 17:59:19.964926 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:19.964910    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wb6f\" (UniqueName: \"kubernetes.io/projected/9273459d-ae23-40b9-9a3b-716ddff44c06-kube-api-access-5wb6f\") pod \"console-6545f6b45-r5zn8\" (UID: \"9273459d-ae23-40b9-9a3b-716ddff44c06\") " pod="openshift-console/console-6545f6b45-r5zn8"
Sep 17 17:59:19.965610 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:19.965584    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9273459d-ae23-40b9-9a3b-716ddff44c06-service-ca\") pod \"console-6545f6b45-r5zn8\" (UID: \"9273459d-ae23-40b9-9a3b-716ddff44c06\") " pod="openshift-console/console-6545f6b45-r5zn8"
Sep 17 17:59:19.965610 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:19.965597    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9273459d-ae23-40b9-9a3b-716ddff44c06-oauth-serving-cert\") pod \"console-6545f6b45-r5zn8\" (UID: \"9273459d-ae23-40b9-9a3b-716ddff44c06\") " pod="openshift-console/console-6545f6b45-r5zn8"
Sep 17 17:59:19.965785 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:19.965607    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9273459d-ae23-40b9-9a3b-716ddff44c06-console-config\") pod \"console-6545f6b45-r5zn8\" (UID: \"9273459d-ae23-40b9-9a3b-716ddff44c06\") " pod="openshift-console/console-6545f6b45-r5zn8"
Sep 17 17:59:19.965836 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:19.965818    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9273459d-ae23-40b9-9a3b-716ddff44c06-trusted-ca-bundle\") pod \"console-6545f6b45-r5zn8\" (UID: \"9273459d-ae23-40b9-9a3b-716ddff44c06\") " pod="openshift-console/console-6545f6b45-r5zn8"
Sep 17 17:59:19.967919 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:19.967893    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9273459d-ae23-40b9-9a3b-716ddff44c06-console-oauth-config\") pod \"console-6545f6b45-r5zn8\" (UID: \"9273459d-ae23-40b9-9a3b-716ddff44c06\") " pod="openshift-console/console-6545f6b45-r5zn8"
Sep 17 17:59:19.968041 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:19.967961    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9273459d-ae23-40b9-9a3b-716ddff44c06-console-serving-cert\") pod \"console-6545f6b45-r5zn8\" (UID: \"9273459d-ae23-40b9-9a3b-716ddff44c06\") " pod="openshift-console/console-6545f6b45-r5zn8"
Sep 17 17:59:19.977865 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:19.977845    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wb6f\" (UniqueName: \"kubernetes.io/projected/9273459d-ae23-40b9-9a3b-716ddff44c06-kube-api-access-5wb6f\") pod \"console-6545f6b45-r5zn8\" (UID: \"9273459d-ae23-40b9-9a3b-716ddff44c06\") " pod="openshift-console/console-6545f6b45-r5zn8"
Sep 17 17:59:20.064053 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:20.063970    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6545f6b45-r5zn8"
Sep 17 17:59:20.237602 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:20.237567    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7bf79c54bd-nr8jr"
Sep 17 17:59:20.237814 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:20.237643    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7bf79c54bd-nr8jr"
Sep 17 17:59:20.242525 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:20.242500    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7bf79c54bd-nr8jr"
Sep 17 17:59:20.355863 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:20.355837    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6545f6b45-r5zn8"]
Sep 17 17:59:20.358305 ip-10-0-137-95 kubenswrapper[2553]: W0917 17:59:20.358282    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9273459d_ae23_40b9_9a3b_716ddff44c06.slice/crio-9c62a5e4ca653d70d7198dc27706ac3fc736cb82f8e066e1a89f4afa44a258d9 WatchSource:0}: Error finding container 9c62a5e4ca653d70d7198dc27706ac3fc736cb82f8e066e1a89f4afa44a258d9: Status 404 returned error can't find the container with id 9c62a5e4ca653d70d7198dc27706ac3fc736cb82f8e066e1a89f4afa44a258d9
Sep 17 17:59:21.164322 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:21.164278    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6545f6b45-r5zn8" event={"ID":"9273459d-ae23-40b9-9a3b-716ddff44c06","Type":"ContainerStarted","Data":"ddc22b3618280633aab685a924d7f9c379e7ee5b89761b49526da5e1bb3a920d"}
Sep 17 17:59:21.164904 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:21.164881    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6545f6b45-r5zn8" event={"ID":"9273459d-ae23-40b9-9a3b-716ddff44c06","Type":"ContainerStarted","Data":"9c62a5e4ca653d70d7198dc27706ac3fc736cb82f8e066e1a89f4afa44a258d9"}
Sep 17 17:59:21.169032 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:21.169011    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7bf79c54bd-nr8jr"
Sep 17 17:59:21.201336 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:21.201287    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6545f6b45-r5zn8" podStartSLOduration=2.201272302 podStartE2EDuration="2.201272302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:59:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:59:21.198549115 +0000 UTC m=+155.894639799" watchObservedRunningTime="2026-09-17 17:59:21.201272302 +0000 UTC m=+155.897362986"
Sep 17 17:59:22.169765 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:22.169729    2553 generic.go:356] "Generic (PLEG): container finished" podID="1ec26660-d938-4072-b5da-eaa607ea5ba5" containerID="c603b90c7cb6a0356947c6a102499a7b1b9d054f636ff860f8158feb02b58ec9" exitCode=0
Sep 17 17:59:22.170139 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:22.169801    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-sxqrz" event={"ID":"1ec26660-d938-4072-b5da-eaa607ea5ba5","Type":"ContainerDied","Data":"c603b90c7cb6a0356947c6a102499a7b1b9d054f636ff860f8158feb02b58ec9"}
Sep 17 17:59:22.170191 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:22.170174    2553 scope.go:117] "RemoveContainer" containerID="c603b90c7cb6a0356947c6a102499a7b1b9d054f636ff860f8158feb02b58ec9"
Sep 17 17:59:23.175499 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:23.175461    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-sxqrz" event={"ID":"1ec26660-d938-4072-b5da-eaa607ea5ba5","Type":"ContainerStarted","Data":"f975434d93303e8376353d3be7fba55f81d6a391777741bb13cb0fe1e44f1f70"}
Sep 17 17:59:30.064722 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:30.064635    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6545f6b45-r5zn8"
Sep 17 17:59:30.064722 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:30.064693    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6545f6b45-r5zn8"
Sep 17 17:59:30.071033 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:30.071009    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6545f6b45-r5zn8"
Sep 17 17:59:30.208303 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:30.208273    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6545f6b45-r5zn8"
Sep 17 17:59:30.314909 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:30.314768    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bf79c54bd-nr8jr"]
Sep 17 17:59:55.339887 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:55.339848    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7bf79c54bd-nr8jr" podUID="35595709-c4a6-473a-aff6-154a3f5fe81f" containerName="console" containerID="cri-o://2087724a4f7aede7b1b2eecec19da0046e3dcb3ece3f90f00dc19705b8b38936" gracePeriod=15
Sep 17 17:59:55.587793 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:55.587768    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bf79c54bd-nr8jr"
Sep 17 17:59:55.656684 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:55.656641    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/35595709-c4a6-473a-aff6-154a3f5fe81f-console-oauth-config\") pod \"35595709-c4a6-473a-aff6-154a3f5fe81f\" (UID: \"35595709-c4a6-473a-aff6-154a3f5fe81f\") "
Sep 17 17:59:55.656891 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:55.656713    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/35595709-c4a6-473a-aff6-154a3f5fe81f-oauth-serving-cert\") pod \"35595709-c4a6-473a-aff6-154a3f5fe81f\" (UID: \"35595709-c4a6-473a-aff6-154a3f5fe81f\") "
Sep 17 17:59:55.656891 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:55.656758    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/35595709-c4a6-473a-aff6-154a3f5fe81f-console-serving-cert\") pod \"35595709-c4a6-473a-aff6-154a3f5fe81f\" (UID: \"35595709-c4a6-473a-aff6-154a3f5fe81f\") "
Sep 17 17:59:55.656891 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:55.656786    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/35595709-c4a6-473a-aff6-154a3f5fe81f-service-ca\") pod \"35595709-c4a6-473a-aff6-154a3f5fe81f\" (UID: \"35595709-c4a6-473a-aff6-154a3f5fe81f\") "
Sep 17 17:59:55.656891 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:55.656819    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xxks5\" (UniqueName: \"kubernetes.io/projected/35595709-c4a6-473a-aff6-154a3f5fe81f-kube-api-access-xxks5\") pod \"35595709-c4a6-473a-aff6-154a3f5fe81f\" (UID: \"35595709-c4a6-473a-aff6-154a3f5fe81f\") "
Sep 17 17:59:55.656891 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:55.656854    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/35595709-c4a6-473a-aff6-154a3f5fe81f-console-config\") pod \"35595709-c4a6-473a-aff6-154a3f5fe81f\" (UID: \"35595709-c4a6-473a-aff6-154a3f5fe81f\") "
Sep 17 17:59:55.657243 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:55.657208    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35595709-c4a6-473a-aff6-154a3f5fe81f-service-ca" (OuterVolumeSpecName: "service-ca") pod "35595709-c4a6-473a-aff6-154a3f5fe81f" (UID: "35595709-c4a6-473a-aff6-154a3f5fe81f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:59:55.657357 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:55.657217    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35595709-c4a6-473a-aff6-154a3f5fe81f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "35595709-c4a6-473a-aff6-154a3f5fe81f" (UID: "35595709-c4a6-473a-aff6-154a3f5fe81f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:59:55.657357 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:55.657273    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35595709-c4a6-473a-aff6-154a3f5fe81f-console-config" (OuterVolumeSpecName: "console-config") pod "35595709-c4a6-473a-aff6-154a3f5fe81f" (UID: "35595709-c4a6-473a-aff6-154a3f5fe81f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:59:55.659100 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:55.659065    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35595709-c4a6-473a-aff6-154a3f5fe81f-kube-api-access-xxks5" (OuterVolumeSpecName: "kube-api-access-xxks5") pod "35595709-c4a6-473a-aff6-154a3f5fe81f" (UID: "35595709-c4a6-473a-aff6-154a3f5fe81f"). InnerVolumeSpecName "kube-api-access-xxks5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:59:55.659100 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:55.659084    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35595709-c4a6-473a-aff6-154a3f5fe81f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "35595709-c4a6-473a-aff6-154a3f5fe81f" (UID: "35595709-c4a6-473a-aff6-154a3f5fe81f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:59:55.659234 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:55.659157    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35595709-c4a6-473a-aff6-154a3f5fe81f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "35595709-c4a6-473a-aff6-154a3f5fe81f" (UID: "35595709-c4a6-473a-aff6-154a3f5fe81f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:59:55.758167 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:55.758127    2553 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/35595709-c4a6-473a-aff6-154a3f5fe81f-console-config\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 17:59:55.758167 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:55.758158    2553 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/35595709-c4a6-473a-aff6-154a3f5fe81f-console-oauth-config\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 17:59:55.758167 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:55.758168    2553 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/35595709-c4a6-473a-aff6-154a3f5fe81f-oauth-serving-cert\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 17:59:55.758167 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:55.758177    2553 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/35595709-c4a6-473a-aff6-154a3f5fe81f-console-serving-cert\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 17:59:55.758423 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:55.758185    2553 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/35595709-c4a6-473a-aff6-154a3f5fe81f-service-ca\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 17:59:55.758423 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:55.758193    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxks5\" (UniqueName: \"kubernetes.io/projected/35595709-c4a6-473a-aff6-154a3f5fe81f-kube-api-access-xxks5\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 17:59:56.307271 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:56.307237    2553 generic.go:356] "Generic (PLEG): container finished" podID="35595709-c4a6-473a-aff6-154a3f5fe81f" containerID="2087724a4f7aede7b1b2eecec19da0046e3dcb3ece3f90f00dc19705b8b38936" exitCode=0
Sep 17 17:59:56.307445 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:56.307280    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bf79c54bd-nr8jr"
Sep 17 17:59:56.307445 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:56.307318    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bf79c54bd-nr8jr" event={"ID":"35595709-c4a6-473a-aff6-154a3f5fe81f","Type":"ContainerDied","Data":"2087724a4f7aede7b1b2eecec19da0046e3dcb3ece3f90f00dc19705b8b38936"}
Sep 17 17:59:56.307445 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:56.307368    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bf79c54bd-nr8jr" event={"ID":"35595709-c4a6-473a-aff6-154a3f5fe81f","Type":"ContainerDied","Data":"c0220e7177b5df6de7da4bf5c790659d0bd25b2fae83e7ba33816ab994b667cb"}
Sep 17 17:59:56.307445 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:56.307389    2553 scope.go:117] "RemoveContainer" containerID="2087724a4f7aede7b1b2eecec19da0046e3dcb3ece3f90f00dc19705b8b38936"
Sep 17 17:59:56.316223 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:56.316203    2553 scope.go:117] "RemoveContainer" containerID="2087724a4f7aede7b1b2eecec19da0046e3dcb3ece3f90f00dc19705b8b38936"
Sep 17 17:59:56.316527 ip-10-0-137-95 kubenswrapper[2553]: E0917 17:59:56.316504    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2087724a4f7aede7b1b2eecec19da0046e3dcb3ece3f90f00dc19705b8b38936\": container with ID starting with 2087724a4f7aede7b1b2eecec19da0046e3dcb3ece3f90f00dc19705b8b38936 not found: ID does not exist" containerID="2087724a4f7aede7b1b2eecec19da0046e3dcb3ece3f90f00dc19705b8b38936"
Sep 17 17:59:56.316574 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:56.316536    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2087724a4f7aede7b1b2eecec19da0046e3dcb3ece3f90f00dc19705b8b38936"} err="failed to get container status \"2087724a4f7aede7b1b2eecec19da0046e3dcb3ece3f90f00dc19705b8b38936\": rpc error: code = NotFound desc = could not find container \"2087724a4f7aede7b1b2eecec19da0046e3dcb3ece3f90f00dc19705b8b38936\": container with ID starting with 2087724a4f7aede7b1b2eecec19da0046e3dcb3ece3f90f00dc19705b8b38936 not found: ID does not exist"
Sep 17 17:59:56.339204 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:56.339168    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bf79c54bd-nr8jr"]
Sep 17 17:59:56.349133 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:56.349103    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7bf79c54bd-nr8jr"]
Sep 17 17:59:57.375112 ip-10-0-137-95 kubenswrapper[2553]: I0917 17:59:57.375078    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35595709-c4a6-473a-aff6-154a3f5fe81f" path="/var/lib/kubelet/pods/35595709-c4a6-473a-aff6-154a3f5fe81f/volumes"
Sep 17 18:00:15.147270 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:00:15.147237    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6775d46968-dvgfd"]
Sep 17 18:00:15.147664 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:00:15.147516    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35595709-c4a6-473a-aff6-154a3f5fe81f" containerName="console"
Sep 17 18:00:15.147664 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:00:15.147527    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="35595709-c4a6-473a-aff6-154a3f5fe81f" containerName="console"
Sep 17 18:00:15.147664 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:00:15.147596    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="35595709-c4a6-473a-aff6-154a3f5fe81f" containerName="console"
Sep 17 18:00:15.149891 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:00:15.149871    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6775d46968-dvgfd"
Sep 17 18:00:15.172047 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:00:15.172024    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6775d46968-dvgfd"]
Sep 17 18:00:15.212015 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:00:15.211990    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/02d48b2a-50ec-4c99-9cd4-8ebed7a24968-oauth-serving-cert\") pod \"console-6775d46968-dvgfd\" (UID: \"02d48b2a-50ec-4c99-9cd4-8ebed7a24968\") " pod="openshift-console/console-6775d46968-dvgfd"
Sep 17 18:00:15.212128 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:00:15.212026    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/02d48b2a-50ec-4c99-9cd4-8ebed7a24968-service-ca\") pod \"console-6775d46968-dvgfd\" (UID: \"02d48b2a-50ec-4c99-9cd4-8ebed7a24968\") " pod="openshift-console/console-6775d46968-dvgfd"
Sep 17 18:00:15.212128 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:00:15.212048    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/02d48b2a-50ec-4c99-9cd4-8ebed7a24968-console-oauth-config\") pod \"console-6775d46968-dvgfd\" (UID: \"02d48b2a-50ec-4c99-9cd4-8ebed7a24968\") " pod="openshift-console/console-6775d46968-dvgfd"
Sep 17 18:00:15.212128 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:00:15.212070    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02d48b2a-50ec-4c99-9cd4-8ebed7a24968-trusted-ca-bundle\") pod \"console-6775d46968-dvgfd\" (UID: \"02d48b2a-50ec-4c99-9cd4-8ebed7a24968\") " pod="openshift-console/console-6775d46968-dvgfd"
Sep 17 18:00:15.212228 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:00:15.212139    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/02d48b2a-50ec-4c99-9cd4-8ebed7a24968-console-serving-cert\") pod \"console-6775d46968-dvgfd\" (UID: \"02d48b2a-50ec-4c99-9cd4-8ebed7a24968\") " pod="openshift-console/console-6775d46968-dvgfd"
Sep 17 18:00:15.212228 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:00:15.212179    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fghx8\" (UniqueName: \"kubernetes.io/projected/02d48b2a-50ec-4c99-9cd4-8ebed7a24968-kube-api-access-fghx8\") pod \"console-6775d46968-dvgfd\" (UID: \"02d48b2a-50ec-4c99-9cd4-8ebed7a24968\") " pod="openshift-console/console-6775d46968-dvgfd"
Sep 17 18:00:15.212228 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:00:15.212203    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/02d48b2a-50ec-4c99-9cd4-8ebed7a24968-console-config\") pod \"console-6775d46968-dvgfd\" (UID: \"02d48b2a-50ec-4c99-9cd4-8ebed7a24968\") " pod="openshift-console/console-6775d46968-dvgfd"
Sep 17 18:00:15.313458 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:00:15.313424    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/02d48b2a-50ec-4c99-9cd4-8ebed7a24968-oauth-serving-cert\") pod \"console-6775d46968-dvgfd\" (UID: \"02d48b2a-50ec-4c99-9cd4-8ebed7a24968\") " pod="openshift-console/console-6775d46968-dvgfd"
Sep 17 18:00:15.313458 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:00:15.313465    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/02d48b2a-50ec-4c99-9cd4-8ebed7a24968-service-ca\") pod \"console-6775d46968-dvgfd\" (UID: \"02d48b2a-50ec-4c99-9cd4-8ebed7a24968\") " pod="openshift-console/console-6775d46968-dvgfd"
Sep 17 18:00:15.313769 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:00:15.313494    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/02d48b2a-50ec-4c99-9cd4-8ebed7a24968-console-oauth-config\") pod \"console-6775d46968-dvgfd\" (UID: \"02d48b2a-50ec-4c99-9cd4-8ebed7a24968\") " pod="openshift-console/console-6775d46968-dvgfd"
Sep 17 18:00:15.313769 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:00:15.313521    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02d48b2a-50ec-4c99-9cd4-8ebed7a24968-trusted-ca-bundle\") pod \"console-6775d46968-dvgfd\" (UID: \"02d48b2a-50ec-4c99-9cd4-8ebed7a24968\") " pod="openshift-console/console-6775d46968-dvgfd"
Sep 17 18:00:15.313769 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:00:15.313539    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/02d48b2a-50ec-4c99-9cd4-8ebed7a24968-console-serving-cert\") pod \"console-6775d46968-dvgfd\" (UID: \"02d48b2a-50ec-4c99-9cd4-8ebed7a24968\") " pod="openshift-console/console-6775d46968-dvgfd"
Sep 17 18:00:15.313769 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:00:15.313566    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fghx8\" (UniqueName: \"kubernetes.io/projected/02d48b2a-50ec-4c99-9cd4-8ebed7a24968-kube-api-access-fghx8\") pod \"console-6775d46968-dvgfd\" (UID: \"02d48b2a-50ec-4c99-9cd4-8ebed7a24968\") " pod="openshift-console/console-6775d46968-dvgfd"
Sep 17 18:00:15.313769 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:00:15.313597    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/02d48b2a-50ec-4c99-9cd4-8ebed7a24968-console-config\") pod \"console-6775d46968-dvgfd\" (UID: \"02d48b2a-50ec-4c99-9cd4-8ebed7a24968\") " pod="openshift-console/console-6775d46968-dvgfd"
Sep 17 18:00:15.314363 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:00:15.314337    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/02d48b2a-50ec-4c99-9cd4-8ebed7a24968-service-ca\") pod \"console-6775d46968-dvgfd\" (UID: \"02d48b2a-50ec-4c99-9cd4-8ebed7a24968\") " pod="openshift-console/console-6775d46968-dvgfd"
Sep 17 18:00:15.314464 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:00:15.314396    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/02d48b2a-50ec-4c99-9cd4-8ebed7a24968-oauth-serving-cert\") pod \"console-6775d46968-dvgfd\" (UID: \"02d48b2a-50ec-4c99-9cd4-8ebed7a24968\") " pod="openshift-console/console-6775d46968-dvgfd"
Sep 17 18:00:15.314464 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:00:15.314415    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/02d48b2a-50ec-4c99-9cd4-8ebed7a24968-console-config\") pod \"console-6775d46968-dvgfd\" (UID: \"02d48b2a-50ec-4c99-9cd4-8ebed7a24968\") " pod="openshift-console/console-6775d46968-dvgfd"
Sep 17 18:00:15.314538 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:00:15.314475    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02d48b2a-50ec-4c99-9cd4-8ebed7a24968-trusted-ca-bundle\") pod \"console-6775d46968-dvgfd\" (UID: \"02d48b2a-50ec-4c99-9cd4-8ebed7a24968\") " pod="openshift-console/console-6775d46968-dvgfd"
Sep 17 18:00:15.316067 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:00:15.316039    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/02d48b2a-50ec-4c99-9cd4-8ebed7a24968-console-serving-cert\") pod \"console-6775d46968-dvgfd\" (UID: \"02d48b2a-50ec-4c99-9cd4-8ebed7a24968\") " pod="openshift-console/console-6775d46968-dvgfd"
Sep 17 18:00:15.316162 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:00:15.316076    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/02d48b2a-50ec-4c99-9cd4-8ebed7a24968-console-oauth-config\") pod \"console-6775d46968-dvgfd\" (UID: \"02d48b2a-50ec-4c99-9cd4-8ebed7a24968\") " pod="openshift-console/console-6775d46968-dvgfd"
Sep 17 18:00:15.322739 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:00:15.322720    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fghx8\" (UniqueName: \"kubernetes.io/projected/02d48b2a-50ec-4c99-9cd4-8ebed7a24968-kube-api-access-fghx8\") pod \"console-6775d46968-dvgfd\" (UID: \"02d48b2a-50ec-4c99-9cd4-8ebed7a24968\") " pod="openshift-console/console-6775d46968-dvgfd"
Sep 17 18:00:15.458519 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:00:15.458423    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6775d46968-dvgfd"
Sep 17 18:00:15.758759 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:00:15.758732    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6775d46968-dvgfd"]
Sep 17 18:00:15.761208 ip-10-0-137-95 kubenswrapper[2553]: W0917 18:00:15.761173    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02d48b2a_50ec_4c99_9cd4_8ebed7a24968.slice/crio-09ea56d79b0448989ecd0e3aa645555a6e818383f2148ff3182d3ab8bdb96bf2 WatchSource:0}: Error finding container 09ea56d79b0448989ecd0e3aa645555a6e818383f2148ff3182d3ab8bdb96bf2: Status 404 returned error can't find the container with id 09ea56d79b0448989ecd0e3aa645555a6e818383f2148ff3182d3ab8bdb96bf2
Sep 17 18:00:16.383251 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:00:16.383211    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6775d46968-dvgfd" event={"ID":"02d48b2a-50ec-4c99-9cd4-8ebed7a24968","Type":"ContainerStarted","Data":"02cf2f02eb21afee20bd9b9f0f835a0ce78a72b8c809bdbca535749c84b2ad57"}
Sep 17 18:00:16.383665 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:00:16.383271    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6775d46968-dvgfd" event={"ID":"02d48b2a-50ec-4c99-9cd4-8ebed7a24968","Type":"ContainerStarted","Data":"09ea56d79b0448989ecd0e3aa645555a6e818383f2148ff3182d3ab8bdb96bf2"}
Sep 17 18:00:16.402947 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:00:16.402897    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6775d46968-dvgfd" podStartSLOduration=1.402880361 podStartE2EDuration="1.402880361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:00:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:00:16.40240056 +0000 UTC m=+211.098491244" watchObservedRunningTime="2026-09-17 18:00:16.402880361 +0000 UTC m=+211.098971044"
Sep 17 18:00:25.459298 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:00:25.459267    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6775d46968-dvgfd"
Sep 17 18:00:25.459738 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:00:25.459408    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6775d46968-dvgfd"
Sep 17 18:00:25.464265 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:00:25.464241    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6775d46968-dvgfd"
Sep 17 18:00:26.421962 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:00:26.421923    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6775d46968-dvgfd"
Sep 17 18:00:26.471715 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:00:26.471677    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6545f6b45-r5zn8"]
Sep 17 18:00:51.491239 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:00:51.491186    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6545f6b45-r5zn8" podUID="9273459d-ae23-40b9-9a3b-716ddff44c06" containerName="console" containerID="cri-o://ddc22b3618280633aab685a924d7f9c379e7ee5b89761b49526da5e1bb3a920d" gracePeriod=15
Sep 17 18:00:51.731509 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:00:51.731475    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6545f6b45-r5zn8"
Sep 17 18:00:51.827867 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:00:51.827776    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9273459d-ae23-40b9-9a3b-716ddff44c06-console-oauth-config\") pod \"9273459d-ae23-40b9-9a3b-716ddff44c06\" (UID: \"9273459d-ae23-40b9-9a3b-716ddff44c06\") "
Sep 17 18:00:51.827867 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:00:51.827820    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9273459d-ae23-40b9-9a3b-716ddff44c06-service-ca\") pod \"9273459d-ae23-40b9-9a3b-716ddff44c06\" (UID: \"9273459d-ae23-40b9-9a3b-716ddff44c06\") "
Sep 17 18:00:51.827867 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:00:51.827848    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9273459d-ae23-40b9-9a3b-716ddff44c06-trusted-ca-bundle\") pod \"9273459d-ae23-40b9-9a3b-716ddff44c06\" (UID: \"9273459d-ae23-40b9-9a3b-716ddff44c06\") "
Sep 17 18:00:51.828136 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:00:51.827883    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9273459d-ae23-40b9-9a3b-716ddff44c06-oauth-serving-cert\") pod \"9273459d-ae23-40b9-9a3b-716ddff44c06\" (UID: \"9273459d-ae23-40b9-9a3b-716ddff44c06\") "
Sep 17 18:00:51.828136 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:00:51.827899    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9273459d-ae23-40b9-9a3b-716ddff44c06-console-serving-cert\") pod \"9273459d-ae23-40b9-9a3b-716ddff44c06\" (UID: \"9273459d-ae23-40b9-9a3b-716ddff44c06\") "
Sep 17 18:00:51.828136 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:00:51.827921    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5wb6f\" (UniqueName: \"kubernetes.io/projected/9273459d-ae23-40b9-9a3b-716ddff44c06-kube-api-access-5wb6f\") pod \"9273459d-ae23-40b9-9a3b-716ddff44c06\" (UID: \"9273459d-ae23-40b9-9a3b-716ddff44c06\") "
Sep 17 18:00:51.828136 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:00:51.827966    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9273459d-ae23-40b9-9a3b-716ddff44c06-console-config\") pod \"9273459d-ae23-40b9-9a3b-716ddff44c06\" (UID: \"9273459d-ae23-40b9-9a3b-716ddff44c06\") "
Sep 17 18:00:51.828337 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:00:51.828228    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9273459d-ae23-40b9-9a3b-716ddff44c06-service-ca" (OuterVolumeSpecName: "service-ca") pod "9273459d-ae23-40b9-9a3b-716ddff44c06" (UID: "9273459d-ae23-40b9-9a3b-716ddff44c06"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:00:51.828337 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:00:51.828293    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9273459d-ae23-40b9-9a3b-716ddff44c06-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "9273459d-ae23-40b9-9a3b-716ddff44c06" (UID: "9273459d-ae23-40b9-9a3b-716ddff44c06"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:00:51.828442 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:00:51.828377    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9273459d-ae23-40b9-9a3b-716ddff44c06-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9273459d-ae23-40b9-9a3b-716ddff44c06" (UID: "9273459d-ae23-40b9-9a3b-716ddff44c06"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:00:51.828442 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:00:51.828411    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9273459d-ae23-40b9-9a3b-716ddff44c06-console-config" (OuterVolumeSpecName: "console-config") pod "9273459d-ae23-40b9-9a3b-716ddff44c06" (UID: "9273459d-ae23-40b9-9a3b-716ddff44c06"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:00:51.830230 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:00:51.830206    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9273459d-ae23-40b9-9a3b-716ddff44c06-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9273459d-ae23-40b9-9a3b-716ddff44c06" (UID: "9273459d-ae23-40b9-9a3b-716ddff44c06"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:00:51.830325 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:00:51.830203    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9273459d-ae23-40b9-9a3b-716ddff44c06-kube-api-access-5wb6f" (OuterVolumeSpecName: "kube-api-access-5wb6f") pod "9273459d-ae23-40b9-9a3b-716ddff44c06" (UID: "9273459d-ae23-40b9-9a3b-716ddff44c06"). InnerVolumeSpecName "kube-api-access-5wb6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:00:51.830325 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:00:51.830282    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9273459d-ae23-40b9-9a3b-716ddff44c06-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9273459d-ae23-40b9-9a3b-716ddff44c06" (UID: "9273459d-ae23-40b9-9a3b-716ddff44c06"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:00:51.928961 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:00:51.928927    2553 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9273459d-ae23-40b9-9a3b-716ddff44c06-console-oauth-config\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:00:51.928961 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:00:51.928956    2553 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9273459d-ae23-40b9-9a3b-716ddff44c06-service-ca\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:00:51.928961 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:00:51.928966    2553 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9273459d-ae23-40b9-9a3b-716ddff44c06-trusted-ca-bundle\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:00:51.929168 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:00:51.928974    2553 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9273459d-ae23-40b9-9a3b-716ddff44c06-oauth-serving-cert\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:00:51.929168 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:00:51.928983    2553 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9273459d-ae23-40b9-9a3b-716ddff44c06-console-serving-cert\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:00:51.929168 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:00:51.928991    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wb6f\" (UniqueName: \"kubernetes.io/projected/9273459d-ae23-40b9-9a3b-716ddff44c06-kube-api-access-5wb6f\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:00:51.929168 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:00:51.928999    2553 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9273459d-ae23-40b9-9a3b-716ddff44c06-console-config\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:00:52.515166 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:00:52.515134    2553 generic.go:356] "Generic (PLEG): container finished" podID="9273459d-ae23-40b9-9a3b-716ddff44c06" containerID="ddc22b3618280633aab685a924d7f9c379e7ee5b89761b49526da5e1bb3a920d" exitCode=0
Sep 17 18:00:52.515577 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:00:52.515170    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6545f6b45-r5zn8"
Sep 17 18:00:52.515577 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:00:52.515219    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6545f6b45-r5zn8" event={"ID":"9273459d-ae23-40b9-9a3b-716ddff44c06","Type":"ContainerDied","Data":"ddc22b3618280633aab685a924d7f9c379e7ee5b89761b49526da5e1bb3a920d"}
Sep 17 18:00:52.515577 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:00:52.515278    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6545f6b45-r5zn8" event={"ID":"9273459d-ae23-40b9-9a3b-716ddff44c06","Type":"ContainerDied","Data":"9c62a5e4ca653d70d7198dc27706ac3fc736cb82f8e066e1a89f4afa44a258d9"}
Sep 17 18:00:52.515577 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:00:52.515304    2553 scope.go:117] "RemoveContainer" containerID="ddc22b3618280633aab685a924d7f9c379e7ee5b89761b49526da5e1bb3a920d"
Sep 17 18:00:52.523577 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:00:52.523554    2553 scope.go:117] "RemoveContainer" containerID="ddc22b3618280633aab685a924d7f9c379e7ee5b89761b49526da5e1bb3a920d"
Sep 17 18:00:52.523905 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:00:52.523886    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddc22b3618280633aab685a924d7f9c379e7ee5b89761b49526da5e1bb3a920d\": container with ID starting with ddc22b3618280633aab685a924d7f9c379e7ee5b89761b49526da5e1bb3a920d not found: ID does not exist" containerID="ddc22b3618280633aab685a924d7f9c379e7ee5b89761b49526da5e1bb3a920d"
Sep 17 18:00:52.523953 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:00:52.523915    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddc22b3618280633aab685a924d7f9c379e7ee5b89761b49526da5e1bb3a920d"} err="failed to get container status \"ddc22b3618280633aab685a924d7f9c379e7ee5b89761b49526da5e1bb3a920d\": rpc error: code = NotFound desc = could not find container \"ddc22b3618280633aab685a924d7f9c379e7ee5b89761b49526da5e1bb3a920d\": container with ID starting with ddc22b3618280633aab685a924d7f9c379e7ee5b89761b49526da5e1bb3a920d not found: ID does not exist"
Sep 17 18:00:52.536096 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:00:52.536068    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6545f6b45-r5zn8"]
Sep 17 18:00:52.539358 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:00:52.539332    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6545f6b45-r5zn8"]
Sep 17 18:00:53.375377 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:00:53.375344    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9273459d-ae23-40b9-9a3b-716ddff44c06" path="/var/lib/kubelet/pods/9273459d-ae23-40b9-9a3b-716ddff44c06/volumes"
Sep 17 18:01:26.415005 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:01:26.414968    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5f6fd68cf5-95h2z"]
Sep 17 18:01:26.415443 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:01:26.415235    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9273459d-ae23-40b9-9a3b-716ddff44c06" containerName="console"
Sep 17 18:01:26.415443 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:01:26.415248    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="9273459d-ae23-40b9-9a3b-716ddff44c06" containerName="console"
Sep 17 18:01:26.415443 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:01:26.415312    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="9273459d-ae23-40b9-9a3b-716ddff44c06" containerName="console"
Sep 17 18:01:26.418397 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:01:26.418374    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f6fd68cf5-95h2z"
Sep 17 18:01:26.431087 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:01:26.431065    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f6fd68cf5-95h2z"]
Sep 17 18:01:26.490277 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:01:26.490238    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f76634d3-2649-4c0d-bd7d-9c697344b3b5-trusted-ca-bundle\") pod \"console-5f6fd68cf5-95h2z\" (UID: \"f76634d3-2649-4c0d-bd7d-9c697344b3b5\") " pod="openshift-console/console-5f6fd68cf5-95h2z"
Sep 17 18:01:26.490277 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:01:26.490275    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f76634d3-2649-4c0d-bd7d-9c697344b3b5-oauth-serving-cert\") pod \"console-5f6fd68cf5-95h2z\" (UID: \"f76634d3-2649-4c0d-bd7d-9c697344b3b5\") " pod="openshift-console/console-5f6fd68cf5-95h2z"
Sep 17 18:01:26.490530 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:01:26.490295    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f76634d3-2649-4c0d-bd7d-9c697344b3b5-console-serving-cert\") pod \"console-5f6fd68cf5-95h2z\" (UID: \"f76634d3-2649-4c0d-bd7d-9c697344b3b5\") " pod="openshift-console/console-5f6fd68cf5-95h2z"
Sep 17 18:01:26.490530 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:01:26.490320    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f76634d3-2649-4c0d-bd7d-9c697344b3b5-console-oauth-config\") pod \"console-5f6fd68cf5-95h2z\" (UID: \"f76634d3-2649-4c0d-bd7d-9c697344b3b5\") " pod="openshift-console/console-5f6fd68cf5-95h2z"
Sep 17 18:01:26.490530 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:01:26.490433    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f76634d3-2649-4c0d-bd7d-9c697344b3b5-console-config\") pod \"console-5f6fd68cf5-95h2z\" (UID: \"f76634d3-2649-4c0d-bd7d-9c697344b3b5\") " pod="openshift-console/console-5f6fd68cf5-95h2z"
Sep 17 18:01:26.490530 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:01:26.490474    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f76634d3-2649-4c0d-bd7d-9c697344b3b5-service-ca\") pod \"console-5f6fd68cf5-95h2z\" (UID: \"f76634d3-2649-4c0d-bd7d-9c697344b3b5\") " pod="openshift-console/console-5f6fd68cf5-95h2z"
Sep 17 18:01:26.490530 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:01:26.490513    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjd4l\" (UniqueName: \"kubernetes.io/projected/f76634d3-2649-4c0d-bd7d-9c697344b3b5-kube-api-access-xjd4l\") pod \"console-5f6fd68cf5-95h2z\" (UID: \"f76634d3-2649-4c0d-bd7d-9c697344b3b5\") " pod="openshift-console/console-5f6fd68cf5-95h2z"
Sep 17 18:01:26.591681 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:01:26.591644    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f76634d3-2649-4c0d-bd7d-9c697344b3b5-trusted-ca-bundle\") pod \"console-5f6fd68cf5-95h2z\" (UID: \"f76634d3-2649-4c0d-bd7d-9c697344b3b5\") " pod="openshift-console/console-5f6fd68cf5-95h2z"
Sep 17 18:01:26.591889 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:01:26.591692    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f76634d3-2649-4c0d-bd7d-9c697344b3b5-oauth-serving-cert\") pod \"console-5f6fd68cf5-95h2z\" (UID: \"f76634d3-2649-4c0d-bd7d-9c697344b3b5\") " pod="openshift-console/console-5f6fd68cf5-95h2z"
Sep 17 18:01:26.591889 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:01:26.591722    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f76634d3-2649-4c0d-bd7d-9c697344b3b5-console-serving-cert\") pod \"console-5f6fd68cf5-95h2z\" (UID: \"f76634d3-2649-4c0d-bd7d-9c697344b3b5\") " pod="openshift-console/console-5f6fd68cf5-95h2z"
Sep 17 18:01:26.591889 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:01:26.591756    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f76634d3-2649-4c0d-bd7d-9c697344b3b5-console-oauth-config\") pod \"console-5f6fd68cf5-95h2z\" (UID: \"f76634d3-2649-4c0d-bd7d-9c697344b3b5\") " pod="openshift-console/console-5f6fd68cf5-95h2z"
Sep 17 18:01:26.591889 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:01:26.591799    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f76634d3-2649-4c0d-bd7d-9c697344b3b5-console-config\") pod \"console-5f6fd68cf5-95h2z\" (UID: \"f76634d3-2649-4c0d-bd7d-9c697344b3b5\") " pod="openshift-console/console-5f6fd68cf5-95h2z"
Sep 17 18:01:26.591889 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:01:26.591823    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f76634d3-2649-4c0d-bd7d-9c697344b3b5-service-ca\") pod \"console-5f6fd68cf5-95h2z\" (UID: \"f76634d3-2649-4c0d-bd7d-9c697344b3b5\") " pod="openshift-console/console-5f6fd68cf5-95h2z"
Sep 17 18:01:26.591889 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:01:26.591851    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjd4l\" (UniqueName: \"kubernetes.io/projected/f76634d3-2649-4c0d-bd7d-9c697344b3b5-kube-api-access-xjd4l\") pod \"console-5f6fd68cf5-95h2z\" (UID: \"f76634d3-2649-4c0d-bd7d-9c697344b3b5\") " pod="openshift-console/console-5f6fd68cf5-95h2z"
Sep 17 18:01:26.592505 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:01:26.592476    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f76634d3-2649-4c0d-bd7d-9c697344b3b5-oauth-serving-cert\") pod \"console-5f6fd68cf5-95h2z\" (UID: \"f76634d3-2649-4c0d-bd7d-9c697344b3b5\") " pod="openshift-console/console-5f6fd68cf5-95h2z"
Sep 17 18:01:26.592638 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:01:26.592547    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f76634d3-2649-4c0d-bd7d-9c697344b3b5-service-ca\") pod \"console-5f6fd68cf5-95h2z\" (UID: \"f76634d3-2649-4c0d-bd7d-9c697344b3b5\") " pod="openshift-console/console-5f6fd68cf5-95h2z"
Sep 17 18:01:26.592638 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:01:26.592587    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f76634d3-2649-4c0d-bd7d-9c697344b3b5-console-config\") pod \"console-5f6fd68cf5-95h2z\" (UID: \"f76634d3-2649-4c0d-bd7d-9c697344b3b5\") " pod="openshift-console/console-5f6fd68cf5-95h2z"
Sep 17 18:01:26.592820 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:01:26.592798    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f76634d3-2649-4c0d-bd7d-9c697344b3b5-trusted-ca-bundle\") pod \"console-5f6fd68cf5-95h2z\" (UID: \"f76634d3-2649-4c0d-bd7d-9c697344b3b5\") " pod="openshift-console/console-5f6fd68cf5-95h2z"
Sep 17 18:01:26.594309 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:01:26.594289    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f76634d3-2649-4c0d-bd7d-9c697344b3b5-console-serving-cert\") pod \"console-5f6fd68cf5-95h2z\" (UID: \"f76634d3-2649-4c0d-bd7d-9c697344b3b5\") " pod="openshift-console/console-5f6fd68cf5-95h2z"
Sep 17 18:01:26.594387 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:01:26.594327    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f76634d3-2649-4c0d-bd7d-9c697344b3b5-console-oauth-config\") pod \"console-5f6fd68cf5-95h2z\" (UID: \"f76634d3-2649-4c0d-bd7d-9c697344b3b5\") " pod="openshift-console/console-5f6fd68cf5-95h2z"
Sep 17 18:01:26.600746 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:01:26.600727    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjd4l\" (UniqueName: \"kubernetes.io/projected/f76634d3-2649-4c0d-bd7d-9c697344b3b5-kube-api-access-xjd4l\") pod \"console-5f6fd68cf5-95h2z\" (UID: \"f76634d3-2649-4c0d-bd7d-9c697344b3b5\") " pod="openshift-console/console-5f6fd68cf5-95h2z"
Sep 17 18:01:26.728721 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:01:26.728633    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f6fd68cf5-95h2z"
Sep 17 18:01:26.999287 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:01:26.999073    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f6fd68cf5-95h2z"]
Sep 17 18:01:27.001872 ip-10-0-137-95 kubenswrapper[2553]: W0917 18:01:27.001845    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf76634d3_2649_4c0d_bd7d_9c697344b3b5.slice/crio-16250abdb2a00eff59d0d440f87594a53cd8a49b7e03b11c9705a6620bf6d3e8 WatchSource:0}: Error finding container 16250abdb2a00eff59d0d440f87594a53cd8a49b7e03b11c9705a6620bf6d3e8: Status 404 returned error can't find the container with id 16250abdb2a00eff59d0d440f87594a53cd8a49b7e03b11c9705a6620bf6d3e8
Sep 17 18:01:27.649550 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:01:27.649516    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f6fd68cf5-95h2z" event={"ID":"f76634d3-2649-4c0d-bd7d-9c697344b3b5","Type":"ContainerStarted","Data":"051f92e2a19dfe59f3c33632a38b18aa3af7a99c8a552ff1355176a2aaa51e0f"}
Sep 17 18:01:27.649955 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:01:27.649564    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f6fd68cf5-95h2z" event={"ID":"f76634d3-2649-4c0d-bd7d-9c697344b3b5","Type":"ContainerStarted","Data":"16250abdb2a00eff59d0d440f87594a53cd8a49b7e03b11c9705a6620bf6d3e8"}
Sep 17 18:01:27.672734 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:01:27.672688    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5f6fd68cf5-95h2z" podStartSLOduration=1.672672 podStartE2EDuration="1.672672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:01:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:01:27.671345659 +0000 UTC m=+282.367436343" watchObservedRunningTime="2026-09-17 18:01:27.672672 +0000 UTC m=+282.368762683"
Sep 17 18:01:36.729612 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:01:36.729571    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5f6fd68cf5-95h2z"
Sep 17 18:01:36.730120 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:01:36.729701    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5f6fd68cf5-95h2z"
Sep 17 18:01:36.734449 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:01:36.734425    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5f6fd68cf5-95h2z"
Sep 17 18:01:37.690014 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:01:37.689984    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5f6fd68cf5-95h2z"
Sep 17 18:01:37.743985 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:01:37.743953    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6775d46968-dvgfd"]
Sep 17 18:01:47.263696 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:01:47.263665    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-btmqz_622ee5be-fb50-42b6-ae23-42900b7c2d98/ovn-acl-logging/0.log"
Sep 17 18:01:47.266970 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:01:47.266948    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-btmqz_622ee5be-fb50-42b6-ae23-42900b7c2d98/ovn-acl-logging/0.log"
Sep 17 18:01:47.269953 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:01:47.269928    2553 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 18:02:02.763232 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:02.763166    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6775d46968-dvgfd" podUID="02d48b2a-50ec-4c99-9cd4-8ebed7a24968" containerName="console" containerID="cri-o://02cf2f02eb21afee20bd9b9f0f835a0ce78a72b8c809bdbca535749c84b2ad57" gracePeriod=15
Sep 17 18:02:03.009652 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:03.009605    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6775d46968-dvgfd"
Sep 17 18:02:03.092754 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:03.092658    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/02d48b2a-50ec-4c99-9cd4-8ebed7a24968-console-config\") pod \"02d48b2a-50ec-4c99-9cd4-8ebed7a24968\" (UID: \"02d48b2a-50ec-4c99-9cd4-8ebed7a24968\") "
Sep 17 18:02:03.092754 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:03.092711    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/02d48b2a-50ec-4c99-9cd4-8ebed7a24968-oauth-serving-cert\") pod \"02d48b2a-50ec-4c99-9cd4-8ebed7a24968\" (UID: \"02d48b2a-50ec-4c99-9cd4-8ebed7a24968\") "
Sep 17 18:02:03.092754 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:03.092732    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/02d48b2a-50ec-4c99-9cd4-8ebed7a24968-console-serving-cert\") pod \"02d48b2a-50ec-4c99-9cd4-8ebed7a24968\" (UID: \"02d48b2a-50ec-4c99-9cd4-8ebed7a24968\") "
Sep 17 18:02:03.093033 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:03.092762    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fghx8\" (UniqueName: \"kubernetes.io/projected/02d48b2a-50ec-4c99-9cd4-8ebed7a24968-kube-api-access-fghx8\") pod \"02d48b2a-50ec-4c99-9cd4-8ebed7a24968\" (UID: \"02d48b2a-50ec-4c99-9cd4-8ebed7a24968\") "
Sep 17 18:02:03.093033 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:03.092805    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/02d48b2a-50ec-4c99-9cd4-8ebed7a24968-console-oauth-config\") pod \"02d48b2a-50ec-4c99-9cd4-8ebed7a24968\" (UID: \"02d48b2a-50ec-4c99-9cd4-8ebed7a24968\") "
Sep 17 18:02:03.093033 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:03.092836    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02d48b2a-50ec-4c99-9cd4-8ebed7a24968-trusted-ca-bundle\") pod \"02d48b2a-50ec-4c99-9cd4-8ebed7a24968\" (UID: \"02d48b2a-50ec-4c99-9cd4-8ebed7a24968\") "
Sep 17 18:02:03.093033 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:03.092894    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/02d48b2a-50ec-4c99-9cd4-8ebed7a24968-service-ca\") pod \"02d48b2a-50ec-4c99-9cd4-8ebed7a24968\" (UID: \"02d48b2a-50ec-4c99-9cd4-8ebed7a24968\") "
Sep 17 18:02:03.093233 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:03.093091    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02d48b2a-50ec-4c99-9cd4-8ebed7a24968-console-config" (OuterVolumeSpecName: "console-config") pod "02d48b2a-50ec-4c99-9cd4-8ebed7a24968" (UID: "02d48b2a-50ec-4c99-9cd4-8ebed7a24968"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:02:03.093291 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:03.093251    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02d48b2a-50ec-4c99-9cd4-8ebed7a24968-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "02d48b2a-50ec-4c99-9cd4-8ebed7a24968" (UID: "02d48b2a-50ec-4c99-9cd4-8ebed7a24968"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:02:03.093399 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:03.093369    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02d48b2a-50ec-4c99-9cd4-8ebed7a24968-service-ca" (OuterVolumeSpecName: "service-ca") pod "02d48b2a-50ec-4c99-9cd4-8ebed7a24968" (UID: "02d48b2a-50ec-4c99-9cd4-8ebed7a24968"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:02:03.093513 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:03.093393    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02d48b2a-50ec-4c99-9cd4-8ebed7a24968-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "02d48b2a-50ec-4c99-9cd4-8ebed7a24968" (UID: "02d48b2a-50ec-4c99-9cd4-8ebed7a24968"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:02:03.095144 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:03.095119    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02d48b2a-50ec-4c99-9cd4-8ebed7a24968-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "02d48b2a-50ec-4c99-9cd4-8ebed7a24968" (UID: "02d48b2a-50ec-4c99-9cd4-8ebed7a24968"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:02:03.095480 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:03.095458    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02d48b2a-50ec-4c99-9cd4-8ebed7a24968-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "02d48b2a-50ec-4c99-9cd4-8ebed7a24968" (UID: "02d48b2a-50ec-4c99-9cd4-8ebed7a24968"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:02:03.095526 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:03.095476    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02d48b2a-50ec-4c99-9cd4-8ebed7a24968-kube-api-access-fghx8" (OuterVolumeSpecName: "kube-api-access-fghx8") pod "02d48b2a-50ec-4c99-9cd4-8ebed7a24968" (UID: "02d48b2a-50ec-4c99-9cd4-8ebed7a24968"). InnerVolumeSpecName "kube-api-access-fghx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:02:03.194008 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:03.193972    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fghx8\" (UniqueName: \"kubernetes.io/projected/02d48b2a-50ec-4c99-9cd4-8ebed7a24968-kube-api-access-fghx8\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:02:03.194008 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:03.194005    2553 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/02d48b2a-50ec-4c99-9cd4-8ebed7a24968-console-oauth-config\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:02:03.194008 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:03.194014    2553 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02d48b2a-50ec-4c99-9cd4-8ebed7a24968-trusted-ca-bundle\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:02:03.194236 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:03.194023    2553 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/02d48b2a-50ec-4c99-9cd4-8ebed7a24968-service-ca\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:02:03.194236 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:03.194033    2553 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/02d48b2a-50ec-4c99-9cd4-8ebed7a24968-console-config\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:02:03.194236 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:03.194041    2553 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/02d48b2a-50ec-4c99-9cd4-8ebed7a24968-oauth-serving-cert\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:02:03.194236 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:03.194051    2553 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/02d48b2a-50ec-4c99-9cd4-8ebed7a24968-console-serving-cert\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:02:03.780679 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:03.780643    2553 generic.go:356] "Generic (PLEG): container finished" podID="02d48b2a-50ec-4c99-9cd4-8ebed7a24968" containerID="02cf2f02eb21afee20bd9b9f0f835a0ce78a72b8c809bdbca535749c84b2ad57" exitCode=0
Sep 17 18:02:03.780679 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:03.780676    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6775d46968-dvgfd"
Sep 17 18:02:03.781162 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:03.780697    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6775d46968-dvgfd" event={"ID":"02d48b2a-50ec-4c99-9cd4-8ebed7a24968","Type":"ContainerDied","Data":"02cf2f02eb21afee20bd9b9f0f835a0ce78a72b8c809bdbca535749c84b2ad57"}
Sep 17 18:02:03.781162 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:03.780742    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6775d46968-dvgfd" event={"ID":"02d48b2a-50ec-4c99-9cd4-8ebed7a24968","Type":"ContainerDied","Data":"09ea56d79b0448989ecd0e3aa645555a6e818383f2148ff3182d3ab8bdb96bf2"}
Sep 17 18:02:03.781162 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:03.780762    2553 scope.go:117] "RemoveContainer" containerID="02cf2f02eb21afee20bd9b9f0f835a0ce78a72b8c809bdbca535749c84b2ad57"
Sep 17 18:02:03.793561 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:03.793536    2553 scope.go:117] "RemoveContainer" containerID="02cf2f02eb21afee20bd9b9f0f835a0ce78a72b8c809bdbca535749c84b2ad57"
Sep 17 18:02:03.793930 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:02:03.793907    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02cf2f02eb21afee20bd9b9f0f835a0ce78a72b8c809bdbca535749c84b2ad57\": container with ID starting with 02cf2f02eb21afee20bd9b9f0f835a0ce78a72b8c809bdbca535749c84b2ad57 not found: ID does not exist" containerID="02cf2f02eb21afee20bd9b9f0f835a0ce78a72b8c809bdbca535749c84b2ad57"
Sep 17 18:02:03.793994 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:03.793939    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02cf2f02eb21afee20bd9b9f0f835a0ce78a72b8c809bdbca535749c84b2ad57"} err="failed to get container status \"02cf2f02eb21afee20bd9b9f0f835a0ce78a72b8c809bdbca535749c84b2ad57\": rpc error: code = NotFound desc = could not find container \"02cf2f02eb21afee20bd9b9f0f835a0ce78a72b8c809bdbca535749c84b2ad57\": container with ID starting with 02cf2f02eb21afee20bd9b9f0f835a0ce78a72b8c809bdbca535749c84b2ad57 not found: ID does not exist"
Sep 17 18:02:03.805870 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:03.805763    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6775d46968-dvgfd"]
Sep 17 18:02:03.819669 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:03.819636    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6775d46968-dvgfd"]
Sep 17 18:02:05.375275 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:05.375240    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02d48b2a-50ec-4c99-9cd4-8ebed7a24968" path="/var/lib/kubelet/pods/02d48b2a-50ec-4c99-9cd4-8ebed7a24968/volumes"
Sep 17 18:02:35.086536 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:35.086456    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adqhdn"]
Sep 17 18:02:35.087026 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:35.086736    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02d48b2a-50ec-4c99-9cd4-8ebed7a24968" containerName="console"
Sep 17 18:02:35.087026 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:35.086747    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="02d48b2a-50ec-4c99-9cd4-8ebed7a24968" containerName="console"
Sep 17 18:02:35.087026 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:35.086805    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="02d48b2a-50ec-4c99-9cd4-8ebed7a24968" containerName="console"
Sep 17 18:02:35.089596 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:35.089578    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adqhdn"
Sep 17 18:02:35.092036 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:35.092012    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 18:02:35.092154 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:35.092013    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 18:02:35.092154 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:35.092078    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-xllnw\""
Sep 17 18:02:35.097755 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:35.097734    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adqhdn"]
Sep 17 18:02:35.142509 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:35.142475    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjhxh\" (UniqueName: \"kubernetes.io/projected/d0172ca1-281e-413b-8f3e-617be9989064-kube-api-access-bjhxh\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adqhdn\" (UID: \"d0172ca1-281e-413b-8f3e-617be9989064\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adqhdn"
Sep 17 18:02:35.142696 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:35.142535    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d0172ca1-281e-413b-8f3e-617be9989064-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adqhdn\" (UID: \"d0172ca1-281e-413b-8f3e-617be9989064\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adqhdn"
Sep 17 18:02:35.142696 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:35.142575    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d0172ca1-281e-413b-8f3e-617be9989064-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adqhdn\" (UID: \"d0172ca1-281e-413b-8f3e-617be9989064\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adqhdn"
Sep 17 18:02:35.243954 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:35.243917    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjhxh\" (UniqueName: \"kubernetes.io/projected/d0172ca1-281e-413b-8f3e-617be9989064-kube-api-access-bjhxh\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adqhdn\" (UID: \"d0172ca1-281e-413b-8f3e-617be9989064\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adqhdn"
Sep 17 18:02:35.244087 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:35.243966    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d0172ca1-281e-413b-8f3e-617be9989064-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adqhdn\" (UID: \"d0172ca1-281e-413b-8f3e-617be9989064\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adqhdn"
Sep 17 18:02:35.244087 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:35.244000    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d0172ca1-281e-413b-8f3e-617be9989064-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adqhdn\" (UID: \"d0172ca1-281e-413b-8f3e-617be9989064\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adqhdn"
Sep 17 18:02:35.244430 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:35.244408    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d0172ca1-281e-413b-8f3e-617be9989064-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adqhdn\" (UID: \"d0172ca1-281e-413b-8f3e-617be9989064\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adqhdn"
Sep 17 18:02:35.244465 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:35.244422    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d0172ca1-281e-413b-8f3e-617be9989064-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adqhdn\" (UID: \"d0172ca1-281e-413b-8f3e-617be9989064\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adqhdn"
Sep 17 18:02:35.252489 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:35.252467    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjhxh\" (UniqueName: \"kubernetes.io/projected/d0172ca1-281e-413b-8f3e-617be9989064-kube-api-access-bjhxh\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adqhdn\" (UID: \"d0172ca1-281e-413b-8f3e-617be9989064\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adqhdn"
Sep 17 18:02:35.398738 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:35.398708    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adqhdn"
Sep 17 18:02:35.668858 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:35.668784    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adqhdn"]
Sep 17 18:02:35.672072 ip-10-0-137-95 kubenswrapper[2553]: W0917 18:02:35.672045    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0172ca1_281e_413b_8f3e_617be9989064.slice/crio-e4c6af648bb00219ec5d02a8cc29cec843cd0b62728594757e6ace790247cfdb WatchSource:0}: Error finding container e4c6af648bb00219ec5d02a8cc29cec843cd0b62728594757e6ace790247cfdb: Status 404 returned error can't find the container with id e4c6af648bb00219ec5d02a8cc29cec843cd0b62728594757e6ace790247cfdb
Sep 17 18:02:35.676794 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:35.676772    2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:02:35.900182 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:35.900145    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adqhdn" event={"ID":"d0172ca1-281e-413b-8f3e-617be9989064","Type":"ContainerStarted","Data":"e4c6af648bb00219ec5d02a8cc29cec843cd0b62728594757e6ace790247cfdb"}
Sep 17 18:02:41.924633 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:41.924587    2553 generic.go:356] "Generic (PLEG): container finished" podID="d0172ca1-281e-413b-8f3e-617be9989064" containerID="771430683f93d0ab202341123b23a6ac6deaf9d3f5222b55b1db04a893444de8" exitCode=0
Sep 17 18:02:41.924942 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:41.924641    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adqhdn" event={"ID":"d0172ca1-281e-413b-8f3e-617be9989064","Type":"ContainerDied","Data":"771430683f93d0ab202341123b23a6ac6deaf9d3f5222b55b1db04a893444de8"}
Sep 17 18:02:49.957304 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:49.957268    2553 generic.go:356] "Generic (PLEG): container finished" podID="d0172ca1-281e-413b-8f3e-617be9989064" containerID="5970319e73629f7136a9b26585c98c0e790ca9ad1b19e967749876defc252823" exitCode=0
Sep 17 18:02:49.957708 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:49.957328    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adqhdn" event={"ID":"d0172ca1-281e-413b-8f3e-617be9989064","Type":"ContainerDied","Data":"5970319e73629f7136a9b26585c98c0e790ca9ad1b19e967749876defc252823"}
Sep 17 18:02:56.990315 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:56.990279    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adqhdn" event={"ID":"d0172ca1-281e-413b-8f3e-617be9989064","Type":"ContainerStarted","Data":"ad06887c6e189926221900c926f4e4bf5ec832ed7d4a16d079b2dbb3a46d293b"}
Sep 17 18:02:57.021020 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:57.020962    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adqhdn" podStartSLOduration=1.478952088 podStartE2EDuration="22.020943737s" totalImagesPullingTime="20.541991649s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:02:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:02:57.019936641 +0000 UTC m=+371.716027326" watchObservedRunningTime="2026-09-17 18:02:57.020943737 +0000 UTC m=+371.717034426"
Sep 17 18:02:57.996492 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:57.996450    2553 generic.go:356] "Generic (PLEG): container finished" podID="d0172ca1-281e-413b-8f3e-617be9989064" containerID="ad06887c6e189926221900c926f4e4bf5ec832ed7d4a16d079b2dbb3a46d293b" exitCode=0
Sep 17 18:02:57.996940 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:57.996536    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adqhdn" event={"ID":"d0172ca1-281e-413b-8f3e-617be9989064","Type":"ContainerDied","Data":"ad06887c6e189926221900c926f4e4bf5ec832ed7d4a16d079b2dbb3a46d293b"}
Sep 17 18:02:59.121022 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:59.120995    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adqhdn"
Sep 17 18:02:59.151836 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:59.151805    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d0172ca1-281e-413b-8f3e-617be9989064-util\") pod \"d0172ca1-281e-413b-8f3e-617be9989064\" (UID: \"d0172ca1-281e-413b-8f3e-617be9989064\") "
Sep 17 18:02:59.151979 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:59.151847    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d0172ca1-281e-413b-8f3e-617be9989064-bundle\") pod \"d0172ca1-281e-413b-8f3e-617be9989064\" (UID: \"d0172ca1-281e-413b-8f3e-617be9989064\") "
Sep 17 18:02:59.151979 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:59.151896    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bjhxh\" (UniqueName: \"kubernetes.io/projected/d0172ca1-281e-413b-8f3e-617be9989064-kube-api-access-bjhxh\") pod \"d0172ca1-281e-413b-8f3e-617be9989064\" (UID: \"d0172ca1-281e-413b-8f3e-617be9989064\") "
Sep 17 18:02:59.152495 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:59.152472    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0172ca1-281e-413b-8f3e-617be9989064-bundle" (OuterVolumeSpecName: "bundle") pod "d0172ca1-281e-413b-8f3e-617be9989064" (UID: "d0172ca1-281e-413b-8f3e-617be9989064"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:02:59.154112 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:59.154086    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0172ca1-281e-413b-8f3e-617be9989064-kube-api-access-bjhxh" (OuterVolumeSpecName: "kube-api-access-bjhxh") pod "d0172ca1-281e-413b-8f3e-617be9989064" (UID: "d0172ca1-281e-413b-8f3e-617be9989064"). InnerVolumeSpecName "kube-api-access-bjhxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:02:59.158491 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:59.158462    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0172ca1-281e-413b-8f3e-617be9989064-util" (OuterVolumeSpecName: "util") pod "d0172ca1-281e-413b-8f3e-617be9989064" (UID: "d0172ca1-281e-413b-8f3e-617be9989064"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:02:59.252916 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:59.252827    2553 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d0172ca1-281e-413b-8f3e-617be9989064-bundle\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:02:59.252916 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:59.252862    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjhxh\" (UniqueName: \"kubernetes.io/projected/d0172ca1-281e-413b-8f3e-617be9989064-kube-api-access-bjhxh\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:02:59.252916 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:02:59.252873    2553 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d0172ca1-281e-413b-8f3e-617be9989064-util\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:03:00.005600 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:03:00.005568    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adqhdn" event={"ID":"d0172ca1-281e-413b-8f3e-617be9989064","Type":"ContainerDied","Data":"e4c6af648bb00219ec5d02a8cc29cec843cd0b62728594757e6ace790247cfdb"}
Sep 17 18:03:00.005600 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:03:00.005586    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adqhdn"
Sep 17 18:03:00.005864 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:03:00.005614    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4c6af648bb00219ec5d02a8cc29cec843cd0b62728594757e6ace790247cfdb"
Sep 17 18:03:03.704896 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:03:03.704860    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-lrqs7"]
Sep 17 18:03:03.705284 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:03:03.705165    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0172ca1-281e-413b-8f3e-617be9989064" containerName="extract"
Sep 17 18:03:03.705284 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:03:03.705176    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0172ca1-281e-413b-8f3e-617be9989064" containerName="extract"
Sep 17 18:03:03.705284 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:03:03.705198    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0172ca1-281e-413b-8f3e-617be9989064" containerName="util"
Sep 17 18:03:03.705284 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:03:03.705205    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0172ca1-281e-413b-8f3e-617be9989064" containerName="util"
Sep 17 18:03:03.705284 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:03:03.705214    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0172ca1-281e-413b-8f3e-617be9989064" containerName="pull"
Sep 17 18:03:03.705284 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:03:03.705218    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0172ca1-281e-413b-8f3e-617be9989064" containerName="pull"
Sep 17 18:03:03.705284 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:03:03.705276    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0172ca1-281e-413b-8f3e-617be9989064" containerName="extract"
Sep 17 18:03:03.711285 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:03:03.711254    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-lrqs7"
Sep 17 18:03:03.716026 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:03:03.715989    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 17 18:03:03.716026 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:03:03.715989    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 17 18:03:03.716220 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:03:03.716067    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-ljwg8\""
Sep 17 18:03:03.716977 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:03:03.716957    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 17 18:03:03.736325 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:03:03.736297    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-lrqs7"]
Sep 17 18:03:03.786278 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:03:03.786239    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/077831e7-55ab-4b48-acdd-7d2f7dd4ad18-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-lrqs7\" (UID: \"077831e7-55ab-4b48-acdd-7d2f7dd4ad18\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-lrqs7"
Sep 17 18:03:03.786474 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:03:03.786338    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sstm\" (UniqueName: \"kubernetes.io/projected/077831e7-55ab-4b48-acdd-7d2f7dd4ad18-kube-api-access-7sstm\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-lrqs7\" (UID: \"077831e7-55ab-4b48-acdd-7d2f7dd4ad18\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-lrqs7"
Sep 17 18:03:03.886754 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:03:03.886715    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/077831e7-55ab-4b48-acdd-7d2f7dd4ad18-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-lrqs7\" (UID: \"077831e7-55ab-4b48-acdd-7d2f7dd4ad18\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-lrqs7"
Sep 17 18:03:03.886917 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:03:03.886801    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sstm\" (UniqueName: \"kubernetes.io/projected/077831e7-55ab-4b48-acdd-7d2f7dd4ad18-kube-api-access-7sstm\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-lrqs7\" (UID: \"077831e7-55ab-4b48-acdd-7d2f7dd4ad18\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-lrqs7"
Sep 17 18:03:03.889116 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:03:03.889092    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/077831e7-55ab-4b48-acdd-7d2f7dd4ad18-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-lrqs7\" (UID: \"077831e7-55ab-4b48-acdd-7d2f7dd4ad18\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-lrqs7"
Sep 17 18:03:03.899676 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:03:03.899652    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sstm\" (UniqueName: \"kubernetes.io/projected/077831e7-55ab-4b48-acdd-7d2f7dd4ad18-kube-api-access-7sstm\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-lrqs7\" (UID: \"077831e7-55ab-4b48-acdd-7d2f7dd4ad18\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-lrqs7"
Sep 17 18:03:04.020933 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:03:04.020837    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-lrqs7"
Sep 17 18:03:04.345175 ip-10-0-137-95 kubenswrapper[2553]: W0917 18:03:04.345120    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod077831e7_55ab_4b48_acdd_7d2f7dd4ad18.slice/crio-d2541c1784d0188cce34cbe224dcc338cb8bb68a55cc47956dbb45d3782cd513 WatchSource:0}: Error finding container d2541c1784d0188cce34cbe224dcc338cb8bb68a55cc47956dbb45d3782cd513: Status 404 returned error can't find the container with id d2541c1784d0188cce34cbe224dcc338cb8bb68a55cc47956dbb45d3782cd513
Sep 17 18:03:04.347472 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:03:04.347449    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-lrqs7"]
Sep 17 18:03:05.025872 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:03:05.025837    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-lrqs7" event={"ID":"077831e7-55ab-4b48-acdd-7d2f7dd4ad18","Type":"ContainerStarted","Data":"d2541c1784d0188cce34cbe224dcc338cb8bb68a55cc47956dbb45d3782cd513"}
Sep 17 18:03:08.041066 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:03:08.041028    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-lrqs7" event={"ID":"077831e7-55ab-4b48-acdd-7d2f7dd4ad18","Type":"ContainerStarted","Data":"34a481f90b8d2aed63a9b7bcde510f0df1e68325c23c669bef7254c4e04d2738"}
Sep 17 18:03:08.041516 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:03:08.041157    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-lrqs7"
Sep 17 18:03:08.066144 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:03:08.066088    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-lrqs7" podStartSLOduration=2.125752789 podStartE2EDuration="5.06607282s" totalImagesPullingTime="2.940320031s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:03:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:03:08.064924928 +0000 UTC m=+382.761015614" watchObservedRunningTime="2026-09-17 18:03:08.06607282 +0000 UTC m=+382.762163505"
Sep 17 18:03:08.190931 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:03:08.190897    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-46zsh"]
Sep 17 18:03:08.193143 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:03:08.193126    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-46zsh"
Sep 17 18:03:08.195860 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:03:08.195835    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-4tnpl\""
Sep 17 18:03:08.196004 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:03:08.195856    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 17 18:03:08.196004 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:03:08.195868    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 17 18:03:08.206971 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:03:08.206944    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-46zsh"]
Sep 17 18:03:08.220376 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:03:08.220346    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdgr9\" (UniqueName: \"kubernetes.io/projected/c7ebcbc7-9dca-457a-a9d2-34af450fa39b-kube-api-access-jdgr9\") pod \"keda-operator-579c8958f7-46zsh\" (UID: \"c7ebcbc7-9dca-457a-a9d2-34af450fa39b\") " pod="openshift-keda/keda-operator-579c8958f7-46zsh"
Sep 17 18:03:08.220518 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:03:08.220398    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/c7ebcbc7-9dca-457a-a9d2-34af450fa39b-cabundle0\") pod \"keda-operator-579c8958f7-46zsh\" (UID: \"c7ebcbc7-9dca-457a-a9d2-34af450fa39b\") " pod="openshift-keda/keda-operator-579c8958f7-46zsh"
Sep 17 18:03:08.220518 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:03:08.220459    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c7ebcbc7-9dca-457a-a9d2-34af450fa39b-certificates\") pod \"keda-operator-579c8958f7-46zsh\" (UID: \"c7ebcbc7-9dca-457a-a9d2-34af450fa39b\") " pod="openshift-keda/keda-operator-579c8958f7-46zsh"
Sep 17 18:03:08.321321 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:03:08.321222    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/c7ebcbc7-9dca-457a-a9d2-34af450fa39b-cabundle0\") pod \"keda-operator-579c8958f7-46zsh\" (UID: \"c7ebcbc7-9dca-457a-a9d2-34af450fa39b\") " pod="openshift-keda/keda-operator-579c8958f7-46zsh"
Sep 17 18:03:08.321321 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:03:08.321266    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c7ebcbc7-9dca-457a-a9d2-34af450fa39b-certificates\") pod \"keda-operator-579c8958f7-46zsh\" (UID: \"c7ebcbc7-9dca-457a-a9d2-34af450fa39b\") " pod="openshift-keda/keda-operator-579c8958f7-46zsh"
Sep 17 18:03:08.321321 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:03:08.321308    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdgr9\" (UniqueName: \"kubernetes.io/projected/c7ebcbc7-9dca-457a-a9d2-34af450fa39b-kube-api-access-jdgr9\") pod \"keda-operator-579c8958f7-46zsh\" (UID: \"c7ebcbc7-9dca-457a-a9d2-34af450fa39b\") " pod="openshift-keda/keda-operator-579c8958f7-46zsh"
Sep 17 18:03:08.321614 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:03:08.321436    2553 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
Sep 17 18:03:08.321614 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:03:08.321461    2553 secret.go:281] references non-existent secret key: ca.crt
Sep 17 18:03:08.321614 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:03:08.321469    2553 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 17 18:03:08.321614 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:03:08.321494    2553 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-46zsh: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Sep 17 18:03:08.321614 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:03:08.321545    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7ebcbc7-9dca-457a-a9d2-34af450fa39b-certificates podName:c7ebcbc7-9dca-457a-a9d2-34af450fa39b nodeName:}" failed. No retries permitted until 2026-09-17 18:03:08.82152943 +0000 UTC m=+383.517620092 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/c7ebcbc7-9dca-457a-a9d2-34af450fa39b-certificates") pod "keda-operator-579c8958f7-46zsh" (UID: "c7ebcbc7-9dca-457a-a9d2-34af450fa39b") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Sep 17 18:03:08.322111 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:03:08.322086    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/c7ebcbc7-9dca-457a-a9d2-34af450fa39b-cabundle0\") pod \"keda-operator-579c8958f7-46zsh\" (UID: \"c7ebcbc7-9dca-457a-a9d2-34af450fa39b\") " pod="openshift-keda/keda-operator-579c8958f7-46zsh"
Sep 17 18:03:08.330676 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:03:08.330651    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdgr9\" (UniqueName: \"kubernetes.io/projected/c7ebcbc7-9dca-457a-a9d2-34af450fa39b-kube-api-access-jdgr9\") pod \"keda-operator-579c8958f7-46zsh\" (UID: \"c7ebcbc7-9dca-457a-a9d2-34af450fa39b\") " pod="openshift-keda/keda-operator-579c8958f7-46zsh"
Sep 17 18:03:08.825607 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:03:08.825575    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c7ebcbc7-9dca-457a-a9d2-34af450fa39b-certificates\") pod \"keda-operator-579c8958f7-46zsh\" (UID: \"c7ebcbc7-9dca-457a-a9d2-34af450fa39b\") " pod="openshift-keda/keda-operator-579c8958f7-46zsh"
Sep 17 18:03:08.827993 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:03:08.827971    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c7ebcbc7-9dca-457a-a9d2-34af450fa39b-certificates\") pod \"keda-operator-579c8958f7-46zsh\" (UID: \"c7ebcbc7-9dca-457a-a9d2-34af450fa39b\") " pod="openshift-keda/keda-operator-579c8958f7-46zsh"
Sep 17 18:03:08.924149 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:03:08.924116    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-wxsk8"]
Sep 17 18:03:08.926875 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:03:08.926852    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-wxsk8"
Sep 17 18:03:08.929926 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:03:08.929904    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 17 18:03:08.940245 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:03:08.940223    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-wxsk8"]
Sep 17 18:03:09.027577 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:03:09.027530    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grb7t\" (UniqueName: \"kubernetes.io/projected/12921c95-6d60-487c-bb3f-c080fc25c795-kube-api-access-grb7t\") pod \"keda-admission-6c4d864b86-wxsk8\" (UID: \"12921c95-6d60-487c-bb3f-c080fc25c795\") " pod="openshift-keda/keda-admission-6c4d864b86-wxsk8"
Sep 17 18:03:09.027790 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:03:09.027585    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/12921c95-6d60-487c-bb3f-c080fc25c795-certificates\") pod \"keda-admission-6c4d864b86-wxsk8\" (UID: \"12921c95-6d60-487c-bb3f-c080fc25c795\") " pod="openshift-keda/keda-admission-6c4d864b86-wxsk8"
Sep 17 18:03:09.103598 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:03:09.103509    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-46zsh"
Sep 17 18:03:09.128128 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:03:09.128099    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grb7t\" (UniqueName: \"kubernetes.io/projected/12921c95-6d60-487c-bb3f-c080fc25c795-kube-api-access-grb7t\") pod \"keda-admission-6c4d864b86-wxsk8\" (UID: \"12921c95-6d60-487c-bb3f-c080fc25c795\") " pod="openshift-keda/keda-admission-6c4d864b86-wxsk8"
Sep 17 18:03:09.128282 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:03:09.128138    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/12921c95-6d60-487c-bb3f-c080fc25c795-certificates\") pod \"keda-admission-6c4d864b86-wxsk8\" (UID: \"12921c95-6d60-487c-bb3f-c080fc25c795\") " pod="openshift-keda/keda-admission-6c4d864b86-wxsk8"
Sep 17 18:03:09.131254 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:03:09.131204    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/12921c95-6d60-487c-bb3f-c080fc25c795-certificates\") pod \"keda-admission-6c4d864b86-wxsk8\" (UID: \"12921c95-6d60-487c-bb3f-c080fc25c795\") " pod="openshift-keda/keda-admission-6c4d864b86-wxsk8"
Sep 17 18:03:09.140743 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:03:09.140713    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grb7t\" (UniqueName: \"kubernetes.io/projected/12921c95-6d60-487c-bb3f-c080fc25c795-kube-api-access-grb7t\") pod \"keda-admission-6c4d864b86-wxsk8\" (UID: \"12921c95-6d60-487c-bb3f-c080fc25c795\") " pod="openshift-keda/keda-admission-6c4d864b86-wxsk8"
Sep 17 18:03:09.237036 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:03:09.236995    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-wxsk8"
Sep 17 18:03:09.409312 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:03:09.409278    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-46zsh"]
Sep 17 18:03:09.417145 ip-10-0-137-95 kubenswrapper[2553]: W0917 18:03:09.417116    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7ebcbc7_9dca_457a_a9d2_34af450fa39b.slice/crio-843d596d55d61c4f01609a25bce5147769b22ee2db60e23c66048c09a6159129 WatchSource:0}: Error finding container 843d596d55d61c4f01609a25bce5147769b22ee2db60e23c66048c09a6159129: Status 404 returned error can't find the container with id 843d596d55d61c4f01609a25bce5147769b22ee2db60e23c66048c09a6159129
Sep 17 18:03:09.529156 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:03:09.529122    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-wxsk8"]
Sep 17 18:03:10.058061 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:03:10.058027    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-wxsk8" event={"ID":"12921c95-6d60-487c-bb3f-c080fc25c795","Type":"ContainerStarted","Data":"74ee3834d18688e0264983d581719dba60489078f149d422aa73831727ef9e46"}
Sep 17 18:03:10.059165 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:03:10.059143    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-46zsh" event={"ID":"c7ebcbc7-9dca-457a-a9d2-34af450fa39b","Type":"ContainerStarted","Data":"843d596d55d61c4f01609a25bce5147769b22ee2db60e23c66048c09a6159129"}
Sep 17 18:03:12.070532 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:03:12.070495    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-wxsk8" event={"ID":"12921c95-6d60-487c-bb3f-c080fc25c795","Type":"ContainerStarted","Data":"f64296c26fb7f426668f5b3e5d333f1cf43c301e45c8a19be9495b69ce569ad0"}
Sep 17 18:03:12.070918 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:03:12.070615    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-wxsk8"
Sep 17 18:03:12.088757 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:03:12.088700    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-wxsk8" podStartSLOduration=2.559757776 podStartE2EDuration="4.088679421s" totalImagesPullingTime="1.528921645s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:03:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:03:12.087740341 +0000 UTC m=+386.783831027" watchObservedRunningTime="2026-09-17 18:03:12.088679421 +0000 UTC m=+386.784770106"
Sep 17 18:03:21.112550 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:03:21.112508    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-46zsh" event={"ID":"c7ebcbc7-9dca-457a-a9d2-34af450fa39b","Type":"ContainerStarted","Data":"7bef3f5f55a74098acd95e2b682f1b8d0faf1bebe0e70139badcc0c15b6472de"}
Sep 17 18:03:21.112998 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:03:21.112610    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-46zsh"
Sep 17 18:03:21.136186 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:03:21.136134    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-46zsh" podStartSLOduration=2.151099451 podStartE2EDuration="13.136117386s" totalImagesPullingTime="10.985017935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:03:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:03:21.135422946 +0000 UTC m=+395.831513629" watchObservedRunningTime="2026-09-17 18:03:21.136117386 +0000 UTC m=+395.832208070"
Sep 17 18:03:29.054477 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:03:29.054444    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-lrqs7"
Sep 17 18:03:33.077526 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:03:33.077492    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-wxsk8"
Sep 17 18:03:42.119503 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:03:42.119464    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-46zsh"
Sep 17 18:04:14.802766 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:14.802685    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6d7695cc68-g67w9"]
Sep 17 18:04:14.805093 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:14.805075    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d7695cc68-g67w9"
Sep 17 18:04:14.818909 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:14.818881    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d7695cc68-g67w9"]
Sep 17 18:04:14.850436 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:14.850403    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3717632b-23a0-4096-98a3-f9ee8352293b-console-serving-cert\") pod \"console-6d7695cc68-g67w9\" (UID: \"3717632b-23a0-4096-98a3-f9ee8352293b\") " pod="openshift-console/console-6d7695cc68-g67w9"
Sep 17 18:04:14.850589 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:14.850443    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3717632b-23a0-4096-98a3-f9ee8352293b-console-oauth-config\") pod \"console-6d7695cc68-g67w9\" (UID: \"3717632b-23a0-4096-98a3-f9ee8352293b\") " pod="openshift-console/console-6d7695cc68-g67w9"
Sep 17 18:04:14.850589 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:14.850536    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3717632b-23a0-4096-98a3-f9ee8352293b-console-config\") pod \"console-6d7695cc68-g67w9\" (UID: \"3717632b-23a0-4096-98a3-f9ee8352293b\") " pod="openshift-console/console-6d7695cc68-g67w9"
Sep 17 18:04:14.850589 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:14.850578    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3717632b-23a0-4096-98a3-f9ee8352293b-service-ca\") pod \"console-6d7695cc68-g67w9\" (UID: \"3717632b-23a0-4096-98a3-f9ee8352293b\") " pod="openshift-console/console-6d7695cc68-g67w9"
Sep 17 18:04:14.850716 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:14.850607    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qwjq\" (UniqueName: \"kubernetes.io/projected/3717632b-23a0-4096-98a3-f9ee8352293b-kube-api-access-6qwjq\") pod \"console-6d7695cc68-g67w9\" (UID: \"3717632b-23a0-4096-98a3-f9ee8352293b\") " pod="openshift-console/console-6d7695cc68-g67w9"
Sep 17 18:04:14.850716 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:14.850667    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3717632b-23a0-4096-98a3-f9ee8352293b-trusted-ca-bundle\") pod \"console-6d7695cc68-g67w9\" (UID: \"3717632b-23a0-4096-98a3-f9ee8352293b\") " pod="openshift-console/console-6d7695cc68-g67w9"
Sep 17 18:04:14.850716 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:14.850688    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3717632b-23a0-4096-98a3-f9ee8352293b-oauth-serving-cert\") pod \"console-6d7695cc68-g67w9\" (UID: \"3717632b-23a0-4096-98a3-f9ee8352293b\") " pod="openshift-console/console-6d7695cc68-g67w9"
Sep 17 18:04:14.951974 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:14.951931    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3717632b-23a0-4096-98a3-f9ee8352293b-trusted-ca-bundle\") pod \"console-6d7695cc68-g67w9\" (UID: \"3717632b-23a0-4096-98a3-f9ee8352293b\") " pod="openshift-console/console-6d7695cc68-g67w9"
Sep 17 18:04:14.951974 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:14.951971    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3717632b-23a0-4096-98a3-f9ee8352293b-oauth-serving-cert\") pod \"console-6d7695cc68-g67w9\" (UID: \"3717632b-23a0-4096-98a3-f9ee8352293b\") " pod="openshift-console/console-6d7695cc68-g67w9"
Sep 17 18:04:14.952212 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:14.952001    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3717632b-23a0-4096-98a3-f9ee8352293b-console-serving-cert\") pod \"console-6d7695cc68-g67w9\" (UID: \"3717632b-23a0-4096-98a3-f9ee8352293b\") " pod="openshift-console/console-6d7695cc68-g67w9"
Sep 17 18:04:14.952212 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:14.952032    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3717632b-23a0-4096-98a3-f9ee8352293b-console-oauth-config\") pod \"console-6d7695cc68-g67w9\" (UID: \"3717632b-23a0-4096-98a3-f9ee8352293b\") " pod="openshift-console/console-6d7695cc68-g67w9"
Sep 17 18:04:14.952212 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:14.952061    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3717632b-23a0-4096-98a3-f9ee8352293b-console-config\") pod \"console-6d7695cc68-g67w9\" (UID: \"3717632b-23a0-4096-98a3-f9ee8352293b\") " pod="openshift-console/console-6d7695cc68-g67w9"
Sep 17 18:04:14.952212 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:14.952082    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3717632b-23a0-4096-98a3-f9ee8352293b-service-ca\") pod \"console-6d7695cc68-g67w9\" (UID: \"3717632b-23a0-4096-98a3-f9ee8352293b\") " pod="openshift-console/console-6d7695cc68-g67w9"
Sep 17 18:04:14.952212 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:14.952115    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qwjq\" (UniqueName: \"kubernetes.io/projected/3717632b-23a0-4096-98a3-f9ee8352293b-kube-api-access-6qwjq\") pod \"console-6d7695cc68-g67w9\" (UID: \"3717632b-23a0-4096-98a3-f9ee8352293b\") " pod="openshift-console/console-6d7695cc68-g67w9"
Sep 17 18:04:14.952838 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:14.952788    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3717632b-23a0-4096-98a3-f9ee8352293b-oauth-serving-cert\") pod \"console-6d7695cc68-g67w9\" (UID: \"3717632b-23a0-4096-98a3-f9ee8352293b\") " pod="openshift-console/console-6d7695cc68-g67w9"
Sep 17 18:04:14.952992 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:14.952965    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3717632b-23a0-4096-98a3-f9ee8352293b-console-config\") pod \"console-6d7695cc68-g67w9\" (UID: \"3717632b-23a0-4096-98a3-f9ee8352293b\") " pod="openshift-console/console-6d7695cc68-g67w9"
Sep 17 18:04:14.953035 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:14.952975    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3717632b-23a0-4096-98a3-f9ee8352293b-service-ca\") pod \"console-6d7695cc68-g67w9\" (UID: \"3717632b-23a0-4096-98a3-f9ee8352293b\") " pod="openshift-console/console-6d7695cc68-g67w9"
Sep 17 18:04:14.953331 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:14.953306    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3717632b-23a0-4096-98a3-f9ee8352293b-trusted-ca-bundle\") pod \"console-6d7695cc68-g67w9\" (UID: \"3717632b-23a0-4096-98a3-f9ee8352293b\") " pod="openshift-console/console-6d7695cc68-g67w9"
Sep 17 18:04:14.954570 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:14.954549    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3717632b-23a0-4096-98a3-f9ee8352293b-console-oauth-config\") pod \"console-6d7695cc68-g67w9\" (UID: \"3717632b-23a0-4096-98a3-f9ee8352293b\") " pod="openshift-console/console-6d7695cc68-g67w9"
Sep 17 18:04:14.954713 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:14.954693    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3717632b-23a0-4096-98a3-f9ee8352293b-console-serving-cert\") pod \"console-6d7695cc68-g67w9\" (UID: \"3717632b-23a0-4096-98a3-f9ee8352293b\") " pod="openshift-console/console-6d7695cc68-g67w9"
Sep 17 18:04:14.961970 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:14.961943    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qwjq\" (UniqueName: \"kubernetes.io/projected/3717632b-23a0-4096-98a3-f9ee8352293b-kube-api-access-6qwjq\") pod \"console-6d7695cc68-g67w9\" (UID: \"3717632b-23a0-4096-98a3-f9ee8352293b\") " pod="openshift-console/console-6d7695cc68-g67w9"
Sep 17 18:04:15.113665 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:15.113568    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d7695cc68-g67w9"
Sep 17 18:04:15.392289 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:15.392259    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d7695cc68-g67w9"]
Sep 17 18:04:15.395547 ip-10-0-137-95 kubenswrapper[2553]: W0917 18:04:15.395523    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3717632b_23a0_4096_98a3_f9ee8352293b.slice/crio-c56d1ae6ba7630a1309da82180cd10584b437dfde70d028040ad4e08deddfc60 WatchSource:0}: Error finding container c56d1ae6ba7630a1309da82180cd10584b437dfde70d028040ad4e08deddfc60: Status 404 returned error can't find the container with id c56d1ae6ba7630a1309da82180cd10584b437dfde70d028040ad4e08deddfc60
Sep 17 18:04:16.345656 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:16.345598    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d7695cc68-g67w9" event={"ID":"3717632b-23a0-4096-98a3-f9ee8352293b","Type":"ContainerStarted","Data":"556506f73b4f08d3ae0a844cfa49bcf8183e5a1ce14f1a1394dfaba3b4487b2c"}
Sep 17 18:04:16.346070 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:16.345680    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d7695cc68-g67w9" event={"ID":"3717632b-23a0-4096-98a3-f9ee8352293b","Type":"ContainerStarted","Data":"c56d1ae6ba7630a1309da82180cd10584b437dfde70d028040ad4e08deddfc60"}
Sep 17 18:04:16.367789 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:16.367724    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6d7695cc68-g67w9" podStartSLOduration=2.367687309 podStartE2EDuration="2.367687309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:04:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:04:16.366312173 +0000 UTC m=+451.062402857" watchObservedRunningTime="2026-09-17 18:04:16.367687309 +0000 UTC m=+451.063778027"
Sep 17 18:04:25.114129 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:25.114086    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6d7695cc68-g67w9"
Sep 17 18:04:25.114578 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:25.114158    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6d7695cc68-g67w9"
Sep 17 18:04:25.118954 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:25.118934    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6d7695cc68-g67w9"
Sep 17 18:04:25.386592 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:25.386559    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6d7695cc68-g67w9"
Sep 17 18:04:25.449296 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:25.449262    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f6fd68cf5-95h2z"]
Sep 17 18:04:29.709698 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:29.709662    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-bt5l6"]
Sep 17 18:04:29.711941 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:29.711923    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-bt5l6"
Sep 17 18:04:29.715120 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:29.715099    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 18:04:29.716134 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:29.716106    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 18:04:29.716252 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:29.716114    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 17 18:04:29.716252 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:29.716195    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-8nglh\""
Sep 17 18:04:29.726665 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:29.726642    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-bt5l6"]
Sep 17 18:04:29.778069 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:29.778030    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/82cc79ef-12c8-4304-934c-0528003d61e2-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-bt5l6\" (UID: \"82cc79ef-12c8-4304-934c-0528003d61e2\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-bt5l6"
Sep 17 18:04:29.778239 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:29.778168    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvgl4\" (UniqueName: \"kubernetes.io/projected/82cc79ef-12c8-4304-934c-0528003d61e2-kube-api-access-jvgl4\") pod \"llmisvc-controller-manager-645f9c8fd6-bt5l6\" (UID: \"82cc79ef-12c8-4304-934c-0528003d61e2\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-bt5l6"
Sep 17 18:04:29.879508 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:29.879467    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvgl4\" (UniqueName: \"kubernetes.io/projected/82cc79ef-12c8-4304-934c-0528003d61e2-kube-api-access-jvgl4\") pod \"llmisvc-controller-manager-645f9c8fd6-bt5l6\" (UID: \"82cc79ef-12c8-4304-934c-0528003d61e2\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-bt5l6"
Sep 17 18:04:29.879508 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:29.879513    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/82cc79ef-12c8-4304-934c-0528003d61e2-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-bt5l6\" (UID: \"82cc79ef-12c8-4304-934c-0528003d61e2\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-bt5l6"
Sep 17 18:04:29.882104 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:29.882078    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/82cc79ef-12c8-4304-934c-0528003d61e2-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-bt5l6\" (UID: \"82cc79ef-12c8-4304-934c-0528003d61e2\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-bt5l6"
Sep 17 18:04:29.890193 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:29.890168    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvgl4\" (UniqueName: \"kubernetes.io/projected/82cc79ef-12c8-4304-934c-0528003d61e2-kube-api-access-jvgl4\") pod \"llmisvc-controller-manager-645f9c8fd6-bt5l6\" (UID: \"82cc79ef-12c8-4304-934c-0528003d61e2\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-bt5l6"
Sep 17 18:04:30.022406 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:30.022312    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-bt5l6"
Sep 17 18:04:30.295483 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:30.295401    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-bt5l6"]
Sep 17 18:04:30.300120 ip-10-0-137-95 kubenswrapper[2553]: W0917 18:04:30.300084    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod82cc79ef_12c8_4304_934c_0528003d61e2.slice/crio-4a603bb3eeed17fbe921cb818e236257d3d8d3163d75a98c4c8a6b995ce07f13 WatchSource:0}: Error finding container 4a603bb3eeed17fbe921cb818e236257d3d8d3163d75a98c4c8a6b995ce07f13: Status 404 returned error can't find the container with id 4a603bb3eeed17fbe921cb818e236257d3d8d3163d75a98c4c8a6b995ce07f13
Sep 17 18:04:30.406508 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:30.406470    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-bt5l6" event={"ID":"82cc79ef-12c8-4304-934c-0528003d61e2","Type":"ContainerStarted","Data":"4a603bb3eeed17fbe921cb818e236257d3d8d3163d75a98c4c8a6b995ce07f13"}
Sep 17 18:04:33.424252 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:33.424210    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-bt5l6" event={"ID":"82cc79ef-12c8-4304-934c-0528003d61e2","Type":"ContainerStarted","Data":"e98b94a0e68db9c09f27c9cd991f4f9ba08152fc3732f8b861dd64ab208767e0"}
Sep 17 18:04:33.424725 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:33.424287    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-bt5l6"
Sep 17 18:04:33.444116 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:33.444061    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-645f9c8fd6-bt5l6" podStartSLOduration=1.9873765909999999 podStartE2EDuration="4.444045188s" totalImagesPullingTime="2.456668597s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:04:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:04:33.442498132 +0000 UTC m=+468.138588815" watchObservedRunningTime="2026-09-17 18:04:33.444045188 +0000 UTC m=+468.140135871"
Sep 17 18:04:50.468868 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:50.468805    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5f6fd68cf5-95h2z" podUID="f76634d3-2649-4c0d-bd7d-9c697344b3b5" containerName="console" containerID="cri-o://051f92e2a19dfe59f3c33632a38b18aa3af7a99c8a552ff1355176a2aaa51e0f" gracePeriod=15
Sep 17 18:04:50.705241 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:50.705213    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f6fd68cf5-95h2z"
Sep 17 18:04:50.867274 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:50.867174    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f76634d3-2649-4c0d-bd7d-9c697344b3b5-oauth-serving-cert\") pod \"f76634d3-2649-4c0d-bd7d-9c697344b3b5\" (UID: \"f76634d3-2649-4c0d-bd7d-9c697344b3b5\") "
Sep 17 18:04:50.867274 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:50.867230    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f76634d3-2649-4c0d-bd7d-9c697344b3b5-console-oauth-config\") pod \"f76634d3-2649-4c0d-bd7d-9c697344b3b5\" (UID: \"f76634d3-2649-4c0d-bd7d-9c697344b3b5\") "
Sep 17 18:04:50.867274 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:50.867252    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjd4l\" (UniqueName: \"kubernetes.io/projected/f76634d3-2649-4c0d-bd7d-9c697344b3b5-kube-api-access-xjd4l\") pod \"f76634d3-2649-4c0d-bd7d-9c697344b3b5\" (UID: \"f76634d3-2649-4c0d-bd7d-9c697344b3b5\") "
Sep 17 18:04:50.867563 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:50.867403    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f76634d3-2649-4c0d-bd7d-9c697344b3b5-trusted-ca-bundle\") pod \"f76634d3-2649-4c0d-bd7d-9c697344b3b5\" (UID: \"f76634d3-2649-4c0d-bd7d-9c697344b3b5\") "
Sep 17 18:04:50.867563 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:50.867508    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f76634d3-2649-4c0d-bd7d-9c697344b3b5-console-serving-cert\") pod \"f76634d3-2649-4c0d-bd7d-9c697344b3b5\" (UID: \"f76634d3-2649-4c0d-bd7d-9c697344b3b5\") "
Sep 17 18:04:50.867563 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:50.867540    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f76634d3-2649-4c0d-bd7d-9c697344b3b5-console-config\") pod \"f76634d3-2649-4c0d-bd7d-9c697344b3b5\" (UID: \"f76634d3-2649-4c0d-bd7d-9c697344b3b5\") "
Sep 17 18:04:50.867563 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:50.867553    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f76634d3-2649-4c0d-bd7d-9c697344b3b5-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f76634d3-2649-4c0d-bd7d-9c697344b3b5" (UID: "f76634d3-2649-4c0d-bd7d-9c697344b3b5"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:04:50.867799 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:50.867583    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f76634d3-2649-4c0d-bd7d-9c697344b3b5-service-ca\") pod \"f76634d3-2649-4c0d-bd7d-9c697344b3b5\" (UID: \"f76634d3-2649-4c0d-bd7d-9c697344b3b5\") "
Sep 17 18:04:50.867890 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:50.867869    2553 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f76634d3-2649-4c0d-bd7d-9c697344b3b5-oauth-serving-cert\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:04:50.868015 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:50.867945    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f76634d3-2649-4c0d-bd7d-9c697344b3b5-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "f76634d3-2649-4c0d-bd7d-9c697344b3b5" (UID: "f76634d3-2649-4c0d-bd7d-9c697344b3b5"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:04:50.868015 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:50.867967    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f76634d3-2649-4c0d-bd7d-9c697344b3b5-console-config" (OuterVolumeSpecName: "console-config") pod "f76634d3-2649-4c0d-bd7d-9c697344b3b5" (UID: "f76634d3-2649-4c0d-bd7d-9c697344b3b5"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:04:50.868015 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:50.867981    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f76634d3-2649-4c0d-bd7d-9c697344b3b5-service-ca" (OuterVolumeSpecName: "service-ca") pod "f76634d3-2649-4c0d-bd7d-9c697344b3b5" (UID: "f76634d3-2649-4c0d-bd7d-9c697344b3b5"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:04:50.869558 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:50.869528    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f76634d3-2649-4c0d-bd7d-9c697344b3b5-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f76634d3-2649-4c0d-bd7d-9c697344b3b5" (UID: "f76634d3-2649-4c0d-bd7d-9c697344b3b5"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:04:50.869929 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:50.869907    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f76634d3-2649-4c0d-bd7d-9c697344b3b5-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f76634d3-2649-4c0d-bd7d-9c697344b3b5" (UID: "f76634d3-2649-4c0d-bd7d-9c697344b3b5"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:04:50.869991 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:50.869936    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f76634d3-2649-4c0d-bd7d-9c697344b3b5-kube-api-access-xjd4l" (OuterVolumeSpecName: "kube-api-access-xjd4l") pod "f76634d3-2649-4c0d-bd7d-9c697344b3b5" (UID: "f76634d3-2649-4c0d-bd7d-9c697344b3b5"). InnerVolumeSpecName "kube-api-access-xjd4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:04:50.969079 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:50.969031    2553 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f76634d3-2649-4c0d-bd7d-9c697344b3b5-console-oauth-config\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:04:50.969079 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:50.969074    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjd4l\" (UniqueName: \"kubernetes.io/projected/f76634d3-2649-4c0d-bd7d-9c697344b3b5-kube-api-access-xjd4l\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:04:50.969079 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:50.969084    2553 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f76634d3-2649-4c0d-bd7d-9c697344b3b5-trusted-ca-bundle\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:04:50.969079 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:50.969092    2553 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f76634d3-2649-4c0d-bd7d-9c697344b3b5-console-serving-cert\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:04:50.969344 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:50.969102    2553 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f76634d3-2649-4c0d-bd7d-9c697344b3b5-console-config\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:04:50.969344 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:50.969111    2553 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f76634d3-2649-4c0d-bd7d-9c697344b3b5-service-ca\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:04:51.505681 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:51.505640    2553 generic.go:356] "Generic (PLEG): container finished" podID="f76634d3-2649-4c0d-bd7d-9c697344b3b5" containerID="051f92e2a19dfe59f3c33632a38b18aa3af7a99c8a552ff1355176a2aaa51e0f" exitCode=0
Sep 17 18:04:51.505681 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:51.505682    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f6fd68cf5-95h2z"
Sep 17 18:04:51.506129 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:51.505693    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f6fd68cf5-95h2z" event={"ID":"f76634d3-2649-4c0d-bd7d-9c697344b3b5","Type":"ContainerDied","Data":"051f92e2a19dfe59f3c33632a38b18aa3af7a99c8a552ff1355176a2aaa51e0f"}
Sep 17 18:04:51.506129 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:51.505741    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f6fd68cf5-95h2z" event={"ID":"f76634d3-2649-4c0d-bd7d-9c697344b3b5","Type":"ContainerDied","Data":"16250abdb2a00eff59d0d440f87594a53cd8a49b7e03b11c9705a6620bf6d3e8"}
Sep 17 18:04:51.506129 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:51.505762    2553 scope.go:117] "RemoveContainer" containerID="051f92e2a19dfe59f3c33632a38b18aa3af7a99c8a552ff1355176a2aaa51e0f"
Sep 17 18:04:51.513809 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:51.513792    2553 scope.go:117] "RemoveContainer" containerID="051f92e2a19dfe59f3c33632a38b18aa3af7a99c8a552ff1355176a2aaa51e0f"
Sep 17 18:04:51.514104 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:04:51.514083    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"051f92e2a19dfe59f3c33632a38b18aa3af7a99c8a552ff1355176a2aaa51e0f\": container with ID starting with 051f92e2a19dfe59f3c33632a38b18aa3af7a99c8a552ff1355176a2aaa51e0f not found: ID does not exist" containerID="051f92e2a19dfe59f3c33632a38b18aa3af7a99c8a552ff1355176a2aaa51e0f"
Sep 17 18:04:51.514150 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:51.514112    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"051f92e2a19dfe59f3c33632a38b18aa3af7a99c8a552ff1355176a2aaa51e0f"} err="failed to get container status \"051f92e2a19dfe59f3c33632a38b18aa3af7a99c8a552ff1355176a2aaa51e0f\": rpc error: code = NotFound desc = could not find container \"051f92e2a19dfe59f3c33632a38b18aa3af7a99c8a552ff1355176a2aaa51e0f\": container with ID starting with 051f92e2a19dfe59f3c33632a38b18aa3af7a99c8a552ff1355176a2aaa51e0f not found: ID does not exist"
Sep 17 18:04:51.523557 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:51.523531    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f6fd68cf5-95h2z"]
Sep 17 18:04:51.529537 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:51.529513    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5f6fd68cf5-95h2z"]
Sep 17 18:04:53.376687 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:04:53.376649    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f76634d3-2649-4c0d-bd7d-9c697344b3b5" path="/var/lib/kubelet/pods/f76634d3-2649-4c0d-bd7d-9c697344b3b5/volumes"
Sep 17 18:05:04.432314 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:05:04.432276    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-bt5l6"
Sep 17 18:05:48.757892 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:05:48.757843    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-p7rj6"]
Sep 17 18:05:48.758308 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:05:48.758209    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f76634d3-2649-4c0d-bd7d-9c697344b3b5" containerName="console"
Sep 17 18:05:48.758308 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:05:48.758221    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="f76634d3-2649-4c0d-bd7d-9c697344b3b5" containerName="console"
Sep 17 18:05:48.758308 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:05:48.758282    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="f76634d3-2649-4c0d-bd7d-9c697344b3b5" containerName="console"
Sep 17 18:05:48.762416 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:05:48.762388    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-p7rj6"
Sep 17 18:05:48.765187 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:05:48.765156    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 17 18:05:48.766219 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:05:48.766200    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-kszbv\""
Sep 17 18:05:48.780579 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:05:48.780544    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-p7rj6"]
Sep 17 18:05:48.954789 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:05:48.954721    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a72ece66-1bea-49e9-95c4-791546aab450-tls-certs\") pod \"model-serving-api-7b954ffc95-p7rj6\" (UID: \"a72ece66-1bea-49e9-95c4-791546aab450\") " pod="kserve/model-serving-api-7b954ffc95-p7rj6"
Sep 17 18:05:48.954940 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:05:48.954823    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw55n\" (UniqueName: \"kubernetes.io/projected/a72ece66-1bea-49e9-95c4-791546aab450-kube-api-access-fw55n\") pod \"model-serving-api-7b954ffc95-p7rj6\" (UID: \"a72ece66-1bea-49e9-95c4-791546aab450\") " pod="kserve/model-serving-api-7b954ffc95-p7rj6"
Sep 17 18:05:49.055927 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:05:49.055840    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a72ece66-1bea-49e9-95c4-791546aab450-tls-certs\") pod \"model-serving-api-7b954ffc95-p7rj6\" (UID: \"a72ece66-1bea-49e9-95c4-791546aab450\") " pod="kserve/model-serving-api-7b954ffc95-p7rj6"
Sep 17 18:05:49.055927 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:05:49.055906    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw55n\" (UniqueName: \"kubernetes.io/projected/a72ece66-1bea-49e9-95c4-791546aab450-kube-api-access-fw55n\") pod \"model-serving-api-7b954ffc95-p7rj6\" (UID: \"a72ece66-1bea-49e9-95c4-791546aab450\") " pod="kserve/model-serving-api-7b954ffc95-p7rj6"
Sep 17 18:05:49.058329 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:05:49.058304    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a72ece66-1bea-49e9-95c4-791546aab450-tls-certs\") pod \"model-serving-api-7b954ffc95-p7rj6\" (UID: \"a72ece66-1bea-49e9-95c4-791546aab450\") " pod="kserve/model-serving-api-7b954ffc95-p7rj6"
Sep 17 18:05:49.076513 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:05:49.076480    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw55n\" (UniqueName: \"kubernetes.io/projected/a72ece66-1bea-49e9-95c4-791546aab450-kube-api-access-fw55n\") pod \"model-serving-api-7b954ffc95-p7rj6\" (UID: \"a72ece66-1bea-49e9-95c4-791546aab450\") " pod="kserve/model-serving-api-7b954ffc95-p7rj6"
Sep 17 18:05:49.373037 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:05:49.372915    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-p7rj6"
Sep 17 18:05:49.660183 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:05:49.660157    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-p7rj6"]
Sep 17 18:05:49.662952 ip-10-0-137-95 kubenswrapper[2553]: W0917 18:05:49.662922    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda72ece66_1bea_49e9_95c4_791546aab450.slice/crio-ccc18efb2f656652445616701a742df69c3d1b8f8523997b638ea51bdc254fd9 WatchSource:0}: Error finding container ccc18efb2f656652445616701a742df69c3d1b8f8523997b638ea51bdc254fd9: Status 404 returned error can't find the container with id ccc18efb2f656652445616701a742df69c3d1b8f8523997b638ea51bdc254fd9
Sep 17 18:05:49.775649 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:05:49.775589    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-p7rj6" event={"ID":"a72ece66-1bea-49e9-95c4-791546aab450","Type":"ContainerStarted","Data":"ccc18efb2f656652445616701a742df69c3d1b8f8523997b638ea51bdc254fd9"}
Sep 17 18:05:52.793485 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:05:52.793448    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-p7rj6" event={"ID":"a72ece66-1bea-49e9-95c4-791546aab450","Type":"ContainerStarted","Data":"d9c5c2bb1a4e7a3c3d9bf5932c6fccca5fea9c3469f81a75bedf42564042b551"}
Sep 17 18:05:52.793930 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:05:52.793539    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-p7rj6"
Sep 17 18:05:52.832714 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:05:52.832654    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-p7rj6" podStartSLOduration=2.266262202 podStartE2EDuration="4.832636165s" totalImagesPullingTime="2.566373963s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:05:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:05:52.831467752 +0000 UTC m=+547.527558436" watchObservedRunningTime="2026-09-17 18:05:52.832636165 +0000 UTC m=+547.528726843"
Sep 17 18:06:03.801736 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:06:03.801702    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-p7rj6"
Sep 17 18:06:47.291667 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:06:47.291609    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-btmqz_622ee5be-fb50-42b6-ae23-42900b7c2d98/ovn-acl-logging/0.log"
Sep 17 18:06:47.292304 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:06:47.292283    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-btmqz_622ee5be-fb50-42b6-ae23-42900b7c2d98/ovn-acl-logging/0.log"
Sep 17 18:08:13.551131 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:08:13.551094    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-vs9tv"]
Sep 17 18:08:13.554285 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:08:13.554268    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-vs9tv"
Sep 17 18:08:13.556746 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:08:13.556725    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Sep 17 18:08:13.556890 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:08:13.556759    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving\""
Sep 17 18:08:13.557002 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:08:13.556986    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-qgczn\""
Sep 17 18:08:13.564707 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:08:13.564684    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-vs9tv"]
Sep 17 18:08:13.636649 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:08:13.636585    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/97400e98-445c-42f0-8b53-b98a8e9ddfd2-data\") pod \"seaweedfs-tls-serving-67b7477c6d-vs9tv\" (UID: \"97400e98-445c-42f0-8b53-b98a8e9ddfd2\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-vs9tv"
Sep 17 18:08:13.636867 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:08:13.636718    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/97400e98-445c-42f0-8b53-b98a8e9ddfd2-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-vs9tv\" (UID: \"97400e98-445c-42f0-8b53-b98a8e9ddfd2\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-vs9tv"
Sep 17 18:08:13.636867 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:08:13.636769    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjhbl\" (UniqueName: \"kubernetes.io/projected/97400e98-445c-42f0-8b53-b98a8e9ddfd2-kube-api-access-pjhbl\") pod \"seaweedfs-tls-serving-67b7477c6d-vs9tv\" (UID: \"97400e98-445c-42f0-8b53-b98a8e9ddfd2\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-vs9tv"
Sep 17 18:08:13.737635 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:08:13.737579    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/97400e98-445c-42f0-8b53-b98a8e9ddfd2-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-vs9tv\" (UID: \"97400e98-445c-42f0-8b53-b98a8e9ddfd2\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-vs9tv"
Sep 17 18:08:13.737635 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:08:13.737637    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjhbl\" (UniqueName: \"kubernetes.io/projected/97400e98-445c-42f0-8b53-b98a8e9ddfd2-kube-api-access-pjhbl\") pod \"seaweedfs-tls-serving-67b7477c6d-vs9tv\" (UID: \"97400e98-445c-42f0-8b53-b98a8e9ddfd2\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-vs9tv"
Sep 17 18:08:13.737850 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:08:13.737688    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/97400e98-445c-42f0-8b53-b98a8e9ddfd2-data\") pod \"seaweedfs-tls-serving-67b7477c6d-vs9tv\" (UID: \"97400e98-445c-42f0-8b53-b98a8e9ddfd2\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-vs9tv"
Sep 17 18:08:13.738123 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:08:13.738103    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/97400e98-445c-42f0-8b53-b98a8e9ddfd2-data\") pod \"seaweedfs-tls-serving-67b7477c6d-vs9tv\" (UID: \"97400e98-445c-42f0-8b53-b98a8e9ddfd2\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-vs9tv"
Sep 17 18:08:13.740015 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:08:13.739993    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/97400e98-445c-42f0-8b53-b98a8e9ddfd2-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-vs9tv\" (UID: \"97400e98-445c-42f0-8b53-b98a8e9ddfd2\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-vs9tv"
Sep 17 18:08:13.748446 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:08:13.748416    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjhbl\" (UniqueName: \"kubernetes.io/projected/97400e98-445c-42f0-8b53-b98a8e9ddfd2-kube-api-access-pjhbl\") pod \"seaweedfs-tls-serving-67b7477c6d-vs9tv\" (UID: \"97400e98-445c-42f0-8b53-b98a8e9ddfd2\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-vs9tv"
Sep 17 18:08:13.862933 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:08:13.862835    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-vs9tv"
Sep 17 18:08:14.134664 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:08:14.134637    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-vs9tv"]
Sep 17 18:08:14.137195 ip-10-0-137-95 kubenswrapper[2553]: W0917 18:08:14.137171    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97400e98_445c_42f0_8b53_b98a8e9ddfd2.slice/crio-d948db261527ba6480a9c5dbcd752bd6614f9d90bfe9c92b2d26225bc7344c6e WatchSource:0}: Error finding container d948db261527ba6480a9c5dbcd752bd6614f9d90bfe9c92b2d26225bc7344c6e: Status 404 returned error can't find the container with id d948db261527ba6480a9c5dbcd752bd6614f9d90bfe9c92b2d26225bc7344c6e
Sep 17 18:08:14.138585 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:08:14.138570    2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:08:14.397403 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:08:14.397314    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-vs9tv" event={"ID":"97400e98-445c-42f0-8b53-b98a8e9ddfd2","Type":"ContainerStarted","Data":"d948db261527ba6480a9c5dbcd752bd6614f9d90bfe9c92b2d26225bc7344c6e"}
Sep 17 18:08:17.412144 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:08:17.412043    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-vs9tv" event={"ID":"97400e98-445c-42f0-8b53-b98a8e9ddfd2","Type":"ContainerStarted","Data":"855a8f13723ecd9ae09dc7c3ef07132019f8417e10951a09e386521e6b4e9bdc"}
Sep 17 18:08:17.431735 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:08:17.431682    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-serving-67b7477c6d-vs9tv" podStartSLOduration=1.432743405 podStartE2EDuration="4.431664959s" totalImagesPullingTime="2.998921554s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:08:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:08:17.430301824 +0000 UTC m=+692.126392532" watchObservedRunningTime="2026-09-17 18:08:17.431664959 +0000 UTC m=+692.127755643"
Sep 17 18:11:47.317953 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:11:47.317924    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-btmqz_622ee5be-fb50-42b6-ae23-42900b7c2d98/ovn-acl-logging/0.log"
Sep 17 18:11:47.318790 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:11:47.318772    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-btmqz_622ee5be-fb50-42b6-ae23-42900b7c2d98/ovn-acl-logging/0.log"
Sep 17 18:12:10.329862 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:12:10.329823    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gq4sf"]
Sep 17 18:12:10.333127 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:12:10.333108    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gq4sf"
Sep 17 18:12:10.335564 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:12:10.335537    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
Sep 17 18:12:10.335564 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:12:10.335552    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:12:10.335777 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:12:10.335581    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-7qcmd\""
Sep 17 18:12:10.336649 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:12:10.336631    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Sep 17 18:12:10.336732 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:12:10.336612    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:12:10.344249 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:12:10.344230    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gq4sf"]
Sep 17 18:12:10.452912 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:12:10.452879    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/efb2856d-2c13-455a-b2b3-0f35609d8c23-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-gq4sf\" (UID: \"efb2856d-2c13-455a-b2b3-0f35609d8c23\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gq4sf"
Sep 17 18:12:10.453087 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:12:10.452971    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgdn7\" (UniqueName: \"kubernetes.io/projected/efb2856d-2c13-455a-b2b3-0f35609d8c23-kube-api-access-fgdn7\") pod \"message-dumper-predictor-6575bcfdfd-gq4sf\" (UID: \"efb2856d-2c13-455a-b2b3-0f35609d8c23\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gq4sf"
Sep 17 18:12:10.453087 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:12:10.453013    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/efb2856d-2c13-455a-b2b3-0f35609d8c23-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-gq4sf\" (UID: \"efb2856d-2c13-455a-b2b3-0f35609d8c23\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gq4sf"
Sep 17 18:12:10.553662 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:12:10.553606    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/efb2856d-2c13-455a-b2b3-0f35609d8c23-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-gq4sf\" (UID: \"efb2856d-2c13-455a-b2b3-0f35609d8c23\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gq4sf"
Sep 17 18:12:10.553781 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:12:10.553706    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgdn7\" (UniqueName: \"kubernetes.io/projected/efb2856d-2c13-455a-b2b3-0f35609d8c23-kube-api-access-fgdn7\") pod \"message-dumper-predictor-6575bcfdfd-gq4sf\" (UID: \"efb2856d-2c13-455a-b2b3-0f35609d8c23\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gq4sf"
Sep 17 18:12:10.553781 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:12:10.553751    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/efb2856d-2c13-455a-b2b3-0f35609d8c23-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-gq4sf\" (UID: \"efb2856d-2c13-455a-b2b3-0f35609d8c23\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gq4sf"
Sep 17 18:12:10.553862 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:12:10.553791    2553 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-predictor-serving-cert: secret "message-dumper-predictor-serving-cert" not found
Sep 17 18:12:10.553909 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:12:10.553865    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efb2856d-2c13-455a-b2b3-0f35609d8c23-proxy-tls podName:efb2856d-2c13-455a-b2b3-0f35609d8c23 nodeName:}" failed. No retries permitted until 2026-09-17 18:12:11.053839906 +0000 UTC m=+925.749930576 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/efb2856d-2c13-455a-b2b3-0f35609d8c23-proxy-tls") pod "message-dumper-predictor-6575bcfdfd-gq4sf" (UID: "efb2856d-2c13-455a-b2b3-0f35609d8c23") : secret "message-dumper-predictor-serving-cert" not found
Sep 17 18:12:10.554394 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:12:10.554372    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/efb2856d-2c13-455a-b2b3-0f35609d8c23-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-gq4sf\" (UID: \"efb2856d-2c13-455a-b2b3-0f35609d8c23\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gq4sf"
Sep 17 18:12:10.562886 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:12:10.562860    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgdn7\" (UniqueName: \"kubernetes.io/projected/efb2856d-2c13-455a-b2b3-0f35609d8c23-kube-api-access-fgdn7\") pod \"message-dumper-predictor-6575bcfdfd-gq4sf\" (UID: \"efb2856d-2c13-455a-b2b3-0f35609d8c23\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gq4sf"
Sep 17 18:12:11.058691 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:12:11.058649    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/efb2856d-2c13-455a-b2b3-0f35609d8c23-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-gq4sf\" (UID: \"efb2856d-2c13-455a-b2b3-0f35609d8c23\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gq4sf"
Sep 17 18:12:11.061166 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:12:11.061130    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/efb2856d-2c13-455a-b2b3-0f35609d8c23-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-gq4sf\" (UID: \"efb2856d-2c13-455a-b2b3-0f35609d8c23\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gq4sf"
Sep 17 18:12:11.243740 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:12:11.243699    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gq4sf"
Sep 17 18:12:11.518118 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:12:11.518075    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gq4sf"]
Sep 17 18:12:11.520381 ip-10-0-137-95 kubenswrapper[2553]: W0917 18:12:11.520351    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefb2856d_2c13_455a_b2b3_0f35609d8c23.slice/crio-dff6dcfb54fa6ebeea467b8c12e8a69154bceaaff9ea1f1ffc496022fddfb35e WatchSource:0}: Error finding container dff6dcfb54fa6ebeea467b8c12e8a69154bceaaff9ea1f1ffc496022fddfb35e: Status 404 returned error can't find the container with id dff6dcfb54fa6ebeea467b8c12e8a69154bceaaff9ea1f1ffc496022fddfb35e
Sep 17 18:12:12.457278 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:12:12.457232    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gq4sf" event={"ID":"efb2856d-2c13-455a-b2b3-0f35609d8c23","Type":"ContainerStarted","Data":"dff6dcfb54fa6ebeea467b8c12e8a69154bceaaff9ea1f1ffc496022fddfb35e"}
Sep 17 18:12:13.465731 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:12:13.465687    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gq4sf" event={"ID":"efb2856d-2c13-455a-b2b3-0f35609d8c23","Type":"ContainerStarted","Data":"9d6a68bf3ca6b1fb54a21bdedac34e475730552571ec6994f26969b0bc712e66"}
Sep 17 18:12:15.476502 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:12:15.476460    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gq4sf" event={"ID":"efb2856d-2c13-455a-b2b3-0f35609d8c23","Type":"ContainerStarted","Data":"d14de384f3ae56038c7bdcf22b146b9c74a8cc4223d993e6cf1fdcaa953a5a87"}
Sep 17 18:12:15.476928 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:12:15.476716    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gq4sf"
Sep 17 18:12:15.497812 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:12:15.497758    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gq4sf" podStartSLOduration=2.095430193 podStartE2EDuration="5.497742151s" totalImagesPullingTime="3.402311958s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:12:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:12:15.495794336 +0000 UTC m=+930.191885020" watchObservedRunningTime="2026-09-17 18:12:15.497742151 +0000 UTC m=+930.193832834"
Sep 17 18:12:16.481288 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:12:16.481254    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gq4sf"
Sep 17 18:12:16.483378 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:12:16.483353    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gq4sf"
Sep 17 18:12:23.493890 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:12:23.493854    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gq4sf"
Sep 17 18:14:05.385515 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:05.385469    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-6575bcfdfd-gq4sf_efb2856d-2c13-455a-b2b3-0f35609d8c23/kserve-container/0.log"
Sep 17 18:14:05.619830 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:05.619798    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gq4sf"]
Sep 17 18:14:05.620073 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:05.620033    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gq4sf" podUID="efb2856d-2c13-455a-b2b3-0f35609d8c23" containerName="kserve-container" containerID="cri-o://9d6a68bf3ca6b1fb54a21bdedac34e475730552571ec6994f26969b0bc712e66" gracePeriod=30
Sep 17 18:14:05.620159 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:05.620085    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gq4sf" podUID="efb2856d-2c13-455a-b2b3-0f35609d8c23" containerName="kube-rbac-proxy" containerID="cri-o://d14de384f3ae56038c7bdcf22b146b9c74a8cc4223d993e6cf1fdcaa953a5a87" gracePeriod=30
Sep 17 18:14:05.872488 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:05.872458    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gq4sf"
Sep 17 18:14:05.965246 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:05.965150    2553 generic.go:356] "Generic (PLEG): container finished" podID="efb2856d-2c13-455a-b2b3-0f35609d8c23" containerID="d14de384f3ae56038c7bdcf22b146b9c74a8cc4223d993e6cf1fdcaa953a5a87" exitCode=2
Sep 17 18:14:05.965246 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:05.965178    2553 generic.go:356] "Generic (PLEG): container finished" podID="efb2856d-2c13-455a-b2b3-0f35609d8c23" containerID="9d6a68bf3ca6b1fb54a21bdedac34e475730552571ec6994f26969b0bc712e66" exitCode=2
Sep 17 18:14:05.965246 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:05.965203    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gq4sf"
Sep 17 18:14:05.965487 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:05.965246    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gq4sf" event={"ID":"efb2856d-2c13-455a-b2b3-0f35609d8c23","Type":"ContainerDied","Data":"d14de384f3ae56038c7bdcf22b146b9c74a8cc4223d993e6cf1fdcaa953a5a87"}
Sep 17 18:14:05.965487 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:05.965314    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gq4sf" event={"ID":"efb2856d-2c13-455a-b2b3-0f35609d8c23","Type":"ContainerDied","Data":"9d6a68bf3ca6b1fb54a21bdedac34e475730552571ec6994f26969b0bc712e66"}
Sep 17 18:14:05.965487 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:05.965342    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gq4sf" event={"ID":"efb2856d-2c13-455a-b2b3-0f35609d8c23","Type":"ContainerDied","Data":"dff6dcfb54fa6ebeea467b8c12e8a69154bceaaff9ea1f1ffc496022fddfb35e"}
Sep 17 18:14:05.965487 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:05.965367    2553 scope.go:117] "RemoveContainer" containerID="d14de384f3ae56038c7bdcf22b146b9c74a8cc4223d993e6cf1fdcaa953a5a87"
Sep 17 18:14:05.973542 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:05.973524    2553 scope.go:117] "RemoveContainer" containerID="9d6a68bf3ca6b1fb54a21bdedac34e475730552571ec6994f26969b0bc712e66"
Sep 17 18:14:05.981356 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:05.981338    2553 scope.go:117] "RemoveContainer" containerID="d14de384f3ae56038c7bdcf22b146b9c74a8cc4223d993e6cf1fdcaa953a5a87"
Sep 17 18:14:05.981645 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:14:05.981608    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d14de384f3ae56038c7bdcf22b146b9c74a8cc4223d993e6cf1fdcaa953a5a87\": container with ID starting with d14de384f3ae56038c7bdcf22b146b9c74a8cc4223d993e6cf1fdcaa953a5a87 not found: ID does not exist" containerID="d14de384f3ae56038c7bdcf22b146b9c74a8cc4223d993e6cf1fdcaa953a5a87"
Sep 17 18:14:05.981712 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:05.981657    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d14de384f3ae56038c7bdcf22b146b9c74a8cc4223d993e6cf1fdcaa953a5a87"} err="failed to get container status \"d14de384f3ae56038c7bdcf22b146b9c74a8cc4223d993e6cf1fdcaa953a5a87\": rpc error: code = NotFound desc = could not find container \"d14de384f3ae56038c7bdcf22b146b9c74a8cc4223d993e6cf1fdcaa953a5a87\": container with ID starting with d14de384f3ae56038c7bdcf22b146b9c74a8cc4223d993e6cf1fdcaa953a5a87 not found: ID does not exist"
Sep 17 18:14:05.981712 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:05.981678    2553 scope.go:117] "RemoveContainer" containerID="9d6a68bf3ca6b1fb54a21bdedac34e475730552571ec6994f26969b0bc712e66"
Sep 17 18:14:05.981933 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:14:05.981912    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d6a68bf3ca6b1fb54a21bdedac34e475730552571ec6994f26969b0bc712e66\": container with ID starting with 9d6a68bf3ca6b1fb54a21bdedac34e475730552571ec6994f26969b0bc712e66 not found: ID does not exist" containerID="9d6a68bf3ca6b1fb54a21bdedac34e475730552571ec6994f26969b0bc712e66"
Sep 17 18:14:05.981997 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:05.981946    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d6a68bf3ca6b1fb54a21bdedac34e475730552571ec6994f26969b0bc712e66"} err="failed to get container status \"9d6a68bf3ca6b1fb54a21bdedac34e475730552571ec6994f26969b0bc712e66\": rpc error: code = NotFound desc = could not find container \"9d6a68bf3ca6b1fb54a21bdedac34e475730552571ec6994f26969b0bc712e66\": container with ID starting with 9d6a68bf3ca6b1fb54a21bdedac34e475730552571ec6994f26969b0bc712e66 not found: ID does not exist"
Sep 17 18:14:05.981997 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:05.981970    2553 scope.go:117] "RemoveContainer" containerID="d14de384f3ae56038c7bdcf22b146b9c74a8cc4223d993e6cf1fdcaa953a5a87"
Sep 17 18:14:05.982251 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:05.982231    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d14de384f3ae56038c7bdcf22b146b9c74a8cc4223d993e6cf1fdcaa953a5a87"} err="failed to get container status \"d14de384f3ae56038c7bdcf22b146b9c74a8cc4223d993e6cf1fdcaa953a5a87\": rpc error: code = NotFound desc = could not find container \"d14de384f3ae56038c7bdcf22b146b9c74a8cc4223d993e6cf1fdcaa953a5a87\": container with ID starting with d14de384f3ae56038c7bdcf22b146b9c74a8cc4223d993e6cf1fdcaa953a5a87 not found: ID does not exist"
Sep 17 18:14:05.982298 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:05.982253    2553 scope.go:117] "RemoveContainer" containerID="9d6a68bf3ca6b1fb54a21bdedac34e475730552571ec6994f26969b0bc712e66"
Sep 17 18:14:05.982506 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:05.982486    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d6a68bf3ca6b1fb54a21bdedac34e475730552571ec6994f26969b0bc712e66"} err="failed to get container status \"9d6a68bf3ca6b1fb54a21bdedac34e475730552571ec6994f26969b0bc712e66\": rpc error: code = NotFound desc = could not find container \"9d6a68bf3ca6b1fb54a21bdedac34e475730552571ec6994f26969b0bc712e66\": container with ID starting with 9d6a68bf3ca6b1fb54a21bdedac34e475730552571ec6994f26969b0bc712e66 not found: ID does not exist"
Sep 17 18:14:06.035849 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:06.035814    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/efb2856d-2c13-455a-b2b3-0f35609d8c23-kube-rbac-proxy-sar-config\") pod \"efb2856d-2c13-455a-b2b3-0f35609d8c23\" (UID: \"efb2856d-2c13-455a-b2b3-0f35609d8c23\") "
Sep 17 18:14:06.036019 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:06.035903    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fgdn7\" (UniqueName: \"kubernetes.io/projected/efb2856d-2c13-455a-b2b3-0f35609d8c23-kube-api-access-fgdn7\") pod \"efb2856d-2c13-455a-b2b3-0f35609d8c23\" (UID: \"efb2856d-2c13-455a-b2b3-0f35609d8c23\") "
Sep 17 18:14:06.036019 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:06.035939    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/efb2856d-2c13-455a-b2b3-0f35609d8c23-proxy-tls\") pod \"efb2856d-2c13-455a-b2b3-0f35609d8c23\" (UID: \"efb2856d-2c13-455a-b2b3-0f35609d8c23\") "
Sep 17 18:14:06.036201 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:06.036176    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efb2856d-2c13-455a-b2b3-0f35609d8c23-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "efb2856d-2c13-455a-b2b3-0f35609d8c23" (UID: "efb2856d-2c13-455a-b2b3-0f35609d8c23"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:14:06.038167 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:06.038143    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efb2856d-2c13-455a-b2b3-0f35609d8c23-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "efb2856d-2c13-455a-b2b3-0f35609d8c23" (UID: "efb2856d-2c13-455a-b2b3-0f35609d8c23"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:14:06.038273 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:06.038181    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efb2856d-2c13-455a-b2b3-0f35609d8c23-kube-api-access-fgdn7" (OuterVolumeSpecName: "kube-api-access-fgdn7") pod "efb2856d-2c13-455a-b2b3-0f35609d8c23" (UID: "efb2856d-2c13-455a-b2b3-0f35609d8c23"). InnerVolumeSpecName "kube-api-access-fgdn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:14:06.136561 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:06.136516    2553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/efb2856d-2c13-455a-b2b3-0f35609d8c23-proxy-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:14:06.136561 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:06.136553    2553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/efb2856d-2c13-455a-b2b3-0f35609d8c23-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:14:06.136561 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:06.136565    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgdn7\" (UniqueName: \"kubernetes.io/projected/efb2856d-2c13-455a-b2b3-0f35609d8c23-kube-api-access-fgdn7\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:14:06.288873 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:06.288836    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gq4sf"]
Sep 17 18:14:06.294708 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:06.294674    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gq4sf"]
Sep 17 18:14:07.376666 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:07.376631    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efb2856d-2c13-455a-b2b3-0f35609d8c23" path="/var/lib/kubelet/pods/efb2856d-2c13-455a-b2b3-0f35609d8c23/volumes"
Sep 17 18:14:17.191225 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:17.191189    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-t8rnq"]
Sep 17 18:14:17.191603 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:17.191536    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efb2856d-2c13-455a-b2b3-0f35609d8c23" containerName="kserve-container"
Sep 17 18:14:17.191603 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:17.191547    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="efb2856d-2c13-455a-b2b3-0f35609d8c23" containerName="kserve-container"
Sep 17 18:14:17.191603 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:17.191559    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efb2856d-2c13-455a-b2b3-0f35609d8c23" containerName="kube-rbac-proxy"
Sep 17 18:14:17.191603 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:17.191565    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="efb2856d-2c13-455a-b2b3-0f35609d8c23" containerName="kube-rbac-proxy"
Sep 17 18:14:17.191761 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:17.191637    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="efb2856d-2c13-455a-b2b3-0f35609d8c23" containerName="kube-rbac-proxy"
Sep 17 18:14:17.191761 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:17.191650    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="efb2856d-2c13-455a-b2b3-0f35609d8c23" containerName="kserve-container"
Sep 17 18:14:17.196086 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:17.196067    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-t8rnq"
Sep 17 18:14:17.198874 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:17.198847    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-predictor-serving-cert\""
Sep 17 18:14:17.199039 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:17.198850    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\""
Sep 17 18:14:17.199039 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:17.198886    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:14:17.199039 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:17.198915    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:14:17.200262 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:17.200233    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:14:17.200374 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:17.200268    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-7qcmd\""
Sep 17 18:14:17.200374 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:17.200241    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:14:17.205553 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:17.205529    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-t8rnq"]
Sep 17 18:14:17.231994 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:17.231947    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-t8rnq\" (UID: \"512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-t8rnq"
Sep 17 18:14:17.231994 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:17.232003    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-t8rnq\" (UID: \"512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-t8rnq"
Sep 17 18:14:17.232238 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:17.232072    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr9nz\" (UniqueName: \"kubernetes.io/projected/512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a-kube-api-access-mr9nz\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-t8rnq\" (UID: \"512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-t8rnq"
Sep 17 18:14:17.232238 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:17.232109    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-t8rnq\" (UID: \"512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-t8rnq"
Sep 17 18:14:17.232238 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:17.232137    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-t8rnq\" (UID: \"512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-t8rnq"
Sep 17 18:14:17.332946 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:17.332898    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-t8rnq\" (UID: \"512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-t8rnq"
Sep 17 18:14:17.332946 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:17.332951    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-t8rnq\" (UID: \"512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-t8rnq"
Sep 17 18:14:17.333215 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:17.332977    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr9nz\" (UniqueName: \"kubernetes.io/projected/512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a-kube-api-access-mr9nz\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-t8rnq\" (UID: \"512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-t8rnq"
Sep 17 18:14:17.333215 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:17.332999    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-t8rnq\" (UID: \"512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-t8rnq"
Sep 17 18:14:17.333215 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:17.333022    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-t8rnq\" (UID: \"512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-t8rnq"
Sep 17 18:14:17.333584 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:17.333558    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-t8rnq\" (UID: \"512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-t8rnq"
Sep 17 18:14:17.333849 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:17.333828    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-t8rnq\" (UID: \"512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-t8rnq"
Sep 17 18:14:17.333897 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:17.333852    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-t8rnq\" (UID: \"512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-t8rnq"
Sep 17 18:14:17.335556 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:17.335534    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-t8rnq\" (UID: \"512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-t8rnq"
Sep 17 18:14:17.341309 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:17.341284    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr9nz\" (UniqueName: \"kubernetes.io/projected/512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a-kube-api-access-mr9nz\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-t8rnq\" (UID: \"512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-t8rnq"
Sep 17 18:14:17.506677 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:17.506561    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-t8rnq"
Sep 17 18:14:17.782259 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:17.782232    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-t8rnq"]
Sep 17 18:14:17.784373 ip-10-0-137-95 kubenswrapper[2553]: W0917 18:14:17.784345    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod512bbd8c_c5e4_41d2_8cdc_ebe2cfcb891a.slice/crio-727626fb0661f7ee2680ead0d249b7b2cb593bae1ae7f5ef8bd56a854050896d WatchSource:0}: Error finding container 727626fb0661f7ee2680ead0d249b7b2cb593bae1ae7f5ef8bd56a854050896d: Status 404 returned error can't find the container with id 727626fb0661f7ee2680ead0d249b7b2cb593bae1ae7f5ef8bd56a854050896d
Sep 17 18:14:17.786596 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:17.786575    2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:14:18.024452 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:18.024413    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-t8rnq" event={"ID":"512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a","Type":"ContainerStarted","Data":"727626fb0661f7ee2680ead0d249b7b2cb593bae1ae7f5ef8bd56a854050896d"}
Sep 17 18:14:22.050906 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:22.050857    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-t8rnq" event={"ID":"512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a","Type":"ContainerStarted","Data":"048948318a6d88d292adadc47d6126e78d02e8a80b4e524c7c53f9f21898f470"}
Sep 17 18:14:26.071948 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:26.071918    2553 generic.go:356] "Generic (PLEG): container finished" podID="512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a" containerID="048948318a6d88d292adadc47d6126e78d02e8a80b4e524c7c53f9f21898f470" exitCode=0
Sep 17 18:14:26.072337 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:26.071979    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-t8rnq" event={"ID":"512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a","Type":"ContainerDied","Data":"048948318a6d88d292adadc47d6126e78d02e8a80b4e524c7c53f9f21898f470"}
Sep 17 18:14:53.222416 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:53.222380    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-t8rnq" event={"ID":"512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a","Type":"ContainerStarted","Data":"8b60b6d03bea51c78d0ca37ce2b6e24fffab16576e7aef00747445ecb08e4455"}
Sep 17 18:14:53.222877 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:53.222445    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-t8rnq" event={"ID":"512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a","Type":"ContainerStarted","Data":"c3234cbf062ceb21200ac0224f69dd24956aa550aa4175992161a72a43d2c102"}
Sep 17 18:14:53.222877 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:53.222667    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-t8rnq"
Sep 17 18:14:53.243157 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:53.243095    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-t8rnq" podStartSLOduration=2.415281157 podStartE2EDuration="36.243080139s" totalImagesPullingTime="29.954903215s" totalInitContainerRuntime="3.872895767s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:14:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:14:53.241796076 +0000 UTC m=+1087.937886759" watchObservedRunningTime="2026-09-17 18:14:53.243080139 +0000 UTC m=+1087.939170822"
Sep 17 18:14:54.226851 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:14:54.226809    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-t8rnq"
Sep 17 18:15:00.242902 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:15:00.242868    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-t8rnq"
Sep 17 18:15:30.243597 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:15:30.243558    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-t8rnq" podUID="512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.32:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 18:15:40.243527 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:15:40.243478    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-t8rnq" podUID="512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.32:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 18:15:50.244246 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:15:50.244199    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-t8rnq" podUID="512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.32:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 18:16:00.244398 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:00.244306    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-t8rnq" podUID="512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.32:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 18:16:10.243391 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:10.243343    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-t8rnq" podUID="512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.32:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 18:16:18.946848 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:18.946805    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-t8rnq"
Sep 17 18:16:27.336997 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:27.336959    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-t8rnq"]
Sep 17 18:16:27.337798 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:27.337305    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-t8rnq" podUID="512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a" containerName="kserve-container" containerID="cri-o://c3234cbf062ceb21200ac0224f69dd24956aa550aa4175992161a72a43d2c102" gracePeriod=30
Sep 17 18:16:27.337798 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:27.337358    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-t8rnq" podUID="512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a" containerName="kube-rbac-proxy" containerID="cri-o://8b60b6d03bea51c78d0ca37ce2b6e24fffab16576e7aef00747445ecb08e4455" gracePeriod=30
Sep 17 18:16:27.642308 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:27.642272    2553 generic.go:356] "Generic (PLEG): container finished" podID="512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a" containerID="8b60b6d03bea51c78d0ca37ce2b6e24fffab16576e7aef00747445ecb08e4455" exitCode=2
Sep 17 18:16:27.642474 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:27.642341    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-t8rnq" event={"ID":"512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a","Type":"ContainerDied","Data":"8b60b6d03bea51c78d0ca37ce2b6e24fffab16576e7aef00747445ecb08e4455"}
Sep 17 18:16:28.371311 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:28.371261    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-t8rnq" podUID="512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.32:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 18:16:30.238479 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:30.238433    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-t8rnq" podUID="512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Sep 17 18:16:32.488285 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:32.488248    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd"]
Sep 17 18:16:32.490614 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:32.490596    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd"
Sep 17 18:16:32.493047 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:32.493028    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\""
Sep 17 18:16:32.493309 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:32.493285    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-predictor-serving-cert\""
Sep 17 18:16:32.502988 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:32.502959    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd"]
Sep 17 18:16:32.610182 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:32.610141    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1eb569d7-9288-4e73-bc7e-de8b8180abcd-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd\" (UID: \"1eb569d7-9288-4e73-bc7e-de8b8180abcd\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd"
Sep 17 18:16:32.610332 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:32.610285    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1eb569d7-9288-4e73-bc7e-de8b8180abcd-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd\" (UID: \"1eb569d7-9288-4e73-bc7e-de8b8180abcd\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd"
Sep 17 18:16:32.610399 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:32.610332    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1eb569d7-9288-4e73-bc7e-de8b8180abcd-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd\" (UID: \"1eb569d7-9288-4e73-bc7e-de8b8180abcd\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd"
Sep 17 18:16:32.610460 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:32.610415    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1eb569d7-9288-4e73-bc7e-de8b8180abcd-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd\" (UID: \"1eb569d7-9288-4e73-bc7e-de8b8180abcd\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd"
Sep 17 18:16:32.610515 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:32.610464    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxlht\" (UniqueName: \"kubernetes.io/projected/1eb569d7-9288-4e73-bc7e-de8b8180abcd-kube-api-access-wxlht\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd\" (UID: \"1eb569d7-9288-4e73-bc7e-de8b8180abcd\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd"
Sep 17 18:16:32.667154 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:32.667124    2553 generic.go:356] "Generic (PLEG): container finished" podID="512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a" containerID="c3234cbf062ceb21200ac0224f69dd24956aa550aa4175992161a72a43d2c102" exitCode=0
Sep 17 18:16:32.667286 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:32.667206    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-t8rnq" event={"ID":"512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a","Type":"ContainerDied","Data":"c3234cbf062ceb21200ac0224f69dd24956aa550aa4175992161a72a43d2c102"}
Sep 17 18:16:32.680929 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:32.680911    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-t8rnq"
Sep 17 18:16:32.711261 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:32.711184    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1eb569d7-9288-4e73-bc7e-de8b8180abcd-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd\" (UID: \"1eb569d7-9288-4e73-bc7e-de8b8180abcd\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd"
Sep 17 18:16:32.711261 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:32.711236    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1eb569d7-9288-4e73-bc7e-de8b8180abcd-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd\" (UID: \"1eb569d7-9288-4e73-bc7e-de8b8180abcd\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd"
Sep 17 18:16:32.711440 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:32.711355    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1eb569d7-9288-4e73-bc7e-de8b8180abcd-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd\" (UID: \"1eb569d7-9288-4e73-bc7e-de8b8180abcd\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd"
Sep 17 18:16:32.711497 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:32.711435    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1eb569d7-9288-4e73-bc7e-de8b8180abcd-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd\" (UID: \"1eb569d7-9288-4e73-bc7e-de8b8180abcd\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd"
Sep 17 18:16:32.711497 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:32.711480    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxlht\" (UniqueName: \"kubernetes.io/projected/1eb569d7-9288-4e73-bc7e-de8b8180abcd-kube-api-access-wxlht\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd\" (UID: \"1eb569d7-9288-4e73-bc7e-de8b8180abcd\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd"
Sep 17 18:16:32.711672 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:32.711651    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1eb569d7-9288-4e73-bc7e-de8b8180abcd-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd\" (UID: \"1eb569d7-9288-4e73-bc7e-de8b8180abcd\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd"
Sep 17 18:16:32.711938 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:32.711912    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1eb569d7-9288-4e73-bc7e-de8b8180abcd-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd\" (UID: \"1eb569d7-9288-4e73-bc7e-de8b8180abcd\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd"
Sep 17 18:16:32.712173 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:32.712152    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1eb569d7-9288-4e73-bc7e-de8b8180abcd-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd\" (UID: \"1eb569d7-9288-4e73-bc7e-de8b8180abcd\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd"
Sep 17 18:16:32.714417 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:32.714397    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1eb569d7-9288-4e73-bc7e-de8b8180abcd-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd\" (UID: \"1eb569d7-9288-4e73-bc7e-de8b8180abcd\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd"
Sep 17 18:16:32.719801 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:32.719777    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxlht\" (UniqueName: \"kubernetes.io/projected/1eb569d7-9288-4e73-bc7e-de8b8180abcd-kube-api-access-wxlht\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd\" (UID: \"1eb569d7-9288-4e73-bc7e-de8b8180abcd\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd"
Sep 17 18:16:32.812033 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:32.811997    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a-kube-rbac-proxy-sar-config\") pod \"512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a\" (UID: \"512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a\") "
Sep 17 18:16:32.812207 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:32.812053    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a-cabundle-cert\") pod \"512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a\" (UID: \"512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a\") "
Sep 17 18:16:32.812207 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:32.812100    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a-proxy-tls\") pod \"512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a\" (UID: \"512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a\") "
Sep 17 18:16:32.812207 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:32.812127    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mr9nz\" (UniqueName: \"kubernetes.io/projected/512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a-kube-api-access-mr9nz\") pod \"512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a\" (UID: \"512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a\") "
Sep 17 18:16:32.812207 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:32.812164    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a-kserve-provision-location\") pod \"512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a\" (UID: \"512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a\") "
Sep 17 18:16:32.812538 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:32.812515    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a" (UID: "512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:16:32.812612 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:32.812511    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a" (UID: "512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:16:32.812612 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:32.812517    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a" (UID: "512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:16:32.814215 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:32.814193    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a" (UID: "512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:16:32.814269 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:32.814194    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a-kube-api-access-mr9nz" (OuterVolumeSpecName: "kube-api-access-mr9nz") pod "512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a" (UID: "512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a"). InnerVolumeSpecName "kube-api-access-mr9nz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:16:32.814269 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:32.814235    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd"
Sep 17 18:16:32.913213 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:32.913169    2553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a-proxy-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:16:32.913213 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:32.913207    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mr9nz\" (UniqueName: \"kubernetes.io/projected/512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a-kube-api-access-mr9nz\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:16:32.913213 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:32.913216    2553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a-kserve-provision-location\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:16:32.913491 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:32.913224    2553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:16:32.913491 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:32.913233    2553 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a-cabundle-cert\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:16:33.087477 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:33.087452    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd"]
Sep 17 18:16:33.090038 ip-10-0-137-95 kubenswrapper[2553]: W0917 18:16:33.090010    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eb569d7_9288_4e73_bc7e_de8b8180abcd.slice/crio-cbd23a842933a90418a191a8de7562c11020ac7e25f154b1568bcaa23759bac1 WatchSource:0}: Error finding container cbd23a842933a90418a191a8de7562c11020ac7e25f154b1568bcaa23759bac1: Status 404 returned error can't find the container with id cbd23a842933a90418a191a8de7562c11020ac7e25f154b1568bcaa23759bac1
Sep 17 18:16:33.674535 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:33.674439    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd" event={"ID":"1eb569d7-9288-4e73-bc7e-de8b8180abcd","Type":"ContainerStarted","Data":"9d21b1fc7bce5a2d1b398448b8f1256336f294fb606b1df4e6521e9cb684a234"}
Sep 17 18:16:33.674535 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:33.674509    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd" event={"ID":"1eb569d7-9288-4e73-bc7e-de8b8180abcd","Type":"ContainerStarted","Data":"cbd23a842933a90418a191a8de7562c11020ac7e25f154b1568bcaa23759bac1"}
Sep 17 18:16:33.676579 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:33.676557    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-t8rnq" event={"ID":"512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a","Type":"ContainerDied","Data":"727626fb0661f7ee2680ead0d249b7b2cb593bae1ae7f5ef8bd56a854050896d"}
Sep 17 18:16:33.676728 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:33.676606    2553 scope.go:117] "RemoveContainer" containerID="8b60b6d03bea51c78d0ca37ce2b6e24fffab16576e7aef00747445ecb08e4455"
Sep 17 18:16:33.676728 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:33.676565    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-t8rnq"
Sep 17 18:16:33.684929 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:33.684910    2553 scope.go:117] "RemoveContainer" containerID="c3234cbf062ceb21200ac0224f69dd24956aa550aa4175992161a72a43d2c102"
Sep 17 18:16:33.693655 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:33.693633    2553 scope.go:117] "RemoveContainer" containerID="048948318a6d88d292adadc47d6126e78d02e8a80b4e524c7c53f9f21898f470"
Sep 17 18:16:33.704296 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:33.704274    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-t8rnq"]
Sep 17 18:16:33.710800 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:33.710779    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-t8rnq"]
Sep 17 18:16:35.375938 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:35.375903    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a" path="/var/lib/kubelet/pods/512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a/volumes"
Sep 17 18:16:37.695700 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:37.695667    2553 generic.go:356] "Generic (PLEG): container finished" podID="1eb569d7-9288-4e73-bc7e-de8b8180abcd" containerID="9d21b1fc7bce5a2d1b398448b8f1256336f294fb606b1df4e6521e9cb684a234" exitCode=0
Sep 17 18:16:37.696093 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:37.695722    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd" event={"ID":"1eb569d7-9288-4e73-bc7e-de8b8180abcd","Type":"ContainerDied","Data":"9d21b1fc7bce5a2d1b398448b8f1256336f294fb606b1df4e6521e9cb684a234"}
Sep 17 18:16:38.703306 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:38.703267    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd" event={"ID":"1eb569d7-9288-4e73-bc7e-de8b8180abcd","Type":"ContainerStarted","Data":"ca3539e77e95bf706ee88807045a2bae814c3543626fc5ac21530744ecb4b02e"}
Sep 17 18:16:38.703684 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:38.703338    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd" event={"ID":"1eb569d7-9288-4e73-bc7e-de8b8180abcd","Type":"ContainerStarted","Data":"e20ad7c22f56412a33932d628ebf17faa4009b9a06ea9b0ba6b076d22e32343f"}
Sep 17 18:16:38.703684 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:38.703528    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd"
Sep 17 18:16:38.723225 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:38.723173    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd" podStartSLOduration=2.756529814 podStartE2EDuration="6.72315259s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.966622776s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:16:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:16:38.721530787 +0000 UTC m=+1193.417621472" watchObservedRunningTime="2026-09-17 18:16:38.72315259 +0000 UTC m=+1193.419243342"
Sep 17 18:16:39.708235 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:39.708192    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd"
Sep 17 18:16:45.718063 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:45.718033    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd"
Sep 17 18:16:47.344226 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:47.344198    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-btmqz_622ee5be-fb50-42b6-ae23-42900b7c2d98/ovn-acl-logging/0.log"
Sep 17 18:16:47.346855 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:47.346833    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-btmqz_622ee5be-fb50-42b6-ae23-42900b7c2d98/ovn-acl-logging/0.log"
Sep 17 18:16:51.090568 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:51.090530    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-w86n9"]
Sep 17 18:16:51.090949 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:51.090893    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a" containerName="kube-rbac-proxy"
Sep 17 18:16:51.090949 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:51.090904    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a" containerName="kube-rbac-proxy"
Sep 17 18:16:51.090949 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:51.090918    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a" containerName="storage-initializer"
Sep 17 18:16:51.090949 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:51.090923    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a" containerName="storage-initializer"
Sep 17 18:16:51.090949 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:51.090938    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a" containerName="kserve-container"
Sep 17 18:16:51.090949 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:51.090944    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a" containerName="kserve-container"
Sep 17 18:16:51.091143 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:51.090997    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a" containerName="kube-rbac-proxy"
Sep 17 18:16:51.091143 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:51.091006    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="512bbd8c-c5e4-41d2-8cdc-ebe2cfcb891a" containerName="kserve-container"
Sep 17 18:16:51.093965 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:51.093945    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-w86n9"
Sep 17 18:16:51.096356 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:51.096321    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Sep 17 18:16:51.096501 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:51.096369    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Sep 17 18:16:51.103727 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:51.103699    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-w86n9"]
Sep 17 18:16:51.162943 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:51.162902    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szzkw\" (UniqueName: \"kubernetes.io/projected/7f25e4b3-1100-4dfc-805d-608712fa295c-kube-api-access-szzkw\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-w86n9\" (UID: \"7f25e4b3-1100-4dfc-805d-608712fa295c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-w86n9"
Sep 17 18:16:51.163115 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:51.162957    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f25e4b3-1100-4dfc-805d-608712fa295c-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-w86n9\" (UID: \"7f25e4b3-1100-4dfc-805d-608712fa295c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-w86n9"
Sep 17 18:16:51.163115 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:51.163021    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f25e4b3-1100-4dfc-805d-608712fa295c-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-w86n9\" (UID: \"7f25e4b3-1100-4dfc-805d-608712fa295c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-w86n9"
Sep 17 18:16:51.163115 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:51.163068    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f25e4b3-1100-4dfc-805d-608712fa295c-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-w86n9\" (UID: \"7f25e4b3-1100-4dfc-805d-608712fa295c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-w86n9"
Sep 17 18:16:51.163115 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:51.163099    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f25e4b3-1100-4dfc-805d-608712fa295c-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-w86n9\" (UID: \"7f25e4b3-1100-4dfc-805d-608712fa295c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-w86n9"
Sep 17 18:16:51.263536 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:51.263492    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szzkw\" (UniqueName: \"kubernetes.io/projected/7f25e4b3-1100-4dfc-805d-608712fa295c-kube-api-access-szzkw\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-w86n9\" (UID: \"7f25e4b3-1100-4dfc-805d-608712fa295c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-w86n9"
Sep 17 18:16:51.263761 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:51.263549    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f25e4b3-1100-4dfc-805d-608712fa295c-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-w86n9\" (UID: \"7f25e4b3-1100-4dfc-805d-608712fa295c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-w86n9"
Sep 17 18:16:51.263761 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:51.263592    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f25e4b3-1100-4dfc-805d-608712fa295c-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-w86n9\" (UID: \"7f25e4b3-1100-4dfc-805d-608712fa295c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-w86n9"
Sep 17 18:16:51.263886 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:51.263757    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f25e4b3-1100-4dfc-805d-608712fa295c-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-w86n9\" (UID: \"7f25e4b3-1100-4dfc-805d-608712fa295c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-w86n9"
Sep 17 18:16:51.263886 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:16:51.263805    2553 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v2-predictor-serving-cert: secret "isvc-autogluon-v2-predictor-serving-cert" not found
Sep 17 18:16:51.263886 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:16:51.263879    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f25e4b3-1100-4dfc-805d-608712fa295c-proxy-tls podName:7f25e4b3-1100-4dfc-805d-608712fa295c nodeName:}" failed. No retries permitted until 2026-09-17 18:16:51.763857645 +0000 UTC m=+1206.459948307 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7f25e4b3-1100-4dfc-805d-608712fa295c-proxy-tls") pod "isvc-autogluon-v2-predictor-9684b4cd9-w86n9" (UID: "7f25e4b3-1100-4dfc-805d-608712fa295c") : secret "isvc-autogluon-v2-predictor-serving-cert" not found
Sep 17 18:16:51.264049 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:51.263809    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f25e4b3-1100-4dfc-805d-608712fa295c-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-w86n9\" (UID: \"7f25e4b3-1100-4dfc-805d-608712fa295c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-w86n9"
Sep 17 18:16:51.264104 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:51.264053    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f25e4b3-1100-4dfc-805d-608712fa295c-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-w86n9\" (UID: \"7f25e4b3-1100-4dfc-805d-608712fa295c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-w86n9"
Sep 17 18:16:51.264360 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:51.264340    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f25e4b3-1100-4dfc-805d-608712fa295c-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-w86n9\" (UID: \"7f25e4b3-1100-4dfc-805d-608712fa295c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-w86n9"
Sep 17 18:16:51.264395 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:51.264372    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f25e4b3-1100-4dfc-805d-608712fa295c-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-w86n9\" (UID: \"7f25e4b3-1100-4dfc-805d-608712fa295c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-w86n9"
Sep 17 18:16:51.272724 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:51.272698    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szzkw\" (UniqueName: \"kubernetes.io/projected/7f25e4b3-1100-4dfc-805d-608712fa295c-kube-api-access-szzkw\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-w86n9\" (UID: \"7f25e4b3-1100-4dfc-805d-608712fa295c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-w86n9"
Sep 17 18:16:51.768313 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:51.768271    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f25e4b3-1100-4dfc-805d-608712fa295c-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-w86n9\" (UID: \"7f25e4b3-1100-4dfc-805d-608712fa295c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-w86n9"
Sep 17 18:16:51.768524 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:16:51.768434    2553 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v2-predictor-serving-cert: secret "isvc-autogluon-v2-predictor-serving-cert" not found
Sep 17 18:16:51.768524 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:16:51.768505    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f25e4b3-1100-4dfc-805d-608712fa295c-proxy-tls podName:7f25e4b3-1100-4dfc-805d-608712fa295c nodeName:}" failed. No retries permitted until 2026-09-17 18:16:52.768486624 +0000 UTC m=+1207.464577311 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7f25e4b3-1100-4dfc-805d-608712fa295c-proxy-tls") pod "isvc-autogluon-v2-predictor-9684b4cd9-w86n9" (UID: "7f25e4b3-1100-4dfc-805d-608712fa295c") : secret "isvc-autogluon-v2-predictor-serving-cert" not found
Sep 17 18:16:52.776898 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:52.776859    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f25e4b3-1100-4dfc-805d-608712fa295c-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-w86n9\" (UID: \"7f25e4b3-1100-4dfc-805d-608712fa295c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-w86n9"
Sep 17 18:16:52.779303 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:52.779277    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f25e4b3-1100-4dfc-805d-608712fa295c-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-w86n9\" (UID: \"7f25e4b3-1100-4dfc-805d-608712fa295c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-w86n9"
Sep 17 18:16:52.904728 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:52.904667    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-w86n9"
Sep 17 18:16:53.177059 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:53.177030    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-w86n9"]
Sep 17 18:16:53.178998 ip-10-0-137-95 kubenswrapper[2553]: W0917 18:16:53.178969    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f25e4b3_1100_4dfc_805d_608712fa295c.slice/crio-236bcc9a4026ff6e86df1cab9f846eb09af1e9d7973f6edd692ec17aa2ab73c8 WatchSource:0}: Error finding container 236bcc9a4026ff6e86df1cab9f846eb09af1e9d7973f6edd692ec17aa2ab73c8: Status 404 returned error can't find the container with id 236bcc9a4026ff6e86df1cab9f846eb09af1e9d7973f6edd692ec17aa2ab73c8
Sep 17 18:16:53.773880 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:53.773828    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-w86n9" event={"ID":"7f25e4b3-1100-4dfc-805d-608712fa295c","Type":"ContainerStarted","Data":"b6740ba726c55700930f5388a40e459a5692a9fa850999b9f39d4f7e3902347b"}
Sep 17 18:16:53.774141 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:53.773924    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-w86n9" event={"ID":"7f25e4b3-1100-4dfc-805d-608712fa295c","Type":"ContainerStarted","Data":"236bcc9a4026ff6e86df1cab9f846eb09af1e9d7973f6edd692ec17aa2ab73c8"}
Sep 17 18:16:58.797784 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:58.797750    2553 generic.go:356] "Generic (PLEG): container finished" podID="7f25e4b3-1100-4dfc-805d-608712fa295c" containerID="b6740ba726c55700930f5388a40e459a5692a9fa850999b9f39d4f7e3902347b" exitCode=0
Sep 17 18:16:58.798166 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:16:58.797822    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-w86n9" event={"ID":"7f25e4b3-1100-4dfc-805d-608712fa295c","Type":"ContainerDied","Data":"b6740ba726c55700930f5388a40e459a5692a9fa850999b9f39d4f7e3902347b"}
Sep 17 18:17:15.719876 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:17:15.719832    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd" podUID="1eb569d7-9288-4e73-bc7e-de8b8180abcd" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.33:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 18:17:25.719216 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:17:25.719172    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd" podUID="1eb569d7-9288-4e73-bc7e-de8b8180abcd" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.33:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 18:17:35.719465 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:17:35.719418    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd" podUID="1eb569d7-9288-4e73-bc7e-de8b8180abcd" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.33:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 18:17:45.719517 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:17:45.719474    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd" podUID="1eb569d7-9288-4e73-bc7e-de8b8180abcd" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.33:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 18:17:55.718603 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:17:55.718554    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd" podUID="1eb569d7-9288-4e73-bc7e-de8b8180abcd" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.33:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 18:18:05.719453 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:18:05.719394    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd" podUID="1eb569d7-9288-4e73-bc7e-de8b8180abcd" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.33:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 18:18:06.371866 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:18:06.371816    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd" podUID="1eb569d7-9288-4e73-bc7e-de8b8180abcd" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.33:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 18:18:17.044716 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:18:17.044669    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd"
Sep 17 18:18:22.584972 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:18:22.584932    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd"]
Sep 17 18:18:22.585416 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:18:22.585250    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd" podUID="1eb569d7-9288-4e73-bc7e-de8b8180abcd" containerName="kserve-container" containerID="cri-o://e20ad7c22f56412a33932d628ebf17faa4009b9a06ea9b0ba6b076d22e32343f" gracePeriod=30
Sep 17 18:18:22.585416 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:18:22.585327    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd" podUID="1eb569d7-9288-4e73-bc7e-de8b8180abcd" containerName="kube-rbac-proxy" containerID="cri-o://ca3539e77e95bf706ee88807045a2bae814c3543626fc5ac21530744ecb4b02e" gracePeriod=30
Sep 17 18:18:25.713504 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:18:25.713447    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd" podUID="1eb569d7-9288-4e73-bc7e-de8b8180abcd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Sep 17 18:18:26.372541 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:18:26.372489    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd" podUID="1eb569d7-9288-4e73-bc7e-de8b8180abcd" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.33:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 18:18:30.713751 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:18:30.713697    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd" podUID="1eb569d7-9288-4e73-bc7e-de8b8180abcd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Sep 17 18:18:35.713315 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:18:35.713251    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd" podUID="1eb569d7-9288-4e73-bc7e-de8b8180abcd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Sep 17 18:18:35.713831 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:18:35.713421    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd"
Sep 17 18:18:36.372202 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:18:36.372158    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd" podUID="1eb569d7-9288-4e73-bc7e-de8b8180abcd" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.33:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 18:18:40.713069 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:18:40.713019    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd" podUID="1eb569d7-9288-4e73-bc7e-de8b8180abcd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Sep 17 18:18:45.713186 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:18:45.713131    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd" podUID="1eb569d7-9288-4e73-bc7e-de8b8180abcd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Sep 17 18:18:46.371835 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:18:46.371785    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd" podUID="1eb569d7-9288-4e73-bc7e-de8b8180abcd" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.33:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 18:18:46.372143 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:18:46.371953    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd"
Sep 17 18:18:50.712855 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:18:50.712804    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd" podUID="1eb569d7-9288-4e73-bc7e-de8b8180abcd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Sep 17 18:18:55.819718 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:18:55.819684    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd"
Sep 17 18:18:55.912662 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:18:55.912605    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wxlht\" (UniqueName: \"kubernetes.io/projected/1eb569d7-9288-4e73-bc7e-de8b8180abcd-kube-api-access-wxlht\") pod \"1eb569d7-9288-4e73-bc7e-de8b8180abcd\" (UID: \"1eb569d7-9288-4e73-bc7e-de8b8180abcd\") "
Sep 17 18:18:55.912865 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:18:55.912675    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1eb569d7-9288-4e73-bc7e-de8b8180abcd-kube-rbac-proxy-sar-config\") pod \"1eb569d7-9288-4e73-bc7e-de8b8180abcd\" (UID: \"1eb569d7-9288-4e73-bc7e-de8b8180abcd\") "
Sep 17 18:18:55.912865 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:18:55.912718    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1eb569d7-9288-4e73-bc7e-de8b8180abcd-proxy-tls\") pod \"1eb569d7-9288-4e73-bc7e-de8b8180abcd\" (UID: \"1eb569d7-9288-4e73-bc7e-de8b8180abcd\") "
Sep 17 18:18:55.912865 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:18:55.912783    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1eb569d7-9288-4e73-bc7e-de8b8180abcd-kserve-provision-location\") pod \"1eb569d7-9288-4e73-bc7e-de8b8180abcd\" (UID: \"1eb569d7-9288-4e73-bc7e-de8b8180abcd\") "
Sep 17 18:18:55.912865 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:18:55.912837    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1eb569d7-9288-4e73-bc7e-de8b8180abcd-cabundle-cert\") pod \"1eb569d7-9288-4e73-bc7e-de8b8180abcd\" (UID: \"1eb569d7-9288-4e73-bc7e-de8b8180abcd\") "
Sep 17 18:18:55.913213 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:18:55.913098    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eb569d7-9288-4e73-bc7e-de8b8180abcd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1eb569d7-9288-4e73-bc7e-de8b8180abcd" (UID: "1eb569d7-9288-4e73-bc7e-de8b8180abcd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:18:55.913332 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:18:55.913302    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eb569d7-9288-4e73-bc7e-de8b8180abcd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1eb569d7-9288-4e73-bc7e-de8b8180abcd" (UID: "1eb569d7-9288-4e73-bc7e-de8b8180abcd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:18:55.913396 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:18:55.913338    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eb569d7-9288-4e73-bc7e-de8b8180abcd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1eb569d7-9288-4e73-bc7e-de8b8180abcd" (UID: "1eb569d7-9288-4e73-bc7e-de8b8180abcd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:18:55.914916 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:18:55.914892    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1eb569d7-9288-4e73-bc7e-de8b8180abcd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1eb569d7-9288-4e73-bc7e-de8b8180abcd" (UID: "1eb569d7-9288-4e73-bc7e-de8b8180abcd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:18:55.915013 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:18:55.914925    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eb569d7-9288-4e73-bc7e-de8b8180abcd-kube-api-access-wxlht" (OuterVolumeSpecName: "kube-api-access-wxlht") pod "1eb569d7-9288-4e73-bc7e-de8b8180abcd" (UID: "1eb569d7-9288-4e73-bc7e-de8b8180abcd"). InnerVolumeSpecName "kube-api-access-wxlht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:18:56.013940 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:18:56.013900    2553 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1eb569d7-9288-4e73-bc7e-de8b8180abcd-cabundle-cert\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:18:56.013940 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:18:56.013937    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxlht\" (UniqueName: \"kubernetes.io/projected/1eb569d7-9288-4e73-bc7e-de8b8180abcd-kube-api-access-wxlht\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:18:56.014141 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:18:56.013950    2553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1eb569d7-9288-4e73-bc7e-de8b8180abcd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:18:56.014141 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:18:56.013963    2553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1eb569d7-9288-4e73-bc7e-de8b8180abcd-proxy-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:18:56.014141 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:18:56.013976    2553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1eb569d7-9288-4e73-bc7e-de8b8180abcd-kserve-provision-location\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:18:56.408381 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:18:56.408350    2553 generic.go:356] "Generic (PLEG): container finished" podID="1eb569d7-9288-4e73-bc7e-de8b8180abcd" containerID="ca3539e77e95bf706ee88807045a2bae814c3543626fc5ac21530744ecb4b02e" exitCode=2
Sep 17 18:18:56.408381 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:18:56.408377    2553 generic.go:356] "Generic (PLEG): container finished" podID="1eb569d7-9288-4e73-bc7e-de8b8180abcd" containerID="e20ad7c22f56412a33932d628ebf17faa4009b9a06ea9b0ba6b076d22e32343f" exitCode=0
Sep 17 18:18:56.408729 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:18:56.408399    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd" event={"ID":"1eb569d7-9288-4e73-bc7e-de8b8180abcd","Type":"ContainerDied","Data":"ca3539e77e95bf706ee88807045a2bae814c3543626fc5ac21530744ecb4b02e"}
Sep 17 18:18:56.408729 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:18:56.408403    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd"
Sep 17 18:18:56.408729 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:18:56.408451    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd" event={"ID":"1eb569d7-9288-4e73-bc7e-de8b8180abcd","Type":"ContainerDied","Data":"e20ad7c22f56412a33932d628ebf17faa4009b9a06ea9b0ba6b076d22e32343f"}
Sep 17 18:18:56.408729 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:18:56.408476    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd" event={"ID":"1eb569d7-9288-4e73-bc7e-de8b8180abcd","Type":"ContainerDied","Data":"cbd23a842933a90418a191a8de7562c11020ac7e25f154b1568bcaa23759bac1"}
Sep 17 18:18:56.408729 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:18:56.408500    2553 scope.go:117] "RemoveContainer" containerID="ca3539e77e95bf706ee88807045a2bae814c3543626fc5ac21530744ecb4b02e"
Sep 17 18:18:56.420656 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:18:56.420630    2553 scope.go:117] "RemoveContainer" containerID="e20ad7c22f56412a33932d628ebf17faa4009b9a06ea9b0ba6b076d22e32343f"
Sep 17 18:18:56.428395 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:18:56.428375    2553 scope.go:117] "RemoveContainer" containerID="9d21b1fc7bce5a2d1b398448b8f1256336f294fb606b1df4e6521e9cb684a234"
Sep 17 18:18:56.436188 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:18:56.436169    2553 scope.go:117] "RemoveContainer" containerID="ca3539e77e95bf706ee88807045a2bae814c3543626fc5ac21530744ecb4b02e"
Sep 17 18:18:56.436507 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:18:56.436489    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca3539e77e95bf706ee88807045a2bae814c3543626fc5ac21530744ecb4b02e\": container with ID starting with ca3539e77e95bf706ee88807045a2bae814c3543626fc5ac21530744ecb4b02e not found: ID does not exist" containerID="ca3539e77e95bf706ee88807045a2bae814c3543626fc5ac21530744ecb4b02e"
Sep 17 18:18:56.436583 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:18:56.436519    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca3539e77e95bf706ee88807045a2bae814c3543626fc5ac21530744ecb4b02e"} err="failed to get container status \"ca3539e77e95bf706ee88807045a2bae814c3543626fc5ac21530744ecb4b02e\": rpc error: code = NotFound desc = could not find container \"ca3539e77e95bf706ee88807045a2bae814c3543626fc5ac21530744ecb4b02e\": container with ID starting with ca3539e77e95bf706ee88807045a2bae814c3543626fc5ac21530744ecb4b02e not found: ID does not exist"
Sep 17 18:18:56.436583 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:18:56.436546    2553 scope.go:117] "RemoveContainer" containerID="e20ad7c22f56412a33932d628ebf17faa4009b9a06ea9b0ba6b076d22e32343f"
Sep 17 18:18:56.436864 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:18:56.436841    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e20ad7c22f56412a33932d628ebf17faa4009b9a06ea9b0ba6b076d22e32343f\": container with ID starting with e20ad7c22f56412a33932d628ebf17faa4009b9a06ea9b0ba6b076d22e32343f not found: ID does not exist" containerID="e20ad7c22f56412a33932d628ebf17faa4009b9a06ea9b0ba6b076d22e32343f"
Sep 17 18:18:56.436914 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:18:56.436873    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e20ad7c22f56412a33932d628ebf17faa4009b9a06ea9b0ba6b076d22e32343f"} err="failed to get container status \"e20ad7c22f56412a33932d628ebf17faa4009b9a06ea9b0ba6b076d22e32343f\": rpc error: code = NotFound desc = could not find container \"e20ad7c22f56412a33932d628ebf17faa4009b9a06ea9b0ba6b076d22e32343f\": container with ID starting with e20ad7c22f56412a33932d628ebf17faa4009b9a06ea9b0ba6b076d22e32343f not found: ID does not exist"
Sep 17 18:18:56.436914 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:18:56.436894    2553 scope.go:117] "RemoveContainer" containerID="9d21b1fc7bce5a2d1b398448b8f1256336f294fb606b1df4e6521e9cb684a234"
Sep 17 18:18:56.437138 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:18:56.437119    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d21b1fc7bce5a2d1b398448b8f1256336f294fb606b1df4e6521e9cb684a234\": container with ID starting with 9d21b1fc7bce5a2d1b398448b8f1256336f294fb606b1df4e6521e9cb684a234 not found: ID does not exist" containerID="9d21b1fc7bce5a2d1b398448b8f1256336f294fb606b1df4e6521e9cb684a234"
Sep 17 18:18:56.437201 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:18:56.437146    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d21b1fc7bce5a2d1b398448b8f1256336f294fb606b1df4e6521e9cb684a234"} err="failed to get container status \"9d21b1fc7bce5a2d1b398448b8f1256336f294fb606b1df4e6521e9cb684a234\": rpc error: code = NotFound desc = could not find container \"9d21b1fc7bce5a2d1b398448b8f1256336f294fb606b1df4e6521e9cb684a234\": container with ID starting with 9d21b1fc7bce5a2d1b398448b8f1256336f294fb606b1df4e6521e9cb684a234 not found: ID does not exist"
Sep 17 18:18:56.437201 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:18:56.437167    2553 scope.go:117] "RemoveContainer" containerID="ca3539e77e95bf706ee88807045a2bae814c3543626fc5ac21530744ecb4b02e"
Sep 17 18:18:56.437421 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:18:56.437402    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca3539e77e95bf706ee88807045a2bae814c3543626fc5ac21530744ecb4b02e"} err="failed to get container status \"ca3539e77e95bf706ee88807045a2bae814c3543626fc5ac21530744ecb4b02e\": rpc error: code = NotFound desc = could not find container \"ca3539e77e95bf706ee88807045a2bae814c3543626fc5ac21530744ecb4b02e\": container with ID starting with ca3539e77e95bf706ee88807045a2bae814c3543626fc5ac21530744ecb4b02e not found: ID does not exist"
Sep 17 18:18:56.437473 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:18:56.437422    2553 scope.go:117] "RemoveContainer" containerID="e20ad7c22f56412a33932d628ebf17faa4009b9a06ea9b0ba6b076d22e32343f"
Sep 17 18:18:56.437745 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:18:56.437724    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e20ad7c22f56412a33932d628ebf17faa4009b9a06ea9b0ba6b076d22e32343f"} err="failed to get container status \"e20ad7c22f56412a33932d628ebf17faa4009b9a06ea9b0ba6b076d22e32343f\": rpc error: code = NotFound desc = could not find container \"e20ad7c22f56412a33932d628ebf17faa4009b9a06ea9b0ba6b076d22e32343f\": container with ID starting with e20ad7c22f56412a33932d628ebf17faa4009b9a06ea9b0ba6b076d22e32343f not found: ID does not exist"
Sep 17 18:18:56.437820 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:18:56.437746    2553 scope.go:117] "RemoveContainer" containerID="9d21b1fc7bce5a2d1b398448b8f1256336f294fb606b1df4e6521e9cb684a234"
Sep 17 18:18:56.438068 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:18:56.438045    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d21b1fc7bce5a2d1b398448b8f1256336f294fb606b1df4e6521e9cb684a234"} err="failed to get container status \"9d21b1fc7bce5a2d1b398448b8f1256336f294fb606b1df4e6521e9cb684a234\": rpc error: code = NotFound desc = could not find container \"9d21b1fc7bce5a2d1b398448b8f1256336f294fb606b1df4e6521e9cb684a234\": container with ID starting with 9d21b1fc7bce5a2d1b398448b8f1256336f294fb606b1df4e6521e9cb684a234 not found: ID does not exist"
Sep 17 18:18:56.449000 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:18:56.448976    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd"]
Sep 17 18:18:56.460018 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:18:56.459994    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd"]
Sep 17 18:18:56.714333 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:18:56.713552    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-dkgrd" podUID="1eb569d7-9288-4e73-bc7e-de8b8180abcd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 17 18:18:57.377057 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:18:57.376978    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1eb569d7-9288-4e73-bc7e-de8b8180abcd" path="/var/lib/kubelet/pods/1eb569d7-9288-4e73-bc7e-de8b8180abcd/volumes"
Sep 17 18:18:57.415452 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:18:57.415412    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-w86n9" event={"ID":"7f25e4b3-1100-4dfc-805d-608712fa295c","Type":"ContainerStarted","Data":"c5bdf5789f3fbcae2f0b88849b2cfff2a7e5c7f6762d20988cc0328fcfc7d0f2"}
Sep 17 18:18:57.415671 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:18:57.415492    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-w86n9" event={"ID":"7f25e4b3-1100-4dfc-805d-608712fa295c","Type":"ContainerStarted","Data":"49e26e17bee17b305bad89c22b66d1f8b1a0a2e618ae7b12f7d0dc2c97e088f3"}
Sep 17 18:18:57.415671 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:18:57.415533    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-w86n9"
Sep 17 18:18:57.435840 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:18:57.435787    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-w86n9" podStartSLOduration=3.5373649069999997 podStartE2EDuration="2m6.43576936s" totalImagesPullingTime="1m57.967253808s" totalInitContainerRuntime="4.931150645s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:16:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:18:57.434214282 +0000 UTC m=+1332.130305048" watchObservedRunningTime="2026-09-17 18:18:57.43576936 +0000 UTC m=+1332.131860045"
Sep 17 18:18:58.421844 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:18:58.421809    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-w86n9"
Sep 17 18:19:04.433352 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:19:04.433266    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-w86n9"
Sep 17 18:20:28.411793 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:28.411749    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-w86n9"
Sep 17 18:20:37.439942 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:37.439904    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-w86n9"]
Sep 17 18:20:37.440368 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:37.440162    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-w86n9" podUID="7f25e4b3-1100-4dfc-805d-608712fa295c" containerName="kserve-container" containerID="cri-o://49e26e17bee17b305bad89c22b66d1f8b1a0a2e618ae7b12f7d0dc2c97e088f3" gracePeriod=30
Sep 17 18:20:37.440368 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:37.440176    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-w86n9" podUID="7f25e4b3-1100-4dfc-805d-608712fa295c" containerName="kube-rbac-proxy" containerID="cri-o://c5bdf5789f3fbcae2f0b88849b2cfff2a7e5c7f6762d20988cc0328fcfc7d0f2" gracePeriod=30
Sep 17 18:20:37.860770 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:37.860680    2553 generic.go:356] "Generic (PLEG): container finished" podID="7f25e4b3-1100-4dfc-805d-608712fa295c" containerID="c5bdf5789f3fbcae2f0b88849b2cfff2a7e5c7f6762d20988cc0328fcfc7d0f2" exitCode=2
Sep 17 18:20:37.860770 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:37.860749    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-w86n9" event={"ID":"7f25e4b3-1100-4dfc-805d-608712fa295c","Type":"ContainerDied","Data":"c5bdf5789f3fbcae2f0b88849b2cfff2a7e5c7f6762d20988cc0328fcfc7d0f2"}
Sep 17 18:20:38.372070 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:38.372026    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-w86n9" podUID="7f25e4b3-1100-4dfc-805d-608712fa295c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.34:8080/v2/models/isvc-autogluon-v2/ready\": dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 18:20:38.868748 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:38.868704    2553 generic.go:356] "Generic (PLEG): container finished" podID="7f25e4b3-1100-4dfc-805d-608712fa295c" containerID="49e26e17bee17b305bad89c22b66d1f8b1a0a2e618ae7b12f7d0dc2c97e088f3" exitCode=0
Sep 17 18:20:38.869088 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:38.868774    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-w86n9" event={"ID":"7f25e4b3-1100-4dfc-805d-608712fa295c","Type":"ContainerDied","Data":"49e26e17bee17b305bad89c22b66d1f8b1a0a2e618ae7b12f7d0dc2c97e088f3"}
Sep 17 18:20:38.987114 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:38.987084    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-w86n9"
Sep 17 18:20:39.087726 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:39.087614    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-szzkw\" (UniqueName: \"kubernetes.io/projected/7f25e4b3-1100-4dfc-805d-608712fa295c-kube-api-access-szzkw\") pod \"7f25e4b3-1100-4dfc-805d-608712fa295c\" (UID: \"7f25e4b3-1100-4dfc-805d-608712fa295c\") "
Sep 17 18:20:39.087726 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:39.087704    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f25e4b3-1100-4dfc-805d-608712fa295c-proxy-tls\") pod \"7f25e4b3-1100-4dfc-805d-608712fa295c\" (UID: \"7f25e4b3-1100-4dfc-805d-608712fa295c\") "
Sep 17 18:20:39.087949 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:39.087749    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f25e4b3-1100-4dfc-805d-608712fa295c-kserve-provision-location\") pod \"7f25e4b3-1100-4dfc-805d-608712fa295c\" (UID: \"7f25e4b3-1100-4dfc-805d-608712fa295c\") "
Sep 17 18:20:39.087949 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:39.087799    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f25e4b3-1100-4dfc-805d-608712fa295c-kube-rbac-proxy-sar-config\") pod \"7f25e4b3-1100-4dfc-805d-608712fa295c\" (UID: \"7f25e4b3-1100-4dfc-805d-608712fa295c\") "
Sep 17 18:20:39.087949 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:39.087834    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f25e4b3-1100-4dfc-805d-608712fa295c-cabundle-cert\") pod \"7f25e4b3-1100-4dfc-805d-608712fa295c\" (UID: \"7f25e4b3-1100-4dfc-805d-608712fa295c\") "
Sep 17 18:20:39.088194 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:39.088159    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f25e4b3-1100-4dfc-805d-608712fa295c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7f25e4b3-1100-4dfc-805d-608712fa295c" (UID: "7f25e4b3-1100-4dfc-805d-608712fa295c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:20:39.088269 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:39.088246    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f25e4b3-1100-4dfc-805d-608712fa295c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7f25e4b3-1100-4dfc-805d-608712fa295c" (UID: "7f25e4b3-1100-4dfc-805d-608712fa295c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:20:39.089850 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:39.089826    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f25e4b3-1100-4dfc-805d-608712fa295c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7f25e4b3-1100-4dfc-805d-608712fa295c" (UID: "7f25e4b3-1100-4dfc-805d-608712fa295c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:20:39.089962 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:39.089900    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f25e4b3-1100-4dfc-805d-608712fa295c-kube-api-access-szzkw" (OuterVolumeSpecName: "kube-api-access-szzkw") pod "7f25e4b3-1100-4dfc-805d-608712fa295c" (UID: "7f25e4b3-1100-4dfc-805d-608712fa295c"). InnerVolumeSpecName "kube-api-access-szzkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:20:39.092527 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:39.092500    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f25e4b3-1100-4dfc-805d-608712fa295c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7f25e4b3-1100-4dfc-805d-608712fa295c" (UID: "7f25e4b3-1100-4dfc-805d-608712fa295c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:20:39.188968 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:39.188925    2553 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f25e4b3-1100-4dfc-805d-608712fa295c-cabundle-cert\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:20:39.188968 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:39.188959    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szzkw\" (UniqueName: \"kubernetes.io/projected/7f25e4b3-1100-4dfc-805d-608712fa295c-kube-api-access-szzkw\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:20:39.188968 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:39.188973    2553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f25e4b3-1100-4dfc-805d-608712fa295c-proxy-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:20:39.189202 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:39.188987    2553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f25e4b3-1100-4dfc-805d-608712fa295c-kserve-provision-location\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:20:39.189202 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:39.188999    2553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f25e4b3-1100-4dfc-805d-608712fa295c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:20:39.875700 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:39.875600    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-w86n9" event={"ID":"7f25e4b3-1100-4dfc-805d-608712fa295c","Type":"ContainerDied","Data":"236bcc9a4026ff6e86df1cab9f846eb09af1e9d7973f6edd692ec17aa2ab73c8"}
Sep 17 18:20:39.875700 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:39.875633    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-w86n9"
Sep 17 18:20:39.875700 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:39.875689    2553 scope.go:117] "RemoveContainer" containerID="c5bdf5789f3fbcae2f0b88849b2cfff2a7e5c7f6762d20988cc0328fcfc7d0f2"
Sep 17 18:20:39.884806 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:39.884777    2553 scope.go:117] "RemoveContainer" containerID="49e26e17bee17b305bad89c22b66d1f8b1a0a2e618ae7b12f7d0dc2c97e088f3"
Sep 17 18:20:39.893050 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:39.893022    2553 scope.go:117] "RemoveContainer" containerID="b6740ba726c55700930f5388a40e459a5692a9fa850999b9f39d4f7e3902347b"
Sep 17 18:20:39.900248 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:39.894897    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-w86n9"]
Sep 17 18:20:39.900248 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:39.900190    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-w86n9"]
Sep 17 18:20:41.376406 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:41.376369    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f25e4b3-1100-4dfc-805d-608712fa295c" path="/var/lib/kubelet/pods/7f25e4b3-1100-4dfc-805d-608712fa295c/volumes"
Sep 17 18:20:42.721063 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:42.721028    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tk5fl"]
Sep 17 18:20:42.721451 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:42.721340    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1eb569d7-9288-4e73-bc7e-de8b8180abcd" containerName="storage-initializer"
Sep 17 18:20:42.721451 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:42.721351    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="1eb569d7-9288-4e73-bc7e-de8b8180abcd" containerName="storage-initializer"
Sep 17 18:20:42.721451 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:42.721361    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f25e4b3-1100-4dfc-805d-608712fa295c" containerName="kube-rbac-proxy"
Sep 17 18:20:42.721451 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:42.721367    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f25e4b3-1100-4dfc-805d-608712fa295c" containerName="kube-rbac-proxy"
Sep 17 18:20:42.721451 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:42.721375    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f25e4b3-1100-4dfc-805d-608712fa295c" containerName="storage-initializer"
Sep 17 18:20:42.721451 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:42.721381    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f25e4b3-1100-4dfc-805d-608712fa295c" containerName="storage-initializer"
Sep 17 18:20:42.721451 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:42.721391    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1eb569d7-9288-4e73-bc7e-de8b8180abcd" containerName="kserve-container"
Sep 17 18:20:42.721451 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:42.721396    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="1eb569d7-9288-4e73-bc7e-de8b8180abcd" containerName="kserve-container"
Sep 17 18:20:42.721451 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:42.721406    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1eb569d7-9288-4e73-bc7e-de8b8180abcd" containerName="kube-rbac-proxy"
Sep 17 18:20:42.721451 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:42.721411    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="1eb569d7-9288-4e73-bc7e-de8b8180abcd" containerName="kube-rbac-proxy"
Sep 17 18:20:42.721451 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:42.721417    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f25e4b3-1100-4dfc-805d-608712fa295c" containerName="kserve-container"
Sep 17 18:20:42.721451 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:42.721422    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f25e4b3-1100-4dfc-805d-608712fa295c" containerName="kserve-container"
Sep 17 18:20:42.721829 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:42.721472    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f25e4b3-1100-4dfc-805d-608712fa295c" containerName="kube-rbac-proxy"
Sep 17 18:20:42.721829 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:42.721480    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="1eb569d7-9288-4e73-bc7e-de8b8180abcd" containerName="kube-rbac-proxy"
Sep 17 18:20:42.721829 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:42.721486    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f25e4b3-1100-4dfc-805d-608712fa295c" containerName="kserve-container"
Sep 17 18:20:42.721829 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:42.721493    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="1eb569d7-9288-4e73-bc7e-de8b8180abcd" containerName="kserve-container"
Sep 17 18:20:42.725028 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:42.725012    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tk5fl"
Sep 17 18:20:42.732019 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:42.731997    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Sep 17 18:20:42.732131 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:42.731997    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:20:42.732220 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:42.732198    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Sep 17 18:20:42.732952 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:42.732932    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:20:42.733019 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:42.733004    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:20:42.733054 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:42.733035    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:20:42.733386 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:42.733374    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-7qcmd\""
Sep 17 18:20:42.752964 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:42.752941    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tk5fl"]
Sep 17 18:20:42.818002 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:42.817963    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d449572c-c0ea-49ff-8430-248f25128703-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tk5fl\" (UID: \"d449572c-c0ea-49ff-8430-248f25128703\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tk5fl"
Sep 17 18:20:42.818176 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:42.818024    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d449572c-c0ea-49ff-8430-248f25128703-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tk5fl\" (UID: \"d449572c-c0ea-49ff-8430-248f25128703\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tk5fl"
Sep 17 18:20:42.818176 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:42.818115    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7td2p\" (UniqueName: \"kubernetes.io/projected/d449572c-c0ea-49ff-8430-248f25128703-kube-api-access-7td2p\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tk5fl\" (UID: \"d449572c-c0ea-49ff-8430-248f25128703\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tk5fl"
Sep 17 18:20:42.818176 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:42.818153    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d449572c-c0ea-49ff-8430-248f25128703-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tk5fl\" (UID: \"d449572c-c0ea-49ff-8430-248f25128703\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tk5fl"
Sep 17 18:20:42.818176 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:42.818175    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d449572c-c0ea-49ff-8430-248f25128703-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tk5fl\" (UID: \"d449572c-c0ea-49ff-8430-248f25128703\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tk5fl"
Sep 17 18:20:42.919430 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:42.919391    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d449572c-c0ea-49ff-8430-248f25128703-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tk5fl\" (UID: \"d449572c-c0ea-49ff-8430-248f25128703\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tk5fl"
Sep 17 18:20:42.919613 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:42.919440    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d449572c-c0ea-49ff-8430-248f25128703-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tk5fl\" (UID: \"d449572c-c0ea-49ff-8430-248f25128703\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tk5fl"
Sep 17 18:20:42.919613 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:42.919490    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7td2p\" (UniqueName: \"kubernetes.io/projected/d449572c-c0ea-49ff-8430-248f25128703-kube-api-access-7td2p\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tk5fl\" (UID: \"d449572c-c0ea-49ff-8430-248f25128703\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tk5fl"
Sep 17 18:20:42.919613 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:42.919512    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d449572c-c0ea-49ff-8430-248f25128703-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tk5fl\" (UID: \"d449572c-c0ea-49ff-8430-248f25128703\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tk5fl"
Sep 17 18:20:42.919613 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:42.919529    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d449572c-c0ea-49ff-8430-248f25128703-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tk5fl\" (UID: \"d449572c-c0ea-49ff-8430-248f25128703\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tk5fl"
Sep 17 18:20:42.919933 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:42.919912    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d449572c-c0ea-49ff-8430-248f25128703-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tk5fl\" (UID: \"d449572c-c0ea-49ff-8430-248f25128703\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tk5fl"
Sep 17 18:20:42.920263 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:42.920229    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d449572c-c0ea-49ff-8430-248f25128703-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tk5fl\" (UID: \"d449572c-c0ea-49ff-8430-248f25128703\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tk5fl"
Sep 17 18:20:42.920376 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:42.920295    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d449572c-c0ea-49ff-8430-248f25128703-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tk5fl\" (UID: \"d449572c-c0ea-49ff-8430-248f25128703\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tk5fl"
Sep 17 18:20:42.921988 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:42.921971    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d449572c-c0ea-49ff-8430-248f25128703-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tk5fl\" (UID: \"d449572c-c0ea-49ff-8430-248f25128703\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tk5fl"
Sep 17 18:20:42.932809 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:42.932787    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7td2p\" (UniqueName: \"kubernetes.io/projected/d449572c-c0ea-49ff-8430-248f25128703-kube-api-access-7td2p\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tk5fl\" (UID: \"d449572c-c0ea-49ff-8430-248f25128703\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tk5fl"
Sep 17 18:20:43.035278 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:43.035185    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tk5fl"
Sep 17 18:20:43.307283 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:43.307172    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tk5fl"]
Sep 17 18:20:43.309644 ip-10-0-137-95 kubenswrapper[2553]: W0917 18:20:43.309591    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd449572c_c0ea_49ff_8430_248f25128703.slice/crio-d99aca6aad6554f6080321710bc116300d2db5a533d4a511d60fab7be84152d2 WatchSource:0}: Error finding container d99aca6aad6554f6080321710bc116300d2db5a533d4a511d60fab7be84152d2: Status 404 returned error can't find the container with id d99aca6aad6554f6080321710bc116300d2db5a533d4a511d60fab7be84152d2
Sep 17 18:20:43.895993 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:43.895950    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tk5fl" event={"ID":"d449572c-c0ea-49ff-8430-248f25128703","Type":"ContainerStarted","Data":"7cd50df4ee062d1fe437d29fca8b45cc9d9699b4f3947fb2484accb883fff98d"}
Sep 17 18:20:43.896440 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:20:43.896018    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tk5fl" event={"ID":"d449572c-c0ea-49ff-8430-248f25128703","Type":"ContainerStarted","Data":"d99aca6aad6554f6080321710bc116300d2db5a533d4a511d60fab7be84152d2"}
Sep 17 18:21:03.984396 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:21:03.984358    2553 generic.go:356] "Generic (PLEG): container finished" podID="d449572c-c0ea-49ff-8430-248f25128703" containerID="7cd50df4ee062d1fe437d29fca8b45cc9d9699b4f3947fb2484accb883fff98d" exitCode=0
Sep 17 18:21:03.984872 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:21:03.984430    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tk5fl" event={"ID":"d449572c-c0ea-49ff-8430-248f25128703","Type":"ContainerDied","Data":"7cd50df4ee062d1fe437d29fca8b45cc9d9699b4f3947fb2484accb883fff98d"}
Sep 17 18:21:04.991760 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:21:04.991720    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tk5fl" event={"ID":"d449572c-c0ea-49ff-8430-248f25128703","Type":"ContainerStarted","Data":"d10d839b2bae9b2155c54b1ddf1f9a0c6d9bd320e2f3434a45cfa791c2400725"}
Sep 17 18:21:04.992230 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:21:04.991793    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tk5fl" event={"ID":"d449572c-c0ea-49ff-8430-248f25128703","Type":"ContainerStarted","Data":"84da0cc1e9290b28a468f274f6291416d1630fcb9b9b3ba2fbeaff292f670cae"}
Sep 17 18:21:04.992230 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:21:04.992011    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tk5fl"
Sep 17 18:21:05.012025 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:21:05.011969    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tk5fl" podStartSLOduration=2.579198561 podStartE2EDuration="23.011953203s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.432754642s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:20:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:21:05.011015546 +0000 UTC m=+1459.707106255" watchObservedRunningTime="2026-09-17 18:21:05.011953203 +0000 UTC m=+1459.708043888"
Sep 17 18:21:05.998807 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:21:05.998775    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tk5fl"
Sep 17 18:21:12.008713 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:21:12.008677    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tk5fl"
Sep 17 18:21:47.371609 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:21:47.371575    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-btmqz_622ee5be-fb50-42b6-ae23-42900b7c2d98/ovn-acl-logging/0.log"
Sep 17 18:21:47.375907 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:21:47.375886    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-btmqz_622ee5be-fb50-42b6-ae23-42900b7c2d98/ovn-acl-logging/0.log"
Sep 17 18:22:38.407608 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:38.407573    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tk5fl"
Sep 17 18:22:49.525090 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:49.525052    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tk5fl"]
Sep 17 18:22:49.525587 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:49.525279    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tk5fl" podUID="d449572c-c0ea-49ff-8430-248f25128703" containerName="kserve-container" containerID="cri-o://84da0cc1e9290b28a468f274f6291416d1630fcb9b9b3ba2fbeaff292f670cae" gracePeriod=30
Sep 17 18:22:49.525587 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:49.525353    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tk5fl" podUID="d449572c-c0ea-49ff-8430-248f25128703" containerName="kube-rbac-proxy" containerID="cri-o://d10d839b2bae9b2155c54b1ddf1f9a0c6d9bd320e2f3434a45cfa791c2400725" gracePeriod=30
Sep 17 18:22:50.474493 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:50.474450    2553 generic.go:356] "Generic (PLEG): container finished" podID="d449572c-c0ea-49ff-8430-248f25128703" containerID="d10d839b2bae9b2155c54b1ddf1f9a0c6d9bd320e2f3434a45cfa791c2400725" exitCode=2
Sep 17 18:22:50.474712 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:50.474523    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tk5fl" event={"ID":"d449572c-c0ea-49ff-8430-248f25128703","Type":"ContainerDied","Data":"d10d839b2bae9b2155c54b1ddf1f9a0c6d9bd320e2f3434a45cfa791c2400725"}
Sep 17 18:22:51.168326 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:51.168300    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tk5fl"
Sep 17 18:22:51.295430 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:51.295337    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d449572c-c0ea-49ff-8430-248f25128703-cabundle-cert\") pod \"d449572c-c0ea-49ff-8430-248f25128703\" (UID: \"d449572c-c0ea-49ff-8430-248f25128703\") "
Sep 17 18:22:51.295430 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:51.295384    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d449572c-c0ea-49ff-8430-248f25128703-kserve-provision-location\") pod \"d449572c-c0ea-49ff-8430-248f25128703\" (UID: \"d449572c-c0ea-49ff-8430-248f25128703\") "
Sep 17 18:22:51.295430 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:51.295408    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d449572c-c0ea-49ff-8430-248f25128703-kube-rbac-proxy-sar-config\") pod \"d449572c-c0ea-49ff-8430-248f25128703\" (UID: \"d449572c-c0ea-49ff-8430-248f25128703\") "
Sep 17 18:22:51.295430 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:51.295429    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7td2p\" (UniqueName: \"kubernetes.io/projected/d449572c-c0ea-49ff-8430-248f25128703-kube-api-access-7td2p\") pod \"d449572c-c0ea-49ff-8430-248f25128703\" (UID: \"d449572c-c0ea-49ff-8430-248f25128703\") "
Sep 17 18:22:51.295798 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:51.295461    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d449572c-c0ea-49ff-8430-248f25128703-proxy-tls\") pod \"d449572c-c0ea-49ff-8430-248f25128703\" (UID: \"d449572c-c0ea-49ff-8430-248f25128703\") "
Sep 17 18:22:51.295866 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:51.295828    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d449572c-c0ea-49ff-8430-248f25128703-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d449572c-c0ea-49ff-8430-248f25128703" (UID: "d449572c-c0ea-49ff-8430-248f25128703"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:22:51.295866 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:51.295835    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d449572c-c0ea-49ff-8430-248f25128703-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d449572c-c0ea-49ff-8430-248f25128703" (UID: "d449572c-c0ea-49ff-8430-248f25128703"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:22:51.297705 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:51.297675    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d449572c-c0ea-49ff-8430-248f25128703-kube-api-access-7td2p" (OuterVolumeSpecName: "kube-api-access-7td2p") pod "d449572c-c0ea-49ff-8430-248f25128703" (UID: "d449572c-c0ea-49ff-8430-248f25128703"). InnerVolumeSpecName "kube-api-access-7td2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:22:51.297822 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:51.297683    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d449572c-c0ea-49ff-8430-248f25128703-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d449572c-c0ea-49ff-8430-248f25128703" (UID: "d449572c-c0ea-49ff-8430-248f25128703"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:22:51.299838 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:51.299810    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d449572c-c0ea-49ff-8430-248f25128703-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d449572c-c0ea-49ff-8430-248f25128703" (UID: "d449572c-c0ea-49ff-8430-248f25128703"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:22:51.395948 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:51.395918    2553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d449572c-c0ea-49ff-8430-248f25128703-proxy-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:22:51.395948 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:51.395946    2553 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d449572c-c0ea-49ff-8430-248f25128703-cabundle-cert\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:22:51.396104 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:51.395955    2553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d449572c-c0ea-49ff-8430-248f25128703-kserve-provision-location\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:22:51.396104 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:51.395964    2553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d449572c-c0ea-49ff-8430-248f25128703-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:22:51.396104 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:51.395973    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7td2p\" (UniqueName: \"kubernetes.io/projected/d449572c-c0ea-49ff-8430-248f25128703-kube-api-access-7td2p\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:22:51.480993 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:51.480957    2553 generic.go:356] "Generic (PLEG): container finished" podID="d449572c-c0ea-49ff-8430-248f25128703" containerID="84da0cc1e9290b28a468f274f6291416d1630fcb9b9b3ba2fbeaff292f670cae" exitCode=0
Sep 17 18:22:51.481161 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:51.481013    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tk5fl"
Sep 17 18:22:51.481161 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:51.481030    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tk5fl" event={"ID":"d449572c-c0ea-49ff-8430-248f25128703","Type":"ContainerDied","Data":"84da0cc1e9290b28a468f274f6291416d1630fcb9b9b3ba2fbeaff292f670cae"}
Sep 17 18:22:51.481161 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:51.481102    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tk5fl" event={"ID":"d449572c-c0ea-49ff-8430-248f25128703","Type":"ContainerDied","Data":"d99aca6aad6554f6080321710bc116300d2db5a533d4a511d60fab7be84152d2"}
Sep 17 18:22:51.481161 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:51.481141    2553 scope.go:117] "RemoveContainer" containerID="d10d839b2bae9b2155c54b1ddf1f9a0c6d9bd320e2f3434a45cfa791c2400725"
Sep 17 18:22:51.489573 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:51.489552    2553 scope.go:117] "RemoveContainer" containerID="84da0cc1e9290b28a468f274f6291416d1630fcb9b9b3ba2fbeaff292f670cae"
Sep 17 18:22:51.497643 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:51.497600    2553 scope.go:117] "RemoveContainer" containerID="7cd50df4ee062d1fe437d29fca8b45cc9d9699b4f3947fb2484accb883fff98d"
Sep 17 18:22:51.498812 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:51.498790    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tk5fl"]
Sep 17 18:22:51.502325 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:51.502304    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tk5fl"]
Sep 17 18:22:51.505269 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:51.505252    2553 scope.go:117] "RemoveContainer" containerID="d10d839b2bae9b2155c54b1ddf1f9a0c6d9bd320e2f3434a45cfa791c2400725"
Sep 17 18:22:51.505576 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:22:51.505557    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d10d839b2bae9b2155c54b1ddf1f9a0c6d9bd320e2f3434a45cfa791c2400725\": container with ID starting with d10d839b2bae9b2155c54b1ddf1f9a0c6d9bd320e2f3434a45cfa791c2400725 not found: ID does not exist" containerID="d10d839b2bae9b2155c54b1ddf1f9a0c6d9bd320e2f3434a45cfa791c2400725"
Sep 17 18:22:51.505636 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:51.505583    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d10d839b2bae9b2155c54b1ddf1f9a0c6d9bd320e2f3434a45cfa791c2400725"} err="failed to get container status \"d10d839b2bae9b2155c54b1ddf1f9a0c6d9bd320e2f3434a45cfa791c2400725\": rpc error: code = NotFound desc = could not find container \"d10d839b2bae9b2155c54b1ddf1f9a0c6d9bd320e2f3434a45cfa791c2400725\": container with ID starting with d10d839b2bae9b2155c54b1ddf1f9a0c6d9bd320e2f3434a45cfa791c2400725 not found: ID does not exist"
Sep 17 18:22:51.505636 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:51.505601    2553 scope.go:117] "RemoveContainer" containerID="84da0cc1e9290b28a468f274f6291416d1630fcb9b9b3ba2fbeaff292f670cae"
Sep 17 18:22:51.505875 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:22:51.505858    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84da0cc1e9290b28a468f274f6291416d1630fcb9b9b3ba2fbeaff292f670cae\": container with ID starting with 84da0cc1e9290b28a468f274f6291416d1630fcb9b9b3ba2fbeaff292f670cae not found: ID does not exist" containerID="84da0cc1e9290b28a468f274f6291416d1630fcb9b9b3ba2fbeaff292f670cae"
Sep 17 18:22:51.505913 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:51.505882    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84da0cc1e9290b28a468f274f6291416d1630fcb9b9b3ba2fbeaff292f670cae"} err="failed to get container status \"84da0cc1e9290b28a468f274f6291416d1630fcb9b9b3ba2fbeaff292f670cae\": rpc error: code = NotFound desc = could not find container \"84da0cc1e9290b28a468f274f6291416d1630fcb9b9b3ba2fbeaff292f670cae\": container with ID starting with 84da0cc1e9290b28a468f274f6291416d1630fcb9b9b3ba2fbeaff292f670cae not found: ID does not exist"
Sep 17 18:22:51.505913 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:51.505896    2553 scope.go:117] "RemoveContainer" containerID="7cd50df4ee062d1fe437d29fca8b45cc9d9699b4f3947fb2484accb883fff98d"
Sep 17 18:22:51.506128 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:22:51.506111    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cd50df4ee062d1fe437d29fca8b45cc9d9699b4f3947fb2484accb883fff98d\": container with ID starting with 7cd50df4ee062d1fe437d29fca8b45cc9d9699b4f3947fb2484accb883fff98d not found: ID does not exist" containerID="7cd50df4ee062d1fe437d29fca8b45cc9d9699b4f3947fb2484accb883fff98d"
Sep 17 18:22:51.506167 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:51.506133    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cd50df4ee062d1fe437d29fca8b45cc9d9699b4f3947fb2484accb883fff98d"} err="failed to get container status \"7cd50df4ee062d1fe437d29fca8b45cc9d9699b4f3947fb2484accb883fff98d\": rpc error: code = NotFound desc = could not find container \"7cd50df4ee062d1fe437d29fca8b45cc9d9699b4f3947fb2484accb883fff98d\": container with ID starting with 7cd50df4ee062d1fe437d29fca8b45cc9d9699b4f3947fb2484accb883fff98d not found: ID does not exist"
Sep 17 18:22:53.376386 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:53.376349    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d449572c-c0ea-49ff-8430-248f25128703" path="/var/lib/kubelet/pods/d449572c-c0ea-49ff-8430-248f25128703/volumes"
Sep 17 18:22:54.710233 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:54.710195    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gxclc"]
Sep 17 18:22:54.710606 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:54.710514    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d449572c-c0ea-49ff-8430-248f25128703" containerName="kube-rbac-proxy"
Sep 17 18:22:54.710606 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:54.710566    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="d449572c-c0ea-49ff-8430-248f25128703" containerName="kube-rbac-proxy"
Sep 17 18:22:54.710606 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:54.710581    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d449572c-c0ea-49ff-8430-248f25128703" containerName="kserve-container"
Sep 17 18:22:54.710606 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:54.710587    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="d449572c-c0ea-49ff-8430-248f25128703" containerName="kserve-container"
Sep 17 18:22:54.710606 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:54.710596    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d449572c-c0ea-49ff-8430-248f25128703" containerName="storage-initializer"
Sep 17 18:22:54.710606 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:54.710603    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="d449572c-c0ea-49ff-8430-248f25128703" containerName="storage-initializer"
Sep 17 18:22:54.710879 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:54.710669    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="d449572c-c0ea-49ff-8430-248f25128703" containerName="kube-rbac-proxy"
Sep 17 18:22:54.710879 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:54.710681    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="d449572c-c0ea-49ff-8430-248f25128703" containerName="kserve-container"
Sep 17 18:22:54.714017 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:54.713998    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gxclc"
Sep 17 18:22:54.716604 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:54.716576    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Sep 17 18:22:54.716723 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:54.716635    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-7qcmd\""
Sep 17 18:22:54.717153 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:54.717137    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Sep 17 18:22:54.717609 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:54.717596    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:22:54.717701 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:54.717610    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:22:54.717701 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:54.717681    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:22:54.717701 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:54.717690    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:22:54.721480 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:54.721460    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da33c43e-b614-4888-8027-d64b189a01e8-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-gxclc\" (UID: \"da33c43e-b614-4888-8027-d64b189a01e8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gxclc"
Sep 17 18:22:54.721568 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:54.721501    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/da33c43e-b614-4888-8027-d64b189a01e8-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-gxclc\" (UID: \"da33c43e-b614-4888-8027-d64b189a01e8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gxclc"
Sep 17 18:22:54.721611 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:54.721586    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da33c43e-b614-4888-8027-d64b189a01e8-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-gxclc\" (UID: \"da33c43e-b614-4888-8027-d64b189a01e8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gxclc"
Sep 17 18:22:54.721672 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:54.721649    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/da33c43e-b614-4888-8027-d64b189a01e8-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-gxclc\" (UID: \"da33c43e-b614-4888-8027-d64b189a01e8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gxclc"
Sep 17 18:22:54.721716 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:54.721693    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8nss\" (UniqueName: \"kubernetes.io/projected/da33c43e-b614-4888-8027-d64b189a01e8-kube-api-access-p8nss\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-gxclc\" (UID: \"da33c43e-b614-4888-8027-d64b189a01e8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gxclc"
Sep 17 18:22:54.724257 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:54.724237    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gxclc"]
Sep 17 18:22:54.822340 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:54.822303    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da33c43e-b614-4888-8027-d64b189a01e8-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-gxclc\" (UID: \"da33c43e-b614-4888-8027-d64b189a01e8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gxclc"
Sep 17 18:22:54.822340 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:54.822347    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/da33c43e-b614-4888-8027-d64b189a01e8-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-gxclc\" (UID: \"da33c43e-b614-4888-8027-d64b189a01e8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gxclc"
Sep 17 18:22:54.822550 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:54.822386    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8nss\" (UniqueName: \"kubernetes.io/projected/da33c43e-b614-4888-8027-d64b189a01e8-kube-api-access-p8nss\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-gxclc\" (UID: \"da33c43e-b614-4888-8027-d64b189a01e8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gxclc"
Sep 17 18:22:54.822550 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:54.822410    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da33c43e-b614-4888-8027-d64b189a01e8-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-gxclc\" (UID: \"da33c43e-b614-4888-8027-d64b189a01e8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gxclc"
Sep 17 18:22:54.822550 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:54.822432    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/da33c43e-b614-4888-8027-d64b189a01e8-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-gxclc\" (UID: \"da33c43e-b614-4888-8027-d64b189a01e8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gxclc"
Sep 17 18:22:54.822733 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:22:54.822565    2553 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-serving-cert: secret "isvc-autogluon-v2-noslash-predictor-serving-cert" not found
Sep 17 18:22:54.822733 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:22:54.822684    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da33c43e-b614-4888-8027-d64b189a01e8-proxy-tls podName:da33c43e-b614-4888-8027-d64b189a01e8 nodeName:}" failed. No retries permitted until 2026-09-17 18:22:55.322661459 +0000 UTC m=+1570.018752128 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/da33c43e-b614-4888-8027-d64b189a01e8-proxy-tls") pod "isvc-autogluon-v2-noslash-predictor-5499c79f9f-gxclc" (UID: "da33c43e-b614-4888-8027-d64b189a01e8") : secret "isvc-autogluon-v2-noslash-predictor-serving-cert" not found
Sep 17 18:22:54.822824 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:54.822790    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da33c43e-b614-4888-8027-d64b189a01e8-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-gxclc\" (UID: \"da33c43e-b614-4888-8027-d64b189a01e8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gxclc"
Sep 17 18:22:54.823114 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:54.823096    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/da33c43e-b614-4888-8027-d64b189a01e8-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-gxclc\" (UID: \"da33c43e-b614-4888-8027-d64b189a01e8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gxclc"
Sep 17 18:22:54.823240 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:54.823224    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/da33c43e-b614-4888-8027-d64b189a01e8-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-gxclc\" (UID: \"da33c43e-b614-4888-8027-d64b189a01e8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gxclc"
Sep 17 18:22:54.835138 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:54.835114    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8nss\" (UniqueName: \"kubernetes.io/projected/da33c43e-b614-4888-8027-d64b189a01e8-kube-api-access-p8nss\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-gxclc\" (UID: \"da33c43e-b614-4888-8027-d64b189a01e8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gxclc"
Sep 17 18:22:55.326681 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:55.326643    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da33c43e-b614-4888-8027-d64b189a01e8-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-gxclc\" (UID: \"da33c43e-b614-4888-8027-d64b189a01e8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gxclc"
Sep 17 18:22:55.329043 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:55.329014    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da33c43e-b614-4888-8027-d64b189a01e8-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-gxclc\" (UID: \"da33c43e-b614-4888-8027-d64b189a01e8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gxclc"
Sep 17 18:22:55.625239 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:55.625127    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gxclc"
Sep 17 18:22:55.897490 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:55.897455    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gxclc"]
Sep 17 18:22:55.900069 ip-10-0-137-95 kubenswrapper[2553]: W0917 18:22:55.900033    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda33c43e_b614_4888_8027_d64b189a01e8.slice/crio-59ebea70025224e11b90fa0cbb7d09b3345c25a98296afcc94230094971164f9 WatchSource:0}: Error finding container 59ebea70025224e11b90fa0cbb7d09b3345c25a98296afcc94230094971164f9: Status 404 returned error can't find the container with id 59ebea70025224e11b90fa0cbb7d09b3345c25a98296afcc94230094971164f9
Sep 17 18:22:56.507424 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:56.507386    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gxclc" event={"ID":"da33c43e-b614-4888-8027-d64b189a01e8","Type":"ContainerStarted","Data":"3c85bf9e7a6b7e7e1ab44cc8bd5a12f448a053c6e88021f995678350cb21a510"}
Sep 17 18:22:56.507604 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:22:56.507446    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gxclc" event={"ID":"da33c43e-b614-4888-8027-d64b189a01e8","Type":"ContainerStarted","Data":"59ebea70025224e11b90fa0cbb7d09b3345c25a98296afcc94230094971164f9"}
Sep 17 18:23:01.529665 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:23:01.529567    2553 generic.go:356] "Generic (PLEG): container finished" podID="da33c43e-b614-4888-8027-d64b189a01e8" containerID="3c85bf9e7a6b7e7e1ab44cc8bd5a12f448a053c6e88021f995678350cb21a510" exitCode=0
Sep 17 18:23:01.529665 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:23:01.529610    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gxclc" event={"ID":"da33c43e-b614-4888-8027-d64b189a01e8","Type":"ContainerDied","Data":"3c85bf9e7a6b7e7e1ab44cc8bd5a12f448a053c6e88021f995678350cb21a510"}
Sep 17 18:23:02.536707 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:23:02.536666    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gxclc" event={"ID":"da33c43e-b614-4888-8027-d64b189a01e8","Type":"ContainerStarted","Data":"74d9e11d17aa936e50b8e7dd9229be5d251c91ca012811bacdb0b27828724a8c"}
Sep 17 18:23:02.537142 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:23:02.536754    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gxclc" event={"ID":"da33c43e-b614-4888-8027-d64b189a01e8","Type":"ContainerStarted","Data":"1f3f012f80d7c6f515410f9f4e480f4133935125e0478c2fb89e0f1bd02ef03e"}
Sep 17 18:23:02.537142 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:23:02.537010    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gxclc"
Sep 17 18:23:02.537142 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:23:02.537059    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gxclc"
Sep 17 18:23:02.556501 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:23:02.556444    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gxclc" podStartSLOduration=3.361335787 podStartE2EDuration="8.556425702s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.195089915s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:22:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:23:02.555294937 +0000 UTC m=+1577.251385621" watchObservedRunningTime="2026-09-17 18:23:02.556425702 +0000 UTC m=+1577.252516387"
Sep 17 18:23:08.548275 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:23:08.548239    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gxclc"
Sep 17 18:24:36.406154 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:36.406120    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gxclc"
Sep 17 18:24:47.411899 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:47.411832    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gxclc" podUID="da33c43e-b614-4888-8027-d64b189a01e8" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.36:8080/v2/models/isvc-autogluon-v2-noslash/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 17 18:24:51.079437 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:51.079402    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gxclc"]
Sep 17 18:24:51.079854 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:51.079739    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gxclc" podUID="da33c43e-b614-4888-8027-d64b189a01e8" containerName="kserve-container" containerID="cri-o://1f3f012f80d7c6f515410f9f4e480f4133935125e0478c2fb89e0f1bd02ef03e" gracePeriod=30
Sep 17 18:24:51.079854 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:51.079800    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gxclc" podUID="da33c43e-b614-4888-8027-d64b189a01e8" containerName="kube-rbac-proxy" containerID="cri-o://74d9e11d17aa936e50b8e7dd9229be5d251c91ca012811bacdb0b27828724a8c" gracePeriod=30
Sep 17 18:24:52.030666 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:52.030610    2553 generic.go:356] "Generic (PLEG): container finished" podID="da33c43e-b614-4888-8027-d64b189a01e8" containerID="74d9e11d17aa936e50b8e7dd9229be5d251c91ca012811bacdb0b27828724a8c" exitCode=2
Sep 17 18:24:52.030912 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:52.030678    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gxclc" event={"ID":"da33c43e-b614-4888-8027-d64b189a01e8","Type":"ContainerDied","Data":"74d9e11d17aa936e50b8e7dd9229be5d251c91ca012811bacdb0b27828724a8c"}
Sep 17 18:24:52.621226 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:52.621198    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gxclc"
Sep 17 18:24:52.731645 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:52.731537    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/da33c43e-b614-4888-8027-d64b189a01e8-cabundle-cert\") pod \"da33c43e-b614-4888-8027-d64b189a01e8\" (UID: \"da33c43e-b614-4888-8027-d64b189a01e8\") "
Sep 17 18:24:52.731645 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:52.731580    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da33c43e-b614-4888-8027-d64b189a01e8-proxy-tls\") pod \"da33c43e-b614-4888-8027-d64b189a01e8\" (UID: \"da33c43e-b614-4888-8027-d64b189a01e8\") "
Sep 17 18:24:52.731645 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:52.731599    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da33c43e-b614-4888-8027-d64b189a01e8-kserve-provision-location\") pod \"da33c43e-b614-4888-8027-d64b189a01e8\" (UID: \"da33c43e-b614-4888-8027-d64b189a01e8\") "
Sep 17 18:24:52.732042 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:52.731651    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p8nss\" (UniqueName: \"kubernetes.io/projected/da33c43e-b614-4888-8027-d64b189a01e8-kube-api-access-p8nss\") pod \"da33c43e-b614-4888-8027-d64b189a01e8\" (UID: \"da33c43e-b614-4888-8027-d64b189a01e8\") "
Sep 17 18:24:52.732042 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:52.731683    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/da33c43e-b614-4888-8027-d64b189a01e8-kube-rbac-proxy-sar-config\") pod \"da33c43e-b614-4888-8027-d64b189a01e8\" (UID: \"da33c43e-b614-4888-8027-d64b189a01e8\") "
Sep 17 18:24:52.732121 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:52.732063    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da33c43e-b614-4888-8027-d64b189a01e8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "da33c43e-b614-4888-8027-d64b189a01e8" (UID: "da33c43e-b614-4888-8027-d64b189a01e8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:24:52.732121 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:52.732099    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da33c43e-b614-4888-8027-d64b189a01e8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "da33c43e-b614-4888-8027-d64b189a01e8" (UID: "da33c43e-b614-4888-8027-d64b189a01e8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:24:52.733759 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:52.733733    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da33c43e-b614-4888-8027-d64b189a01e8-kube-api-access-p8nss" (OuterVolumeSpecName: "kube-api-access-p8nss") pod "da33c43e-b614-4888-8027-d64b189a01e8" (UID: "da33c43e-b614-4888-8027-d64b189a01e8"). InnerVolumeSpecName "kube-api-access-p8nss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:24:52.734125 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:52.734110    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da33c43e-b614-4888-8027-d64b189a01e8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "da33c43e-b614-4888-8027-d64b189a01e8" (UID: "da33c43e-b614-4888-8027-d64b189a01e8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:24:52.737377 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:52.737351    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da33c43e-b614-4888-8027-d64b189a01e8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "da33c43e-b614-4888-8027-d64b189a01e8" (UID: "da33c43e-b614-4888-8027-d64b189a01e8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:24:52.832402 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:52.832361    2553 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/da33c43e-b614-4888-8027-d64b189a01e8-cabundle-cert\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:24:52.832402 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:52.832396    2553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da33c43e-b614-4888-8027-d64b189a01e8-proxy-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:24:52.832402 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:52.832405    2553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da33c43e-b614-4888-8027-d64b189a01e8-kserve-provision-location\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:24:52.832402 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:52.832415    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8nss\" (UniqueName: \"kubernetes.io/projected/da33c43e-b614-4888-8027-d64b189a01e8-kube-api-access-p8nss\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:24:52.832699 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:52.832424    2553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/da33c43e-b614-4888-8027-d64b189a01e8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:24:53.037072 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:53.036989    2553 generic.go:356] "Generic (PLEG): container finished" podID="da33c43e-b614-4888-8027-d64b189a01e8" containerID="1f3f012f80d7c6f515410f9f4e480f4133935125e0478c2fb89e0f1bd02ef03e" exitCode=0
Sep 17 18:24:53.037072 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:53.037043    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gxclc"
Sep 17 18:24:53.037261 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:53.037068    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gxclc" event={"ID":"da33c43e-b614-4888-8027-d64b189a01e8","Type":"ContainerDied","Data":"1f3f012f80d7c6f515410f9f4e480f4133935125e0478c2fb89e0f1bd02ef03e"}
Sep 17 18:24:53.037261 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:53.037130    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gxclc" event={"ID":"da33c43e-b614-4888-8027-d64b189a01e8","Type":"ContainerDied","Data":"59ebea70025224e11b90fa0cbb7d09b3345c25a98296afcc94230094971164f9"}
Sep 17 18:24:53.037261 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:53.037157    2553 scope.go:117] "RemoveContainer" containerID="74d9e11d17aa936e50b8e7dd9229be5d251c91ca012811bacdb0b27828724a8c"
Sep 17 18:24:53.046399 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:53.046381    2553 scope.go:117] "RemoveContainer" containerID="1f3f012f80d7c6f515410f9f4e480f4133935125e0478c2fb89e0f1bd02ef03e"
Sep 17 18:24:53.054166 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:53.054150    2553 scope.go:117] "RemoveContainer" containerID="3c85bf9e7a6b7e7e1ab44cc8bd5a12f448a053c6e88021f995678350cb21a510"
Sep 17 18:24:53.059502 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:53.059476    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gxclc"]
Sep 17 18:24:53.062827 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:53.062804    2553 scope.go:117] "RemoveContainer" containerID="74d9e11d17aa936e50b8e7dd9229be5d251c91ca012811bacdb0b27828724a8c"
Sep 17 18:24:53.063158 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:24:53.063139    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74d9e11d17aa936e50b8e7dd9229be5d251c91ca012811bacdb0b27828724a8c\": container with ID starting with 74d9e11d17aa936e50b8e7dd9229be5d251c91ca012811bacdb0b27828724a8c not found: ID does not exist" containerID="74d9e11d17aa936e50b8e7dd9229be5d251c91ca012811bacdb0b27828724a8c"
Sep 17 18:24:53.063216 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:53.063167    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74d9e11d17aa936e50b8e7dd9229be5d251c91ca012811bacdb0b27828724a8c"} err="failed to get container status \"74d9e11d17aa936e50b8e7dd9229be5d251c91ca012811bacdb0b27828724a8c\": rpc error: code = NotFound desc = could not find container \"74d9e11d17aa936e50b8e7dd9229be5d251c91ca012811bacdb0b27828724a8c\": container with ID starting with 74d9e11d17aa936e50b8e7dd9229be5d251c91ca012811bacdb0b27828724a8c not found: ID does not exist"
Sep 17 18:24:53.063216 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:53.063187    2553 scope.go:117] "RemoveContainer" containerID="1f3f012f80d7c6f515410f9f4e480f4133935125e0478c2fb89e0f1bd02ef03e"
Sep 17 18:24:53.063293 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:53.063231    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gxclc"]
Sep 17 18:24:53.063446 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:24:53.063428    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f3f012f80d7c6f515410f9f4e480f4133935125e0478c2fb89e0f1bd02ef03e\": container with ID starting with 1f3f012f80d7c6f515410f9f4e480f4133935125e0478c2fb89e0f1bd02ef03e not found: ID does not exist" containerID="1f3f012f80d7c6f515410f9f4e480f4133935125e0478c2fb89e0f1bd02ef03e"
Sep 17 18:24:53.063489 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:53.063452    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f3f012f80d7c6f515410f9f4e480f4133935125e0478c2fb89e0f1bd02ef03e"} err="failed to get container status \"1f3f012f80d7c6f515410f9f4e480f4133935125e0478c2fb89e0f1bd02ef03e\": rpc error: code = NotFound desc = could not find container \"1f3f012f80d7c6f515410f9f4e480f4133935125e0478c2fb89e0f1bd02ef03e\": container with ID starting with 1f3f012f80d7c6f515410f9f4e480f4133935125e0478c2fb89e0f1bd02ef03e not found: ID does not exist"
Sep 17 18:24:53.063528 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:53.063489    2553 scope.go:117] "RemoveContainer" containerID="3c85bf9e7a6b7e7e1ab44cc8bd5a12f448a053c6e88021f995678350cb21a510"
Sep 17 18:24:53.063762 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:24:53.063745    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c85bf9e7a6b7e7e1ab44cc8bd5a12f448a053c6e88021f995678350cb21a510\": container with ID starting with 3c85bf9e7a6b7e7e1ab44cc8bd5a12f448a053c6e88021f995678350cb21a510 not found: ID does not exist" containerID="3c85bf9e7a6b7e7e1ab44cc8bd5a12f448a053c6e88021f995678350cb21a510"
Sep 17 18:24:53.063815 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:53.063765    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c85bf9e7a6b7e7e1ab44cc8bd5a12f448a053c6e88021f995678350cb21a510"} err="failed to get container status \"3c85bf9e7a6b7e7e1ab44cc8bd5a12f448a053c6e88021f995678350cb21a510\": rpc error: code = NotFound desc = could not find container \"3c85bf9e7a6b7e7e1ab44cc8bd5a12f448a053c6e88021f995678350cb21a510\": container with ID starting with 3c85bf9e7a6b7e7e1ab44cc8bd5a12f448a053c6e88021f995678350cb21a510 not found: ID does not exist"
Sep 17 18:24:53.377295 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:53.377206    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da33c43e-b614-4888-8027-d64b189a01e8" path="/var/lib/kubelet/pods/da33c43e-b614-4888-8027-d64b189a01e8/volumes"
Sep 17 18:24:56.263038 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:56.262999    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fd2d4"]
Sep 17 18:24:56.263414 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:56.263366    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da33c43e-b614-4888-8027-d64b189a01e8" containerName="storage-initializer"
Sep 17 18:24:56.263414 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:56.263378    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="da33c43e-b614-4888-8027-d64b189a01e8" containerName="storage-initializer"
Sep 17 18:24:56.263414 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:56.263392    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da33c43e-b614-4888-8027-d64b189a01e8" containerName="kserve-container"
Sep 17 18:24:56.263414 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:56.263398    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="da33c43e-b614-4888-8027-d64b189a01e8" containerName="kserve-container"
Sep 17 18:24:56.263414 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:56.263416    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da33c43e-b614-4888-8027-d64b189a01e8" containerName="kube-rbac-proxy"
Sep 17 18:24:56.263766 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:56.263423    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="da33c43e-b614-4888-8027-d64b189a01e8" containerName="kube-rbac-proxy"
Sep 17 18:24:56.263766 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:56.263468    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="da33c43e-b614-4888-8027-d64b189a01e8" containerName="kserve-container"
Sep 17 18:24:56.263766 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:56.263477    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="da33c43e-b614-4888-8027-d64b189a01e8" containerName="kube-rbac-proxy"
Sep 17 18:24:56.268233 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:56.268212    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fd2d4"
Sep 17 18:24:56.270834 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:56.270812    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Sep 17 18:24:56.270938 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:56.270833    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:24:56.271131 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:56.271113    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:24:56.271206 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:56.271124    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:24:56.271206 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:56.271155    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-7qcmd\""
Sep 17 18:24:56.271980 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:56.271964    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:24:56.272075 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:56.272060    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Sep 17 18:24:56.277505 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:56.277482    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fd2d4"]
Sep 17 18:24:56.357386 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:56.357350    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8ddddf7-d988-4e7d-9727-8fd44fe67713-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fd2d4\" (UID: \"e8ddddf7-d988-4e7d-9727-8fd44fe67713\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fd2d4"
Sep 17 18:24:56.357568 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:56.357402    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8ddddf7-d988-4e7d-9727-8fd44fe67713-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fd2d4\" (UID: \"e8ddddf7-d988-4e7d-9727-8fd44fe67713\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fd2d4"
Sep 17 18:24:56.357568 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:56.357422    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8ddddf7-d988-4e7d-9727-8fd44fe67713-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fd2d4\" (UID: \"e8ddddf7-d988-4e7d-9727-8fd44fe67713\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fd2d4"
Sep 17 18:24:56.357568 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:56.357476    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7565p\" (UniqueName: \"kubernetes.io/projected/e8ddddf7-d988-4e7d-9727-8fd44fe67713-kube-api-access-7565p\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fd2d4\" (UID: \"e8ddddf7-d988-4e7d-9727-8fd44fe67713\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fd2d4"
Sep 17 18:24:56.357568 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:56.357512    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8ddddf7-d988-4e7d-9727-8fd44fe67713-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fd2d4\" (UID: \"e8ddddf7-d988-4e7d-9727-8fd44fe67713\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fd2d4"
Sep 17 18:24:56.458841 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:56.458795    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8ddddf7-d988-4e7d-9727-8fd44fe67713-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fd2d4\" (UID: \"e8ddddf7-d988-4e7d-9727-8fd44fe67713\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fd2d4"
Sep 17 18:24:56.459058 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:56.458861    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8ddddf7-d988-4e7d-9727-8fd44fe67713-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fd2d4\" (UID: \"e8ddddf7-d988-4e7d-9727-8fd44fe67713\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fd2d4"
Sep 17 18:24:56.459058 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:56.458883    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8ddddf7-d988-4e7d-9727-8fd44fe67713-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fd2d4\" (UID: \"e8ddddf7-d988-4e7d-9727-8fd44fe67713\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fd2d4"
Sep 17 18:24:56.459058 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:56.458913    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7565p\" (UniqueName: \"kubernetes.io/projected/e8ddddf7-d988-4e7d-9727-8fd44fe67713-kube-api-access-7565p\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fd2d4\" (UID: \"e8ddddf7-d988-4e7d-9727-8fd44fe67713\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fd2d4"
Sep 17 18:24:56.459058 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:56.458941    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8ddddf7-d988-4e7d-9727-8fd44fe67713-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fd2d4\" (UID: \"e8ddddf7-d988-4e7d-9727-8fd44fe67713\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fd2d4"
Sep 17 18:24:56.459058 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:24:56.458960    2553 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-serving-cert: secret "isvc-autogluon-ts-v1-predictor-serving-cert" not found
Sep 17 18:24:56.459058 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:24:56.459037    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8ddddf7-d988-4e7d-9727-8fd44fe67713-proxy-tls podName:e8ddddf7-d988-4e7d-9727-8fd44fe67713 nodeName:}" failed. No retries permitted until 2026-09-17 18:24:56.959016527 +0000 UTC m=+1691.655107208 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e8ddddf7-d988-4e7d-9727-8fd44fe67713-proxy-tls") pod "isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fd2d4" (UID: "e8ddddf7-d988-4e7d-9727-8fd44fe67713") : secret "isvc-autogluon-ts-v1-predictor-serving-cert" not found
Sep 17 18:24:56.459414 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:56.459316    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8ddddf7-d988-4e7d-9727-8fd44fe67713-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fd2d4\" (UID: \"e8ddddf7-d988-4e7d-9727-8fd44fe67713\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fd2d4"
Sep 17 18:24:56.459597 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:56.459577    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8ddddf7-d988-4e7d-9727-8fd44fe67713-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fd2d4\" (UID: \"e8ddddf7-d988-4e7d-9727-8fd44fe67713\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fd2d4"
Sep 17 18:24:56.459697 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:56.459652    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8ddddf7-d988-4e7d-9727-8fd44fe67713-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fd2d4\" (UID: \"e8ddddf7-d988-4e7d-9727-8fd44fe67713\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fd2d4"
Sep 17 18:24:56.467745 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:56.467722    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7565p\" (UniqueName: \"kubernetes.io/projected/e8ddddf7-d988-4e7d-9727-8fd44fe67713-kube-api-access-7565p\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fd2d4\" (UID: \"e8ddddf7-d988-4e7d-9727-8fd44fe67713\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fd2d4"
Sep 17 18:24:56.963070 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:56.963022    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8ddddf7-d988-4e7d-9727-8fd44fe67713-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fd2d4\" (UID: \"e8ddddf7-d988-4e7d-9727-8fd44fe67713\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fd2d4"
Sep 17 18:24:56.965465 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:56.965431    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8ddddf7-d988-4e7d-9727-8fd44fe67713-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fd2d4\" (UID: \"e8ddddf7-d988-4e7d-9727-8fd44fe67713\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fd2d4"
Sep 17 18:24:57.179160 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:57.179125    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fd2d4"
Sep 17 18:24:57.454970 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:57.454939    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fd2d4"]
Sep 17 18:24:57.457457 ip-10-0-137-95 kubenswrapper[2553]: W0917 18:24:57.457432    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8ddddf7_d988_4e7d_9727_8fd44fe67713.slice/crio-6954a831858d7a074fc5be56653d825d4c6da2ce9498a01102de6f8231ed1bd7 WatchSource:0}: Error finding container 6954a831858d7a074fc5be56653d825d4c6da2ce9498a01102de6f8231ed1bd7: Status 404 returned error can't find the container with id 6954a831858d7a074fc5be56653d825d4c6da2ce9498a01102de6f8231ed1bd7
Sep 17 18:24:58.064418 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:58.064376    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fd2d4" event={"ID":"e8ddddf7-d988-4e7d-9727-8fd44fe67713","Type":"ContainerStarted","Data":"60c74e4cd34f7a5e735f36ff3846fa319198689fe7d5ccd0e0f0735717c03ca6"}
Sep 17 18:24:58.064595 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:24:58.064450    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fd2d4" event={"ID":"e8ddddf7-d988-4e7d-9727-8fd44fe67713","Type":"ContainerStarted","Data":"6954a831858d7a074fc5be56653d825d4c6da2ce9498a01102de6f8231ed1bd7"}
Sep 17 18:25:03.088170 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:03.088135    2553 generic.go:356] "Generic (PLEG): container finished" podID="e8ddddf7-d988-4e7d-9727-8fd44fe67713" containerID="60c74e4cd34f7a5e735f36ff3846fa319198689fe7d5ccd0e0f0735717c03ca6" exitCode=0
Sep 17 18:25:03.088535 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:03.088206    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fd2d4" event={"ID":"e8ddddf7-d988-4e7d-9727-8fd44fe67713","Type":"ContainerDied","Data":"60c74e4cd34f7a5e735f36ff3846fa319198689fe7d5ccd0e0f0735717c03ca6"}
Sep 17 18:25:04.097675 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:04.097639    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fd2d4" event={"ID":"e8ddddf7-d988-4e7d-9727-8fd44fe67713","Type":"ContainerStarted","Data":"408b475fbfed1beb0ea56c8545d69cea0498b56e4c76f45f30c27a5725d70b08"}
Sep 17 18:25:04.098173 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:04.097706    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fd2d4" event={"ID":"e8ddddf7-d988-4e7d-9727-8fd44fe67713","Type":"ContainerStarted","Data":"3501332a2a11702fc16112d9867d2a6236f934f6368b7ebb5ebb388a05aa0552"}
Sep 17 18:25:04.098173 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:04.097884    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fd2d4"
Sep 17 18:25:04.119120 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:04.119053    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fd2d4" podStartSLOduration=3.508811986 podStartE2EDuration="8.119035541s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.610223555s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:24:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:25:04.11722811 +0000 UTC m=+1698.813318793" watchObservedRunningTime="2026-09-17 18:25:04.119035541 +0000 UTC m=+1698.815126224"
Sep 17 18:25:05.103397 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:05.103351    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fd2d4"
Sep 17 18:25:05.104879 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:05.104846    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fd2d4" podUID="e8ddddf7-d988-4e7d-9727-8fd44fe67713" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 18:25:06.108480 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:06.108440    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fd2d4" podUID="e8ddddf7-d988-4e7d-9727-8fd44fe67713" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 18:25:11.119884 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:11.119850    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fd2d4"
Sep 17 18:25:11.120529 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:11.120495    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fd2d4" podUID="e8ddddf7-d988-4e7d-9727-8fd44fe67713" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 18:25:21.121753 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:21.121719    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fd2d4"
Sep 17 18:25:39.050804 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:39.050771    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fd2d4"]
Sep 17 18:25:39.051333 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:39.051013    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fd2d4" podUID="e8ddddf7-d988-4e7d-9727-8fd44fe67713" containerName="kserve-container" containerID="cri-o://3501332a2a11702fc16112d9867d2a6236f934f6368b7ebb5ebb388a05aa0552" gracePeriod=30
Sep 17 18:25:39.051333 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:39.051081    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fd2d4" podUID="e8ddddf7-d988-4e7d-9727-8fd44fe67713" containerName="kube-rbac-proxy" containerID="cri-o://408b475fbfed1beb0ea56c8545d69cea0498b56e4c76f45f30c27a5725d70b08" gracePeriod=30
Sep 17 18:25:39.279249 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:39.279211    2553 generic.go:356] "Generic (PLEG): container finished" podID="e8ddddf7-d988-4e7d-9727-8fd44fe67713" containerID="408b475fbfed1beb0ea56c8545d69cea0498b56e4c76f45f30c27a5725d70b08" exitCode=2
Sep 17 18:25:39.279418 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:39.279264    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fd2d4" event={"ID":"e8ddddf7-d988-4e7d-9727-8fd44fe67713","Type":"ContainerDied","Data":"408b475fbfed1beb0ea56c8545d69cea0498b56e4c76f45f30c27a5725d70b08"}
Sep 17 18:25:40.996944 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:40.996914    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fd2d4"
Sep 17 18:25:41.064727 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:41.064643    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8ddddf7-d988-4e7d-9727-8fd44fe67713-kube-rbac-proxy-sar-config\") pod \"e8ddddf7-d988-4e7d-9727-8fd44fe67713\" (UID: \"e8ddddf7-d988-4e7d-9727-8fd44fe67713\") "
Sep 17 18:25:41.064888 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:41.064730    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8ddddf7-d988-4e7d-9727-8fd44fe67713-proxy-tls\") pod \"e8ddddf7-d988-4e7d-9727-8fd44fe67713\" (UID: \"e8ddddf7-d988-4e7d-9727-8fd44fe67713\") "
Sep 17 18:25:41.064888 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:41.064755    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8ddddf7-d988-4e7d-9727-8fd44fe67713-kserve-provision-location\") pod \"e8ddddf7-d988-4e7d-9727-8fd44fe67713\" (UID: \"e8ddddf7-d988-4e7d-9727-8fd44fe67713\") "
Sep 17 18:25:41.064888 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:41.064791    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8ddddf7-d988-4e7d-9727-8fd44fe67713-cabundle-cert\") pod \"e8ddddf7-d988-4e7d-9727-8fd44fe67713\" (UID: \"e8ddddf7-d988-4e7d-9727-8fd44fe67713\") "
Sep 17 18:25:41.064888 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:41.064815    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7565p\" (UniqueName: \"kubernetes.io/projected/e8ddddf7-d988-4e7d-9727-8fd44fe67713-kube-api-access-7565p\") pod \"e8ddddf7-d988-4e7d-9727-8fd44fe67713\" (UID: \"e8ddddf7-d988-4e7d-9727-8fd44fe67713\") "
Sep 17 18:25:41.065097 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:41.065071    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8ddddf7-d988-4e7d-9727-8fd44fe67713-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e8ddddf7-d988-4e7d-9727-8fd44fe67713" (UID: "e8ddddf7-d988-4e7d-9727-8fd44fe67713"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:25:41.065163 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:41.065128    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8ddddf7-d988-4e7d-9727-8fd44fe67713-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e8ddddf7-d988-4e7d-9727-8fd44fe67713" (UID: "e8ddddf7-d988-4e7d-9727-8fd44fe67713"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:25:41.065793 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:41.065768    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8ddddf7-d988-4e7d-9727-8fd44fe67713-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e8ddddf7-d988-4e7d-9727-8fd44fe67713" (UID: "e8ddddf7-d988-4e7d-9727-8fd44fe67713"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:25:41.066971 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:41.066946    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8ddddf7-d988-4e7d-9727-8fd44fe67713-kube-api-access-7565p" (OuterVolumeSpecName: "kube-api-access-7565p") pod "e8ddddf7-d988-4e7d-9727-8fd44fe67713" (UID: "e8ddddf7-d988-4e7d-9727-8fd44fe67713"). InnerVolumeSpecName "kube-api-access-7565p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:25:41.066971 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:41.066965    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8ddddf7-d988-4e7d-9727-8fd44fe67713-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e8ddddf7-d988-4e7d-9727-8fd44fe67713" (UID: "e8ddddf7-d988-4e7d-9727-8fd44fe67713"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:25:41.166116 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:41.166059    2553 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8ddddf7-d988-4e7d-9727-8fd44fe67713-cabundle-cert\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:25:41.166116 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:41.166107    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7565p\" (UniqueName: \"kubernetes.io/projected/e8ddddf7-d988-4e7d-9727-8fd44fe67713-kube-api-access-7565p\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:25:41.166116 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:41.166118    2553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8ddddf7-d988-4e7d-9727-8fd44fe67713-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:25:41.166116 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:41.166130    2553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8ddddf7-d988-4e7d-9727-8fd44fe67713-proxy-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:25:41.166436 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:41.166141    2553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8ddddf7-d988-4e7d-9727-8fd44fe67713-kserve-provision-location\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:25:41.291207 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:41.291169    2553 generic.go:356] "Generic (PLEG): container finished" podID="e8ddddf7-d988-4e7d-9727-8fd44fe67713" containerID="3501332a2a11702fc16112d9867d2a6236f934f6368b7ebb5ebb388a05aa0552" exitCode=0
Sep 17 18:25:41.291361 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:41.291230    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fd2d4"
Sep 17 18:25:41.291361 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:41.291252    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fd2d4" event={"ID":"e8ddddf7-d988-4e7d-9727-8fd44fe67713","Type":"ContainerDied","Data":"3501332a2a11702fc16112d9867d2a6236f934f6368b7ebb5ebb388a05aa0552"}
Sep 17 18:25:41.291361 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:41.291325    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fd2d4" event={"ID":"e8ddddf7-d988-4e7d-9727-8fd44fe67713","Type":"ContainerDied","Data":"6954a831858d7a074fc5be56653d825d4c6da2ce9498a01102de6f8231ed1bd7"}
Sep 17 18:25:41.291465 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:41.291364    2553 scope.go:117] "RemoveContainer" containerID="408b475fbfed1beb0ea56c8545d69cea0498b56e4c76f45f30c27a5725d70b08"
Sep 17 18:25:41.301965 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:41.301895    2553 scope.go:117] "RemoveContainer" containerID="3501332a2a11702fc16112d9867d2a6236f934f6368b7ebb5ebb388a05aa0552"
Sep 17 18:25:41.309883 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:41.309861    2553 scope.go:117] "RemoveContainer" containerID="60c74e4cd34f7a5e735f36ff3846fa319198689fe7d5ccd0e0f0735717c03ca6"
Sep 17 18:25:41.313814 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:41.313793    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fd2d4"]
Sep 17 18:25:41.317464 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:41.317407    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-fd2d4"]
Sep 17 18:25:41.317739 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:41.317723    2553 scope.go:117] "RemoveContainer" containerID="408b475fbfed1beb0ea56c8545d69cea0498b56e4c76f45f30c27a5725d70b08"
Sep 17 18:25:41.318035 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:25:41.318016    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"408b475fbfed1beb0ea56c8545d69cea0498b56e4c76f45f30c27a5725d70b08\": container with ID starting with 408b475fbfed1beb0ea56c8545d69cea0498b56e4c76f45f30c27a5725d70b08 not found: ID does not exist" containerID="408b475fbfed1beb0ea56c8545d69cea0498b56e4c76f45f30c27a5725d70b08"
Sep 17 18:25:41.318105 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:41.318047    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"408b475fbfed1beb0ea56c8545d69cea0498b56e4c76f45f30c27a5725d70b08"} err="failed to get container status \"408b475fbfed1beb0ea56c8545d69cea0498b56e4c76f45f30c27a5725d70b08\": rpc error: code = NotFound desc = could not find container \"408b475fbfed1beb0ea56c8545d69cea0498b56e4c76f45f30c27a5725d70b08\": container with ID starting with 408b475fbfed1beb0ea56c8545d69cea0498b56e4c76f45f30c27a5725d70b08 not found: ID does not exist"
Sep 17 18:25:41.318105 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:41.318073    2553 scope.go:117] "RemoveContainer" containerID="3501332a2a11702fc16112d9867d2a6236f934f6368b7ebb5ebb388a05aa0552"
Sep 17 18:25:41.318381 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:25:41.318363    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3501332a2a11702fc16112d9867d2a6236f934f6368b7ebb5ebb388a05aa0552\": container with ID starting with 3501332a2a11702fc16112d9867d2a6236f934f6368b7ebb5ebb388a05aa0552 not found: ID does not exist" containerID="3501332a2a11702fc16112d9867d2a6236f934f6368b7ebb5ebb388a05aa0552"
Sep 17 18:25:41.318420 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:41.318387    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3501332a2a11702fc16112d9867d2a6236f934f6368b7ebb5ebb388a05aa0552"} err="failed to get container status \"3501332a2a11702fc16112d9867d2a6236f934f6368b7ebb5ebb388a05aa0552\": rpc error: code = NotFound desc = could not find container \"3501332a2a11702fc16112d9867d2a6236f934f6368b7ebb5ebb388a05aa0552\": container with ID starting with 3501332a2a11702fc16112d9867d2a6236f934f6368b7ebb5ebb388a05aa0552 not found: ID does not exist"
Sep 17 18:25:41.318420 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:41.318404    2553 scope.go:117] "RemoveContainer" containerID="60c74e4cd34f7a5e735f36ff3846fa319198689fe7d5ccd0e0f0735717c03ca6"
Sep 17 18:25:41.318674 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:25:41.318657    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60c74e4cd34f7a5e735f36ff3846fa319198689fe7d5ccd0e0f0735717c03ca6\": container with ID starting with 60c74e4cd34f7a5e735f36ff3846fa319198689fe7d5ccd0e0f0735717c03ca6 not found: ID does not exist" containerID="60c74e4cd34f7a5e735f36ff3846fa319198689fe7d5ccd0e0f0735717c03ca6"
Sep 17 18:25:41.318724 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:41.318679    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60c74e4cd34f7a5e735f36ff3846fa319198689fe7d5ccd0e0f0735717c03ca6"} err="failed to get container status \"60c74e4cd34f7a5e735f36ff3846fa319198689fe7d5ccd0e0f0735717c03ca6\": rpc error: code = NotFound desc = could not find container \"60c74e4cd34f7a5e735f36ff3846fa319198689fe7d5ccd0e0f0735717c03ca6\": container with ID starting with 60c74e4cd34f7a5e735f36ff3846fa319198689fe7d5ccd0e0f0735717c03ca6 not found: ID does not exist"
Sep 17 18:25:41.375902 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:41.375869    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8ddddf7-d988-4e7d-9727-8fd44fe67713" path="/var/lib/kubelet/pods/e8ddddf7-d988-4e7d-9727-8fd44fe67713/volumes"
Sep 17 18:25:44.242064 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:44.242026    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-l7jdv"]
Sep 17 18:25:44.242456 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:44.242379    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8ddddf7-d988-4e7d-9727-8fd44fe67713" containerName="kserve-container"
Sep 17 18:25:44.242456 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:44.242391    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8ddddf7-d988-4e7d-9727-8fd44fe67713" containerName="kserve-container"
Sep 17 18:25:44.242456 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:44.242428    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8ddddf7-d988-4e7d-9727-8fd44fe67713" containerName="kube-rbac-proxy"
Sep 17 18:25:44.242456 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:44.242434    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8ddddf7-d988-4e7d-9727-8fd44fe67713" containerName="kube-rbac-proxy"
Sep 17 18:25:44.242456 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:44.242443    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8ddddf7-d988-4e7d-9727-8fd44fe67713" containerName="storage-initializer"
Sep 17 18:25:44.242456 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:44.242450    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8ddddf7-d988-4e7d-9727-8fd44fe67713" containerName="storage-initializer"
Sep 17 18:25:44.242706 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:44.242521    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8ddddf7-d988-4e7d-9727-8fd44fe67713" containerName="kube-rbac-proxy"
Sep 17 18:25:44.242706 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:44.242529    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8ddddf7-d988-4e7d-9727-8fd44fe67713" containerName="kserve-container"
Sep 17 18:25:44.246400 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:44.246377    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-l7jdv"
Sep 17 18:25:44.249238 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:44.249209    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-predictor-serving-cert\""
Sep 17 18:25:44.249371 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:44.249221    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-kube-rbac-proxy-sar-config\""
Sep 17 18:25:44.249371 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:44.249267    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:25:44.249371 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:44.249224    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:25:44.249558 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:44.249375    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-7qcmd\""
Sep 17 18:25:44.249558 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:44.249379    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:25:44.250265 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:44.250240    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:25:44.254746 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:44.254718    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-l7jdv"]
Sep 17 18:25:44.293222 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:44.293193    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-l7jdv\" (UID: \"5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-l7jdv"
Sep 17 18:25:44.293439 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:44.293233    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fgz9\" (UniqueName: \"kubernetes.io/projected/5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f-kube-api-access-6fgz9\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-l7jdv\" (UID: \"5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-l7jdv"
Sep 17 18:25:44.293439 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:44.293262    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-l7jdv\" (UID: \"5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-l7jdv"
Sep 17 18:25:44.293439 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:44.293309    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-l7jdv\" (UID: \"5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-l7jdv"
Sep 17 18:25:44.293439 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:44.293332    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-l7jdv\" (UID: \"5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-l7jdv"
Sep 17 18:25:44.394000 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:44.393963    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-l7jdv\" (UID: \"5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-l7jdv"
Sep 17 18:25:44.394262 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:44.394005    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fgz9\" (UniqueName: \"kubernetes.io/projected/5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f-kube-api-access-6fgz9\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-l7jdv\" (UID: \"5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-l7jdv"
Sep 17 18:25:44.394262 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:44.394033    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-l7jdv\" (UID: \"5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-l7jdv"
Sep 17 18:25:44.394262 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:44.394063    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-l7jdv\" (UID: \"5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-l7jdv"
Sep 17 18:25:44.394262 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:44.394083    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-l7jdv\" (UID: \"5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-l7jdv"
Sep 17 18:25:44.394262 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:25:44.394130    2553 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-serving-cert: secret "isvc-autogluon-ts-v1-noslash-predictor-serving-cert" not found
Sep 17 18:25:44.394262 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:25:44.394209    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f-proxy-tls podName:5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f nodeName:}" failed. No retries permitted until 2026-09-17 18:25:44.894188395 +0000 UTC m=+1739.590279060 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f-proxy-tls") pod "isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-l7jdv" (UID: "5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f") : secret "isvc-autogluon-ts-v1-noslash-predictor-serving-cert" not found
Sep 17 18:25:44.394689 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:44.394572    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-l7jdv\" (UID: \"5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-l7jdv"
Sep 17 18:25:44.394839 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:44.394819    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-l7jdv\" (UID: \"5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-l7jdv"
Sep 17 18:25:44.394961 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:44.394943    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-l7jdv\" (UID: \"5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-l7jdv"
Sep 17 18:25:44.402710 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:44.402687    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fgz9\" (UniqueName: \"kubernetes.io/projected/5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f-kube-api-access-6fgz9\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-l7jdv\" (UID: \"5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-l7jdv"
Sep 17 18:25:44.898210 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:44.898158    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-l7jdv\" (UID: \"5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-l7jdv"
Sep 17 18:25:44.900605 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:44.900574    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-l7jdv\" (UID: \"5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-l7jdv"
Sep 17 18:25:45.157193 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:45.157102    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-l7jdv"
Sep 17 18:25:45.427315 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:45.427279    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-l7jdv"]
Sep 17 18:25:45.429547 ip-10-0-137-95 kubenswrapper[2553]: W0917 18:25:45.429519    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e8a45cd_0d2f_4bb8_b62e_4176fb75d23f.slice/crio-caa147a68b3657407757294dbba75b8c23f247a2630931f571d3743e7167dc74 WatchSource:0}: Error finding container caa147a68b3657407757294dbba75b8c23f247a2630931f571d3743e7167dc74: Status 404 returned error can't find the container with id caa147a68b3657407757294dbba75b8c23f247a2630931f571d3743e7167dc74
Sep 17 18:25:46.316197 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:46.316164    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-l7jdv" event={"ID":"5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f","Type":"ContainerStarted","Data":"fab27b7899715d8a7ac0634daf221de85ba69581dc3885bc3beac819a041bfc5"}
Sep 17 18:25:46.316368 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:46.316223    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-l7jdv" event={"ID":"5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f","Type":"ContainerStarted","Data":"caa147a68b3657407757294dbba75b8c23f247a2630931f571d3743e7167dc74"}
Sep 17 18:25:50.333999 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:50.333961    2553 generic.go:356] "Generic (PLEG): container finished" podID="5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f" containerID="fab27b7899715d8a7ac0634daf221de85ba69581dc3885bc3beac819a041bfc5" exitCode=0
Sep 17 18:25:50.334385 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:50.334039    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-l7jdv" event={"ID":"5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f","Type":"ContainerDied","Data":"fab27b7899715d8a7ac0634daf221de85ba69581dc3885bc3beac819a041bfc5"}
Sep 17 18:25:51.340987 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:51.340953    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-l7jdv" event={"ID":"5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f","Type":"ContainerStarted","Data":"a8c6f38bef9e6440f1768266434849214e3f7a93c764615e17cf23e52cf37ee8"}
Sep 17 18:25:51.341366 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:51.341018    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-l7jdv" event={"ID":"5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f","Type":"ContainerStarted","Data":"f1888cf8172a02dfd5b9f02b199df3dc77643572162677688c019f3cc773c97c"}
Sep 17 18:25:51.341366 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:51.341229    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-l7jdv"
Sep 17 18:25:51.362411 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:51.362356    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-l7jdv" podStartSLOduration=2.746579358 podStartE2EDuration="7.362339755s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.615760397s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:25:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:25:51.359993276 +0000 UTC m=+1746.056083958" watchObservedRunningTime="2026-09-17 18:25:51.362339755 +0000 UTC m=+1746.058430438"
Sep 17 18:25:52.345518 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:52.345478    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-l7jdv"
Sep 17 18:25:52.346962 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:52.346933    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-l7jdv" podUID="5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 18:25:53.350799 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:53.350758    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-l7jdv" podUID="5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 18:25:58.355848 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:58.355813    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-l7jdv"
Sep 17 18:25:58.356517 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:25:58.356491    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-l7jdv" podUID="5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 18:26:08.358055 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:08.358021    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-l7jdv"
Sep 17 18:26:19.624528 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:19.624487    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-l7jdv"]
Sep 17 18:26:19.625112 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:19.624794    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-l7jdv" podUID="5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f" containerName="kube-rbac-proxy" containerID="cri-o://a8c6f38bef9e6440f1768266434849214e3f7a93c764615e17cf23e52cf37ee8" gracePeriod=30
Sep 17 18:26:19.625112 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:19.624794    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-l7jdv" podUID="5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f" containerName="kserve-container" containerID="cri-o://f1888cf8172a02dfd5b9f02b199df3dc77643572162677688c019f3cc773c97c" gracePeriod=30
Sep 17 18:26:20.479054 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:20.479019    2553 generic.go:356] "Generic (PLEG): container finished" podID="5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f" containerID="a8c6f38bef9e6440f1768266434849214e3f7a93c764615e17cf23e52cf37ee8" exitCode=2
Sep 17 18:26:20.479263 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:20.479095    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-l7jdv" event={"ID":"5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f","Type":"ContainerDied","Data":"a8c6f38bef9e6440f1768266434849214e3f7a93c764615e17cf23e52cf37ee8"}
Sep 17 18:26:21.064378 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:21.064352    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-l7jdv"
Sep 17 18:26:21.104012 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:21.103983    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f-kserve-provision-location\") pod \"5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f\" (UID: \"5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f\") "
Sep 17 18:26:21.104182 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:21.104021    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6fgz9\" (UniqueName: \"kubernetes.io/projected/5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f-kube-api-access-6fgz9\") pod \"5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f\" (UID: \"5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f\") "
Sep 17 18:26:21.104182 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:21.104047    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f-proxy-tls\") pod \"5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f\" (UID: \"5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f\") "
Sep 17 18:26:21.104182 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:21.104085    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f-cabundle-cert\") pod \"5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f\" (UID: \"5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f\") "
Sep 17 18:26:21.104182 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:21.104122    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f-kube-rbac-proxy-sar-config\") pod \"5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f\" (UID: \"5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f\") "
Sep 17 18:26:21.104491 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:21.104466    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f" (UID: "5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:26:21.104612 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:21.104549    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f" (UID: "5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:26:21.105162 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:21.105142    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f" (UID: "5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:26:21.106687 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:21.106660    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f" (UID: "5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:26:21.106782 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:21.106701    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f-kube-api-access-6fgz9" (OuterVolumeSpecName: "kube-api-access-6fgz9") pod "5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f" (UID: "5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f"). InnerVolumeSpecName "kube-api-access-6fgz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:26:21.204818 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:21.204722    2553 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f-cabundle-cert\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:26:21.204818 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:21.204764    2553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:26:21.204818 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:21.204775    2553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f-kserve-provision-location\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:26:21.204818 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:21.204784    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fgz9\" (UniqueName: \"kubernetes.io/projected/5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f-kube-api-access-6fgz9\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:26:21.204818 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:21.204794    2553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f-proxy-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:26:21.485288 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:21.485199    2553 generic.go:356] "Generic (PLEG): container finished" podID="5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f" containerID="f1888cf8172a02dfd5b9f02b199df3dc77643572162677688c019f3cc773c97c" exitCode=0
Sep 17 18:26:21.485288 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:21.485244    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-l7jdv" event={"ID":"5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f","Type":"ContainerDied","Data":"f1888cf8172a02dfd5b9f02b199df3dc77643572162677688c019f3cc773c97c"}
Sep 17 18:26:21.485288 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:21.485262    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-l7jdv"
Sep 17 18:26:21.485527 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:21.485305    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-l7jdv" event={"ID":"5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f","Type":"ContainerDied","Data":"caa147a68b3657407757294dbba75b8c23f247a2630931f571d3743e7167dc74"}
Sep 17 18:26:21.485527 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:21.485352    2553 scope.go:117] "RemoveContainer" containerID="a8c6f38bef9e6440f1768266434849214e3f7a93c764615e17cf23e52cf37ee8"
Sep 17 18:26:21.493818 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:21.493800    2553 scope.go:117] "RemoveContainer" containerID="f1888cf8172a02dfd5b9f02b199df3dc77643572162677688c019f3cc773c97c"
Sep 17 18:26:21.501609 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:21.501574    2553 scope.go:117] "RemoveContainer" containerID="fab27b7899715d8a7ac0634daf221de85ba69581dc3885bc3beac819a041bfc5"
Sep 17 18:26:21.503430 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:21.503403    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-l7jdv"]
Sep 17 18:26:21.507209 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:21.507186    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-l7jdv"]
Sep 17 18:26:21.509956 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:21.509943    2553 scope.go:117] "RemoveContainer" containerID="a8c6f38bef9e6440f1768266434849214e3f7a93c764615e17cf23e52cf37ee8"
Sep 17 18:26:21.510244 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:26:21.510227    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8c6f38bef9e6440f1768266434849214e3f7a93c764615e17cf23e52cf37ee8\": container with ID starting with a8c6f38bef9e6440f1768266434849214e3f7a93c764615e17cf23e52cf37ee8 not found: ID does not exist" containerID="a8c6f38bef9e6440f1768266434849214e3f7a93c764615e17cf23e52cf37ee8"
Sep 17 18:26:21.510284 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:21.510253    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8c6f38bef9e6440f1768266434849214e3f7a93c764615e17cf23e52cf37ee8"} err="failed to get container status \"a8c6f38bef9e6440f1768266434849214e3f7a93c764615e17cf23e52cf37ee8\": rpc error: code = NotFound desc = could not find container \"a8c6f38bef9e6440f1768266434849214e3f7a93c764615e17cf23e52cf37ee8\": container with ID starting with a8c6f38bef9e6440f1768266434849214e3f7a93c764615e17cf23e52cf37ee8 not found: ID does not exist"
Sep 17 18:26:21.510284 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:21.510271    2553 scope.go:117] "RemoveContainer" containerID="f1888cf8172a02dfd5b9f02b199df3dc77643572162677688c019f3cc773c97c"
Sep 17 18:26:21.510552 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:26:21.510532    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1888cf8172a02dfd5b9f02b199df3dc77643572162677688c019f3cc773c97c\": container with ID starting with f1888cf8172a02dfd5b9f02b199df3dc77643572162677688c019f3cc773c97c not found: ID does not exist" containerID="f1888cf8172a02dfd5b9f02b199df3dc77643572162677688c019f3cc773c97c"
Sep 17 18:26:21.510612 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:21.510559    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1888cf8172a02dfd5b9f02b199df3dc77643572162677688c019f3cc773c97c"} err="failed to get container status \"f1888cf8172a02dfd5b9f02b199df3dc77643572162677688c019f3cc773c97c\": rpc error: code = NotFound desc = could not find container \"f1888cf8172a02dfd5b9f02b199df3dc77643572162677688c019f3cc773c97c\": container with ID starting with f1888cf8172a02dfd5b9f02b199df3dc77643572162677688c019f3cc773c97c not found: ID does not exist"
Sep 17 18:26:21.510612 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:21.510575    2553 scope.go:117] "RemoveContainer" containerID="fab27b7899715d8a7ac0634daf221de85ba69581dc3885bc3beac819a041bfc5"
Sep 17 18:26:21.510839 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:26:21.510824    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fab27b7899715d8a7ac0634daf221de85ba69581dc3885bc3beac819a041bfc5\": container with ID starting with fab27b7899715d8a7ac0634daf221de85ba69581dc3885bc3beac819a041bfc5 not found: ID does not exist" containerID="fab27b7899715d8a7ac0634daf221de85ba69581dc3885bc3beac819a041bfc5"
Sep 17 18:26:21.510881 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:21.510846    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fab27b7899715d8a7ac0634daf221de85ba69581dc3885bc3beac819a041bfc5"} err="failed to get container status \"fab27b7899715d8a7ac0634daf221de85ba69581dc3885bc3beac819a041bfc5\": rpc error: code = NotFound desc = could not find container \"fab27b7899715d8a7ac0634daf221de85ba69581dc3885bc3beac819a041bfc5\": container with ID starting with fab27b7899715d8a7ac0634daf221de85ba69581dc3885bc3beac819a041bfc5 not found: ID does not exist"
Sep 17 18:26:23.375912 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:23.375880    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f" path="/var/lib/kubelet/pods/5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f/volumes"
Sep 17 18:26:24.924491 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:24.924452    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-dz4nh"]
Sep 17 18:26:24.924979 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:24.924775    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f" containerName="kserve-container"
Sep 17 18:26:24.924979 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:24.924787    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f" containerName="kserve-container"
Sep 17 18:26:24.924979 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:24.924804    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f" containerName="storage-initializer"
Sep 17 18:26:24.924979 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:24.924809    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f" containerName="storage-initializer"
Sep 17 18:26:24.924979 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:24.924824    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f" containerName="kube-rbac-proxy"
Sep 17 18:26:24.924979 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:24.924830    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f" containerName="kube-rbac-proxy"
Sep 17 18:26:24.924979 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:24.924877    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f" containerName="kube-rbac-proxy"
Sep 17 18:26:24.924979 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:24.924887    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e8a45cd-0d2f-4bb8-b62e-4176fb75d23f" containerName="kserve-container"
Sep 17 18:26:24.928046 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:24.928029    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-dz4nh"
Sep 17 18:26:24.930645 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:24.930588    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:26:24.930758 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:24.930661    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f3497fd9-9759-4ba8-ba5f-75e695b5385f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5956b77f85-dz4nh\" (UID: \"f3497fd9-9759-4ba8-ba5f-75e695b5385f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-dz4nh"
Sep 17 18:26:24.930758 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:24.930726    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml2hd\" (UniqueName: \"kubernetes.io/projected/f3497fd9-9759-4ba8-ba5f-75e695b5385f-kube-api-access-ml2hd\") pod \"isvc-canary-create-predictor-5956b77f85-dz4nh\" (UID: \"f3497fd9-9759-4ba8-ba5f-75e695b5385f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-dz4nh"
Sep 17 18:26:24.930758 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:24.930737    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 17 18:26:24.930901 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:24.930770    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3497fd9-9759-4ba8-ba5f-75e695b5385f-kserve-provision-location\") pod \"isvc-canary-create-predictor-5956b77f85-dz4nh\" (UID: \"f3497fd9-9759-4ba8-ba5f-75e695b5385f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-dz4nh"
Sep 17 18:26:24.930901 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:24.930827    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f3497fd9-9759-4ba8-ba5f-75e695b5385f-cabundle-cert\") pod \"isvc-canary-create-predictor-5956b77f85-dz4nh\" (UID: \"f3497fd9-9759-4ba8-ba5f-75e695b5385f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-dz4nh"
Sep 17 18:26:24.930901 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:24.930849    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3497fd9-9759-4ba8-ba5f-75e695b5385f-proxy-tls\") pod \"isvc-canary-create-predictor-5956b77f85-dz4nh\" (UID: \"f3497fd9-9759-4ba8-ba5f-75e695b5385f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-dz4nh"
Sep 17 18:26:24.931557 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:24.931541    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-7qcmd\""
Sep 17 18:26:24.931662 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:24.931555    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 17 18:26:24.931662 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:24.931544    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:26:24.931662 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:24.931573    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:26:24.931662 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:24.931560    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:26:24.939177 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:24.939156    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-dz4nh"]
Sep 17 18:26:25.031763 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:25.031716    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f3497fd9-9759-4ba8-ba5f-75e695b5385f-cabundle-cert\") pod \"isvc-canary-create-predictor-5956b77f85-dz4nh\" (UID: \"f3497fd9-9759-4ba8-ba5f-75e695b5385f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-dz4nh"
Sep 17 18:26:25.031763 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:25.031768    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3497fd9-9759-4ba8-ba5f-75e695b5385f-proxy-tls\") pod \"isvc-canary-create-predictor-5956b77f85-dz4nh\" (UID: \"f3497fd9-9759-4ba8-ba5f-75e695b5385f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-dz4nh"
Sep 17 18:26:25.031994 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:25.031832    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f3497fd9-9759-4ba8-ba5f-75e695b5385f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5956b77f85-dz4nh\" (UID: \"f3497fd9-9759-4ba8-ba5f-75e695b5385f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-dz4nh"
Sep 17 18:26:25.031994 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:25.031881    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml2hd\" (UniqueName: \"kubernetes.io/projected/f3497fd9-9759-4ba8-ba5f-75e695b5385f-kube-api-access-ml2hd\") pod \"isvc-canary-create-predictor-5956b77f85-dz4nh\" (UID: \"f3497fd9-9759-4ba8-ba5f-75e695b5385f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-dz4nh"
Sep 17 18:26:25.031994 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:25.031915    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3497fd9-9759-4ba8-ba5f-75e695b5385f-kserve-provision-location\") pod \"isvc-canary-create-predictor-5956b77f85-dz4nh\" (UID: \"f3497fd9-9759-4ba8-ba5f-75e695b5385f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-dz4nh"
Sep 17 18:26:25.032328 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:25.032298    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3497fd9-9759-4ba8-ba5f-75e695b5385f-kserve-provision-location\") pod \"isvc-canary-create-predictor-5956b77f85-dz4nh\" (UID: \"f3497fd9-9759-4ba8-ba5f-75e695b5385f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-dz4nh"
Sep 17 18:26:25.032486 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:25.032467    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f3497fd9-9759-4ba8-ba5f-75e695b5385f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5956b77f85-dz4nh\" (UID: \"f3497fd9-9759-4ba8-ba5f-75e695b5385f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-dz4nh"
Sep 17 18:26:25.032528 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:25.032472    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f3497fd9-9759-4ba8-ba5f-75e695b5385f-cabundle-cert\") pod \"isvc-canary-create-predictor-5956b77f85-dz4nh\" (UID: \"f3497fd9-9759-4ba8-ba5f-75e695b5385f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-dz4nh"
Sep 17 18:26:25.034332 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:25.034313    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3497fd9-9759-4ba8-ba5f-75e695b5385f-proxy-tls\") pod \"isvc-canary-create-predictor-5956b77f85-dz4nh\" (UID: \"f3497fd9-9759-4ba8-ba5f-75e695b5385f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-dz4nh"
Sep 17 18:26:25.042758 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:25.042730    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml2hd\" (UniqueName: \"kubernetes.io/projected/f3497fd9-9759-4ba8-ba5f-75e695b5385f-kube-api-access-ml2hd\") pod \"isvc-canary-create-predictor-5956b77f85-dz4nh\" (UID: \"f3497fd9-9759-4ba8-ba5f-75e695b5385f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-dz4nh"
Sep 17 18:26:25.237981 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:25.237878    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-dz4nh"
Sep 17 18:26:25.510968 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:25.510946    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-dz4nh"]
Sep 17 18:26:25.513167 ip-10-0-137-95 kubenswrapper[2553]: W0917 18:26:25.513138    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3497fd9_9759_4ba8_ba5f_75e695b5385f.slice/crio-2c4e0c7cfeb7eff719d7a633e0c30efc7fcdb42f47f3b0cdcf24fa38cefadf6f WatchSource:0}: Error finding container 2c4e0c7cfeb7eff719d7a633e0c30efc7fcdb42f47f3b0cdcf24fa38cefadf6f: Status 404 returned error can't find the container with id 2c4e0c7cfeb7eff719d7a633e0c30efc7fcdb42f47f3b0cdcf24fa38cefadf6f
Sep 17 18:26:26.511355 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:26.511318    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-dz4nh" event={"ID":"f3497fd9-9759-4ba8-ba5f-75e695b5385f","Type":"ContainerStarted","Data":"c0032de4b5eb0980dea3fe95b5c66c09fcc784d1aaff27b72369b61462ddc8af"}
Sep 17 18:26:26.511845 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:26.511376    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-dz4nh" event={"ID":"f3497fd9-9759-4ba8-ba5f-75e695b5385f","Type":"ContainerStarted","Data":"2c4e0c7cfeb7eff719d7a633e0c30efc7fcdb42f47f3b0cdcf24fa38cefadf6f"}
Sep 17 18:26:29.525431 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:29.525400    2553 generic.go:356] "Generic (PLEG): container finished" podID="f3497fd9-9759-4ba8-ba5f-75e695b5385f" containerID="c0032de4b5eb0980dea3fe95b5c66c09fcc784d1aaff27b72369b61462ddc8af" exitCode=0
Sep 17 18:26:29.525834 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:29.525454    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-dz4nh" event={"ID":"f3497fd9-9759-4ba8-ba5f-75e695b5385f","Type":"ContainerDied","Data":"c0032de4b5eb0980dea3fe95b5c66c09fcc784d1aaff27b72369b61462ddc8af"}
Sep 17 18:26:29.526589 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:29.526575    2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:26:43.603604 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:43.603560    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-dz4nh" event={"ID":"f3497fd9-9759-4ba8-ba5f-75e695b5385f","Type":"ContainerStarted","Data":"82fb385e1cc14b85858f136294e7647d944244756b350e986231a6b68f27c70e"}
Sep 17 18:26:43.604021 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:43.603654    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-dz4nh" event={"ID":"f3497fd9-9759-4ba8-ba5f-75e695b5385f","Type":"ContainerStarted","Data":"540d26bb26073449bfee0206c4181bf4a12aa68a32bb50eaa51a3d12be760d72"}
Sep 17 18:26:43.604021 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:43.603996    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-dz4nh"
Sep 17 18:26:43.624110 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:43.624055    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-dz4nh" podStartSLOduration=2.360193784 podStartE2EDuration="19.624037499s" totalImagesPullingTime="13.445855024s" totalInitContainerRuntime="3.817988691s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:26:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:26:43.623027567 +0000 UTC m=+1798.319118249" watchObservedRunningTime="2026-09-17 18:26:43.624037499 +0000 UTC m=+1798.320128184"
Sep 17 18:26:44.608380 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:44.608348    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-dz4nh"
Sep 17 18:26:44.609802 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:44.609767    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-dz4nh" podUID="f3497fd9-9759-4ba8-ba5f-75e695b5385f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 18:26:45.613044 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:45.612997    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-dz4nh" podUID="f3497fd9-9759-4ba8-ba5f-75e695b5385f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 18:26:47.399204 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:47.399167    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-btmqz_622ee5be-fb50-42b6-ae23-42900b7c2d98/ovn-acl-logging/0.log"
Sep 17 18:26:47.410935 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:47.410914    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-btmqz_622ee5be-fb50-42b6-ae23-42900b7c2d98/ovn-acl-logging/0.log"
Sep 17 18:26:50.617661 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:50.617615    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-dz4nh"
Sep 17 18:26:50.618184 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:26:50.618157    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-dz4nh" podUID="f3497fd9-9759-4ba8-ba5f-75e695b5385f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 18:27:00.618786 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:27:00.618739    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-dz4nh" podUID="f3497fd9-9759-4ba8-ba5f-75e695b5385f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 18:27:00.647196 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:27:00.647162    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cbb795f56-wn4dw"]
Sep 17 18:27:00.650540 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:27:00.650523    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cbb795f56-wn4dw"
Sep 17 18:27:00.653349 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:27:00.653324    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-kube-rbac-proxy-sar-config\""
Sep 17 18:27:00.653551 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:27:00.653534    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-predictor-serving-cert\""
Sep 17 18:27:00.660919 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:27:00.660895    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cbb795f56-wn4dw"]
Sep 17 18:27:00.738363 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:27:00.738323    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a41f41b-30ae-4e33-99e9-0b4099cb4372-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-7cbb795f56-wn4dw\" (UID: \"1a41f41b-30ae-4e33-99e9-0b4099cb4372\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cbb795f56-wn4dw"
Sep 17 18:27:00.738363 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:27:00.738364    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a41f41b-30ae-4e33-99e9-0b4099cb4372-proxy-tls\") pod \"isvc-sklearn-v2-predictor-7cbb795f56-wn4dw\" (UID: \"1a41f41b-30ae-4e33-99e9-0b4099cb4372\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cbb795f56-wn4dw"
Sep 17 18:27:00.738590 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:27:00.738387    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a41f41b-30ae-4e33-99e9-0b4099cb4372-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-7cbb795f56-wn4dw\" (UID: \"1a41f41b-30ae-4e33-99e9-0b4099cb4372\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cbb795f56-wn4dw"
Sep 17 18:27:00.738590 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:27:00.738527    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a41f41b-30ae-4e33-99e9-0b4099cb4372-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-7cbb795f56-wn4dw\" (UID: \"1a41f41b-30ae-4e33-99e9-0b4099cb4372\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cbb795f56-wn4dw"
Sep 17 18:27:00.738590 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:27:00.738559    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2gmv\" (UniqueName: \"kubernetes.io/projected/1a41f41b-30ae-4e33-99e9-0b4099cb4372-kube-api-access-d2gmv\") pod \"isvc-sklearn-v2-predictor-7cbb795f56-wn4dw\" (UID: \"1a41f41b-30ae-4e33-99e9-0b4099cb4372\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cbb795f56-wn4dw"
Sep 17 18:27:00.839736 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:27:00.839698    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a41f41b-30ae-4e33-99e9-0b4099cb4372-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-7cbb795f56-wn4dw\" (UID: \"1a41f41b-30ae-4e33-99e9-0b4099cb4372\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cbb795f56-wn4dw"
Sep 17 18:27:00.839939 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:27:00.839745    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a41f41b-30ae-4e33-99e9-0b4099cb4372-proxy-tls\") pod \"isvc-sklearn-v2-predictor-7cbb795f56-wn4dw\" (UID: \"1a41f41b-30ae-4e33-99e9-0b4099cb4372\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cbb795f56-wn4dw"
Sep 17 18:27:00.839939 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:27:00.839777    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a41f41b-30ae-4e33-99e9-0b4099cb4372-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-7cbb795f56-wn4dw\" (UID: \"1a41f41b-30ae-4e33-99e9-0b4099cb4372\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cbb795f56-wn4dw"
Sep 17 18:27:00.839939 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:27:00.839873    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a41f41b-30ae-4e33-99e9-0b4099cb4372-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-7cbb795f56-wn4dw\" (UID: \"1a41f41b-30ae-4e33-99e9-0b4099cb4372\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cbb795f56-wn4dw"
Sep 17 18:27:00.839939 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:27:00.839899    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2gmv\" (UniqueName: \"kubernetes.io/projected/1a41f41b-30ae-4e33-99e9-0b4099cb4372-kube-api-access-d2gmv\") pod \"isvc-sklearn-v2-predictor-7cbb795f56-wn4dw\" (UID: \"1a41f41b-30ae-4e33-99e9-0b4099cb4372\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cbb795f56-wn4dw"
Sep 17 18:27:00.840272 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:27:00.840248    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a41f41b-30ae-4e33-99e9-0b4099cb4372-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-7cbb795f56-wn4dw\" (UID: \"1a41f41b-30ae-4e33-99e9-0b4099cb4372\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cbb795f56-wn4dw"
Sep 17 18:27:00.840419 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:27:00.840400    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a41f41b-30ae-4e33-99e9-0b4099cb4372-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-7cbb795f56-wn4dw\" (UID: \"1a41f41b-30ae-4e33-99e9-0b4099cb4372\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cbb795f56-wn4dw"
Sep 17 18:27:00.840484 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:27:00.840467    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a41f41b-30ae-4e33-99e9-0b4099cb4372-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-7cbb795f56-wn4dw\" (UID: \"1a41f41b-30ae-4e33-99e9-0b4099cb4372\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cbb795f56-wn4dw"
Sep 17 18:27:00.842360 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:27:00.842338    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a41f41b-30ae-4e33-99e9-0b4099cb4372-proxy-tls\") pod \"isvc-sklearn-v2-predictor-7cbb795f56-wn4dw\" (UID: \"1a41f41b-30ae-4e33-99e9-0b4099cb4372\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cbb795f56-wn4dw"
Sep 17 18:27:00.848743 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:27:00.848721    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2gmv\" (UniqueName: \"kubernetes.io/projected/1a41f41b-30ae-4e33-99e9-0b4099cb4372-kube-api-access-d2gmv\") pod \"isvc-sklearn-v2-predictor-7cbb795f56-wn4dw\" (UID: \"1a41f41b-30ae-4e33-99e9-0b4099cb4372\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cbb795f56-wn4dw"
Sep 17 18:27:00.959766 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:27:00.959730    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cbb795f56-wn4dw"
Sep 17 18:27:01.230504 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:27:01.230426    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cbb795f56-wn4dw"]
Sep 17 18:27:01.234031 ip-10-0-137-95 kubenswrapper[2553]: W0917 18:27:01.234002    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a41f41b_30ae_4e33_99e9_0b4099cb4372.slice/crio-8538a7908ee580245af5d5ed2ccc8650d8ba61e84fef715d2b3a277c9e3cda67 WatchSource:0}: Error finding container 8538a7908ee580245af5d5ed2ccc8650d8ba61e84fef715d2b3a277c9e3cda67: Status 404 returned error can't find the container with id 8538a7908ee580245af5d5ed2ccc8650d8ba61e84fef715d2b3a277c9e3cda67
Sep 17 18:27:01.685715 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:27:01.685672    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cbb795f56-wn4dw" event={"ID":"1a41f41b-30ae-4e33-99e9-0b4099cb4372","Type":"ContainerStarted","Data":"c5c48523d541658eb84bbb4a17050be2aa537f5d429d850ab8e3a27765f00b52"}
Sep 17 18:27:01.686079 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:27:01.685745    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cbb795f56-wn4dw" event={"ID":"1a41f41b-30ae-4e33-99e9-0b4099cb4372","Type":"ContainerStarted","Data":"8538a7908ee580245af5d5ed2ccc8650d8ba61e84fef715d2b3a277c9e3cda67"}
Sep 17 18:27:05.706009 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:27:05.705974    2553 generic.go:356] "Generic (PLEG): container finished" podID="1a41f41b-30ae-4e33-99e9-0b4099cb4372" containerID="c5c48523d541658eb84bbb4a17050be2aa537f5d429d850ab8e3a27765f00b52" exitCode=0
Sep 17 18:27:05.706386 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:27:05.706045    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cbb795f56-wn4dw" event={"ID":"1a41f41b-30ae-4e33-99e9-0b4099cb4372","Type":"ContainerDied","Data":"c5c48523d541658eb84bbb4a17050be2aa537f5d429d850ab8e3a27765f00b52"}
Sep 17 18:27:06.713637 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:27:06.713580    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cbb795f56-wn4dw" event={"ID":"1a41f41b-30ae-4e33-99e9-0b4099cb4372","Type":"ContainerStarted","Data":"204d345a257744e63f5a869e1525c6283866c881da321796783f421a33743b1e"}
Sep 17 18:27:06.714044 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:27:06.713692    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cbb795f56-wn4dw" event={"ID":"1a41f41b-30ae-4e33-99e9-0b4099cb4372","Type":"ContainerStarted","Data":"63ea000f1565b52fc1a939f563c02e4212b752d7f6b2a105713e42a0a26629a3"}
Sep 17 18:27:06.714044 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:27:06.713978    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cbb795f56-wn4dw"
Sep 17 18:27:06.714160 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:27:06.714136    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cbb795f56-wn4dw"
Sep 17 18:27:06.715367 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:27:06.715343    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cbb795f56-wn4dw" podUID="1a41f41b-30ae-4e33-99e9-0b4099cb4372" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 17 18:27:06.735166 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:27:06.735115    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cbb795f56-wn4dw" podStartSLOduration=3.122924991 podStartE2EDuration="6.735100267s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.612175276s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:27:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:27:06.733684292 +0000 UTC m=+1821.429774975" watchObservedRunningTime="2026-09-17 18:27:06.735100267 +0000 UTC m=+1821.431190950"
Sep 17 18:27:07.717827 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:27:07.717783    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cbb795f56-wn4dw" podUID="1a41f41b-30ae-4e33-99e9-0b4099cb4372" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 17 18:27:10.618434 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:27:10.618387    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-dz4nh" podUID="f3497fd9-9759-4ba8-ba5f-75e695b5385f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 18:27:12.722282 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:27:12.722252    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cbb795f56-wn4dw"
Sep 17 18:27:12.722862 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:27:12.722799    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cbb795f56-wn4dw" podUID="1a41f41b-30ae-4e33-99e9-0b4099cb4372" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 17 18:27:20.618188 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:27:20.618141    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-dz4nh" podUID="f3497fd9-9759-4ba8-ba5f-75e695b5385f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 18:27:22.723675 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:27:22.723638    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cbb795f56-wn4dw" podUID="1a41f41b-30ae-4e33-99e9-0b4099cb4372" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 17 18:27:30.618562 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:27:30.618519    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-dz4nh" podUID="f3497fd9-9759-4ba8-ba5f-75e695b5385f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 18:27:32.723234 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:27:32.723187    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cbb795f56-wn4dw" podUID="1a41f41b-30ae-4e33-99e9-0b4099cb4372" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 17 18:27:40.618868 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:27:40.618828    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-dz4nh" podUID="f3497fd9-9759-4ba8-ba5f-75e695b5385f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 18:27:42.722701 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:27:42.722663    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cbb795f56-wn4dw" podUID="1a41f41b-30ae-4e33-99e9-0b4099cb4372" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 17 18:27:47.513248 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:27:47.513208    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-dz4nh"]
Sep 17 18:27:47.513792 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:27:47.513524    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-dz4nh" podUID="f3497fd9-9759-4ba8-ba5f-75e695b5385f" containerName="kserve-container" containerID="cri-o://540d26bb26073449bfee0206c4181bf4a12aa68a32bb50eaa51a3d12be760d72" gracePeriod=30
Sep 17 18:27:47.513792 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:27:47.513567    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-dz4nh" podUID="f3497fd9-9759-4ba8-ba5f-75e695b5385f" containerName="kube-rbac-proxy" containerID="cri-o://82fb385e1cc14b85858f136294e7647d944244756b350e986231a6b68f27c70e" gracePeriod=30
Sep 17 18:27:47.513792 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:27:47.513732    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-dz4nh" podUID="f3497fd9-9759-4ba8-ba5f-75e695b5385f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 18:27:47.900383 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:27:47.900352    2553 generic.go:356] "Generic (PLEG): container finished" podID="f3497fd9-9759-4ba8-ba5f-75e695b5385f" containerID="82fb385e1cc14b85858f136294e7647d944244756b350e986231a6b68f27c70e" exitCode=2
Sep 17 18:27:47.900564 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:27:47.900421    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-dz4nh" event={"ID":"f3497fd9-9759-4ba8-ba5f-75e695b5385f","Type":"ContainerDied","Data":"82fb385e1cc14b85858f136294e7647d944244756b350e986231a6b68f27c70e"}
Sep 17 18:27:50.613212 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:27:50.613169    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-dz4nh" podUID="f3497fd9-9759-4ba8-ba5f-75e695b5385f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Sep 17 18:27:52.723313 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:27:52.723272    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cbb795f56-wn4dw" podUID="1a41f41b-30ae-4e33-99e9-0b4099cb4372" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 17 18:27:55.613518 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:27:55.613480    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-dz4nh" podUID="f3497fd9-9759-4ba8-ba5f-75e695b5385f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Sep 17 18:27:57.515398 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:27:57.515366    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-dz4nh"
Sep 17 18:28:00.154782 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:00.154697    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-km26d"]
Sep 17 18:28:00.160384 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:00.160351    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-km26d"
Sep 17 18:28:00.163364 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:00.163340    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 17 18:28:00.163495 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:00.163358    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 17 18:28:00.168666 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:00.168645    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-km26d"]
Sep 17 18:28:00.248852 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:00.248813    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df9aaec5-4754-4987-a36a-5753010073bc-kserve-provision-location\") pod \"isvc-canary-promote-predictor-7877c757df-km26d\" (UID: \"df9aaec5-4754-4987-a36a-5753010073bc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-km26d"
Sep 17 18:28:00.248852 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:00.248858    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df9aaec5-4754-4987-a36a-5753010073bc-proxy-tls\") pod \"isvc-canary-promote-predictor-7877c757df-km26d\" (UID: \"df9aaec5-4754-4987-a36a-5753010073bc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-km26d"
Sep 17 18:28:00.249089 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:00.248944    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df9aaec5-4754-4987-a36a-5753010073bc-cabundle-cert\") pod \"isvc-canary-promote-predictor-7877c757df-km26d\" (UID: \"df9aaec5-4754-4987-a36a-5753010073bc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-km26d"
Sep 17 18:28:00.249089 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:00.249014    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt2z2\" (UniqueName: \"kubernetes.io/projected/df9aaec5-4754-4987-a36a-5753010073bc-kube-api-access-lt2z2\") pod \"isvc-canary-promote-predictor-7877c757df-km26d\" (UID: \"df9aaec5-4754-4987-a36a-5753010073bc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-km26d"
Sep 17 18:28:00.249089 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:00.249052    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df9aaec5-4754-4987-a36a-5753010073bc-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-7877c757df-km26d\" (UID: \"df9aaec5-4754-4987-a36a-5753010073bc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-km26d"
Sep 17 18:28:00.350419 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:00.350381    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df9aaec5-4754-4987-a36a-5753010073bc-cabundle-cert\") pod \"isvc-canary-promote-predictor-7877c757df-km26d\" (UID: \"df9aaec5-4754-4987-a36a-5753010073bc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-km26d"
Sep 17 18:28:00.350607 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:00.350443    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt2z2\" (UniqueName: \"kubernetes.io/projected/df9aaec5-4754-4987-a36a-5753010073bc-kube-api-access-lt2z2\") pod \"isvc-canary-promote-predictor-7877c757df-km26d\" (UID: \"df9aaec5-4754-4987-a36a-5753010073bc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-km26d"
Sep 17 18:28:00.350607 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:00.350469    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df9aaec5-4754-4987-a36a-5753010073bc-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-7877c757df-km26d\" (UID: \"df9aaec5-4754-4987-a36a-5753010073bc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-km26d"
Sep 17 18:28:00.350607 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:00.350487    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df9aaec5-4754-4987-a36a-5753010073bc-kserve-provision-location\") pod \"isvc-canary-promote-predictor-7877c757df-km26d\" (UID: \"df9aaec5-4754-4987-a36a-5753010073bc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-km26d"
Sep 17 18:28:00.350607 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:00.350504    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df9aaec5-4754-4987-a36a-5753010073bc-proxy-tls\") pod \"isvc-canary-promote-predictor-7877c757df-km26d\" (UID: \"df9aaec5-4754-4987-a36a-5753010073bc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-km26d"
Sep 17 18:28:00.350974 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:00.350947    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df9aaec5-4754-4987-a36a-5753010073bc-kserve-provision-location\") pod \"isvc-canary-promote-predictor-7877c757df-km26d\" (UID: \"df9aaec5-4754-4987-a36a-5753010073bc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-km26d"
Sep 17 18:28:00.351118 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:00.351096    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df9aaec5-4754-4987-a36a-5753010073bc-cabundle-cert\") pod \"isvc-canary-promote-predictor-7877c757df-km26d\" (UID: \"df9aaec5-4754-4987-a36a-5753010073bc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-km26d"
Sep 17 18:28:00.351200 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:00.351180    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df9aaec5-4754-4987-a36a-5753010073bc-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-7877c757df-km26d\" (UID: \"df9aaec5-4754-4987-a36a-5753010073bc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-km26d"
Sep 17 18:28:00.352923 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:00.352904    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df9aaec5-4754-4987-a36a-5753010073bc-proxy-tls\") pod \"isvc-canary-promote-predictor-7877c757df-km26d\" (UID: \"df9aaec5-4754-4987-a36a-5753010073bc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-km26d"
Sep 17 18:28:00.360903 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:00.360883    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt2z2\" (UniqueName: \"kubernetes.io/projected/df9aaec5-4754-4987-a36a-5753010073bc-kube-api-access-lt2z2\") pod \"isvc-canary-promote-predictor-7877c757df-km26d\" (UID: \"df9aaec5-4754-4987-a36a-5753010073bc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-km26d"
Sep 17 18:28:00.472333 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:00.472248    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-km26d"
Sep 17 18:28:00.613250 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:00.613203    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-dz4nh" podUID="f3497fd9-9759-4ba8-ba5f-75e695b5385f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Sep 17 18:28:00.613464 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:00.613346    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-dz4nh"
Sep 17 18:28:00.744555 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:00.744477    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-km26d"]
Sep 17 18:28:00.750947 ip-10-0-137-95 kubenswrapper[2553]: W0917 18:28:00.750913    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf9aaec5_4754_4987_a36a_5753010073bc.slice/crio-95592dce2a82fa3e6185df4b5c05125b1ddb1be741497a64061aeb81e79b4caf WatchSource:0}: Error finding container 95592dce2a82fa3e6185df4b5c05125b1ddb1be741497a64061aeb81e79b4caf: Status 404 returned error can't find the container with id 95592dce2a82fa3e6185df4b5c05125b1ddb1be741497a64061aeb81e79b4caf
Sep 17 18:28:00.960919 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:00.960869    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-km26d" event={"ID":"df9aaec5-4754-4987-a36a-5753010073bc","Type":"ContainerStarted","Data":"a79142dba5eb11938496c152ea735e821e157ac99015c3e12fb773a520743f05"}
Sep 17 18:28:00.961280 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:00.960960    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-km26d" event={"ID":"df9aaec5-4754-4987-a36a-5753010073bc","Type":"ContainerStarted","Data":"95592dce2a82fa3e6185df4b5c05125b1ddb1be741497a64061aeb81e79b4caf"}
Sep 17 18:28:02.723153 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:02.723113    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cbb795f56-wn4dw" podUID="1a41f41b-30ae-4e33-99e9-0b4099cb4372" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 17 18:28:04.980718 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:04.980681    2553 generic.go:356] "Generic (PLEG): container finished" podID="df9aaec5-4754-4987-a36a-5753010073bc" containerID="a79142dba5eb11938496c152ea735e821e157ac99015c3e12fb773a520743f05" exitCode=0
Sep 17 18:28:04.981182 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:04.980727    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-km26d" event={"ID":"df9aaec5-4754-4987-a36a-5753010073bc","Type":"ContainerDied","Data":"a79142dba5eb11938496c152ea735e821e157ac99015c3e12fb773a520743f05"}
Sep 17 18:28:05.613533 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:05.613488    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-dz4nh" podUID="f3497fd9-9759-4ba8-ba5f-75e695b5385f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Sep 17 18:28:05.987659 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:05.987602    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-km26d" event={"ID":"df9aaec5-4754-4987-a36a-5753010073bc","Type":"ContainerStarted","Data":"4d97b432d08b0d01e17b927c8a9657c3dacd92e6d04b4161a756f18a71570171"}
Sep 17 18:28:05.988036 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:05.987709    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-km26d" event={"ID":"df9aaec5-4754-4987-a36a-5753010073bc","Type":"ContainerStarted","Data":"d5ee09113db90993afb9e21d6804721245687d55fec5467b284a68c595d11de9"}
Sep 17 18:28:05.988036 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:05.987905    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-km26d"
Sep 17 18:28:06.007476 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:06.007422    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-km26d" podStartSLOduration=2.220768614 podStartE2EDuration="6.007408278s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.786639664s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:28:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:28:06.005213645 +0000 UTC m=+1880.701304337" watchObservedRunningTime="2026-09-17 18:28:06.007408278 +0000 UTC m=+1880.703498961"
Sep 17 18:28:06.992823 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:06.992786    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-km26d"
Sep 17 18:28:06.994180 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:06.994154    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-km26d" podUID="df9aaec5-4754-4987-a36a-5753010073bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 17 18:28:07.996868 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:07.996829    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-km26d" podUID="df9aaec5-4754-4987-a36a-5753010073bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 17 18:28:10.614124 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:10.614080    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-dz4nh" podUID="f3497fd9-9759-4ba8-ba5f-75e695b5385f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Sep 17 18:28:12.723234 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:12.723196    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cbb795f56-wn4dw" podUID="1a41f41b-30ae-4e33-99e9-0b4099cb4372" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 17 18:28:13.002994 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:13.002908    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-km26d"
Sep 17 18:28:13.003568 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:13.003541    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-km26d" podUID="df9aaec5-4754-4987-a36a-5753010073bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 17 18:28:15.614151 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:15.614107    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-dz4nh" podUID="f3497fd9-9759-4ba8-ba5f-75e695b5385f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Sep 17 18:28:17.373613 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:17.373567    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cbb795f56-wn4dw" podUID="1a41f41b-30ae-4e33-99e9-0b4099cb4372" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 17 18:28:18.046436 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:18.046397    2553 generic.go:356] "Generic (PLEG): container finished" podID="f3497fd9-9759-4ba8-ba5f-75e695b5385f" containerID="540d26bb26073449bfee0206c4181bf4a12aa68a32bb50eaa51a3d12be760d72" exitCode=137
Sep 17 18:28:18.046681 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:18.046462    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-dz4nh" event={"ID":"f3497fd9-9759-4ba8-ba5f-75e695b5385f","Type":"ContainerDied","Data":"540d26bb26073449bfee0206c4181bf4a12aa68a32bb50eaa51a3d12be760d72"}
Sep 17 18:28:18.158609 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:18.158582    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-dz4nh"
Sep 17 18:28:18.306572 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:18.306485    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f3497fd9-9759-4ba8-ba5f-75e695b5385f-kube-rbac-proxy-sar-config\") pod \"f3497fd9-9759-4ba8-ba5f-75e695b5385f\" (UID: \"f3497fd9-9759-4ba8-ba5f-75e695b5385f\") "
Sep 17 18:28:18.306572 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:18.306536    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3497fd9-9759-4ba8-ba5f-75e695b5385f-proxy-tls\") pod \"f3497fd9-9759-4ba8-ba5f-75e695b5385f\" (UID: \"f3497fd9-9759-4ba8-ba5f-75e695b5385f\") "
Sep 17 18:28:18.306824 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:18.306577    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3497fd9-9759-4ba8-ba5f-75e695b5385f-kserve-provision-location\") pod \"f3497fd9-9759-4ba8-ba5f-75e695b5385f\" (UID: \"f3497fd9-9759-4ba8-ba5f-75e695b5385f\") "
Sep 17 18:28:18.306824 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:18.306666    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ml2hd\" (UniqueName: \"kubernetes.io/projected/f3497fd9-9759-4ba8-ba5f-75e695b5385f-kube-api-access-ml2hd\") pod \"f3497fd9-9759-4ba8-ba5f-75e695b5385f\" (UID: \"f3497fd9-9759-4ba8-ba5f-75e695b5385f\") "
Sep 17 18:28:18.306824 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:18.306745    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f3497fd9-9759-4ba8-ba5f-75e695b5385f-cabundle-cert\") pod \"f3497fd9-9759-4ba8-ba5f-75e695b5385f\" (UID: \"f3497fd9-9759-4ba8-ba5f-75e695b5385f\") "
Sep 17 18:28:18.306984 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:18.306911    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3497fd9-9759-4ba8-ba5f-75e695b5385f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f3497fd9-9759-4ba8-ba5f-75e695b5385f" (UID: "f3497fd9-9759-4ba8-ba5f-75e695b5385f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:28:18.306984 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:18.306914    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3497fd9-9759-4ba8-ba5f-75e695b5385f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f3497fd9-9759-4ba8-ba5f-75e695b5385f" (UID: "f3497fd9-9759-4ba8-ba5f-75e695b5385f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:28:18.307091 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:18.307071    2553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f3497fd9-9759-4ba8-ba5f-75e695b5385f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:28:18.307091 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:18.307083    2553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3497fd9-9759-4ba8-ba5f-75e695b5385f-kserve-provision-location\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:28:18.307164 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:18.307112    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3497fd9-9759-4ba8-ba5f-75e695b5385f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f3497fd9-9759-4ba8-ba5f-75e695b5385f" (UID: "f3497fd9-9759-4ba8-ba5f-75e695b5385f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:28:18.308748 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:18.308724    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3497fd9-9759-4ba8-ba5f-75e695b5385f-kube-api-access-ml2hd" (OuterVolumeSpecName: "kube-api-access-ml2hd") pod "f3497fd9-9759-4ba8-ba5f-75e695b5385f" (UID: "f3497fd9-9759-4ba8-ba5f-75e695b5385f"). InnerVolumeSpecName "kube-api-access-ml2hd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:28:18.308830 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:18.308782    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3497fd9-9759-4ba8-ba5f-75e695b5385f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f3497fd9-9759-4ba8-ba5f-75e695b5385f" (UID: "f3497fd9-9759-4ba8-ba5f-75e695b5385f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:28:18.407789 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:18.407746    2553 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f3497fd9-9759-4ba8-ba5f-75e695b5385f-cabundle-cert\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:28:18.407789 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:18.407784    2553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3497fd9-9759-4ba8-ba5f-75e695b5385f-proxy-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:28:18.407789 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:18.407793    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ml2hd\" (UniqueName: \"kubernetes.io/projected/f3497fd9-9759-4ba8-ba5f-75e695b5385f-kube-api-access-ml2hd\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:28:19.053232 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:19.053198    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-dz4nh" event={"ID":"f3497fd9-9759-4ba8-ba5f-75e695b5385f","Type":"ContainerDied","Data":"2c4e0c7cfeb7eff719d7a633e0c30efc7fcdb42f47f3b0cdcf24fa38cefadf6f"}
Sep 17 18:28:19.053409 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:19.053270    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-dz4nh"
Sep 17 18:28:19.053409 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:19.053274    2553 scope.go:117] "RemoveContainer" containerID="82fb385e1cc14b85858f136294e7647d944244756b350e986231a6b68f27c70e"
Sep 17 18:28:19.061684 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:19.061669    2553 scope.go:117] "RemoveContainer" containerID="540d26bb26073449bfee0206c4181bf4a12aa68a32bb50eaa51a3d12be760d72"
Sep 17 18:28:19.069263 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:19.069244    2553 scope.go:117] "RemoveContainer" containerID="c0032de4b5eb0980dea3fe95b5c66c09fcc784d1aaff27b72369b61462ddc8af"
Sep 17 18:28:19.075493 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:19.075463    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-dz4nh"]
Sep 17 18:28:19.079283 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:19.079264    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5956b77f85-dz4nh"]
Sep 17 18:28:19.381721 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:19.378838    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3497fd9-9759-4ba8-ba5f-75e695b5385f" path="/var/lib/kubelet/pods/f3497fd9-9759-4ba8-ba5f-75e695b5385f/volumes"
Sep 17 18:28:23.003850 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:23.003808    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-km26d" podUID="df9aaec5-4754-4987-a36a-5753010073bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 17 18:28:27.375541 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:27.375512    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cbb795f56-wn4dw"
Sep 17 18:28:30.789804 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:30.789766    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cbb795f56-wn4dw"]
Sep 17 18:28:30.790185 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:30.790029    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cbb795f56-wn4dw" podUID="1a41f41b-30ae-4e33-99e9-0b4099cb4372" containerName="kserve-container" containerID="cri-o://63ea000f1565b52fc1a939f563c02e4212b752d7f6b2a105713e42a0a26629a3" gracePeriod=30
Sep 17 18:28:30.790185 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:30.790114    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cbb795f56-wn4dw" podUID="1a41f41b-30ae-4e33-99e9-0b4099cb4372" containerName="kube-rbac-proxy" containerID="cri-o://204d345a257744e63f5a869e1525c6283866c881da321796783f421a33743b1e" gracePeriod=30
Sep 17 18:28:31.109486 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:31.109395    2553 generic.go:356] "Generic (PLEG): container finished" podID="1a41f41b-30ae-4e33-99e9-0b4099cb4372" containerID="204d345a257744e63f5a869e1525c6283866c881da321796783f421a33743b1e" exitCode=2
Sep 17 18:28:31.109486 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:31.109460    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cbb795f56-wn4dw" event={"ID":"1a41f41b-30ae-4e33-99e9-0b4099cb4372","Type":"ContainerDied","Data":"204d345a257744e63f5a869e1525c6283866c881da321796783f421a33743b1e"}
Sep 17 18:28:32.718276 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:32.718226    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cbb795f56-wn4dw" podUID="1a41f41b-30ae-4e33-99e9-0b4099cb4372" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.40:8643/healthz\": dial tcp 10.134.0.40:8643: connect: connection refused"
Sep 17 18:28:33.004247 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:33.004153    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-km26d" podUID="df9aaec5-4754-4987-a36a-5753010073bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 17 18:28:35.941130 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:35.941096    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cbb795f56-wn4dw"
Sep 17 18:28:36.060487 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:36.060388    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a41f41b-30ae-4e33-99e9-0b4099cb4372-kserve-provision-location\") pod \"1a41f41b-30ae-4e33-99e9-0b4099cb4372\" (UID: \"1a41f41b-30ae-4e33-99e9-0b4099cb4372\") "
Sep 17 18:28:36.060487 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:36.060449    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d2gmv\" (UniqueName: \"kubernetes.io/projected/1a41f41b-30ae-4e33-99e9-0b4099cb4372-kube-api-access-d2gmv\") pod \"1a41f41b-30ae-4e33-99e9-0b4099cb4372\" (UID: \"1a41f41b-30ae-4e33-99e9-0b4099cb4372\") "
Sep 17 18:28:36.060487 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:36.060479    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a41f41b-30ae-4e33-99e9-0b4099cb4372-kube-rbac-proxy-sar-config\") pod \"1a41f41b-30ae-4e33-99e9-0b4099cb4372\" (UID: \"1a41f41b-30ae-4e33-99e9-0b4099cb4372\") "
Sep 17 18:28:36.060815 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:36.060513    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a41f41b-30ae-4e33-99e9-0b4099cb4372-cabundle-cert\") pod \"1a41f41b-30ae-4e33-99e9-0b4099cb4372\" (UID: \"1a41f41b-30ae-4e33-99e9-0b4099cb4372\") "
Sep 17 18:28:36.060815 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:36.060682    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a41f41b-30ae-4e33-99e9-0b4099cb4372-proxy-tls\") pod \"1a41f41b-30ae-4e33-99e9-0b4099cb4372\" (UID: \"1a41f41b-30ae-4e33-99e9-0b4099cb4372\") "
Sep 17 18:28:36.060913 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:36.060817    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a41f41b-30ae-4e33-99e9-0b4099cb4372-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1a41f41b-30ae-4e33-99e9-0b4099cb4372" (UID: "1a41f41b-30ae-4e33-99e9-0b4099cb4372"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:28:36.060960 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:36.060922    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a41f41b-30ae-4e33-99e9-0b4099cb4372-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1a41f41b-30ae-4e33-99e9-0b4099cb4372" (UID: "1a41f41b-30ae-4e33-99e9-0b4099cb4372"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:28:36.060960 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:36.060930    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a41f41b-30ae-4e33-99e9-0b4099cb4372-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1a41f41b-30ae-4e33-99e9-0b4099cb4372" (UID: "1a41f41b-30ae-4e33-99e9-0b4099cb4372"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:28:36.061051 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:36.061034    2553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a41f41b-30ae-4e33-99e9-0b4099cb4372-kserve-provision-location\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:28:36.061083 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:36.061055    2553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a41f41b-30ae-4e33-99e9-0b4099cb4372-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:28:36.061083 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:36.061069    2553 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a41f41b-30ae-4e33-99e9-0b4099cb4372-cabundle-cert\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:28:36.062778 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:36.062756    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a41f41b-30ae-4e33-99e9-0b4099cb4372-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1a41f41b-30ae-4e33-99e9-0b4099cb4372" (UID: "1a41f41b-30ae-4e33-99e9-0b4099cb4372"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:28:36.062907 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:36.062779    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a41f41b-30ae-4e33-99e9-0b4099cb4372-kube-api-access-d2gmv" (OuterVolumeSpecName: "kube-api-access-d2gmv") pod "1a41f41b-30ae-4e33-99e9-0b4099cb4372" (UID: "1a41f41b-30ae-4e33-99e9-0b4099cb4372"). InnerVolumeSpecName "kube-api-access-d2gmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:28:36.133373 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:36.133334    2553 generic.go:356] "Generic (PLEG): container finished" podID="1a41f41b-30ae-4e33-99e9-0b4099cb4372" containerID="63ea000f1565b52fc1a939f563c02e4212b752d7f6b2a105713e42a0a26629a3" exitCode=0
Sep 17 18:28:36.133555 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:36.133391    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cbb795f56-wn4dw"
Sep 17 18:28:36.133555 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:36.133415    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cbb795f56-wn4dw" event={"ID":"1a41f41b-30ae-4e33-99e9-0b4099cb4372","Type":"ContainerDied","Data":"63ea000f1565b52fc1a939f563c02e4212b752d7f6b2a105713e42a0a26629a3"}
Sep 17 18:28:36.133555 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:36.133486    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cbb795f56-wn4dw" event={"ID":"1a41f41b-30ae-4e33-99e9-0b4099cb4372","Type":"ContainerDied","Data":"8538a7908ee580245af5d5ed2ccc8650d8ba61e84fef715d2b3a277c9e3cda67"}
Sep 17 18:28:36.133555 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:36.133515    2553 scope.go:117] "RemoveContainer" containerID="204d345a257744e63f5a869e1525c6283866c881da321796783f421a33743b1e"
Sep 17 18:28:36.142353 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:36.142330    2553 scope.go:117] "RemoveContainer" containerID="63ea000f1565b52fc1a939f563c02e4212b752d7f6b2a105713e42a0a26629a3"
Sep 17 18:28:36.150699 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:36.150674    2553 scope.go:117] "RemoveContainer" containerID="c5c48523d541658eb84bbb4a17050be2aa537f5d429d850ab8e3a27765f00b52"
Sep 17 18:28:36.155486 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:36.155457    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cbb795f56-wn4dw"]
Sep 17 18:28:36.159306 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:36.159284    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7cbb795f56-wn4dw"]
Sep 17 18:28:36.159375 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:36.159332    2553 scope.go:117] "RemoveContainer" containerID="204d345a257744e63f5a869e1525c6283866c881da321796783f421a33743b1e"
Sep 17 18:28:36.159719 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:28:36.159699    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"204d345a257744e63f5a869e1525c6283866c881da321796783f421a33743b1e\": container with ID starting with 204d345a257744e63f5a869e1525c6283866c881da321796783f421a33743b1e not found: ID does not exist" containerID="204d345a257744e63f5a869e1525c6283866c881da321796783f421a33743b1e"
Sep 17 18:28:36.159779 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:36.159729    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"204d345a257744e63f5a869e1525c6283866c881da321796783f421a33743b1e"} err="failed to get container status \"204d345a257744e63f5a869e1525c6283866c881da321796783f421a33743b1e\": rpc error: code = NotFound desc = could not find container \"204d345a257744e63f5a869e1525c6283866c881da321796783f421a33743b1e\": container with ID starting with 204d345a257744e63f5a869e1525c6283866c881da321796783f421a33743b1e not found: ID does not exist"
Sep 17 18:28:36.159779 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:36.159748    2553 scope.go:117] "RemoveContainer" containerID="63ea000f1565b52fc1a939f563c02e4212b752d7f6b2a105713e42a0a26629a3"
Sep 17 18:28:36.160081 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:28:36.160065    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63ea000f1565b52fc1a939f563c02e4212b752d7f6b2a105713e42a0a26629a3\": container with ID starting with 63ea000f1565b52fc1a939f563c02e4212b752d7f6b2a105713e42a0a26629a3 not found: ID does not exist" containerID="63ea000f1565b52fc1a939f563c02e4212b752d7f6b2a105713e42a0a26629a3"
Sep 17 18:28:36.160139 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:36.160086    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63ea000f1565b52fc1a939f563c02e4212b752d7f6b2a105713e42a0a26629a3"} err="failed to get container status \"63ea000f1565b52fc1a939f563c02e4212b752d7f6b2a105713e42a0a26629a3\": rpc error: code = NotFound desc = could not find container \"63ea000f1565b52fc1a939f563c02e4212b752d7f6b2a105713e42a0a26629a3\": container with ID starting with 63ea000f1565b52fc1a939f563c02e4212b752d7f6b2a105713e42a0a26629a3 not found: ID does not exist"
Sep 17 18:28:36.160139 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:36.160100    2553 scope.go:117] "RemoveContainer" containerID="c5c48523d541658eb84bbb4a17050be2aa537f5d429d850ab8e3a27765f00b52"
Sep 17 18:28:36.160346 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:28:36.160328    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5c48523d541658eb84bbb4a17050be2aa537f5d429d850ab8e3a27765f00b52\": container with ID starting with c5c48523d541658eb84bbb4a17050be2aa537f5d429d850ab8e3a27765f00b52 not found: ID does not exist" containerID="c5c48523d541658eb84bbb4a17050be2aa537f5d429d850ab8e3a27765f00b52"
Sep 17 18:28:36.160394 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:36.160354    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5c48523d541658eb84bbb4a17050be2aa537f5d429d850ab8e3a27765f00b52"} err="failed to get container status \"c5c48523d541658eb84bbb4a17050be2aa537f5d429d850ab8e3a27765f00b52\": rpc error: code = NotFound desc = could not find container \"c5c48523d541658eb84bbb4a17050be2aa537f5d429d850ab8e3a27765f00b52\": container with ID starting with c5c48523d541658eb84bbb4a17050be2aa537f5d429d850ab8e3a27765f00b52 not found: ID does not exist"
Sep 17 18:28:36.161701 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:36.161684    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2gmv\" (UniqueName: \"kubernetes.io/projected/1a41f41b-30ae-4e33-99e9-0b4099cb4372-kube-api-access-d2gmv\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:28:36.161751 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:36.161703    2553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a41f41b-30ae-4e33-99e9-0b4099cb4372-proxy-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:28:37.377389 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:37.377353    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a41f41b-30ae-4e33-99e9-0b4099cb4372" path="/var/lib/kubelet/pods/1a41f41b-30ae-4e33-99e9-0b4099cb4372/volumes"
Sep 17 18:28:43.004443 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:43.004402    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-km26d" podUID="df9aaec5-4754-4987-a36a-5753010073bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 17 18:28:53.003604 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:28:53.003563    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-km26d" podUID="df9aaec5-4754-4987-a36a-5753010073bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 17 18:29:03.003737 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:03.003694    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-km26d" podUID="df9aaec5-4754-4987-a36a-5753010073bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 17 18:29:13.003600 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:13.003557    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-km26d" podUID="df9aaec5-4754-4987-a36a-5753010073bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 17 18:29:23.004576 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:23.004542    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-km26d"
Sep 17 18:29:30.155755 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:30.155668    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-km26d"]
Sep 17 18:29:30.156198 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:30.155967    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-km26d" podUID="df9aaec5-4754-4987-a36a-5753010073bc" containerName="kserve-container" containerID="cri-o://d5ee09113db90993afb9e21d6804721245687d55fec5467b284a68c595d11de9" gracePeriod=30
Sep 17 18:29:30.156198 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:30.155992    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-km26d" podUID="df9aaec5-4754-4987-a36a-5753010073bc" containerName="kube-rbac-proxy" containerID="cri-o://4d97b432d08b0d01e17b927c8a9657c3dacd92e6d04b4161a756f18a71570171" gracePeriod=30
Sep 17 18:29:30.232415 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:30.232375    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-pxfp9"]
Sep 17 18:29:30.232818 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:30.232798    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a41f41b-30ae-4e33-99e9-0b4099cb4372" containerName="storage-initializer"
Sep 17 18:29:30.232818 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:30.232819    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a41f41b-30ae-4e33-99e9-0b4099cb4372" containerName="storage-initializer"
Sep 17 18:29:30.232967 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:30.232836    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3497fd9-9759-4ba8-ba5f-75e695b5385f" containerName="storage-initializer"
Sep 17 18:29:30.232967 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:30.232843    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3497fd9-9759-4ba8-ba5f-75e695b5385f" containerName="storage-initializer"
Sep 17 18:29:30.232967 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:30.232855    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3497fd9-9759-4ba8-ba5f-75e695b5385f" containerName="kube-rbac-proxy"
Sep 17 18:29:30.232967 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:30.232863    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3497fd9-9759-4ba8-ba5f-75e695b5385f" containerName="kube-rbac-proxy"
Sep 17 18:29:30.232967 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:30.232875    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3497fd9-9759-4ba8-ba5f-75e695b5385f" containerName="kserve-container"
Sep 17 18:29:30.232967 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:30.232883    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3497fd9-9759-4ba8-ba5f-75e695b5385f" containerName="kserve-container"
Sep 17 18:29:30.232967 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:30.232894    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a41f41b-30ae-4e33-99e9-0b4099cb4372" containerName="kube-rbac-proxy"
Sep 17 18:29:30.232967 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:30.232902    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a41f41b-30ae-4e33-99e9-0b4099cb4372" containerName="kube-rbac-proxy"
Sep 17 18:29:30.232967 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:30.232925    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a41f41b-30ae-4e33-99e9-0b4099cb4372" containerName="kserve-container"
Sep 17 18:29:30.232967 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:30.232934    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a41f41b-30ae-4e33-99e9-0b4099cb4372" containerName="kserve-container"
Sep 17 18:29:30.233452 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:30.233011    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3497fd9-9759-4ba8-ba5f-75e695b5385f" containerName="kserve-container"
Sep 17 18:29:30.233452 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:30.233029    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3497fd9-9759-4ba8-ba5f-75e695b5385f" containerName="kube-rbac-proxy"
Sep 17 18:29:30.233452 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:30.233038    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a41f41b-30ae-4e33-99e9-0b4099cb4372" containerName="kube-rbac-proxy"
Sep 17 18:29:30.233452 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:30.233048    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a41f41b-30ae-4e33-99e9-0b4099cb4372" containerName="kserve-container"
Sep 17 18:29:30.236431 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:30.236410    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-pxfp9"
Sep 17 18:29:30.239996 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:30.239975    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 17 18:29:30.256712 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:30.256687    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-pxfp9"]
Sep 17 18:29:30.309574 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:30.309535    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh5kl\" (UniqueName: \"kubernetes.io/projected/2e5d8da2-d685-457a-925a-370375c22a60-kube-api-access-fh5kl\") pod \"isvc-canary-promote-v2-predictor-6b666fbfdb-pxfp9\" (UID: \"2e5d8da2-d685-457a-925a-370375c22a60\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-pxfp9"
Sep 17 18:29:30.309749 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:30.309589    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e5d8da2-d685-457a-925a-370375c22a60-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6b666fbfdb-pxfp9\" (UID: \"2e5d8da2-d685-457a-925a-370375c22a60\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-pxfp9"
Sep 17 18:29:30.309749 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:30.309681    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e5d8da2-d685-457a-925a-370375c22a60-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6b666fbfdb-pxfp9\" (UID: \"2e5d8da2-d685-457a-925a-370375c22a60\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-pxfp9"
Sep 17 18:29:30.309749 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:30.309742    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e5d8da2-d685-457a-925a-370375c22a60-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6b666fbfdb-pxfp9\" (UID: \"2e5d8da2-d685-457a-925a-370375c22a60\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-pxfp9"
Sep 17 18:29:30.309852 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:30.309779    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e5d8da2-d685-457a-925a-370375c22a60-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6b666fbfdb-pxfp9\" (UID: \"2e5d8da2-d685-457a-925a-370375c22a60\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-pxfp9"
Sep 17 18:29:30.364118 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:30.364085    2553 generic.go:356] "Generic (PLEG): container finished" podID="df9aaec5-4754-4987-a36a-5753010073bc" containerID="4d97b432d08b0d01e17b927c8a9657c3dacd92e6d04b4161a756f18a71570171" exitCode=2
Sep 17 18:29:30.364266 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:30.364155    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-km26d" event={"ID":"df9aaec5-4754-4987-a36a-5753010073bc","Type":"ContainerDied","Data":"4d97b432d08b0d01e17b927c8a9657c3dacd92e6d04b4161a756f18a71570171"}
Sep 17 18:29:30.410814 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:30.410742    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh5kl\" (UniqueName: \"kubernetes.io/projected/2e5d8da2-d685-457a-925a-370375c22a60-kube-api-access-fh5kl\") pod \"isvc-canary-promote-v2-predictor-6b666fbfdb-pxfp9\" (UID: \"2e5d8da2-d685-457a-925a-370375c22a60\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-pxfp9"
Sep 17 18:29:30.410814 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:30.410788    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e5d8da2-d685-457a-925a-370375c22a60-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6b666fbfdb-pxfp9\" (UID: \"2e5d8da2-d685-457a-925a-370375c22a60\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-pxfp9"
Sep 17 18:29:30.410984 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:30.410949    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e5d8da2-d685-457a-925a-370375c22a60-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6b666fbfdb-pxfp9\" (UID: \"2e5d8da2-d685-457a-925a-370375c22a60\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-pxfp9"
Sep 17 18:29:30.411121 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:30.411033    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e5d8da2-d685-457a-925a-370375c22a60-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6b666fbfdb-pxfp9\" (UID: \"2e5d8da2-d685-457a-925a-370375c22a60\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-pxfp9"
Sep 17 18:29:30.411121 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:30.411082    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e5d8da2-d685-457a-925a-370375c22a60-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6b666fbfdb-pxfp9\" (UID: \"2e5d8da2-d685-457a-925a-370375c22a60\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-pxfp9"
Sep 17 18:29:30.411287 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:30.411266    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e5d8da2-d685-457a-925a-370375c22a60-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6b666fbfdb-pxfp9\" (UID: \"2e5d8da2-d685-457a-925a-370375c22a60\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-pxfp9"
Sep 17 18:29:30.411533 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:30.411504    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e5d8da2-d685-457a-925a-370375c22a60-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6b666fbfdb-pxfp9\" (UID: \"2e5d8da2-d685-457a-925a-370375c22a60\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-pxfp9"
Sep 17 18:29:30.411694 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:30.411677    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e5d8da2-d685-457a-925a-370375c22a60-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6b666fbfdb-pxfp9\" (UID: \"2e5d8da2-d685-457a-925a-370375c22a60\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-pxfp9"
Sep 17 18:29:30.413389 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:30.413366    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e5d8da2-d685-457a-925a-370375c22a60-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6b666fbfdb-pxfp9\" (UID: \"2e5d8da2-d685-457a-925a-370375c22a60\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-pxfp9"
Sep 17 18:29:30.419141 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:30.419116    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh5kl\" (UniqueName: \"kubernetes.io/projected/2e5d8da2-d685-457a-925a-370375c22a60-kube-api-access-fh5kl\") pod \"isvc-canary-promote-v2-predictor-6b666fbfdb-pxfp9\" (UID: \"2e5d8da2-d685-457a-925a-370375c22a60\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-pxfp9"
Sep 17 18:29:30.546380 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:30.546339    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-pxfp9"
Sep 17 18:29:30.819991 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:30.819965    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-pxfp9"]
Sep 17 18:29:30.822285 ip-10-0-137-95 kubenswrapper[2553]: W0917 18:29:30.822257    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e5d8da2_d685_457a_925a_370375c22a60.slice/crio-3b84e3df8144a920c576c4aa0ea131744e2cb1f080de4b2d4782a2fe925c8894 WatchSource:0}: Error finding container 3b84e3df8144a920c576c4aa0ea131744e2cb1f080de4b2d4782a2fe925c8894: Status 404 returned error can't find the container with id 3b84e3df8144a920c576c4aa0ea131744e2cb1f080de4b2d4782a2fe925c8894
Sep 17 18:29:31.370289 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:31.370247    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-pxfp9" event={"ID":"2e5d8da2-d685-457a-925a-370375c22a60","Type":"ContainerStarted","Data":"61b1696bbac74cbccbf654cbadd2ce534f83e7bf5c24f1c7a39b098bf5a0beed"}
Sep 17 18:29:31.370677 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:31.370325    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-pxfp9" event={"ID":"2e5d8da2-d685-457a-925a-370375c22a60","Type":"ContainerStarted","Data":"3b84e3df8144a920c576c4aa0ea131744e2cb1f080de4b2d4782a2fe925c8894"}
Sep 17 18:29:32.998130 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:32.998082    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-km26d" podUID="df9aaec5-4754-4987-a36a-5753010073bc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
Sep 17 18:29:33.004564 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:33.004525    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-km26d" podUID="df9aaec5-4754-4987-a36a-5753010073bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 17 18:29:35.098950 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:35.098920    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-km26d"
Sep 17 18:29:35.156477 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:35.156438    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df9aaec5-4754-4987-a36a-5753010073bc-proxy-tls\") pod \"df9aaec5-4754-4987-a36a-5753010073bc\" (UID: \"df9aaec5-4754-4987-a36a-5753010073bc\") "
Sep 17 18:29:35.156691 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:35.156494    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df9aaec5-4754-4987-a36a-5753010073bc-kserve-provision-location\") pod \"df9aaec5-4754-4987-a36a-5753010073bc\" (UID: \"df9aaec5-4754-4987-a36a-5753010073bc\") "
Sep 17 18:29:35.156691 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:35.156521    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df9aaec5-4754-4987-a36a-5753010073bc-cabundle-cert\") pod \"df9aaec5-4754-4987-a36a-5753010073bc\" (UID: \"df9aaec5-4754-4987-a36a-5753010073bc\") "
Sep 17 18:29:35.156691 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:35.156541    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lt2z2\" (UniqueName: \"kubernetes.io/projected/df9aaec5-4754-4987-a36a-5753010073bc-kube-api-access-lt2z2\") pod \"df9aaec5-4754-4987-a36a-5753010073bc\" (UID: \"df9aaec5-4754-4987-a36a-5753010073bc\") "
Sep 17 18:29:35.156691 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:35.156584    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df9aaec5-4754-4987-a36a-5753010073bc-kube-rbac-proxy-sar-config\") pod \"df9aaec5-4754-4987-a36a-5753010073bc\" (UID: \"df9aaec5-4754-4987-a36a-5753010073bc\") "
Sep 17 18:29:35.156928 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:35.156864    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df9aaec5-4754-4987-a36a-5753010073bc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "df9aaec5-4754-4987-a36a-5753010073bc" (UID: "df9aaec5-4754-4987-a36a-5753010073bc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:29:35.157019 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:35.156991    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df9aaec5-4754-4987-a36a-5753010073bc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "df9aaec5-4754-4987-a36a-5753010073bc" (UID: "df9aaec5-4754-4987-a36a-5753010073bc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:29:35.157134 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:35.157048    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df9aaec5-4754-4987-a36a-5753010073bc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "df9aaec5-4754-4987-a36a-5753010073bc" (UID: "df9aaec5-4754-4987-a36a-5753010073bc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:29:35.158707 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:35.158677    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df9aaec5-4754-4987-a36a-5753010073bc-kube-api-access-lt2z2" (OuterVolumeSpecName: "kube-api-access-lt2z2") pod "df9aaec5-4754-4987-a36a-5753010073bc" (UID: "df9aaec5-4754-4987-a36a-5753010073bc"). InnerVolumeSpecName "kube-api-access-lt2z2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:29:35.158785 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:35.158756    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df9aaec5-4754-4987-a36a-5753010073bc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "df9aaec5-4754-4987-a36a-5753010073bc" (UID: "df9aaec5-4754-4987-a36a-5753010073bc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:29:35.257211 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:35.257096    2553 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df9aaec5-4754-4987-a36a-5753010073bc-cabundle-cert\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:29:35.257211 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:35.257147    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt2z2\" (UniqueName: \"kubernetes.io/projected/df9aaec5-4754-4987-a36a-5753010073bc-kube-api-access-lt2z2\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:29:35.257211 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:35.257162    2553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df9aaec5-4754-4987-a36a-5753010073bc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:29:35.257211 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:35.257175    2553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df9aaec5-4754-4987-a36a-5753010073bc-proxy-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:29:35.257211 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:35.257189    2553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df9aaec5-4754-4987-a36a-5753010073bc-kserve-provision-location\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:29:35.388375 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:35.388339    2553 generic.go:356] "Generic (PLEG): container finished" podID="df9aaec5-4754-4987-a36a-5753010073bc" containerID="d5ee09113db90993afb9e21d6804721245687d55fec5467b284a68c595d11de9" exitCode=0
Sep 17 18:29:35.388550 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:35.388399    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-km26d"
Sep 17 18:29:35.388550 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:35.388416    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-km26d" event={"ID":"df9aaec5-4754-4987-a36a-5753010073bc","Type":"ContainerDied","Data":"d5ee09113db90993afb9e21d6804721245687d55fec5467b284a68c595d11de9"}
Sep 17 18:29:35.388550 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:35.388480    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-km26d" event={"ID":"df9aaec5-4754-4987-a36a-5753010073bc","Type":"ContainerDied","Data":"95592dce2a82fa3e6185df4b5c05125b1ddb1be741497a64061aeb81e79b4caf"}
Sep 17 18:29:35.388550 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:35.388518    2553 scope.go:117] "RemoveContainer" containerID="4d97b432d08b0d01e17b927c8a9657c3dacd92e6d04b4161a756f18a71570171"
Sep 17 18:29:35.390030 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:35.390007    2553 generic.go:356] "Generic (PLEG): container finished" podID="2e5d8da2-d685-457a-925a-370375c22a60" containerID="61b1696bbac74cbccbf654cbadd2ce534f83e7bf5c24f1c7a39b098bf5a0beed" exitCode=0
Sep 17 18:29:35.390124 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:35.390053    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-pxfp9" event={"ID":"2e5d8da2-d685-457a-925a-370375c22a60","Type":"ContainerDied","Data":"61b1696bbac74cbccbf654cbadd2ce534f83e7bf5c24f1c7a39b098bf5a0beed"}
Sep 17 18:29:35.397253 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:35.397234    2553 scope.go:117] "RemoveContainer" containerID="d5ee09113db90993afb9e21d6804721245687d55fec5467b284a68c595d11de9"
Sep 17 18:29:35.406079 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:35.406054    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-km26d"]
Sep 17 18:29:35.406188 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:35.406162    2553 scope.go:117] "RemoveContainer" containerID="a79142dba5eb11938496c152ea735e821e157ac99015c3e12fb773a520743f05"
Sep 17 18:29:35.411589 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:35.411565    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7877c757df-km26d"]
Sep 17 18:29:35.415420 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:35.415402    2553 scope.go:117] "RemoveContainer" containerID="4d97b432d08b0d01e17b927c8a9657c3dacd92e6d04b4161a756f18a71570171"
Sep 17 18:29:35.415798 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:29:35.415779    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d97b432d08b0d01e17b927c8a9657c3dacd92e6d04b4161a756f18a71570171\": container with ID starting with 4d97b432d08b0d01e17b927c8a9657c3dacd92e6d04b4161a756f18a71570171 not found: ID does not exist" containerID="4d97b432d08b0d01e17b927c8a9657c3dacd92e6d04b4161a756f18a71570171"
Sep 17 18:29:35.415889 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:35.415809    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d97b432d08b0d01e17b927c8a9657c3dacd92e6d04b4161a756f18a71570171"} err="failed to get container status \"4d97b432d08b0d01e17b927c8a9657c3dacd92e6d04b4161a756f18a71570171\": rpc error: code = NotFound desc = could not find container \"4d97b432d08b0d01e17b927c8a9657c3dacd92e6d04b4161a756f18a71570171\": container with ID starting with 4d97b432d08b0d01e17b927c8a9657c3dacd92e6d04b4161a756f18a71570171 not found: ID does not exist"
Sep 17 18:29:35.415889 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:35.415838    2553 scope.go:117] "RemoveContainer" containerID="d5ee09113db90993afb9e21d6804721245687d55fec5467b284a68c595d11de9"
Sep 17 18:29:35.416120 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:29:35.416100    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5ee09113db90993afb9e21d6804721245687d55fec5467b284a68c595d11de9\": container with ID starting with d5ee09113db90993afb9e21d6804721245687d55fec5467b284a68c595d11de9 not found: ID does not exist" containerID="d5ee09113db90993afb9e21d6804721245687d55fec5467b284a68c595d11de9"
Sep 17 18:29:35.416158 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:35.416126    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5ee09113db90993afb9e21d6804721245687d55fec5467b284a68c595d11de9"} err="failed to get container status \"d5ee09113db90993afb9e21d6804721245687d55fec5467b284a68c595d11de9\": rpc error: code = NotFound desc = could not find container \"d5ee09113db90993afb9e21d6804721245687d55fec5467b284a68c595d11de9\": container with ID starting with d5ee09113db90993afb9e21d6804721245687d55fec5467b284a68c595d11de9 not found: ID does not exist"
Sep 17 18:29:35.416158 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:35.416140    2553 scope.go:117] "RemoveContainer" containerID="a79142dba5eb11938496c152ea735e821e157ac99015c3e12fb773a520743f05"
Sep 17 18:29:35.416388 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:29:35.416370    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a79142dba5eb11938496c152ea735e821e157ac99015c3e12fb773a520743f05\": container with ID starting with a79142dba5eb11938496c152ea735e821e157ac99015c3e12fb773a520743f05 not found: ID does not exist" containerID="a79142dba5eb11938496c152ea735e821e157ac99015c3e12fb773a520743f05"
Sep 17 18:29:35.416430 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:35.416393    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a79142dba5eb11938496c152ea735e821e157ac99015c3e12fb773a520743f05"} err="failed to get container status \"a79142dba5eb11938496c152ea735e821e157ac99015c3e12fb773a520743f05\": rpc error: code = NotFound desc = could not find container \"a79142dba5eb11938496c152ea735e821e157ac99015c3e12fb773a520743f05\": container with ID starting with a79142dba5eb11938496c152ea735e821e157ac99015c3e12fb773a520743f05 not found: ID does not exist"
Sep 17 18:29:36.398164 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:36.398128    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-pxfp9" event={"ID":"2e5d8da2-d685-457a-925a-370375c22a60","Type":"ContainerStarted","Data":"119682562fbe631bee39f9d2f45c0426bbe8ebe2186a2ac58df86d5111208f1a"}
Sep 17 18:29:36.398615 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:36.398192    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-pxfp9" event={"ID":"2e5d8da2-d685-457a-925a-370375c22a60","Type":"ContainerStarted","Data":"cdb0e48664a00375bb3f2ab347c4fc460e6049484c6f1edad53bc37ded00fbbf"}
Sep 17 18:29:36.398615 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:36.398468    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-pxfp9"
Sep 17 18:29:36.398615 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:36.398527    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-pxfp9"
Sep 17 18:29:36.399929 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:36.399903    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-pxfp9" podUID="2e5d8da2-d685-457a-925a-370375c22a60" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 18:29:36.416825 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:36.416766    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-pxfp9" podStartSLOduration=2.717910006 podStartE2EDuration="6.416748389s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.698838383s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:36.415828174 +0000 UTC m=+1971.111918855" watchObservedRunningTime="2026-09-17 18:29:36.416748389 +0000 UTC m=+1971.112839100"
Sep 17 18:29:37.376746 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:37.376707    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df9aaec5-4754-4987-a36a-5753010073bc" path="/var/lib/kubelet/pods/df9aaec5-4754-4987-a36a-5753010073bc/volumes"
Sep 17 18:29:37.402613 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:37.402575    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-pxfp9" podUID="2e5d8da2-d685-457a-925a-370375c22a60" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 18:29:42.407467 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:42.407437    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-pxfp9"
Sep 17 18:29:42.408169 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:42.408139    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-pxfp9" podUID="2e5d8da2-d685-457a-925a-370375c22a60" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 18:29:52.408932 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:29:52.408893    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-pxfp9" podUID="2e5d8da2-d685-457a-925a-370375c22a60" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 18:30:02.408482 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:30:02.408438    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-pxfp9" podUID="2e5d8da2-d685-457a-925a-370375c22a60" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 18:30:11.271504 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:30:11.271465    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4fgsf"]
Sep 17 18:30:11.271920 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:30:11.271826    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df9aaec5-4754-4987-a36a-5753010073bc" containerName="storage-initializer"
Sep 17 18:30:11.271920 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:30:11.271839    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="df9aaec5-4754-4987-a36a-5753010073bc" containerName="storage-initializer"
Sep 17 18:30:11.271920 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:30:11.271853    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df9aaec5-4754-4987-a36a-5753010073bc" containerName="kserve-container"
Sep 17 18:30:11.271920 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:30:11.271859    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="df9aaec5-4754-4987-a36a-5753010073bc" containerName="kserve-container"
Sep 17 18:30:11.271920 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:30:11.271879    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df9aaec5-4754-4987-a36a-5753010073bc" containerName="kube-rbac-proxy"
Sep 17 18:30:11.271920 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:30:11.271884    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="df9aaec5-4754-4987-a36a-5753010073bc" containerName="kube-rbac-proxy"
Sep 17 18:30:11.272111 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:30:11.271934    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="df9aaec5-4754-4987-a36a-5753010073bc" containerName="kube-rbac-proxy"
Sep 17 18:30:11.272111 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:30:11.271941    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="df9aaec5-4754-4987-a36a-5753010073bc" containerName="kserve-container"
Sep 17 18:30:11.274902 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:30:11.274884    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4fgsf"
Sep 17 18:30:11.277427 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:30:11.277404    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-predictor-serving-cert\""
Sep 17 18:30:11.277536 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:30:11.277433    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-kube-rbac-proxy-sar-config\""
Sep 17 18:30:11.284396 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:30:11.284372    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4fgsf"]
Sep 17 18:30:11.356300 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:30:11.356262    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be05f6a4-ecbc-4030-a469-a4dd4cf3570c-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-4fgsf\" (UID: \"be05f6a4-ecbc-4030-a469-a4dd4cf3570c\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4fgsf"
Sep 17 18:30:11.356482 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:30:11.356317    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be05f6a4-ecbc-4030-a469-a4dd4cf3570c-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-4fgsf\" (UID: \"be05f6a4-ecbc-4030-a469-a4dd4cf3570c\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4fgsf"
Sep 17 18:30:11.356482 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:30:11.356346    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be05f6a4-ecbc-4030-a469-a4dd4cf3570c-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-4fgsf\" (UID: \"be05f6a4-ecbc-4030-a469-a4dd4cf3570c\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4fgsf"
Sep 17 18:30:11.356482 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:30:11.356372    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g72lx\" (UniqueName: \"kubernetes.io/projected/be05f6a4-ecbc-4030-a469-a4dd4cf3570c-kube-api-access-g72lx\") pod \"isvc-tensorflow-predictor-547dfc5669-4fgsf\" (UID: \"be05f6a4-ecbc-4030-a469-a4dd4cf3570c\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4fgsf"
Sep 17 18:30:11.356482 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:30:11.356405    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be05f6a4-ecbc-4030-a469-a4dd4cf3570c-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-4fgsf\" (UID: \"be05f6a4-ecbc-4030-a469-a4dd4cf3570c\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4fgsf"
Sep 17 18:30:11.456744 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:30:11.456706    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g72lx\" (UniqueName: \"kubernetes.io/projected/be05f6a4-ecbc-4030-a469-a4dd4cf3570c-kube-api-access-g72lx\") pod \"isvc-tensorflow-predictor-547dfc5669-4fgsf\" (UID: \"be05f6a4-ecbc-4030-a469-a4dd4cf3570c\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4fgsf"
Sep 17 18:30:11.456744 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:30:11.456748    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be05f6a4-ecbc-4030-a469-a4dd4cf3570c-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-4fgsf\" (UID: \"be05f6a4-ecbc-4030-a469-a4dd4cf3570c\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4fgsf"
Sep 17 18:30:11.456976 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:30:11.456793    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be05f6a4-ecbc-4030-a469-a4dd4cf3570c-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-4fgsf\" (UID: \"be05f6a4-ecbc-4030-a469-a4dd4cf3570c\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4fgsf"
Sep 17 18:30:11.456976 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:30:11.456836    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be05f6a4-ecbc-4030-a469-a4dd4cf3570c-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-4fgsf\" (UID: \"be05f6a4-ecbc-4030-a469-a4dd4cf3570c\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4fgsf"
Sep 17 18:30:11.456976 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:30:11.456869    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be05f6a4-ecbc-4030-a469-a4dd4cf3570c-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-4fgsf\" (UID: \"be05f6a4-ecbc-4030-a469-a4dd4cf3570c\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4fgsf"
Sep 17 18:30:11.457130 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:30:11.457108    2553 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-tensorflow-predictor-serving-cert: secret "isvc-tensorflow-predictor-serving-cert" not found
Sep 17 18:30:11.457187 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:30:11.457176    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be05f6a4-ecbc-4030-a469-a4dd4cf3570c-proxy-tls podName:be05f6a4-ecbc-4030-a469-a4dd4cf3570c nodeName:}" failed. No retries permitted until 2026-09-17 18:30:11.957157957 +0000 UTC m=+2006.653248632 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/be05f6a4-ecbc-4030-a469-a4dd4cf3570c-proxy-tls") pod "isvc-tensorflow-predictor-547dfc5669-4fgsf" (UID: "be05f6a4-ecbc-4030-a469-a4dd4cf3570c") : secret "isvc-tensorflow-predictor-serving-cert" not found
Sep 17 18:30:11.457319 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:30:11.457212    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be05f6a4-ecbc-4030-a469-a4dd4cf3570c-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-4fgsf\" (UID: \"be05f6a4-ecbc-4030-a469-a4dd4cf3570c\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4fgsf"
Sep 17 18:30:11.457521 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:30:11.457498    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be05f6a4-ecbc-4030-a469-a4dd4cf3570c-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-4fgsf\" (UID: \"be05f6a4-ecbc-4030-a469-a4dd4cf3570c\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4fgsf"
Sep 17 18:30:11.457656 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:30:11.457637    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be05f6a4-ecbc-4030-a469-a4dd4cf3570c-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-4fgsf\" (UID: \"be05f6a4-ecbc-4030-a469-a4dd4cf3570c\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4fgsf"
Sep 17 18:30:11.465891 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:30:11.465865    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g72lx\" (UniqueName: \"kubernetes.io/projected/be05f6a4-ecbc-4030-a469-a4dd4cf3570c-kube-api-access-g72lx\") pod \"isvc-tensorflow-predictor-547dfc5669-4fgsf\" (UID: \"be05f6a4-ecbc-4030-a469-a4dd4cf3570c\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4fgsf"
Sep 17 18:30:11.959049 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:30:11.959004    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be05f6a4-ecbc-4030-a469-a4dd4cf3570c-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-4fgsf\" (UID: \"be05f6a4-ecbc-4030-a469-a4dd4cf3570c\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4fgsf"
Sep 17 18:30:11.961469 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:30:11.961448    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be05f6a4-ecbc-4030-a469-a4dd4cf3570c-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-4fgsf\" (UID: \"be05f6a4-ecbc-4030-a469-a4dd4cf3570c\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4fgsf"
Sep 17 18:30:12.185312 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:30:12.185271    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4fgsf"
Sep 17 18:30:12.408331 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:30:12.408289    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-pxfp9" podUID="2e5d8da2-d685-457a-925a-370375c22a60" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 18:30:12.460069 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:30:12.460042    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4fgsf"]
Sep 17 18:30:12.462303 ip-10-0-137-95 kubenswrapper[2553]: W0917 18:30:12.462274    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe05f6a4_ecbc_4030_a469_a4dd4cf3570c.slice/crio-3cb1b727f1a24dc9c45f9b854cc0c6720366086ec30e7c2b739fe3bc8873b64f WatchSource:0}: Error finding container 3cb1b727f1a24dc9c45f9b854cc0c6720366086ec30e7c2b739fe3bc8873b64f: Status 404 returned error can't find the container with id 3cb1b727f1a24dc9c45f9b854cc0c6720366086ec30e7c2b739fe3bc8873b64f
Sep 17 18:30:12.564723 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:30:12.564682    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4fgsf" event={"ID":"be05f6a4-ecbc-4030-a469-a4dd4cf3570c","Type":"ContainerStarted","Data":"caa8d268b60800a69cac317e26942405ea8e3b2ab024d84682cea8080badf0c7"}
Sep 17 18:30:12.564871 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:30:12.564747    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4fgsf" event={"ID":"be05f6a4-ecbc-4030-a469-a4dd4cf3570c","Type":"ContainerStarted","Data":"3cb1b727f1a24dc9c45f9b854cc0c6720366086ec30e7c2b739fe3bc8873b64f"}
Sep 17 18:30:17.588844 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:30:17.588755    2553 generic.go:356] "Generic (PLEG): container finished" podID="be05f6a4-ecbc-4030-a469-a4dd4cf3570c" containerID="caa8d268b60800a69cac317e26942405ea8e3b2ab024d84682cea8080badf0c7" exitCode=0
Sep 17 18:30:17.588844 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:30:17.588804    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4fgsf" event={"ID":"be05f6a4-ecbc-4030-a469-a4dd4cf3570c","Type":"ContainerDied","Data":"caa8d268b60800a69cac317e26942405ea8e3b2ab024d84682cea8080badf0c7"}
Sep 17 18:30:21.610727 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:30:21.610692    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4fgsf" event={"ID":"be05f6a4-ecbc-4030-a469-a4dd4cf3570c","Type":"ContainerStarted","Data":"9ab7a8d398b4f46ce93bd51c6fda69329ccde3b7725adf14ec0ab7ccf6e92a1c"}
Sep 17 18:30:21.611137 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:30:21.610756    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4fgsf" event={"ID":"be05f6a4-ecbc-4030-a469-a4dd4cf3570c","Type":"ContainerStarted","Data":"6dc9d7325fbfde93b302080ab42ebaf53cd5be3704f8894752ef2a6a8310a505"}
Sep 17 18:30:21.611137 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:30:21.611036    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4fgsf"
Sep 17 18:30:21.611282 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:30:21.611193    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4fgsf"
Sep 17 18:30:21.612436 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:30:21.612407    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4fgsf" podUID="be05f6a4-ecbc-4030-a469-a4dd4cf3570c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 18:30:21.630910 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:30:21.630862    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4fgsf" podStartSLOduration=2.3726055280000002 podStartE2EDuration="10.630846953s" totalImagesPullingTime="3.533693211s" totalInitContainerRuntime="4.724548214s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:21.629133885 +0000 UTC m=+2016.325224575" watchObservedRunningTime="2026-09-17 18:30:21.630846953 +0000 UTC m=+2016.326937635"
Sep 17 18:30:22.408368 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:30:22.408324    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-pxfp9" podUID="2e5d8da2-d685-457a-925a-370375c22a60" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 18:30:22.615141 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:30:22.615103    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4fgsf" podUID="be05f6a4-ecbc-4030-a469-a4dd4cf3570c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 18:30:27.619373 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:30:27.619335    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4fgsf"
Sep 17 18:30:27.619907 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:30:27.619880    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4fgsf" podUID="be05f6a4-ecbc-4030-a469-a4dd4cf3570c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 18:30:32.408781 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:30:32.408738    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-pxfp9" podUID="2e5d8da2-d685-457a-925a-370375c22a60" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 18:30:37.620019 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:30:37.619989    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4fgsf"
Sep 17 18:30:42.408287 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:30:42.408244    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-pxfp9" podUID="2e5d8da2-d685-457a-925a-370375c22a60" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 18:30:52.366819 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:30:52.366785    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4fgsf"]
Sep 17 18:30:52.367239 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:30:52.367013    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4fgsf" podUID="be05f6a4-ecbc-4030-a469-a4dd4cf3570c" containerName="kserve-container" containerID="cri-o://6dc9d7325fbfde93b302080ab42ebaf53cd5be3704f8894752ef2a6a8310a505" gracePeriod=30
Sep 17 18:30:52.367239 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:30:52.367081    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4fgsf" podUID="be05f6a4-ecbc-4030-a469-a4dd4cf3570c" containerName="kube-rbac-proxy" containerID="cri-o://9ab7a8d398b4f46ce93bd51c6fda69329ccde3b7725adf14ec0ab7ccf6e92a1c" gracePeriod=30
Sep 17 18:30:52.408675 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:30:52.408645    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-pxfp9"
Sep 17 18:30:52.615506 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:30:52.615462    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4fgsf" podUID="be05f6a4-ecbc-4030-a469-a4dd4cf3570c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Sep 17 18:30:52.754273 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:30:52.754241    2553 generic.go:356] "Generic (PLEG): container finished" podID="be05f6a4-ecbc-4030-a469-a4dd4cf3570c" containerID="9ab7a8d398b4f46ce93bd51c6fda69329ccde3b7725adf14ec0ab7ccf6e92a1c" exitCode=2
Sep 17 18:30:52.754453 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:30:52.754310    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4fgsf" event={"ID":"be05f6a4-ecbc-4030-a469-a4dd4cf3570c","Type":"ContainerDied","Data":"9ab7a8d398b4f46ce93bd51c6fda69329ccde3b7725adf14ec0ab7ccf6e92a1c"}
Sep 17 18:30:57.615803 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:30:57.615764    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4fgsf" podUID="be05f6a4-ecbc-4030-a469-a4dd4cf3570c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Sep 17 18:31:00.332977 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:00.332898    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-pxfp9"]
Sep 17 18:31:00.333387 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:00.333142    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-pxfp9" podUID="2e5d8da2-d685-457a-925a-370375c22a60" containerName="kserve-container" containerID="cri-o://cdb0e48664a00375bb3f2ab347c4fc460e6049484c6f1edad53bc37ded00fbbf" gracePeriod=30
Sep 17 18:31:00.333387 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:00.333249    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-pxfp9" podUID="2e5d8da2-d685-457a-925a-370375c22a60" containerName="kube-rbac-proxy" containerID="cri-o://119682562fbe631bee39f9d2f45c0426bbe8ebe2186a2ac58df86d5111208f1a" gracePeriod=30
Sep 17 18:31:00.791676 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:00.791639    2553 generic.go:356] "Generic (PLEG): container finished" podID="2e5d8da2-d685-457a-925a-370375c22a60" containerID="119682562fbe631bee39f9d2f45c0426bbe8ebe2186a2ac58df86d5111208f1a" exitCode=2
Sep 17 18:31:00.791857 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:00.791712    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-pxfp9" event={"ID":"2e5d8da2-d685-457a-925a-370375c22a60","Type":"ContainerDied","Data":"119682562fbe631bee39f9d2f45c0426bbe8ebe2186a2ac58df86d5111208f1a"}
Sep 17 18:31:02.402989 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:02.402943    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-pxfp9" podUID="2e5d8da2-d685-457a-925a-370375c22a60" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.42:8643/healthz\": dial tcp 10.134.0.42:8643: connect: connection refused"
Sep 17 18:31:02.408308 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:02.408281    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-pxfp9" podUID="2e5d8da2-d685-457a-925a-370375c22a60" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 18:31:02.615604 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:02.615551    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4fgsf" podUID="be05f6a4-ecbc-4030-a469-a4dd4cf3570c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Sep 17 18:31:02.615829 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:02.615762    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4fgsf"
Sep 17 18:31:05.275679 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:05.275653    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-pxfp9"
Sep 17 18:31:05.414672 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:05.414639    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e5d8da2-d685-457a-925a-370375c22a60-kserve-provision-location\") pod \"2e5d8da2-d685-457a-925a-370375c22a60\" (UID: \"2e5d8da2-d685-457a-925a-370375c22a60\") "
Sep 17 18:31:05.414851 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:05.414698    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e5d8da2-d685-457a-925a-370375c22a60-kube-rbac-proxy-sar-config\") pod \"2e5d8da2-d685-457a-925a-370375c22a60\" (UID: \"2e5d8da2-d685-457a-925a-370375c22a60\") "
Sep 17 18:31:05.414851 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:05.414735    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e5d8da2-d685-457a-925a-370375c22a60-cabundle-cert\") pod \"2e5d8da2-d685-457a-925a-370375c22a60\" (UID: \"2e5d8da2-d685-457a-925a-370375c22a60\") "
Sep 17 18:31:05.414851 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:05.414796    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e5d8da2-d685-457a-925a-370375c22a60-proxy-tls\") pod \"2e5d8da2-d685-457a-925a-370375c22a60\" (UID: \"2e5d8da2-d685-457a-925a-370375c22a60\") "
Sep 17 18:31:05.414851 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:05.414829    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fh5kl\" (UniqueName: \"kubernetes.io/projected/2e5d8da2-d685-457a-925a-370375c22a60-kube-api-access-fh5kl\") pod \"2e5d8da2-d685-457a-925a-370375c22a60\" (UID: \"2e5d8da2-d685-457a-925a-370375c22a60\") "
Sep 17 18:31:05.415014 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:05.414853    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e5d8da2-d685-457a-925a-370375c22a60-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2e5d8da2-d685-457a-925a-370375c22a60" (UID: "2e5d8da2-d685-457a-925a-370375c22a60"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:31:05.415078 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:05.415055    2553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e5d8da2-d685-457a-925a-370375c22a60-kserve-provision-location\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:31:05.415205 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:05.415101    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e5d8da2-d685-457a-925a-370375c22a60-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2e5d8da2-d685-457a-925a-370375c22a60" (UID: "2e5d8da2-d685-457a-925a-370375c22a60"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:31:05.415253 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:05.415207    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e5d8da2-d685-457a-925a-370375c22a60-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2e5d8da2-d685-457a-925a-370375c22a60" (UID: "2e5d8da2-d685-457a-925a-370375c22a60"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:31:05.416926 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:05.416897    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e5d8da2-d685-457a-925a-370375c22a60-kube-api-access-fh5kl" (OuterVolumeSpecName: "kube-api-access-fh5kl") pod "2e5d8da2-d685-457a-925a-370375c22a60" (UID: "2e5d8da2-d685-457a-925a-370375c22a60"). InnerVolumeSpecName "kube-api-access-fh5kl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:31:05.417001 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:05.416967    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e5d8da2-d685-457a-925a-370375c22a60-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2e5d8da2-d685-457a-925a-370375c22a60" (UID: "2e5d8da2-d685-457a-925a-370375c22a60"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:31:05.515936 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:05.515898    2553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e5d8da2-d685-457a-925a-370375c22a60-proxy-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:31:05.515936 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:05.515936    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fh5kl\" (UniqueName: \"kubernetes.io/projected/2e5d8da2-d685-457a-925a-370375c22a60-kube-api-access-fh5kl\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:31:05.515936 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:05.515949    2553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e5d8da2-d685-457a-925a-370375c22a60-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:31:05.516208 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:05.515962    2553 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e5d8da2-d685-457a-925a-370375c22a60-cabundle-cert\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:31:05.562065 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:05.562024    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-v5z84"]
Sep 17 18:31:05.562354 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:05.562342    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e5d8da2-d685-457a-925a-370375c22a60" containerName="storage-initializer"
Sep 17 18:31:05.562396 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:05.562356    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e5d8da2-d685-457a-925a-370375c22a60" containerName="storage-initializer"
Sep 17 18:31:05.562396 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:05.562367    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e5d8da2-d685-457a-925a-370375c22a60" containerName="kube-rbac-proxy"
Sep 17 18:31:05.562396 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:05.562373    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e5d8da2-d685-457a-925a-370375c22a60" containerName="kube-rbac-proxy"
Sep 17 18:31:05.562396 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:05.562383    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e5d8da2-d685-457a-925a-370375c22a60" containerName="kserve-container"
Sep 17 18:31:05.562396 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:05.562388    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e5d8da2-d685-457a-925a-370375c22a60" containerName="kserve-container"
Sep 17 18:31:05.562549 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:05.562436    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e5d8da2-d685-457a-925a-370375c22a60" containerName="kserve-container"
Sep 17 18:31:05.562549 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:05.562446    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e5d8da2-d685-457a-925a-370375c22a60" containerName="kube-rbac-proxy"
Sep 17 18:31:05.565150 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:05.565133    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-v5z84"
Sep 17 18:31:05.567411 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:05.567388    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 17 18:31:05.567530 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:05.567478    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 17 18:31:05.574849 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:05.574816    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-v5z84"]
Sep 17 18:31:05.717800 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:05.717702    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d081844-ec04-4c66-b982-eddd6ac59a46-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-v5z84\" (UID: \"0d081844-ec04-4c66-b982-eddd6ac59a46\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-v5z84"
Sep 17 18:31:05.717800 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:05.717755    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d081844-ec04-4c66-b982-eddd6ac59a46-proxy-tls\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-v5z84\" (UID: \"0d081844-ec04-4c66-b982-eddd6ac59a46\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-v5z84"
Sep 17 18:31:05.717800 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:05.717799    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhlrq\" (UniqueName: \"kubernetes.io/projected/0d081844-ec04-4c66-b982-eddd6ac59a46-kube-api-access-jhlrq\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-v5z84\" (UID: \"0d081844-ec04-4c66-b982-eddd6ac59a46\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-v5z84"
Sep 17 18:31:05.718065 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:05.717817    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d081844-ec04-4c66-b982-eddd6ac59a46-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-v5z84\" (UID: \"0d081844-ec04-4c66-b982-eddd6ac59a46\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-v5z84"
Sep 17 18:31:05.718065 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:05.717835    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d081844-ec04-4c66-b982-eddd6ac59a46-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-v5z84\" (UID: \"0d081844-ec04-4c66-b982-eddd6ac59a46\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-v5z84"
Sep 17 18:31:05.815444 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:05.815413    2553 generic.go:356] "Generic (PLEG): container finished" podID="2e5d8da2-d685-457a-925a-370375c22a60" containerID="cdb0e48664a00375bb3f2ab347c4fc460e6049484c6f1edad53bc37ded00fbbf" exitCode=0
Sep 17 18:31:05.815637 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:05.815466    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-pxfp9"
Sep 17 18:31:05.815637 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:05.815477    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-pxfp9" event={"ID":"2e5d8da2-d685-457a-925a-370375c22a60","Type":"ContainerDied","Data":"cdb0e48664a00375bb3f2ab347c4fc460e6049484c6f1edad53bc37ded00fbbf"}
Sep 17 18:31:05.815637 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:05.815527    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-pxfp9" event={"ID":"2e5d8da2-d685-457a-925a-370375c22a60","Type":"ContainerDied","Data":"3b84e3df8144a920c576c4aa0ea131744e2cb1f080de4b2d4782a2fe925c8894"}
Sep 17 18:31:05.815637 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:05.815555    2553 scope.go:117] "RemoveContainer" containerID="119682562fbe631bee39f9d2f45c0426bbe8ebe2186a2ac58df86d5111208f1a"
Sep 17 18:31:05.818330 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:05.818310    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d081844-ec04-4c66-b982-eddd6ac59a46-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-v5z84\" (UID: \"0d081844-ec04-4c66-b982-eddd6ac59a46\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-v5z84"
Sep 17 18:31:05.818440 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:05.818341    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d081844-ec04-4c66-b982-eddd6ac59a46-proxy-tls\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-v5z84\" (UID: \"0d081844-ec04-4c66-b982-eddd6ac59a46\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-v5z84"
Sep 17 18:31:05.818440 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:05.818363    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhlrq\" (UniqueName: \"kubernetes.io/projected/0d081844-ec04-4c66-b982-eddd6ac59a46-kube-api-access-jhlrq\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-v5z84\" (UID: \"0d081844-ec04-4c66-b982-eddd6ac59a46\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-v5z84"
Sep 17 18:31:05.818553 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:31:05.818445    2553 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 17 18:31:05.818553 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:05.818482    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d081844-ec04-4c66-b982-eddd6ac59a46-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-v5z84\" (UID: \"0d081844-ec04-4c66-b982-eddd6ac59a46\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-v5z84"
Sep 17 18:31:05.818553 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:31:05.818512    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d081844-ec04-4c66-b982-eddd6ac59a46-proxy-tls podName:0d081844-ec04-4c66-b982-eddd6ac59a46 nodeName:}" failed. No retries permitted until 2026-09-17 18:31:06.318491269 +0000 UTC m=+2061.014581944 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0d081844-ec04-4c66-b982-eddd6ac59a46-proxy-tls") pod "isvc-canary-rollback-predictor-7df7cf66bf-v5z84" (UID: "0d081844-ec04-4c66-b982-eddd6ac59a46") : secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 17 18:31:05.818553 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:05.818542    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d081844-ec04-4c66-b982-eddd6ac59a46-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-v5z84\" (UID: \"0d081844-ec04-4c66-b982-eddd6ac59a46\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-v5z84"
Sep 17 18:31:05.819010 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:05.818964    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d081844-ec04-4c66-b982-eddd6ac59a46-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-v5z84\" (UID: \"0d081844-ec04-4c66-b982-eddd6ac59a46\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-v5z84"
Sep 17 18:31:05.819096 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:05.819072    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d081844-ec04-4c66-b982-eddd6ac59a46-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-v5z84\" (UID: \"0d081844-ec04-4c66-b982-eddd6ac59a46\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-v5z84"
Sep 17 18:31:05.819248 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:05.819227    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d081844-ec04-4c66-b982-eddd6ac59a46-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-v5z84\" (UID: \"0d081844-ec04-4c66-b982-eddd6ac59a46\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-v5z84"
Sep 17 18:31:05.824050 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:05.824033    2553 scope.go:117] "RemoveContainer" containerID="cdb0e48664a00375bb3f2ab347c4fc460e6049484c6f1edad53bc37ded00fbbf"
Sep 17 18:31:05.827563 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:05.827547    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhlrq\" (UniqueName: \"kubernetes.io/projected/0d081844-ec04-4c66-b982-eddd6ac59a46-kube-api-access-jhlrq\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-v5z84\" (UID: \"0d081844-ec04-4c66-b982-eddd6ac59a46\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-v5z84"
Sep 17 18:31:05.831755 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:05.831738    2553 scope.go:117] "RemoveContainer" containerID="61b1696bbac74cbccbf654cbadd2ce534f83e7bf5c24f1c7a39b098bf5a0beed"
Sep 17 18:31:05.836786 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:05.836765    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-pxfp9"]
Sep 17 18:31:05.840119 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:05.840103    2553 scope.go:117] "RemoveContainer" containerID="119682562fbe631bee39f9d2f45c0426bbe8ebe2186a2ac58df86d5111208f1a"
Sep 17 18:31:05.840401 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:31:05.840383    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"119682562fbe631bee39f9d2f45c0426bbe8ebe2186a2ac58df86d5111208f1a\": container with ID starting with 119682562fbe631bee39f9d2f45c0426bbe8ebe2186a2ac58df86d5111208f1a not found: ID does not exist" containerID="119682562fbe631bee39f9d2f45c0426bbe8ebe2186a2ac58df86d5111208f1a"
Sep 17 18:31:05.840471 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:05.840413    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"119682562fbe631bee39f9d2f45c0426bbe8ebe2186a2ac58df86d5111208f1a"} err="failed to get container status \"119682562fbe631bee39f9d2f45c0426bbe8ebe2186a2ac58df86d5111208f1a\": rpc error: code = NotFound desc = could not find container \"119682562fbe631bee39f9d2f45c0426bbe8ebe2186a2ac58df86d5111208f1a\": container with ID starting with 119682562fbe631bee39f9d2f45c0426bbe8ebe2186a2ac58df86d5111208f1a not found: ID does not exist"
Sep 17 18:31:05.840471 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:05.840437    2553 scope.go:117] "RemoveContainer" containerID="cdb0e48664a00375bb3f2ab347c4fc460e6049484c6f1edad53bc37ded00fbbf"
Sep 17 18:31:05.840776 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:31:05.840750    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdb0e48664a00375bb3f2ab347c4fc460e6049484c6f1edad53bc37ded00fbbf\": container with ID starting with cdb0e48664a00375bb3f2ab347c4fc460e6049484c6f1edad53bc37ded00fbbf not found: ID does not exist" containerID="cdb0e48664a00375bb3f2ab347c4fc460e6049484c6f1edad53bc37ded00fbbf"
Sep 17 18:31:05.840867 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:05.840785    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdb0e48664a00375bb3f2ab347c4fc460e6049484c6f1edad53bc37ded00fbbf"} err="failed to get container status \"cdb0e48664a00375bb3f2ab347c4fc460e6049484c6f1edad53bc37ded00fbbf\": rpc error: code = NotFound desc = could not find container \"cdb0e48664a00375bb3f2ab347c4fc460e6049484c6f1edad53bc37ded00fbbf\": container with ID starting with cdb0e48664a00375bb3f2ab347c4fc460e6049484c6f1edad53bc37ded00fbbf not found: ID does not exist"
Sep 17 18:31:05.840867 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:05.840804    2553 scope.go:117] "RemoveContainer" containerID="61b1696bbac74cbccbf654cbadd2ce534f83e7bf5c24f1c7a39b098bf5a0beed"
Sep 17 18:31:05.840990 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:05.840971    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6b666fbfdb-pxfp9"]
Sep 17 18:31:05.841055 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:31:05.841035    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61b1696bbac74cbccbf654cbadd2ce534f83e7bf5c24f1c7a39b098bf5a0beed\": container with ID starting with 61b1696bbac74cbccbf654cbadd2ce534f83e7bf5c24f1c7a39b098bf5a0beed not found: ID does not exist" containerID="61b1696bbac74cbccbf654cbadd2ce534f83e7bf5c24f1c7a39b098bf5a0beed"
Sep 17 18:31:05.841106 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:05.841062    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61b1696bbac74cbccbf654cbadd2ce534f83e7bf5c24f1c7a39b098bf5a0beed"} err="failed to get container status \"61b1696bbac74cbccbf654cbadd2ce534f83e7bf5c24f1c7a39b098bf5a0beed\": rpc error: code = NotFound desc = could not find container \"61b1696bbac74cbccbf654cbadd2ce534f83e7bf5c24f1c7a39b098bf5a0beed\": container with ID starting with 61b1696bbac74cbccbf654cbadd2ce534f83e7bf5c24f1c7a39b098bf5a0beed not found: ID does not exist"
Sep 17 18:31:06.321805 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:06.321767    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d081844-ec04-4c66-b982-eddd6ac59a46-proxy-tls\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-v5z84\" (UID: \"0d081844-ec04-4c66-b982-eddd6ac59a46\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-v5z84"
Sep 17 18:31:06.324238 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:06.324203    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d081844-ec04-4c66-b982-eddd6ac59a46-proxy-tls\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-v5z84\" (UID: \"0d081844-ec04-4c66-b982-eddd6ac59a46\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-v5z84"
Sep 17 18:31:06.476888 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:06.476843    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-v5z84"
Sep 17 18:31:06.750695 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:06.750660    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-v5z84"]
Sep 17 18:31:06.753877 ip-10-0-137-95 kubenswrapper[2553]: W0917 18:31:06.753847    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d081844_ec04_4c66_b982_eddd6ac59a46.slice/crio-c7d9541c221b3c842502e23f5a2c14f9f24db5536a869f7ce9fe427b346e96a1 WatchSource:0}: Error finding container c7d9541c221b3c842502e23f5a2c14f9f24db5536a869f7ce9fe427b346e96a1: Status 404 returned error can't find the container with id c7d9541c221b3c842502e23f5a2c14f9f24db5536a869f7ce9fe427b346e96a1
Sep 17 18:31:06.822288 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:06.822237    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-v5z84" event={"ID":"0d081844-ec04-4c66-b982-eddd6ac59a46","Type":"ContainerStarted","Data":"6554c236cf44ac55fe7a65d8bff1e98f1438be8dc825d848a310b062b66cda67"}
Sep 17 18:31:06.822450 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:06.822314    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-v5z84" event={"ID":"0d081844-ec04-4c66-b982-eddd6ac59a46","Type":"ContainerStarted","Data":"c7d9541c221b3c842502e23f5a2c14f9f24db5536a869f7ce9fe427b346e96a1"}
Sep 17 18:31:07.376363 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:07.376329    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e5d8da2-d685-457a-925a-370375c22a60" path="/var/lib/kubelet/pods/2e5d8da2-d685-457a-925a-370375c22a60/volumes"
Sep 17 18:31:07.616120 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:07.616079    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4fgsf" podUID="be05f6a4-ecbc-4030-a469-a4dd4cf3570c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Sep 17 18:31:10.841511 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:10.841479    2553 generic.go:356] "Generic (PLEG): container finished" podID="0d081844-ec04-4c66-b982-eddd6ac59a46" containerID="6554c236cf44ac55fe7a65d8bff1e98f1438be8dc825d848a310b062b66cda67" exitCode=0
Sep 17 18:31:10.841952 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:10.841552    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-v5z84" event={"ID":"0d081844-ec04-4c66-b982-eddd6ac59a46","Type":"ContainerDied","Data":"6554c236cf44ac55fe7a65d8bff1e98f1438be8dc825d848a310b062b66cda67"}
Sep 17 18:31:11.849041 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:11.849007    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-v5z84" event={"ID":"0d081844-ec04-4c66-b982-eddd6ac59a46","Type":"ContainerStarted","Data":"e053f27dc25f065400229c51522471215e03221f95c69635aea2c777827643b9"}
Sep 17 18:31:11.849451 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:11.849067    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-v5z84" event={"ID":"0d081844-ec04-4c66-b982-eddd6ac59a46","Type":"ContainerStarted","Data":"a3baf14c788d8d581e32153fbb3c691b304e339a4124ef01fd5b11656a31f54b"}
Sep 17 18:31:11.849451 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:11.849355    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-v5z84"
Sep 17 18:31:11.849554 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:11.849520    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-v5z84"
Sep 17 18:31:11.851041 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:11.851010    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-v5z84" podUID="0d081844-ec04-4c66-b982-eddd6ac59a46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 17 18:31:11.869941 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:11.869886    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-v5z84" podStartSLOduration=3.037342764 podStartE2EDuration="6.869752415s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.832409651s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:31:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:31:11.866864447 +0000 UTC m=+2066.562955141" watchObservedRunningTime="2026-09-17 18:31:11.869752415 +0000 UTC m=+2066.565843098"
Sep 17 18:31:12.616127 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:12.616080    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4fgsf" podUID="be05f6a4-ecbc-4030-a469-a4dd4cf3570c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Sep 17 18:31:12.853786 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:12.853744    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-v5z84" podUID="0d081844-ec04-4c66-b982-eddd6ac59a46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 17 18:31:17.615818 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:17.615779    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4fgsf" podUID="be05f6a4-ecbc-4030-a469-a4dd4cf3570c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Sep 17 18:31:17.858348 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:17.858318    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-v5z84"
Sep 17 18:31:17.858938 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:17.858905    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-v5z84" podUID="0d081844-ec04-4c66-b982-eddd6ac59a46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 17 18:31:22.615674 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:22.615608    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4fgsf" podUID="be05f6a4-ecbc-4030-a469-a4dd4cf3570c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Sep 17 18:31:22.897936 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:22.897904    2553 generic.go:356] "Generic (PLEG): container finished" podID="be05f6a4-ecbc-4030-a469-a4dd4cf3570c" containerID="6dc9d7325fbfde93b302080ab42ebaf53cd5be3704f8894752ef2a6a8310a505" exitCode=137
Sep 17 18:31:22.898099 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:22.897983    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4fgsf" event={"ID":"be05f6a4-ecbc-4030-a469-a4dd4cf3570c","Type":"ContainerDied","Data":"6dc9d7325fbfde93b302080ab42ebaf53cd5be3704f8894752ef2a6a8310a505"}
Sep 17 18:31:23.013174 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:23.013151    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4fgsf"
Sep 17 18:31:23.058705 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:23.058671    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g72lx\" (UniqueName: \"kubernetes.io/projected/be05f6a4-ecbc-4030-a469-a4dd4cf3570c-kube-api-access-g72lx\") pod \"be05f6a4-ecbc-4030-a469-a4dd4cf3570c\" (UID: \"be05f6a4-ecbc-4030-a469-a4dd4cf3570c\") "
Sep 17 18:31:23.058876 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:23.058757    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be05f6a4-ecbc-4030-a469-a4dd4cf3570c-cabundle-cert\") pod \"be05f6a4-ecbc-4030-a469-a4dd4cf3570c\" (UID: \"be05f6a4-ecbc-4030-a469-a4dd4cf3570c\") "
Sep 17 18:31:23.058876 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:23.058787    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be05f6a4-ecbc-4030-a469-a4dd4cf3570c-kserve-provision-location\") pod \"be05f6a4-ecbc-4030-a469-a4dd4cf3570c\" (UID: \"be05f6a4-ecbc-4030-a469-a4dd4cf3570c\") "
Sep 17 18:31:23.058876 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:23.058804    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be05f6a4-ecbc-4030-a469-a4dd4cf3570c-proxy-tls\") pod \"be05f6a4-ecbc-4030-a469-a4dd4cf3570c\" (UID: \"be05f6a4-ecbc-4030-a469-a4dd4cf3570c\") "
Sep 17 18:31:23.058876 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:23.058818    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be05f6a4-ecbc-4030-a469-a4dd4cf3570c-kube-rbac-proxy-sar-config\") pod \"be05f6a4-ecbc-4030-a469-a4dd4cf3570c\" (UID: \"be05f6a4-ecbc-4030-a469-a4dd4cf3570c\") "
Sep 17 18:31:23.059186 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:23.059159    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be05f6a4-ecbc-4030-a469-a4dd4cf3570c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "be05f6a4-ecbc-4030-a469-a4dd4cf3570c" (UID: "be05f6a4-ecbc-4030-a469-a4dd4cf3570c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:31:23.059264 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:23.059236    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be05f6a4-ecbc-4030-a469-a4dd4cf3570c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "be05f6a4-ecbc-4030-a469-a4dd4cf3570c" (UID: "be05f6a4-ecbc-4030-a469-a4dd4cf3570c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:31:23.060886 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:23.060857    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be05f6a4-ecbc-4030-a469-a4dd4cf3570c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "be05f6a4-ecbc-4030-a469-a4dd4cf3570c" (UID: "be05f6a4-ecbc-4030-a469-a4dd4cf3570c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:31:23.060968 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:23.060939    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be05f6a4-ecbc-4030-a469-a4dd4cf3570c-kube-api-access-g72lx" (OuterVolumeSpecName: "kube-api-access-g72lx") pod "be05f6a4-ecbc-4030-a469-a4dd4cf3570c" (UID: "be05f6a4-ecbc-4030-a469-a4dd4cf3570c"). InnerVolumeSpecName "kube-api-access-g72lx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:31:23.068649 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:23.068612    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be05f6a4-ecbc-4030-a469-a4dd4cf3570c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "be05f6a4-ecbc-4030-a469-a4dd4cf3570c" (UID: "be05f6a4-ecbc-4030-a469-a4dd4cf3570c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:31:23.160166 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:23.160132    2553 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be05f6a4-ecbc-4030-a469-a4dd4cf3570c-cabundle-cert\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:31:23.160166 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:23.160163    2553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be05f6a4-ecbc-4030-a469-a4dd4cf3570c-kserve-provision-location\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:31:23.160166 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:23.160173    2553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be05f6a4-ecbc-4030-a469-a4dd4cf3570c-proxy-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:31:23.160398 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:23.160182    2553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be05f6a4-ecbc-4030-a469-a4dd4cf3570c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:31:23.160398 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:23.160191    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g72lx\" (UniqueName: \"kubernetes.io/projected/be05f6a4-ecbc-4030-a469-a4dd4cf3570c-kube-api-access-g72lx\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:31:23.904763 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:23.904729    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4fgsf" event={"ID":"be05f6a4-ecbc-4030-a469-a4dd4cf3570c","Type":"ContainerDied","Data":"3cb1b727f1a24dc9c45f9b854cc0c6720366086ec30e7c2b739fe3bc8873b64f"}
Sep 17 18:31:23.905151 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:23.904774    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4fgsf"
Sep 17 18:31:23.905151 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:23.904799    2553 scope.go:117] "RemoveContainer" containerID="9ab7a8d398b4f46ce93bd51c6fda69329ccde3b7725adf14ec0ab7ccf6e92a1c"
Sep 17 18:31:23.913208 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:23.913189    2553 scope.go:117] "RemoveContainer" containerID="6dc9d7325fbfde93b302080ab42ebaf53cd5be3704f8894752ef2a6a8310a505"
Sep 17 18:31:23.920296 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:23.920270    2553 scope.go:117] "RemoveContainer" containerID="caa8d268b60800a69cac317e26942405ea8e3b2ab024d84682cea8080badf0c7"
Sep 17 18:31:23.924775 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:23.924756    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4fgsf"]
Sep 17 18:31:23.928460 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:23.928439    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4fgsf"]
Sep 17 18:31:25.376183 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:25.376152    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be05f6a4-ecbc-4030-a469-a4dd4cf3570c" path="/var/lib/kubelet/pods/be05f6a4-ecbc-4030-a469-a4dd4cf3570c/volumes"
Sep 17 18:31:27.858969 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:27.858932    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-v5z84" podUID="0d081844-ec04-4c66-b982-eddd6ac59a46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 17 18:31:37.858988 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:37.858943    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-v5z84" podUID="0d081844-ec04-4c66-b982-eddd6ac59a46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 17 18:31:44.291410 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:44.291372    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7kj5n"]
Sep 17 18:31:44.291917 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:44.291742    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be05f6a4-ecbc-4030-a469-a4dd4cf3570c" containerName="kserve-container"
Sep 17 18:31:44.291917 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:44.291755    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="be05f6a4-ecbc-4030-a469-a4dd4cf3570c" containerName="kserve-container"
Sep 17 18:31:44.291917 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:44.291773    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be05f6a4-ecbc-4030-a469-a4dd4cf3570c" containerName="storage-initializer"
Sep 17 18:31:44.291917 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:44.291780    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="be05f6a4-ecbc-4030-a469-a4dd4cf3570c" containerName="storage-initializer"
Sep 17 18:31:44.291917 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:44.291786    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be05f6a4-ecbc-4030-a469-a4dd4cf3570c" containerName="kube-rbac-proxy"
Sep 17 18:31:44.291917 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:44.291792    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="be05f6a4-ecbc-4030-a469-a4dd4cf3570c" containerName="kube-rbac-proxy"
Sep 17 18:31:44.291917 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:44.291844    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="be05f6a4-ecbc-4030-a469-a4dd4cf3570c" containerName="kube-rbac-proxy"
Sep 17 18:31:44.291917 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:44.291852    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="be05f6a4-ecbc-4030-a469-a4dd4cf3570c" containerName="kserve-container"
Sep 17 18:31:44.294950 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:44.294931    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7kj5n"
Sep 17 18:31:44.297712 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:44.297682    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-predictor-serving-cert\""
Sep 17 18:31:44.297712 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:44.297689    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-kube-rbac-proxy-sar-config\""
Sep 17 18:31:44.306428 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:44.306401    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7kj5n"]
Sep 17 18:31:44.442713 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:44.442671    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-7kj5n\" (UID: \"2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7kj5n"
Sep 17 18:31:44.442873 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:44.442779    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-7kj5n\" (UID: \"2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7kj5n"
Sep 17 18:31:44.442873 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:44.442815    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nblx\" (UniqueName: \"kubernetes.io/projected/2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e-kube-api-access-2nblx\") pod \"isvc-triton-predictor-5d7db58676-7kj5n\" (UID: \"2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7kj5n"
Sep 17 18:31:44.442985 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:44.442882    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-7kj5n\" (UID: \"2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7kj5n"
Sep 17 18:31:44.442985 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:44.442941    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-7kj5n\" (UID: \"2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7kj5n"
Sep 17 18:31:44.544223 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:44.544120    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-7kj5n\" (UID: \"2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7kj5n"
Sep 17 18:31:44.544223 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:44.544169    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nblx\" (UniqueName: \"kubernetes.io/projected/2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e-kube-api-access-2nblx\") pod \"isvc-triton-predictor-5d7db58676-7kj5n\" (UID: \"2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7kj5n"
Sep 17 18:31:44.544223 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:44.544210    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-7kj5n\" (UID: \"2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7kj5n"
Sep 17 18:31:44.544511 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:44.544239    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-7kj5n\" (UID: \"2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7kj5n"
Sep 17 18:31:44.544511 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:44.544293    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-7kj5n\" (UID: \"2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7kj5n"
Sep 17 18:31:44.544614 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:44.544597    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-7kj5n\" (UID: \"2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7kj5n"
Sep 17 18:31:44.545015 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:44.544994    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-7kj5n\" (UID: \"2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7kj5n"
Sep 17 18:31:44.545101 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:44.544993    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-7kj5n\" (UID: \"2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7kj5n"
Sep 17 18:31:44.546929 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:44.546907    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-7kj5n\" (UID: \"2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7kj5n"
Sep 17 18:31:44.552883 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:44.552861    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nblx\" (UniqueName: \"kubernetes.io/projected/2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e-kube-api-access-2nblx\") pod \"isvc-triton-predictor-5d7db58676-7kj5n\" (UID: \"2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7kj5n"
Sep 17 18:31:44.605987 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:44.605933    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7kj5n"
Sep 17 18:31:44.883562 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:44.883533    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7kj5n"]
Sep 17 18:31:44.886215 ip-10-0-137-95 kubenswrapper[2553]: W0917 18:31:44.886189    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2acb8c70_7b6c_4f71_8183_74f0b4eb7c3e.slice/crio-aba1d01b79b0d8ad0c8b4d3650958639b35c4f3706c4c1a4ab7aba3629945b95 WatchSource:0}: Error finding container aba1d01b79b0d8ad0c8b4d3650958639b35c4f3706c4c1a4ab7aba3629945b95: Status 404 returned error can't find the container with id aba1d01b79b0d8ad0c8b4d3650958639b35c4f3706c4c1a4ab7aba3629945b95
Sep 17 18:31:45.004392 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:45.004350    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7kj5n" event={"ID":"2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e","Type":"ContainerStarted","Data":"cb9d9973ced9213afc35791433bb7adb1b38370946c3c556e441f164c619bba8"}
Sep 17 18:31:45.004593 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:45.004414    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7kj5n" event={"ID":"2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e","Type":"ContainerStarted","Data":"aba1d01b79b0d8ad0c8b4d3650958639b35c4f3706c4c1a4ab7aba3629945b95"}
Sep 17 18:31:47.431342 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:47.431310    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-btmqz_622ee5be-fb50-42b6-ae23-42900b7c2d98/ovn-acl-logging/0.log"
Sep 17 18:31:47.438799 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:47.438774    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-btmqz_622ee5be-fb50-42b6-ae23-42900b7c2d98/ovn-acl-logging/0.log"
Sep 17 18:31:47.858988 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:47.858874    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-v5z84" podUID="0d081844-ec04-4c66-b982-eddd6ac59a46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 17 18:31:49.024974 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:49.024938    2553 generic.go:356] "Generic (PLEG): container finished" podID="2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e" containerID="cb9d9973ced9213afc35791433bb7adb1b38370946c3c556e441f164c619bba8" exitCode=0
Sep 17 18:31:49.025381 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:49.024987    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7kj5n" event={"ID":"2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e","Type":"ContainerDied","Data":"cb9d9973ced9213afc35791433bb7adb1b38370946c3c556e441f164c619bba8"}
Sep 17 18:31:49.026138 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:49.026121    2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:31:57.859830 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:31:57.859377    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-v5z84" podUID="0d081844-ec04-4c66-b982-eddd6ac59a46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 17 18:32:07.859023 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:32:07.858977    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-v5z84" podUID="0d081844-ec04-4c66-b982-eddd6ac59a46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 17 18:32:17.859171 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:32:17.859127    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-v5z84" podUID="0d081844-ec04-4c66-b982-eddd6ac59a46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 17 18:32:23.371790 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:32:23.371744    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-v5z84" podUID="0d081844-ec04-4c66-b982-eddd6ac59a46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 17 18:32:28.895198 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:32:28.895160    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-v5z84"]
Sep 17 18:32:28.895800 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:32:28.895506    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-v5z84" podUID="0d081844-ec04-4c66-b982-eddd6ac59a46" containerName="kserve-container" containerID="cri-o://a3baf14c788d8d581e32153fbb3c691b304e339a4124ef01fd5b11656a31f54b" gracePeriod=30
Sep 17 18:32:28.895800 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:32:28.895721    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-v5z84" podUID="0d081844-ec04-4c66-b982-eddd6ac59a46" containerName="kube-rbac-proxy" containerID="cri-o://e053f27dc25f065400229c51522471215e03221f95c69635aea2c777827643b9" gracePeriod=30
Sep 17 18:32:28.895928 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:32:28.895895    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-v5z84" podUID="0d081844-ec04-4c66-b982-eddd6ac59a46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 17 18:32:29.314357 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:32:29.314062    2553 generic.go:356] "Generic (PLEG): container finished" podID="0d081844-ec04-4c66-b982-eddd6ac59a46" containerID="e053f27dc25f065400229c51522471215e03221f95c69635aea2c777827643b9" exitCode=2
Sep 17 18:32:29.314357 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:32:29.314154    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-v5z84" event={"ID":"0d081844-ec04-4c66-b982-eddd6ac59a46","Type":"ContainerDied","Data":"e053f27dc25f065400229c51522471215e03221f95c69635aea2c777827643b9"}
Sep 17 18:32:32.854713 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:32:32.854657    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-v5z84" podUID="0d081844-ec04-4c66-b982-eddd6ac59a46" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Sep 17 18:32:35.596261 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:32:35.596218    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-rp8kx"]
Sep 17 18:32:35.618051 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:32:35.618015    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-rp8kx"]
Sep 17 18:32:35.618240 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:32:35.618165    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-rp8kx"
Sep 17 18:32:35.654097 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:32:35.654051    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/add52769-2b2d-4f30-baba-da8457c1c67f-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-rp8kx\" (UID: \"add52769-2b2d-4f30-baba-da8457c1c67f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-rp8kx"
Sep 17 18:32:35.654320 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:32:35.654113    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9vjj\" (UniqueName: \"kubernetes.io/projected/add52769-2b2d-4f30-baba-da8457c1c67f-kube-api-access-p9vjj\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-rp8kx\" (UID: \"add52769-2b2d-4f30-baba-da8457c1c67f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-rp8kx"
Sep 17 18:32:35.654320 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:32:35.654187    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/add52769-2b2d-4f30-baba-da8457c1c67f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-rp8kx\" (UID: \"add52769-2b2d-4f30-baba-da8457c1c67f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-rp8kx"
Sep 17 18:32:35.654320 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:32:35.654239    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/add52769-2b2d-4f30-baba-da8457c1c67f-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-rp8kx\" (UID: \"add52769-2b2d-4f30-baba-da8457c1c67f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-rp8kx"
Sep 17 18:32:35.654320 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:32:35.654271    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/add52769-2b2d-4f30-baba-da8457c1c67f-proxy-tls\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-rp8kx\" (UID: \"add52769-2b2d-4f30-baba-da8457c1c67f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-rp8kx"
Sep 17 18:32:35.754945 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:32:35.754906    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/add52769-2b2d-4f30-baba-da8457c1c67f-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-rp8kx\" (UID: \"add52769-2b2d-4f30-baba-da8457c1c67f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-rp8kx"
Sep 17 18:32:35.755156 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:32:35.754975    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9vjj\" (UniqueName: \"kubernetes.io/projected/add52769-2b2d-4f30-baba-da8457c1c67f-kube-api-access-p9vjj\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-rp8kx\" (UID: \"add52769-2b2d-4f30-baba-da8457c1c67f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-rp8kx"
Sep 17 18:32:35.755156 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:32:35.755029    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/add52769-2b2d-4f30-baba-da8457c1c67f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-rp8kx\" (UID: \"add52769-2b2d-4f30-baba-da8457c1c67f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-rp8kx"
Sep 17 18:32:35.755156 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:32:35.755075    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/add52769-2b2d-4f30-baba-da8457c1c67f-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-rp8kx\" (UID: \"add52769-2b2d-4f30-baba-da8457c1c67f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-rp8kx"
Sep 17 18:32:35.755156 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:32:35.755123    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/add52769-2b2d-4f30-baba-da8457c1c67f-proxy-tls\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-rp8kx\" (UID: \"add52769-2b2d-4f30-baba-da8457c1c67f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-rp8kx"
Sep 17 18:32:35.755566 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:32:35.755537    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/add52769-2b2d-4f30-baba-da8457c1c67f-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-rp8kx\" (UID: \"add52769-2b2d-4f30-baba-da8457c1c67f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-rp8kx"
Sep 17 18:32:35.755846 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:32:35.755799    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/add52769-2b2d-4f30-baba-da8457c1c67f-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-rp8kx\" (UID: \"add52769-2b2d-4f30-baba-da8457c1c67f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-rp8kx"
Sep 17 18:32:35.755846 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:32:35.755836    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/add52769-2b2d-4f30-baba-da8457c1c67f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-rp8kx\" (UID: \"add52769-2b2d-4f30-baba-da8457c1c67f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-rp8kx"
Sep 17 18:32:35.758179 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:32:35.758155    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/add52769-2b2d-4f30-baba-da8457c1c67f-proxy-tls\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-rp8kx\" (UID: \"add52769-2b2d-4f30-baba-da8457c1c67f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-rp8kx"
Sep 17 18:32:35.763738 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:32:35.763713    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9vjj\" (UniqueName: \"kubernetes.io/projected/add52769-2b2d-4f30-baba-da8457c1c67f-kube-api-access-p9vjj\") pod \"isvc-canary-rollback-predictor-7df7cf66bf-rp8kx\" (UID: \"add52769-2b2d-4f30-baba-da8457c1c67f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-rp8kx"
Sep 17 18:32:35.933305 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:32:35.933194    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-rp8kx"
Sep 17 18:32:37.854224 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:32:37.854157    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-v5z84" podUID="0d081844-ec04-4c66-b982-eddd6ac59a46" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Sep 17 18:32:38.895874 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:32:38.895829    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-v5z84" podUID="0d081844-ec04-4c66-b982-eddd6ac59a46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 17 18:32:42.854186 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:32:42.854142    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-v5z84" podUID="0d081844-ec04-4c66-b982-eddd6ac59a46" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Sep 17 18:32:42.866949 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:32:42.854292    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-v5z84"
Sep 17 18:32:47.854883 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:32:47.854824    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-v5z84" podUID="0d081844-ec04-4c66-b982-eddd6ac59a46" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Sep 17 18:32:48.896858 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:32:48.896825    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-v5z84"
Sep 17 18:32:52.853963 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:32:52.853915    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-v5z84" podUID="0d081844-ec04-4c66-b982-eddd6ac59a46" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Sep 17 18:32:57.854688 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:32:57.854643    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-v5z84" podUID="0d081844-ec04-4c66-b982-eddd6ac59a46" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Sep 17 18:33:01.970613 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:01.970527    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-v5z84"
Sep 17 18:33:02.106713 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:02.106675    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d081844-ec04-4c66-b982-eddd6ac59a46-kserve-provision-location\") pod \"0d081844-ec04-4c66-b982-eddd6ac59a46\" (UID: \"0d081844-ec04-4c66-b982-eddd6ac59a46\") "
Sep 17 18:33:02.106894 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:02.106720    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jhlrq\" (UniqueName: \"kubernetes.io/projected/0d081844-ec04-4c66-b982-eddd6ac59a46-kube-api-access-jhlrq\") pod \"0d081844-ec04-4c66-b982-eddd6ac59a46\" (UID: \"0d081844-ec04-4c66-b982-eddd6ac59a46\") "
Sep 17 18:33:02.106894 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:02.106765    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d081844-ec04-4c66-b982-eddd6ac59a46-kube-rbac-proxy-sar-config\") pod \"0d081844-ec04-4c66-b982-eddd6ac59a46\" (UID: \"0d081844-ec04-4c66-b982-eddd6ac59a46\") "
Sep 17 18:33:02.106894 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:02.106807    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d081844-ec04-4c66-b982-eddd6ac59a46-proxy-tls\") pod \"0d081844-ec04-4c66-b982-eddd6ac59a46\" (UID: \"0d081844-ec04-4c66-b982-eddd6ac59a46\") "
Sep 17 18:33:02.106894 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:02.106827    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d081844-ec04-4c66-b982-eddd6ac59a46-cabundle-cert\") pod \"0d081844-ec04-4c66-b982-eddd6ac59a46\" (UID: \"0d081844-ec04-4c66-b982-eddd6ac59a46\") "
Sep 17 18:33:02.107089 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:02.107054    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d081844-ec04-4c66-b982-eddd6ac59a46-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0d081844-ec04-4c66-b982-eddd6ac59a46" (UID: "0d081844-ec04-4c66-b982-eddd6ac59a46"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:33:02.107192 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:02.107165    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d081844-ec04-4c66-b982-eddd6ac59a46-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0d081844-ec04-4c66-b982-eddd6ac59a46" (UID: "0d081844-ec04-4c66-b982-eddd6ac59a46"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:33:02.107258 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:02.107210    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d081844-ec04-4c66-b982-eddd6ac59a46-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0d081844-ec04-4c66-b982-eddd6ac59a46" (UID: "0d081844-ec04-4c66-b982-eddd6ac59a46"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:33:02.109071 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:02.109051    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d081844-ec04-4c66-b982-eddd6ac59a46-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0d081844-ec04-4c66-b982-eddd6ac59a46" (UID: "0d081844-ec04-4c66-b982-eddd6ac59a46"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:33:02.109163 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:02.109078    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d081844-ec04-4c66-b982-eddd6ac59a46-kube-api-access-jhlrq" (OuterVolumeSpecName: "kube-api-access-jhlrq") pod "0d081844-ec04-4c66-b982-eddd6ac59a46" (UID: "0d081844-ec04-4c66-b982-eddd6ac59a46"). InnerVolumeSpecName "kube-api-access-jhlrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:33:02.208268 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:02.208230    2553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d081844-ec04-4c66-b982-eddd6ac59a46-proxy-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:33:02.208268 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:02.208270    2553 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d081844-ec04-4c66-b982-eddd6ac59a46-cabundle-cert\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:33:02.208474 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:02.208285    2553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d081844-ec04-4c66-b982-eddd6ac59a46-kserve-provision-location\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:33:02.208474 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:02.208300    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhlrq\" (UniqueName: \"kubernetes.io/projected/0d081844-ec04-4c66-b982-eddd6ac59a46-kube-api-access-jhlrq\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:33:02.208474 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:02.208314    2553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d081844-ec04-4c66-b982-eddd6ac59a46-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:33:02.392808 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:02.392777    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-rp8kx"]
Sep 17 18:33:02.504193 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:02.504154    2553 generic.go:356] "Generic (PLEG): container finished" podID="0d081844-ec04-4c66-b982-eddd6ac59a46" containerID="a3baf14c788d8d581e32153fbb3c691b304e339a4124ef01fd5b11656a31f54b" exitCode=137
Sep 17 18:33:02.504438 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:02.504215    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-v5z84"
Sep 17 18:33:02.504438 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:02.504249    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-v5z84" event={"ID":"0d081844-ec04-4c66-b982-eddd6ac59a46","Type":"ContainerDied","Data":"a3baf14c788d8d581e32153fbb3c691b304e339a4124ef01fd5b11656a31f54b"}
Sep 17 18:33:02.504438 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:02.504333    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-v5z84" event={"ID":"0d081844-ec04-4c66-b982-eddd6ac59a46","Type":"ContainerDied","Data":"c7d9541c221b3c842502e23f5a2c14f9f24db5536a869f7ce9fe427b346e96a1"}
Sep 17 18:33:02.504438 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:02.504371    2553 scope.go:117] "RemoveContainer" containerID="e053f27dc25f065400229c51522471215e03221f95c69635aea2c777827643b9"
Sep 17 18:33:02.529726 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:02.529686    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-v5z84"]
Sep 17 18:33:02.533766 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:02.533736    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-v5z84"]
Sep 17 18:33:02.952036 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:02.951727    2553 scope.go:117] "RemoveContainer" containerID="a3baf14c788d8d581e32153fbb3c691b304e339a4124ef01fd5b11656a31f54b"
Sep 17 18:33:02.953190 ip-10-0-137-95 kubenswrapper[2553]: W0917 18:33:02.953169    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadd52769_2b2d_4f30_baba_da8457c1c67f.slice/crio-5ba54f815cb84e89bb50b590025181cde73ac092fd9ca8b434802c4accaa2add WatchSource:0}: Error finding container 5ba54f815cb84e89bb50b590025181cde73ac092fd9ca8b434802c4accaa2add: Status 404 returned error can't find the container with id 5ba54f815cb84e89bb50b590025181cde73ac092fd9ca8b434802c4accaa2add
Sep 17 18:33:03.378199 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:03.378122    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d081844-ec04-4c66-b982-eddd6ac59a46" path="/var/lib/kubelet/pods/0d081844-ec04-4c66-b982-eddd6ac59a46/volumes"
Sep 17 18:33:03.511399 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:03.511359    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-rp8kx" event={"ID":"add52769-2b2d-4f30-baba-da8457c1c67f","Type":"ContainerStarted","Data":"5ba54f815cb84e89bb50b590025181cde73ac092fd9ca8b434802c4accaa2add"}
Sep 17 18:33:04.061641 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:04.061599    2553 scope.go:117] "RemoveContainer" containerID="6554c236cf44ac55fe7a65d8bff1e98f1438be8dc825d848a310b062b66cda67"
Sep 17 18:33:04.168165 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:04.168140    2553 scope.go:117] "RemoveContainer" containerID="e053f27dc25f065400229c51522471215e03221f95c69635aea2c777827643b9"
Sep 17 18:33:04.168498 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:33:04.168477    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e053f27dc25f065400229c51522471215e03221f95c69635aea2c777827643b9\": container with ID starting with e053f27dc25f065400229c51522471215e03221f95c69635aea2c777827643b9 not found: ID does not exist" containerID="e053f27dc25f065400229c51522471215e03221f95c69635aea2c777827643b9"
Sep 17 18:33:04.168576 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:04.168512    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e053f27dc25f065400229c51522471215e03221f95c69635aea2c777827643b9"} err="failed to get container status \"e053f27dc25f065400229c51522471215e03221f95c69635aea2c777827643b9\": rpc error: code = NotFound desc = could not find container \"e053f27dc25f065400229c51522471215e03221f95c69635aea2c777827643b9\": container with ID starting with e053f27dc25f065400229c51522471215e03221f95c69635aea2c777827643b9 not found: ID does not exist"
Sep 17 18:33:04.168576 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:04.168536    2553 scope.go:117] "RemoveContainer" containerID="a3baf14c788d8d581e32153fbb3c691b304e339a4124ef01fd5b11656a31f54b"
Sep 17 18:33:04.168865 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:33:04.168846    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3baf14c788d8d581e32153fbb3c691b304e339a4124ef01fd5b11656a31f54b\": container with ID starting with a3baf14c788d8d581e32153fbb3c691b304e339a4124ef01fd5b11656a31f54b not found: ID does not exist" containerID="a3baf14c788d8d581e32153fbb3c691b304e339a4124ef01fd5b11656a31f54b"
Sep 17 18:33:04.168918 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:04.168870    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3baf14c788d8d581e32153fbb3c691b304e339a4124ef01fd5b11656a31f54b"} err="failed to get container status \"a3baf14c788d8d581e32153fbb3c691b304e339a4124ef01fd5b11656a31f54b\": rpc error: code = NotFound desc = could not find container \"a3baf14c788d8d581e32153fbb3c691b304e339a4124ef01fd5b11656a31f54b\": container with ID starting with a3baf14c788d8d581e32153fbb3c691b304e339a4124ef01fd5b11656a31f54b not found: ID does not exist"
Sep 17 18:33:04.168918 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:04.168887    2553 scope.go:117] "RemoveContainer" containerID="6554c236cf44ac55fe7a65d8bff1e98f1438be8dc825d848a310b062b66cda67"
Sep 17 18:33:04.169097 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:33:04.169083    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6554c236cf44ac55fe7a65d8bff1e98f1438be8dc825d848a310b062b66cda67\": container with ID starting with 6554c236cf44ac55fe7a65d8bff1e98f1438be8dc825d848a310b062b66cda67 not found: ID does not exist" containerID="6554c236cf44ac55fe7a65d8bff1e98f1438be8dc825d848a310b062b66cda67"
Sep 17 18:33:04.169140 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:04.169100    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6554c236cf44ac55fe7a65d8bff1e98f1438be8dc825d848a310b062b66cda67"} err="failed to get container status \"6554c236cf44ac55fe7a65d8bff1e98f1438be8dc825d848a310b062b66cda67\": rpc error: code = NotFound desc = could not find container \"6554c236cf44ac55fe7a65d8bff1e98f1438be8dc825d848a310b062b66cda67\": container with ID starting with 6554c236cf44ac55fe7a65d8bff1e98f1438be8dc825d848a310b062b66cda67 not found: ID does not exist"
Sep 17 18:33:13.569612 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:13.569555    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-rp8kx" event={"ID":"add52769-2b2d-4f30-baba-da8457c1c67f","Type":"ContainerStarted","Data":"3bfbfa1f5c7d0b1e439a83d3696874089f4ede039260d6c8bf7f10121918ae6e"}
Sep 17 18:33:42.716519 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:42.716480    2553 generic.go:356] "Generic (PLEG): container finished" podID="add52769-2b2d-4f30-baba-da8457c1c67f" containerID="3bfbfa1f5c7d0b1e439a83d3696874089f4ede039260d6c8bf7f10121918ae6e" exitCode=0
Sep 17 18:33:42.716970 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:42.716558    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-rp8kx" event={"ID":"add52769-2b2d-4f30-baba-da8457c1c67f","Type":"ContainerDied","Data":"3bfbfa1f5c7d0b1e439a83d3696874089f4ede039260d6c8bf7f10121918ae6e"}
Sep 17 18:33:43.724710 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:43.724493    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-rp8kx" event={"ID":"add52769-2b2d-4f30-baba-da8457c1c67f","Type":"ContainerStarted","Data":"95e958b8489c06e7cd3934793074ceffa088595c23f570f77009a5d749582d3c"}
Sep 17 18:33:43.724710 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:43.724575    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-rp8kx" event={"ID":"add52769-2b2d-4f30-baba-da8457c1c67f","Type":"ContainerStarted","Data":"d1bb4424a356ae318deb75a368402f3cacb3cb52dc63fa258d5dd1d4dbd07665"}
Sep 17 18:33:43.725160 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:43.724954    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-rp8kx"
Sep 17 18:33:43.725160 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:43.725019    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-rp8kx"
Sep 17 18:33:43.727062 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:43.727028    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-rp8kx" podUID="add52769-2b2d-4f30-baba-da8457c1c67f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 18:33:43.757372 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:43.757320    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-rp8kx" podStartSLOduration=64.232667057 podStartE2EDuration="1m8.757302414s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.524635357s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:32:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:33:43.756245139 +0000 UTC m=+2218.452335816" watchObservedRunningTime="2026-09-17 18:33:43.757302414 +0000 UTC m=+2218.453393172"
Sep 17 18:33:44.731049 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:44.730999    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7kj5n" event={"ID":"2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e","Type":"ContainerStarted","Data":"f2dc28af6bed825317af687f66fd33df10cc61f1d26665cb03d209aa8cb7a676"}
Sep 17 18:33:44.731454 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:44.731080    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7kj5n" event={"ID":"2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e","Type":"ContainerStarted","Data":"1733bef34380cb7f7eb4b3c8c330017d3345527b78269413b1d26ca87bc9f395"}
Sep 17 18:33:44.731454 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:44.731270    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-rp8kx" podUID="add52769-2b2d-4f30-baba-da8457c1c67f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 18:33:44.752701 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:44.752636    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7kj5n" podStartSLOduration=2.168020994 podStartE2EDuration="2m0.752603271s" totalImagesPullingTime="1m54.632926815s" totalInitContainerRuntime="3.951655462s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:31:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:33:44.750659522 +0000 UTC m=+2219.446750209" watchObservedRunningTime="2026-09-17 18:33:44.752603271 +0000 UTC m=+2219.448693956"
Sep 17 18:33:45.735703 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:45.735669    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7kj5n"
Sep 17 18:33:45.736120 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:45.735732    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7kj5n"
Sep 17 18:33:45.737250 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:45.737218    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7kj5n" podUID="2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 17 18:33:46.740228 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:46.740136    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7kj5n" podUID="2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 17 18:33:49.736311 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:49.736272    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-rp8kx"
Sep 17 18:33:49.737013 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:49.736984    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-rp8kx" podUID="add52769-2b2d-4f30-baba-da8457c1c67f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 18:33:51.744576 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:51.744544    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7kj5n"
Sep 17 18:33:51.745363 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:51.745332    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7kj5n"
Sep 17 18:33:56.759977 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:56.759939    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7kj5n"]
Sep 17 18:33:56.760427 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:56.760188    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7kj5n" podUID="2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e" containerName="kserve-container" containerID="cri-o://1733bef34380cb7f7eb4b3c8c330017d3345527b78269413b1d26ca87bc9f395" gracePeriod=30
Sep 17 18:33:56.760427 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:56.760212    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7kj5n" podUID="2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e" containerName="kube-rbac-proxy" containerID="cri-o://f2dc28af6bed825317af687f66fd33df10cc61f1d26665cb03d209aa8cb7a676" gracePeriod=30
Sep 17 18:33:57.794800 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:57.794766    2553 generic.go:356] "Generic (PLEG): container finished" podID="2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e" containerID="f2dc28af6bed825317af687f66fd33df10cc61f1d26665cb03d209aa8cb7a676" exitCode=2
Sep 17 18:33:57.795198 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:57.794840    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7kj5n" event={"ID":"2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e","Type":"ContainerDied","Data":"f2dc28af6bed825317af687f66fd33df10cc61f1d26665cb03d209aa8cb7a676"}
Sep 17 18:33:59.180656 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:59.180604    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7kj5n"
Sep 17 18:33:59.238099 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:59.237996    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e-proxy-tls\") pod \"2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e\" (UID: \"2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e\") "
Sep 17 18:33:59.238099 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:59.238048    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e-cabundle-cert\") pod \"2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e\" (UID: \"2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e\") "
Sep 17 18:33:59.238099 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:59.238087    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e-kube-rbac-proxy-sar-config\") pod \"2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e\" (UID: \"2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e\") "
Sep 17 18:33:59.238394 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:59.238184    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2nblx\" (UniqueName: \"kubernetes.io/projected/2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e-kube-api-access-2nblx\") pod \"2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e\" (UID: \"2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e\") "
Sep 17 18:33:59.238394 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:59.238240    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e-kserve-provision-location\") pod \"2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e\" (UID: \"2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e\") "
Sep 17 18:33:59.238514 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:59.238461    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e" (UID: "2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:33:59.238514 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:59.238476    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e" (UID: "2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:33:59.238614 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:59.238552    2553 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e-cabundle-cert\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:33:59.238614 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:59.238576    2553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:33:59.238892 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:59.238869    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e" (UID: "2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:33:59.240447 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:59.240425    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e" (UID: "2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:33:59.240547 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:59.240499    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e-kube-api-access-2nblx" (OuterVolumeSpecName: "kube-api-access-2nblx") pod "2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e" (UID: "2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e"). InnerVolumeSpecName "kube-api-access-2nblx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:33:59.339218 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:59.339174    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nblx\" (UniqueName: \"kubernetes.io/projected/2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e-kube-api-access-2nblx\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:33:59.339218 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:59.339210    2553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e-kserve-provision-location\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:33:59.339218 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:59.339221    2553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e-proxy-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:33:59.737027 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:59.736984    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-rp8kx" podUID="add52769-2b2d-4f30-baba-da8457c1c67f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 18:33:59.805833 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:59.805791    2553 generic.go:356] "Generic (PLEG): container finished" podID="2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e" containerID="1733bef34380cb7f7eb4b3c8c330017d3345527b78269413b1d26ca87bc9f395" exitCode=0
Sep 17 18:33:59.806006 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:59.805845    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7kj5n" event={"ID":"2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e","Type":"ContainerDied","Data":"1733bef34380cb7f7eb4b3c8c330017d3345527b78269413b1d26ca87bc9f395"}
Sep 17 18:33:59.806006 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:59.805873    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7kj5n"
Sep 17 18:33:59.806006 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:59.805910    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7kj5n" event={"ID":"2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e","Type":"ContainerDied","Data":"aba1d01b79b0d8ad0c8b4d3650958639b35c4f3706c4c1a4ab7aba3629945b95"}
Sep 17 18:33:59.806006 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:59.805937    2553 scope.go:117] "RemoveContainer" containerID="f2dc28af6bed825317af687f66fd33df10cc61f1d26665cb03d209aa8cb7a676"
Sep 17 18:33:59.815337 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:59.815315    2553 scope.go:117] "RemoveContainer" containerID="1733bef34380cb7f7eb4b3c8c330017d3345527b78269413b1d26ca87bc9f395"
Sep 17 18:33:59.823413 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:59.823388    2553 scope.go:117] "RemoveContainer" containerID="cb9d9973ced9213afc35791433bb7adb1b38370946c3c556e441f164c619bba8"
Sep 17 18:33:59.824887 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:59.824865    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7kj5n"]
Sep 17 18:33:59.828380 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:59.828354    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-7kj5n"]
Sep 17 18:33:59.831885 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:59.831863    2553 scope.go:117] "RemoveContainer" containerID="f2dc28af6bed825317af687f66fd33df10cc61f1d26665cb03d209aa8cb7a676"
Sep 17 18:33:59.832211 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:33:59.832183    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2dc28af6bed825317af687f66fd33df10cc61f1d26665cb03d209aa8cb7a676\": container with ID starting with f2dc28af6bed825317af687f66fd33df10cc61f1d26665cb03d209aa8cb7a676 not found: ID does not exist" containerID="f2dc28af6bed825317af687f66fd33df10cc61f1d26665cb03d209aa8cb7a676"
Sep 17 18:33:59.832280 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:59.832220    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2dc28af6bed825317af687f66fd33df10cc61f1d26665cb03d209aa8cb7a676"} err="failed to get container status \"f2dc28af6bed825317af687f66fd33df10cc61f1d26665cb03d209aa8cb7a676\": rpc error: code = NotFound desc = could not find container \"f2dc28af6bed825317af687f66fd33df10cc61f1d26665cb03d209aa8cb7a676\": container with ID starting with f2dc28af6bed825317af687f66fd33df10cc61f1d26665cb03d209aa8cb7a676 not found: ID does not exist"
Sep 17 18:33:59.832280 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:59.832239    2553 scope.go:117] "RemoveContainer" containerID="1733bef34380cb7f7eb4b3c8c330017d3345527b78269413b1d26ca87bc9f395"
Sep 17 18:33:59.832505 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:33:59.832487    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1733bef34380cb7f7eb4b3c8c330017d3345527b78269413b1d26ca87bc9f395\": container with ID starting with 1733bef34380cb7f7eb4b3c8c330017d3345527b78269413b1d26ca87bc9f395 not found: ID does not exist" containerID="1733bef34380cb7f7eb4b3c8c330017d3345527b78269413b1d26ca87bc9f395"
Sep 17 18:33:59.832550 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:59.832511    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1733bef34380cb7f7eb4b3c8c330017d3345527b78269413b1d26ca87bc9f395"} err="failed to get container status \"1733bef34380cb7f7eb4b3c8c330017d3345527b78269413b1d26ca87bc9f395\": rpc error: code = NotFound desc = could not find container \"1733bef34380cb7f7eb4b3c8c330017d3345527b78269413b1d26ca87bc9f395\": container with ID starting with 1733bef34380cb7f7eb4b3c8c330017d3345527b78269413b1d26ca87bc9f395 not found: ID does not exist"
Sep 17 18:33:59.832550 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:59.832527    2553 scope.go:117] "RemoveContainer" containerID="cb9d9973ced9213afc35791433bb7adb1b38370946c3c556e441f164c619bba8"
Sep 17 18:33:59.832790 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:33:59.832773    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb9d9973ced9213afc35791433bb7adb1b38370946c3c556e441f164c619bba8\": container with ID starting with cb9d9973ced9213afc35791433bb7adb1b38370946c3c556e441f164c619bba8 not found: ID does not exist" containerID="cb9d9973ced9213afc35791433bb7adb1b38370946c3c556e441f164c619bba8"
Sep 17 18:33:59.832840 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:33:59.832793    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb9d9973ced9213afc35791433bb7adb1b38370946c3c556e441f164c619bba8"} err="failed to get container status \"cb9d9973ced9213afc35791433bb7adb1b38370946c3c556e441f164c619bba8\": rpc error: code = NotFound desc = could not find container \"cb9d9973ced9213afc35791433bb7adb1b38370946c3c556e441f164c619bba8\": container with ID starting with cb9d9973ced9213afc35791433bb7adb1b38370946c3c556e441f164c619bba8 not found: ID does not exist"
Sep 17 18:34:01.376350 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:34:01.376317    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e" path="/var/lib/kubelet/pods/2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e/volumes"
Sep 17 18:34:01.946007 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:34:01.945964    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-xlspt"]
Sep 17 18:34:01.946563 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:34:01.946541    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d081844-ec04-4c66-b982-eddd6ac59a46" containerName="storage-initializer"
Sep 17 18:34:01.946632 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:34:01.946568    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d081844-ec04-4c66-b982-eddd6ac59a46" containerName="storage-initializer"
Sep 17 18:34:01.946632 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:34:01.946582    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d081844-ec04-4c66-b982-eddd6ac59a46" containerName="kserve-container"
Sep 17 18:34:01.946632 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:34:01.946592    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d081844-ec04-4c66-b982-eddd6ac59a46" containerName="kserve-container"
Sep 17 18:34:01.946632 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:34:01.946606    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d081844-ec04-4c66-b982-eddd6ac59a46" containerName="kube-rbac-proxy"
Sep 17 18:34:01.946632 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:34:01.946631    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d081844-ec04-4c66-b982-eddd6ac59a46" containerName="kube-rbac-proxy"
Sep 17 18:34:01.946810 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:34:01.946645    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e" containerName="kserve-container"
Sep 17 18:34:01.946810 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:34:01.946654    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e" containerName="kserve-container"
Sep 17 18:34:01.946810 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:34:01.946677    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e" containerName="kube-rbac-proxy"
Sep 17 18:34:01.946810 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:34:01.946685    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e" containerName="kube-rbac-proxy"
Sep 17 18:34:01.946810 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:34:01.946709    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e" containerName="storage-initializer"
Sep 17 18:34:01.946810 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:34:01.946719    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e" containerName="storage-initializer"
Sep 17 18:34:01.946810 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:34:01.946802    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e" containerName="kube-rbac-proxy"
Sep 17 18:34:01.947030 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:34:01.946816    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="2acb8c70-7b6c-4f71-8183-74f0b4eb7c3e" containerName="kserve-container"
Sep 17 18:34:01.947030 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:34:01.946827    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d081844-ec04-4c66-b982-eddd6ac59a46" containerName="kserve-container"
Sep 17 18:34:01.947030 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:34:01.946838    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d081844-ec04-4c66-b982-eddd6ac59a46" containerName="kube-rbac-proxy"
Sep 17 18:34:01.992974 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:34:01.992931    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-xlspt"]
Sep 17 18:34:01.993159 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:34:01.992987    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-xlspt"
Sep 17 18:34:01.995446 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:34:01.995427    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-predictor-serving-cert\""
Sep 17 18:34:01.995537 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:34:01.995447    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-kube-rbac-proxy-sar-config\""
Sep 17 18:34:02.063440 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:34:02.063404    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41bf207e-2eea-4616-9c26-d20142d1c329-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-xlspt\" (UID: \"41bf207e-2eea-4616-9c26-d20142d1c329\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-xlspt"
Sep 17 18:34:02.063440 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:34:02.063442    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b45wr\" (UniqueName: \"kubernetes.io/projected/41bf207e-2eea-4616-9c26-d20142d1c329-kube-api-access-b45wr\") pod \"isvc-xgboost-predictor-7566d65bf7-xlspt\" (UID: \"41bf207e-2eea-4616-9c26-d20142d1c329\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-xlspt"
Sep 17 18:34:02.063692 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:34:02.063481    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41bf207e-2eea-4616-9c26-d20142d1c329-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-xlspt\" (UID: \"41bf207e-2eea-4616-9c26-d20142d1c329\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-xlspt"
Sep 17 18:34:02.063692 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:34:02.063522    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41bf207e-2eea-4616-9c26-d20142d1c329-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-xlspt\" (UID: \"41bf207e-2eea-4616-9c26-d20142d1c329\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-xlspt"
Sep 17 18:34:02.063692 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:34:02.063566    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41bf207e-2eea-4616-9c26-d20142d1c329-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-xlspt\" (UID: \"41bf207e-2eea-4616-9c26-d20142d1c329\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-xlspt"
Sep 17 18:34:02.164599 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:34:02.164554    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41bf207e-2eea-4616-9c26-d20142d1c329-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-xlspt\" (UID: \"41bf207e-2eea-4616-9c26-d20142d1c329\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-xlspt"
Sep 17 18:34:02.164599 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:34:02.164607    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41bf207e-2eea-4616-9c26-d20142d1c329-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-xlspt\" (UID: \"41bf207e-2eea-4616-9c26-d20142d1c329\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-xlspt"
Sep 17 18:34:02.164876 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:34:02.164683    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41bf207e-2eea-4616-9c26-d20142d1c329-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-xlspt\" (UID: \"41bf207e-2eea-4616-9c26-d20142d1c329\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-xlspt"
Sep 17 18:34:02.164876 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:34:02.164723    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41bf207e-2eea-4616-9c26-d20142d1c329-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-xlspt\" (UID: \"41bf207e-2eea-4616-9c26-d20142d1c329\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-xlspt"
Sep 17 18:34:02.164876 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:34:02.164750    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b45wr\" (UniqueName: \"kubernetes.io/projected/41bf207e-2eea-4616-9c26-d20142d1c329-kube-api-access-b45wr\") pod \"isvc-xgboost-predictor-7566d65bf7-xlspt\" (UID: \"41bf207e-2eea-4616-9c26-d20142d1c329\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-xlspt"
Sep 17 18:34:02.164876 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:34:02.164852    2553 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-predictor-serving-cert: secret "isvc-xgboost-predictor-serving-cert" not found
Sep 17 18:34:02.165093 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:34:02.164946    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41bf207e-2eea-4616-9c26-d20142d1c329-proxy-tls podName:41bf207e-2eea-4616-9c26-d20142d1c329 nodeName:}" failed. No retries permitted until 2026-09-17 18:34:02.66492271 +0000 UTC m=+2237.361013383 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/41bf207e-2eea-4616-9c26-d20142d1c329-proxy-tls") pod "isvc-xgboost-predictor-7566d65bf7-xlspt" (UID: "41bf207e-2eea-4616-9c26-d20142d1c329") : secret "isvc-xgboost-predictor-serving-cert" not found
Sep 17 18:34:02.165157 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:34:02.165108    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41bf207e-2eea-4616-9c26-d20142d1c329-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-xlspt\" (UID: \"41bf207e-2eea-4616-9c26-d20142d1c329\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-xlspt"
Sep 17 18:34:02.165276 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:34:02.165257    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41bf207e-2eea-4616-9c26-d20142d1c329-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-xlspt\" (UID: \"41bf207e-2eea-4616-9c26-d20142d1c329\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-xlspt"
Sep 17 18:34:02.165392 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:34:02.165375    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41bf207e-2eea-4616-9c26-d20142d1c329-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-xlspt\" (UID: \"41bf207e-2eea-4616-9c26-d20142d1c329\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-xlspt"
Sep 17 18:34:02.175409 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:34:02.175381    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b45wr\" (UniqueName: \"kubernetes.io/projected/41bf207e-2eea-4616-9c26-d20142d1c329-kube-api-access-b45wr\") pod \"isvc-xgboost-predictor-7566d65bf7-xlspt\" (UID: \"41bf207e-2eea-4616-9c26-d20142d1c329\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-xlspt"
Sep 17 18:34:02.669191 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:34:02.669149    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41bf207e-2eea-4616-9c26-d20142d1c329-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-xlspt\" (UID: \"41bf207e-2eea-4616-9c26-d20142d1c329\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-xlspt"
Sep 17 18:34:02.671591 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:34:02.671564    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41bf207e-2eea-4616-9c26-d20142d1c329-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-xlspt\" (UID: \"41bf207e-2eea-4616-9c26-d20142d1c329\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-xlspt"
Sep 17 18:34:02.903534 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:34:02.903487    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-xlspt"
Sep 17 18:34:03.205243 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:34:03.205217    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-xlspt"]
Sep 17 18:34:03.207771 ip-10-0-137-95 kubenswrapper[2553]: W0917 18:34:03.207746    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41bf207e_2eea_4616_9c26_d20142d1c329.slice/crio-a56bc1b845aad59ab2bfe3aef9fe9dbd0e421f555d9b1012f21056a8479f163f WatchSource:0}: Error finding container a56bc1b845aad59ab2bfe3aef9fe9dbd0e421f555d9b1012f21056a8479f163f: Status 404 returned error can't find the container with id a56bc1b845aad59ab2bfe3aef9fe9dbd0e421f555d9b1012f21056a8479f163f
Sep 17 18:34:03.828570 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:34:03.828529    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-xlspt" event={"ID":"41bf207e-2eea-4616-9c26-d20142d1c329","Type":"ContainerStarted","Data":"a0e16276af3c631c2899dd5a3ae576fe9dd638db8d1f6b71072f22d319371adb"}
Sep 17 18:34:03.828948 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:34:03.828609    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-xlspt" event={"ID":"41bf207e-2eea-4616-9c26-d20142d1c329","Type":"ContainerStarted","Data":"a56bc1b845aad59ab2bfe3aef9fe9dbd0e421f555d9b1012f21056a8479f163f"}
Sep 17 18:34:07.851312 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:34:07.851277    2553 generic.go:356] "Generic (PLEG): container finished" podID="41bf207e-2eea-4616-9c26-d20142d1c329" containerID="a0e16276af3c631c2899dd5a3ae576fe9dd638db8d1f6b71072f22d319371adb" exitCode=0
Sep 17 18:34:07.851714 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:34:07.851346    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-xlspt" event={"ID":"41bf207e-2eea-4616-9c26-d20142d1c329","Type":"ContainerDied","Data":"a0e16276af3c631c2899dd5a3ae576fe9dd638db8d1f6b71072f22d319371adb"}
Sep 17 18:34:09.737496 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:34:09.737456    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-rp8kx" podUID="add52769-2b2d-4f30-baba-da8457c1c67f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 18:34:19.737879 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:34:19.737839    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-rp8kx" podUID="add52769-2b2d-4f30-baba-da8457c1c67f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 18:34:29.737741 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:34:29.737699    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-rp8kx" podUID="add52769-2b2d-4f30-baba-da8457c1c67f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 18:34:29.972501 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:34:29.972465    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-xlspt" event={"ID":"41bf207e-2eea-4616-9c26-d20142d1c329","Type":"ContainerStarted","Data":"1eb51f9f33ae7d8e0ab8c0f9ff884d1aa751bb87bf26be2031b3173ba55e56f8"}
Sep 17 18:34:29.972681 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:34:29.972536    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-xlspt" event={"ID":"41bf207e-2eea-4616-9c26-d20142d1c329","Type":"ContainerStarted","Data":"634863b265aba6d40287102e6d7077c6041c89c2738be9f47245acc54aa1aeab"}
Sep 17 18:34:29.972830 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:34:29.972808    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-xlspt"
Sep 17 18:34:29.972977 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:34:29.972960    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-xlspt"
Sep 17 18:34:29.974281 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:34:29.974252    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-xlspt" podUID="41bf207e-2eea-4616-9c26-d20142d1c329" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 18:34:29.992525 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:34:29.992434    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-xlspt" podStartSLOduration=3.046265749 podStartE2EDuration="28.992419703s" totalImagesPullingTime="21.942450941s" totalInitContainerRuntime="4.003703013s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:34:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:34:29.991098943 +0000 UTC m=+2264.687189627" watchObservedRunningTime="2026-09-17 18:34:29.992419703 +0000 UTC m=+2264.688510448"
Sep 17 18:34:30.977019 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:34:30.976981    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-xlspt" podUID="41bf207e-2eea-4616-9c26-d20142d1c329" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 18:34:35.981356 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:34:35.981322    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-xlspt"
Sep 17 18:34:35.981804 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:34:35.981778    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-xlspt" podUID="41bf207e-2eea-4616-9c26-d20142d1c329" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 18:34:39.736939 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:34:39.736895    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-rp8kx" podUID="add52769-2b2d-4f30-baba-da8457c1c67f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 18:34:45.982443 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:34:45.982384    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-xlspt" podUID="41bf207e-2eea-4616-9c26-d20142d1c329" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 18:34:49.737347 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:34:49.737305    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-rp8kx" podUID="add52769-2b2d-4f30-baba-da8457c1c67f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 18:34:55.982121 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:34:55.982080    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-xlspt" podUID="41bf207e-2eea-4616-9c26-d20142d1c329" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 18:34:59.737092 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:34:59.737045    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-rp8kx" podUID="add52769-2b2d-4f30-baba-da8457c1c67f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 18:35:01.371180 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:01.371139    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-rp8kx" podUID="add52769-2b2d-4f30-baba-da8457c1c67f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 18:35:05.982671 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:05.982633    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-xlspt" podUID="41bf207e-2eea-4616-9c26-d20142d1c329" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 18:35:11.376807 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:11.376775    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-rp8kx"
Sep 17 18:35:15.743532 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:15.743489    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-rp8kx"]
Sep 17 18:35:15.744043 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:15.743765    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-rp8kx" podUID="add52769-2b2d-4f30-baba-da8457c1c67f" containerName="kserve-container" containerID="cri-o://d1bb4424a356ae318deb75a368402f3cacb3cb52dc63fa258d5dd1d4dbd07665" gracePeriod=30
Sep 17 18:35:15.744043 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:15.743822    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-rp8kx" podUID="add52769-2b2d-4f30-baba-da8457c1c67f" containerName="kube-rbac-proxy" containerID="cri-o://95e958b8489c06e7cd3934793074ceffa088595c23f570f77009a5d749582d3c" gracePeriod=30
Sep 17 18:35:15.982509 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:15.982464    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-xlspt" podUID="41bf207e-2eea-4616-9c26-d20142d1c329" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 18:35:16.185044 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:16.185003    2553 generic.go:356] "Generic (PLEG): container finished" podID="add52769-2b2d-4f30-baba-da8457c1c67f" containerID="95e958b8489c06e7cd3934793074ceffa088595c23f570f77009a5d749582d3c" exitCode=2
Sep 17 18:35:16.185044 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:16.185043    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-rp8kx" event={"ID":"add52769-2b2d-4f30-baba-da8457c1c67f","Type":"ContainerDied","Data":"95e958b8489c06e7cd3934793074ceffa088595c23f570f77009a5d749582d3c"}
Sep 17 18:35:19.732048 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:19.732007    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-rp8kx" podUID="add52769-2b2d-4f30-baba-da8457c1c67f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.46:8643/healthz\": dial tcp 10.134.0.46:8643: connect: connection refused"
Sep 17 18:35:20.592416 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:20.592384    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-rp8kx"
Sep 17 18:35:20.774870 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:20.774779    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/add52769-2b2d-4f30-baba-da8457c1c67f-proxy-tls\") pod \"add52769-2b2d-4f30-baba-da8457c1c67f\" (UID: \"add52769-2b2d-4f30-baba-da8457c1c67f\") "
Sep 17 18:35:20.774870 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:20.774834    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p9vjj\" (UniqueName: \"kubernetes.io/projected/add52769-2b2d-4f30-baba-da8457c1c67f-kube-api-access-p9vjj\") pod \"add52769-2b2d-4f30-baba-da8457c1c67f\" (UID: \"add52769-2b2d-4f30-baba-da8457c1c67f\") "
Sep 17 18:35:20.774870 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:20.774868    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/add52769-2b2d-4f30-baba-da8457c1c67f-kube-rbac-proxy-sar-config\") pod \"add52769-2b2d-4f30-baba-da8457c1c67f\" (UID: \"add52769-2b2d-4f30-baba-da8457c1c67f\") "
Sep 17 18:35:20.775408 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:20.774892    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/add52769-2b2d-4f30-baba-da8457c1c67f-kserve-provision-location\") pod \"add52769-2b2d-4f30-baba-da8457c1c67f\" (UID: \"add52769-2b2d-4f30-baba-da8457c1c67f\") "
Sep 17 18:35:20.775408 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:20.774923    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/add52769-2b2d-4f30-baba-da8457c1c67f-cabundle-cert\") pod \"add52769-2b2d-4f30-baba-da8457c1c67f\" (UID: \"add52769-2b2d-4f30-baba-da8457c1c67f\") "
Sep 17 18:35:20.775408 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:20.775260    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/add52769-2b2d-4f30-baba-da8457c1c67f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "add52769-2b2d-4f30-baba-da8457c1c67f" (UID: "add52769-2b2d-4f30-baba-da8457c1c67f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:35:20.775408 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:20.775287    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/add52769-2b2d-4f30-baba-da8457c1c67f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "add52769-2b2d-4f30-baba-da8457c1c67f" (UID: "add52769-2b2d-4f30-baba-da8457c1c67f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:35:20.775408 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:20.775323    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/add52769-2b2d-4f30-baba-da8457c1c67f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "add52769-2b2d-4f30-baba-da8457c1c67f" (UID: "add52769-2b2d-4f30-baba-da8457c1c67f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:35:20.776963 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:20.776942    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/add52769-2b2d-4f30-baba-da8457c1c67f-kube-api-access-p9vjj" (OuterVolumeSpecName: "kube-api-access-p9vjj") pod "add52769-2b2d-4f30-baba-da8457c1c67f" (UID: "add52769-2b2d-4f30-baba-da8457c1c67f"). InnerVolumeSpecName "kube-api-access-p9vjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:35:20.777056 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:20.776979    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/add52769-2b2d-4f30-baba-da8457c1c67f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "add52769-2b2d-4f30-baba-da8457c1c67f" (UID: "add52769-2b2d-4f30-baba-da8457c1c67f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:35:20.875872 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:20.875836    2553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/add52769-2b2d-4f30-baba-da8457c1c67f-proxy-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:35:20.875872 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:20.875868    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9vjj\" (UniqueName: \"kubernetes.io/projected/add52769-2b2d-4f30-baba-da8457c1c67f-kube-api-access-p9vjj\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:35:20.875872 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:20.875878    2553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/add52769-2b2d-4f30-baba-da8457c1c67f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:35:20.876099 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:20.875887    2553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/add52769-2b2d-4f30-baba-da8457c1c67f-kserve-provision-location\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:35:20.876099 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:20.875896    2553 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/add52769-2b2d-4f30-baba-da8457c1c67f-cabundle-cert\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:35:20.943397 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:20.943363    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-rt7hq"]
Sep 17 18:35:20.943746 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:20.943731    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="add52769-2b2d-4f30-baba-da8457c1c67f" containerName="storage-initializer"
Sep 17 18:35:20.943746 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:20.943747    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="add52769-2b2d-4f30-baba-da8457c1c67f" containerName="storage-initializer"
Sep 17 18:35:20.943886 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:20.943756    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="add52769-2b2d-4f30-baba-da8457c1c67f" containerName="kube-rbac-proxy"
Sep 17 18:35:20.943886 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:20.943762    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="add52769-2b2d-4f30-baba-da8457c1c67f" containerName="kube-rbac-proxy"
Sep 17 18:35:20.943886 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:20.943779    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="add52769-2b2d-4f30-baba-da8457c1c67f" containerName="kserve-container"
Sep 17 18:35:20.943886 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:20.943785    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="add52769-2b2d-4f30-baba-da8457c1c67f" containerName="kserve-container"
Sep 17 18:35:20.943886 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:20.943844    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="add52769-2b2d-4f30-baba-da8457c1c67f" containerName="kserve-container"
Sep 17 18:35:20.943886 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:20.943856    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="add52769-2b2d-4f30-baba-da8457c1c67f" containerName="kube-rbac-proxy"
Sep 17 18:35:20.946704 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:20.946685    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-rt7hq"
Sep 17 18:35:20.949307 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:20.949286    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Sep 17 18:35:20.949401 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:20.949311    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 17 18:35:20.958390 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:20.958363    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-rt7hq"]
Sep 17 18:35:21.021561 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:21.021519    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-hw6dc"]
Sep 17 18:35:21.024653 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:21.024634    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-hw6dc"
Sep 17 18:35:21.026806 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:21.026750    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 17 18:35:21.035311 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:21.035286    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-hw6dc"]
Sep 17 18:35:21.077640 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:21.077582    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/094068e2-76dd-41d0-8682-acddc8dd32f1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-867f674bf6-rt7hq\" (UID: \"094068e2-76dd-41d0-8682-acddc8dd32f1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-rt7hq"
Sep 17 18:35:21.077797 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:21.077695    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/094068e2-76dd-41d0-8682-acddc8dd32f1-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-867f674bf6-rt7hq\" (UID: \"094068e2-76dd-41d0-8682-acddc8dd32f1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-rt7hq"
Sep 17 18:35:21.077797 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:21.077718    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw7vt\" (UniqueName: \"kubernetes.io/projected/094068e2-76dd-41d0-8682-acddc8dd32f1-kube-api-access-vw7vt\") pod \"isvc-canary-stop-v2-predictor-867f674bf6-rt7hq\" (UID: \"094068e2-76dd-41d0-8682-acddc8dd32f1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-rt7hq"
Sep 17 18:35:21.077797 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:21.077760    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/094068e2-76dd-41d0-8682-acddc8dd32f1-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-867f674bf6-rt7hq\" (UID: \"094068e2-76dd-41d0-8682-acddc8dd32f1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-rt7hq"
Sep 17 18:35:21.077940 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:21.077812    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/094068e2-76dd-41d0-8682-acddc8dd32f1-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-867f674bf6-rt7hq\" (UID: \"094068e2-76dd-41d0-8682-acddc8dd32f1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-rt7hq"
Sep 17 18:35:21.178465 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:21.178422    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/094068e2-76dd-41d0-8682-acddc8dd32f1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-867f674bf6-rt7hq\" (UID: \"094068e2-76dd-41d0-8682-acddc8dd32f1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-rt7hq"
Sep 17 18:35:21.178685 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:21.178499    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9290625f-2be6-4370-b08b-8955a51a8d4e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-b84cdfc7-hw6dc\" (UID: \"9290625f-2be6-4370-b08b-8955a51a8d4e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-hw6dc"
Sep 17 18:35:21.178685 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:21.178527    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9290625f-2be6-4370-b08b-8955a51a8d4e-kserve-provision-location\") pod \"isvc-canary-stop-predictor-b84cdfc7-hw6dc\" (UID: \"9290625f-2be6-4370-b08b-8955a51a8d4e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-hw6dc"
Sep 17 18:35:21.178685 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:21.178569    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/094068e2-76dd-41d0-8682-acddc8dd32f1-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-867f674bf6-rt7hq\" (UID: \"094068e2-76dd-41d0-8682-acddc8dd32f1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-rt7hq"
Sep 17 18:35:21.178685 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:21.178662    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw7vt\" (UniqueName: \"kubernetes.io/projected/094068e2-76dd-41d0-8682-acddc8dd32f1-kube-api-access-vw7vt\") pod \"isvc-canary-stop-v2-predictor-867f674bf6-rt7hq\" (UID: \"094068e2-76dd-41d0-8682-acddc8dd32f1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-rt7hq"
Sep 17 18:35:21.178902 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:21.178695    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9290625f-2be6-4370-b08b-8955a51a8d4e-proxy-tls\") pod \"isvc-canary-stop-predictor-b84cdfc7-hw6dc\" (UID: \"9290625f-2be6-4370-b08b-8955a51a8d4e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-hw6dc"
Sep 17 18:35:21.178902 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:21.178726    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/094068e2-76dd-41d0-8682-acddc8dd32f1-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-867f674bf6-rt7hq\" (UID: \"094068e2-76dd-41d0-8682-acddc8dd32f1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-rt7hq"
Sep 17 18:35:21.178902 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:21.178758    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9290625f-2be6-4370-b08b-8955a51a8d4e-cabundle-cert\") pod \"isvc-canary-stop-predictor-b84cdfc7-hw6dc\" (UID: \"9290625f-2be6-4370-b08b-8955a51a8d4e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-hw6dc"
Sep 17 18:35:21.178902 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:21.178780    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xjsd\" (UniqueName: \"kubernetes.io/projected/9290625f-2be6-4370-b08b-8955a51a8d4e-kube-api-access-9xjsd\") pod \"isvc-canary-stop-predictor-b84cdfc7-hw6dc\" (UID: \"9290625f-2be6-4370-b08b-8955a51a8d4e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-hw6dc"
Sep 17 18:35:21.178902 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:21.178824    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/094068e2-76dd-41d0-8682-acddc8dd32f1-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-867f674bf6-rt7hq\" (UID: \"094068e2-76dd-41d0-8682-acddc8dd32f1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-rt7hq"
Sep 17 18:35:21.179159 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:21.179137    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/094068e2-76dd-41d0-8682-acddc8dd32f1-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-867f674bf6-rt7hq\" (UID: \"094068e2-76dd-41d0-8682-acddc8dd32f1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-rt7hq"
Sep 17 18:35:21.179243 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:21.179223    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/094068e2-76dd-41d0-8682-acddc8dd32f1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-867f674bf6-rt7hq\" (UID: \"094068e2-76dd-41d0-8682-acddc8dd32f1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-rt7hq"
Sep 17 18:35:21.179311 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:21.179300    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/094068e2-76dd-41d0-8682-acddc8dd32f1-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-867f674bf6-rt7hq\" (UID: \"094068e2-76dd-41d0-8682-acddc8dd32f1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-rt7hq"
Sep 17 18:35:21.181101 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:21.181081    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/094068e2-76dd-41d0-8682-acddc8dd32f1-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-867f674bf6-rt7hq\" (UID: \"094068e2-76dd-41d0-8682-acddc8dd32f1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-rt7hq"
Sep 17 18:35:21.186440 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:21.186408    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw7vt\" (UniqueName: \"kubernetes.io/projected/094068e2-76dd-41d0-8682-acddc8dd32f1-kube-api-access-vw7vt\") pod \"isvc-canary-stop-v2-predictor-867f674bf6-rt7hq\" (UID: \"094068e2-76dd-41d0-8682-acddc8dd32f1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-rt7hq"
Sep 17 18:35:21.208924 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:21.208882    2553 generic.go:356] "Generic (PLEG): container finished" podID="add52769-2b2d-4f30-baba-da8457c1c67f" containerID="d1bb4424a356ae318deb75a368402f3cacb3cb52dc63fa258d5dd1d4dbd07665" exitCode=0
Sep 17 18:35:21.209084 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:21.208938    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-rp8kx"
Sep 17 18:35:21.209084 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:21.208944    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-rp8kx" event={"ID":"add52769-2b2d-4f30-baba-da8457c1c67f","Type":"ContainerDied","Data":"d1bb4424a356ae318deb75a368402f3cacb3cb52dc63fa258d5dd1d4dbd07665"}
Sep 17 18:35:21.209084 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:21.209006    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-rp8kx" event={"ID":"add52769-2b2d-4f30-baba-da8457c1c67f","Type":"ContainerDied","Data":"5ba54f815cb84e89bb50b590025181cde73ac092fd9ca8b434802c4accaa2add"}
Sep 17 18:35:21.209084 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:21.209034    2553 scope.go:117] "RemoveContainer" containerID="95e958b8489c06e7cd3934793074ceffa088595c23f570f77009a5d749582d3c"
Sep 17 18:35:21.218171 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:21.218154    2553 scope.go:117] "RemoveContainer" containerID="d1bb4424a356ae318deb75a368402f3cacb3cb52dc63fa258d5dd1d4dbd07665"
Sep 17 18:35:21.225755 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:21.225732    2553 scope.go:117] "RemoveContainer" containerID="3bfbfa1f5c7d0b1e439a83d3696874089f4ede039260d6c8bf7f10121918ae6e"
Sep 17 18:35:21.232839 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:21.232812    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-rp8kx"]
Sep 17 18:35:21.234204 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:21.234188    2553 scope.go:117] "RemoveContainer" containerID="95e958b8489c06e7cd3934793074ceffa088595c23f570f77009a5d749582d3c"
Sep 17 18:35:21.234561 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:35:21.234535    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95e958b8489c06e7cd3934793074ceffa088595c23f570f77009a5d749582d3c\": container with ID starting with 95e958b8489c06e7cd3934793074ceffa088595c23f570f77009a5d749582d3c not found: ID does not exist" containerID="95e958b8489c06e7cd3934793074ceffa088595c23f570f77009a5d749582d3c"
Sep 17 18:35:21.234674 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:21.234573    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95e958b8489c06e7cd3934793074ceffa088595c23f570f77009a5d749582d3c"} err="failed to get container status \"95e958b8489c06e7cd3934793074ceffa088595c23f570f77009a5d749582d3c\": rpc error: code = NotFound desc = could not find container \"95e958b8489c06e7cd3934793074ceffa088595c23f570f77009a5d749582d3c\": container with ID starting with 95e958b8489c06e7cd3934793074ceffa088595c23f570f77009a5d749582d3c not found: ID does not exist"
Sep 17 18:35:21.234674 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:21.234603    2553 scope.go:117] "RemoveContainer" containerID="d1bb4424a356ae318deb75a368402f3cacb3cb52dc63fa258d5dd1d4dbd07665"
Sep 17 18:35:21.234914 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:35:21.234894    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1bb4424a356ae318deb75a368402f3cacb3cb52dc63fa258d5dd1d4dbd07665\": container with ID starting with d1bb4424a356ae318deb75a368402f3cacb3cb52dc63fa258d5dd1d4dbd07665 not found: ID does not exist" containerID="d1bb4424a356ae318deb75a368402f3cacb3cb52dc63fa258d5dd1d4dbd07665"
Sep 17 18:35:21.234954 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:21.234921    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1bb4424a356ae318deb75a368402f3cacb3cb52dc63fa258d5dd1d4dbd07665"} err="failed to get container status \"d1bb4424a356ae318deb75a368402f3cacb3cb52dc63fa258d5dd1d4dbd07665\": rpc error: code = NotFound desc = could not find container \"d1bb4424a356ae318deb75a368402f3cacb3cb52dc63fa258d5dd1d4dbd07665\": container with ID starting with d1bb4424a356ae318deb75a368402f3cacb3cb52dc63fa258d5dd1d4dbd07665 not found: ID does not exist"
Sep 17 18:35:21.234954 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:21.234937    2553 scope.go:117] "RemoveContainer" containerID="3bfbfa1f5c7d0b1e439a83d3696874089f4ede039260d6c8bf7f10121918ae6e"
Sep 17 18:35:21.235192 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:35:21.235174    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bfbfa1f5c7d0b1e439a83d3696874089f4ede039260d6c8bf7f10121918ae6e\": container with ID starting with 3bfbfa1f5c7d0b1e439a83d3696874089f4ede039260d6c8bf7f10121918ae6e not found: ID does not exist" containerID="3bfbfa1f5c7d0b1e439a83d3696874089f4ede039260d6c8bf7f10121918ae6e"
Sep 17 18:35:21.235252 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:21.235199    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bfbfa1f5c7d0b1e439a83d3696874089f4ede039260d6c8bf7f10121918ae6e"} err="failed to get container status \"3bfbfa1f5c7d0b1e439a83d3696874089f4ede039260d6c8bf7f10121918ae6e\": rpc error: code = NotFound desc = could not find container \"3bfbfa1f5c7d0b1e439a83d3696874089f4ede039260d6c8bf7f10121918ae6e\": container with ID starting with 3bfbfa1f5c7d0b1e439a83d3696874089f4ede039260d6c8bf7f10121918ae6e not found: ID does not exist"
Sep 17 18:35:21.238840 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:21.238811    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df7cf66bf-rp8kx"]
Sep 17 18:35:21.257242 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:21.257210    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-rt7hq"
Sep 17 18:35:21.280141 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:21.280065    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9290625f-2be6-4370-b08b-8955a51a8d4e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-b84cdfc7-hw6dc\" (UID: \"9290625f-2be6-4370-b08b-8955a51a8d4e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-hw6dc"
Sep 17 18:35:21.280141 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:21.280125    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9290625f-2be6-4370-b08b-8955a51a8d4e-kserve-provision-location\") pod \"isvc-canary-stop-predictor-b84cdfc7-hw6dc\" (UID: \"9290625f-2be6-4370-b08b-8955a51a8d4e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-hw6dc"
Sep 17 18:35:21.280287 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:21.280195    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9290625f-2be6-4370-b08b-8955a51a8d4e-proxy-tls\") pod \"isvc-canary-stop-predictor-b84cdfc7-hw6dc\" (UID: \"9290625f-2be6-4370-b08b-8955a51a8d4e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-hw6dc"
Sep 17 18:35:21.280287 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:21.280236    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9290625f-2be6-4370-b08b-8955a51a8d4e-cabundle-cert\") pod \"isvc-canary-stop-predictor-b84cdfc7-hw6dc\" (UID: \"9290625f-2be6-4370-b08b-8955a51a8d4e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-hw6dc"
Sep 17 18:35:21.280287 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:21.280267    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xjsd\" (UniqueName: \"kubernetes.io/projected/9290625f-2be6-4370-b08b-8955a51a8d4e-kube-api-access-9xjsd\") pod \"isvc-canary-stop-predictor-b84cdfc7-hw6dc\" (UID: \"9290625f-2be6-4370-b08b-8955a51a8d4e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-hw6dc"
Sep 17 18:35:21.280557 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:21.280537    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9290625f-2be6-4370-b08b-8955a51a8d4e-kserve-provision-location\") pod \"isvc-canary-stop-predictor-b84cdfc7-hw6dc\" (UID: \"9290625f-2be6-4370-b08b-8955a51a8d4e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-hw6dc"
Sep 17 18:35:21.280855 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:21.280835    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9290625f-2be6-4370-b08b-8955a51a8d4e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-b84cdfc7-hw6dc\" (UID: \"9290625f-2be6-4370-b08b-8955a51a8d4e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-hw6dc"
Sep 17 18:35:21.280930 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:21.280868    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9290625f-2be6-4370-b08b-8955a51a8d4e-cabundle-cert\") pod \"isvc-canary-stop-predictor-b84cdfc7-hw6dc\" (UID: \"9290625f-2be6-4370-b08b-8955a51a8d4e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-hw6dc"
Sep 17 18:35:21.282649 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:21.282632    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9290625f-2be6-4370-b08b-8955a51a8d4e-proxy-tls\") pod \"isvc-canary-stop-predictor-b84cdfc7-hw6dc\" (UID: \"9290625f-2be6-4370-b08b-8955a51a8d4e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-hw6dc"
Sep 17 18:35:21.290261 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:21.290237    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xjsd\" (UniqueName: \"kubernetes.io/projected/9290625f-2be6-4370-b08b-8955a51a8d4e-kube-api-access-9xjsd\") pod \"isvc-canary-stop-predictor-b84cdfc7-hw6dc\" (UID: \"9290625f-2be6-4370-b08b-8955a51a8d4e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-hw6dc"
Sep 17 18:35:21.335993 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:21.335962    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-hw6dc"
Sep 17 18:35:21.377251 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:21.377214    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="add52769-2b2d-4f30-baba-da8457c1c67f" path="/var/lib/kubelet/pods/add52769-2b2d-4f30-baba-da8457c1c67f/volumes"
Sep 17 18:35:21.531141 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:21.531067    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-rt7hq"]
Sep 17 18:35:21.533772 ip-10-0-137-95 kubenswrapper[2553]: W0917 18:35:21.533742    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod094068e2_76dd_41d0_8682_acddc8dd32f1.slice/crio-989ce651a7a0bd25ad5545ed78f9fd6a598a64046db870de76e6a8c84c9f095a WatchSource:0}: Error finding container 989ce651a7a0bd25ad5545ed78f9fd6a598a64046db870de76e6a8c84c9f095a: Status 404 returned error can't find the container with id 989ce651a7a0bd25ad5545ed78f9fd6a598a64046db870de76e6a8c84c9f095a
Sep 17 18:35:21.615001 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:21.614975    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-hw6dc"]
Sep 17 18:35:21.617827 ip-10-0-137-95 kubenswrapper[2553]: W0917 18:35:21.617796    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9290625f_2be6_4370_b08b_8955a51a8d4e.slice/crio-4cc3dee7318b94d83c2a3f06fdc047acb4637cd312b873e04d6a22568428fcf6 WatchSource:0}: Error finding container 4cc3dee7318b94d83c2a3f06fdc047acb4637cd312b873e04d6a22568428fcf6: Status 404 returned error can't find the container with id 4cc3dee7318b94d83c2a3f06fdc047acb4637cd312b873e04d6a22568428fcf6
Sep 17 18:35:22.216452 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:22.216395    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-rt7hq" event={"ID":"094068e2-76dd-41d0-8682-acddc8dd32f1","Type":"ContainerStarted","Data":"75279d871f2effd7c0e91ab473a2e21c1c39a9646dd5b5fdfeeda30e964feaa2"}
Sep 17 18:35:22.216984 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:22.216481    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-rt7hq" event={"ID":"094068e2-76dd-41d0-8682-acddc8dd32f1","Type":"ContainerStarted","Data":"989ce651a7a0bd25ad5545ed78f9fd6a598a64046db870de76e6a8c84c9f095a"}
Sep 17 18:35:22.219241 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:22.219206    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-hw6dc" event={"ID":"9290625f-2be6-4370-b08b-8955a51a8d4e","Type":"ContainerStarted","Data":"08ed73f1197f10d2112d49ebe040eefa25788348390767c492ab302a346de551"}
Sep 17 18:35:22.219384 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:22.219272    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-hw6dc" event={"ID":"9290625f-2be6-4370-b08b-8955a51a8d4e","Type":"ContainerStarted","Data":"4cc3dee7318b94d83c2a3f06fdc047acb4637cd312b873e04d6a22568428fcf6"}
Sep 17 18:35:25.982646 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:25.982542    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-xlspt" podUID="41bf207e-2eea-4616-9c26-d20142d1c329" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 18:35:26.239835 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:26.239750    2553 generic.go:356] "Generic (PLEG): container finished" podID="9290625f-2be6-4370-b08b-8955a51a8d4e" containerID="08ed73f1197f10d2112d49ebe040eefa25788348390767c492ab302a346de551" exitCode=0
Sep 17 18:35:26.239835 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:26.239820    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-hw6dc" event={"ID":"9290625f-2be6-4370-b08b-8955a51a8d4e","Type":"ContainerDied","Data":"08ed73f1197f10d2112d49ebe040eefa25788348390767c492ab302a346de551"}
Sep 17 18:35:26.241427 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:26.241406    2553 generic.go:356] "Generic (PLEG): container finished" podID="094068e2-76dd-41d0-8682-acddc8dd32f1" containerID="75279d871f2effd7c0e91ab473a2e21c1c39a9646dd5b5fdfeeda30e964feaa2" exitCode=0
Sep 17 18:35:26.241529 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:26.241449    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-rt7hq" event={"ID":"094068e2-76dd-41d0-8682-acddc8dd32f1","Type":"ContainerDied","Data":"75279d871f2effd7c0e91ab473a2e21c1c39a9646dd5b5fdfeeda30e964feaa2"}
Sep 17 18:35:27.252365 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:27.252328    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-hw6dc" event={"ID":"9290625f-2be6-4370-b08b-8955a51a8d4e","Type":"ContainerStarted","Data":"b86c0b56bcce07e0fefecd0430baba7a848c5140153704f02d0882cd5e92a0ee"}
Sep 17 18:35:27.252986 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:27.252404    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-hw6dc" event={"ID":"9290625f-2be6-4370-b08b-8955a51a8d4e","Type":"ContainerStarted","Data":"55339e0e302b81c8683fbf5c775de625657bad6093868739a8d5c89ef030c178"}
Sep 17 18:35:27.252986 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:27.252639    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-hw6dc"
Sep 17 18:35:27.254424 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:27.254398    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-rt7hq" event={"ID":"094068e2-76dd-41d0-8682-acddc8dd32f1","Type":"ContainerStarted","Data":"12546091cb647b6a35c140a7a5a2e3177daed88f90a0be316d97f12f6f841b1d"}
Sep 17 18:35:27.254560 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:27.254455    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-rt7hq" event={"ID":"094068e2-76dd-41d0-8682-acddc8dd32f1","Type":"ContainerStarted","Data":"0b10f5c7bb0a88d0d5234d308a8222addb138ab3c1be84fcb0c6b143b5a78eee"}
Sep 17 18:35:27.254677 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:27.254658    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-rt7hq"
Sep 17 18:35:27.272944 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:27.272877    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-hw6dc" podStartSLOduration=2.254412432 podStartE2EDuration="6.272857563s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.018445131s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:35:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:35:27.27214633 +0000 UTC m=+2321.968237013" watchObservedRunningTime="2026-09-17 18:35:27.272857563 +0000 UTC m=+2321.968948247"
Sep 17 18:35:27.291406 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:27.291352    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-rt7hq" podStartSLOduration=3.602163978 podStartE2EDuration="7.291335092s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.689171114s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:35:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:35:27.289362453 +0000 UTC m=+2321.985453138" watchObservedRunningTime="2026-09-17 18:35:27.291335092 +0000 UTC m=+2321.987425775"
Sep 17 18:35:28.258850 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:28.258813    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-hw6dc"
Sep 17 18:35:28.259332 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:28.258878    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-rt7hq"
Sep 17 18:35:28.259902 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:28.259874    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-rt7hq" podUID="094068e2-76dd-41d0-8682-acddc8dd32f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 18:35:28.260005 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:28.259951    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-hw6dc" podUID="9290625f-2be6-4370-b08b-8955a51a8d4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 18:35:29.262759 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:29.262660    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-hw6dc" podUID="9290625f-2be6-4370-b08b-8955a51a8d4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 18:35:29.263142 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:29.262792    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-rt7hq" podUID="094068e2-76dd-41d0-8682-acddc8dd32f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 18:35:34.267347 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:34.267312    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-rt7hq"
Sep 17 18:35:34.267836 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:34.267772    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-hw6dc"
Sep 17 18:35:34.267900 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:34.267835    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-rt7hq" podUID="094068e2-76dd-41d0-8682-acddc8dd32f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 18:35:34.268170 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:34.268152    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-hw6dc" podUID="9290625f-2be6-4370-b08b-8955a51a8d4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 18:35:35.982780 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:35.982732    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-xlspt" podUID="41bf207e-2eea-4616-9c26-d20142d1c329" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 18:35:44.268654 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:44.268598    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-rt7hq" podUID="094068e2-76dd-41d0-8682-acddc8dd32f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 18:35:44.269142 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:44.268598    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-hw6dc" podUID="9290625f-2be6-4370-b08b-8955a51a8d4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 18:35:45.983245 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:45.983205    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-xlspt"
Sep 17 18:35:52.547967 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:52.547933    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-xlspt"]
Sep 17 18:35:52.548551 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:52.548203    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-xlspt" podUID="41bf207e-2eea-4616-9c26-d20142d1c329" containerName="kserve-container" containerID="cri-o://634863b265aba6d40287102e6d7077c6041c89c2738be9f47245acc54aa1aeab" gracePeriod=30
Sep 17 18:35:52.548551 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:52.548252    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-xlspt" podUID="41bf207e-2eea-4616-9c26-d20142d1c329" containerName="kube-rbac-proxy" containerID="cri-o://1eb51f9f33ae7d8e0ab8c0f9ff884d1aa751bb87bf26be2031b3173ba55e56f8" gracePeriod=30
Sep 17 18:35:53.380486 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:53.380452    2553 generic.go:356] "Generic (PLEG): container finished" podID="41bf207e-2eea-4616-9c26-d20142d1c329" containerID="1eb51f9f33ae7d8e0ab8c0f9ff884d1aa751bb87bf26be2031b3173ba55e56f8" exitCode=2
Sep 17 18:35:53.380691 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:53.380506    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-xlspt" event={"ID":"41bf207e-2eea-4616-9c26-d20142d1c329","Type":"ContainerDied","Data":"1eb51f9f33ae7d8e0ab8c0f9ff884d1aa751bb87bf26be2031b3173ba55e56f8"}
Sep 17 18:35:54.267944 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:54.267905    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-rt7hq" podUID="094068e2-76dd-41d0-8682-acddc8dd32f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 18:35:54.268337 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:54.268152    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-hw6dc" podUID="9290625f-2be6-4370-b08b-8955a51a8d4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 18:35:55.977424 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:55.977378    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-xlspt" podUID="41bf207e-2eea-4616-9c26-d20142d1c329" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.47:8643/healthz\": dial tcp 10.134.0.47:8643: connect: connection refused"
Sep 17 18:35:55.982463 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:55.982428    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-xlspt" podUID="41bf207e-2eea-4616-9c26-d20142d1c329" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 18:35:56.996782 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:56.996756    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-xlspt"
Sep 17 18:35:57.105783 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:57.105677    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41bf207e-2eea-4616-9c26-d20142d1c329-kserve-provision-location\") pod \"41bf207e-2eea-4616-9c26-d20142d1c329\" (UID: \"41bf207e-2eea-4616-9c26-d20142d1c329\") "
Sep 17 18:35:57.105783 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:57.105724    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41bf207e-2eea-4616-9c26-d20142d1c329-proxy-tls\") pod \"41bf207e-2eea-4616-9c26-d20142d1c329\" (UID: \"41bf207e-2eea-4616-9c26-d20142d1c329\") "
Sep 17 18:35:57.105783 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:57.105742    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b45wr\" (UniqueName: \"kubernetes.io/projected/41bf207e-2eea-4616-9c26-d20142d1c329-kube-api-access-b45wr\") pod \"41bf207e-2eea-4616-9c26-d20142d1c329\" (UID: \"41bf207e-2eea-4616-9c26-d20142d1c329\") "
Sep 17 18:35:57.106082 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:57.105798    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41bf207e-2eea-4616-9c26-d20142d1c329-cabundle-cert\") pod \"41bf207e-2eea-4616-9c26-d20142d1c329\" (UID: \"41bf207e-2eea-4616-9c26-d20142d1c329\") "
Sep 17 18:35:57.106082 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:57.105822    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41bf207e-2eea-4616-9c26-d20142d1c329-kube-rbac-proxy-sar-config\") pod \"41bf207e-2eea-4616-9c26-d20142d1c329\" (UID: \"41bf207e-2eea-4616-9c26-d20142d1c329\") "
Sep 17 18:35:57.106196 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:57.106078    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41bf207e-2eea-4616-9c26-d20142d1c329-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "41bf207e-2eea-4616-9c26-d20142d1c329" (UID: "41bf207e-2eea-4616-9c26-d20142d1c329"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:35:57.106244 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:57.106190    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41bf207e-2eea-4616-9c26-d20142d1c329-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "41bf207e-2eea-4616-9c26-d20142d1c329" (UID: "41bf207e-2eea-4616-9c26-d20142d1c329"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:35:57.106282 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:57.106251    2553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41bf207e-2eea-4616-9c26-d20142d1c329-kserve-provision-location\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:35:57.106282 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:57.106257    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41bf207e-2eea-4616-9c26-d20142d1c329-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "41bf207e-2eea-4616-9c26-d20142d1c329" (UID: "41bf207e-2eea-4616-9c26-d20142d1c329"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:35:57.108474 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:57.108448    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41bf207e-2eea-4616-9c26-d20142d1c329-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "41bf207e-2eea-4616-9c26-d20142d1c329" (UID: "41bf207e-2eea-4616-9c26-d20142d1c329"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:35:57.108594 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:57.108450    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41bf207e-2eea-4616-9c26-d20142d1c329-kube-api-access-b45wr" (OuterVolumeSpecName: "kube-api-access-b45wr") pod "41bf207e-2eea-4616-9c26-d20142d1c329" (UID: "41bf207e-2eea-4616-9c26-d20142d1c329"). InnerVolumeSpecName "kube-api-access-b45wr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:35:57.207471 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:57.207430    2553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41bf207e-2eea-4616-9c26-d20142d1c329-proxy-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:35:57.207471 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:57.207464    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b45wr\" (UniqueName: \"kubernetes.io/projected/41bf207e-2eea-4616-9c26-d20142d1c329-kube-api-access-b45wr\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:35:57.207471 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:57.207473    2553 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41bf207e-2eea-4616-9c26-d20142d1c329-cabundle-cert\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:35:57.207752 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:57.207486    2553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41bf207e-2eea-4616-9c26-d20142d1c329-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:35:57.401077 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:57.401045    2553 generic.go:356] "Generic (PLEG): container finished" podID="41bf207e-2eea-4616-9c26-d20142d1c329" containerID="634863b265aba6d40287102e6d7077c6041c89c2738be9f47245acc54aa1aeab" exitCode=0
Sep 17 18:35:57.401236 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:57.401094    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-xlspt"
Sep 17 18:35:57.401236 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:57.401123    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-xlspt" event={"ID":"41bf207e-2eea-4616-9c26-d20142d1c329","Type":"ContainerDied","Data":"634863b265aba6d40287102e6d7077c6041c89c2738be9f47245acc54aa1aeab"}
Sep 17 18:35:57.401236 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:57.401195    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-xlspt" event={"ID":"41bf207e-2eea-4616-9c26-d20142d1c329","Type":"ContainerDied","Data":"a56bc1b845aad59ab2bfe3aef9fe9dbd0e421f555d9b1012f21056a8479f163f"}
Sep 17 18:35:57.401236 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:57.401223    2553 scope.go:117] "RemoveContainer" containerID="1eb51f9f33ae7d8e0ab8c0f9ff884d1aa751bb87bf26be2031b3173ba55e56f8"
Sep 17 18:35:57.409798 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:57.409781    2553 scope.go:117] "RemoveContainer" containerID="634863b265aba6d40287102e6d7077c6041c89c2738be9f47245acc54aa1aeab"
Sep 17 18:35:57.417562 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:57.417537    2553 scope.go:117] "RemoveContainer" containerID="a0e16276af3c631c2899dd5a3ae576fe9dd638db8d1f6b71072f22d319371adb"
Sep 17 18:35:57.418964 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:57.418945    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-xlspt"]
Sep 17 18:35:57.422455 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:57.422434    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-xlspt"]
Sep 17 18:35:57.425927 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:57.425909    2553 scope.go:117] "RemoveContainer" containerID="1eb51f9f33ae7d8e0ab8c0f9ff884d1aa751bb87bf26be2031b3173ba55e56f8"
Sep 17 18:35:57.426241 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:35:57.426223    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1eb51f9f33ae7d8e0ab8c0f9ff884d1aa751bb87bf26be2031b3173ba55e56f8\": container with ID starting with 1eb51f9f33ae7d8e0ab8c0f9ff884d1aa751bb87bf26be2031b3173ba55e56f8 not found: ID does not exist" containerID="1eb51f9f33ae7d8e0ab8c0f9ff884d1aa751bb87bf26be2031b3173ba55e56f8"
Sep 17 18:35:57.426293 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:57.426251    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1eb51f9f33ae7d8e0ab8c0f9ff884d1aa751bb87bf26be2031b3173ba55e56f8"} err="failed to get container status \"1eb51f9f33ae7d8e0ab8c0f9ff884d1aa751bb87bf26be2031b3173ba55e56f8\": rpc error: code = NotFound desc = could not find container \"1eb51f9f33ae7d8e0ab8c0f9ff884d1aa751bb87bf26be2031b3173ba55e56f8\": container with ID starting with 1eb51f9f33ae7d8e0ab8c0f9ff884d1aa751bb87bf26be2031b3173ba55e56f8 not found: ID does not exist"
Sep 17 18:35:57.426293 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:57.426269    2553 scope.go:117] "RemoveContainer" containerID="634863b265aba6d40287102e6d7077c6041c89c2738be9f47245acc54aa1aeab"
Sep 17 18:35:57.426518 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:35:57.426499    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"634863b265aba6d40287102e6d7077c6041c89c2738be9f47245acc54aa1aeab\": container with ID starting with 634863b265aba6d40287102e6d7077c6041c89c2738be9f47245acc54aa1aeab not found: ID does not exist" containerID="634863b265aba6d40287102e6d7077c6041c89c2738be9f47245acc54aa1aeab"
Sep 17 18:35:57.426582 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:57.426527    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"634863b265aba6d40287102e6d7077c6041c89c2738be9f47245acc54aa1aeab"} err="failed to get container status \"634863b265aba6d40287102e6d7077c6041c89c2738be9f47245acc54aa1aeab\": rpc error: code = NotFound desc = could not find container \"634863b265aba6d40287102e6d7077c6041c89c2738be9f47245acc54aa1aeab\": container with ID starting with 634863b265aba6d40287102e6d7077c6041c89c2738be9f47245acc54aa1aeab not found: ID does not exist"
Sep 17 18:35:57.426582 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:57.426549    2553 scope.go:117] "RemoveContainer" containerID="a0e16276af3c631c2899dd5a3ae576fe9dd638db8d1f6b71072f22d319371adb"
Sep 17 18:35:57.426802 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:35:57.426785    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0e16276af3c631c2899dd5a3ae576fe9dd638db8d1f6b71072f22d319371adb\": container with ID starting with a0e16276af3c631c2899dd5a3ae576fe9dd638db8d1f6b71072f22d319371adb not found: ID does not exist" containerID="a0e16276af3c631c2899dd5a3ae576fe9dd638db8d1f6b71072f22d319371adb"
Sep 17 18:35:57.426844 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:57.426806    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0e16276af3c631c2899dd5a3ae576fe9dd638db8d1f6b71072f22d319371adb"} err="failed to get container status \"a0e16276af3c631c2899dd5a3ae576fe9dd638db8d1f6b71072f22d319371adb\": rpc error: code = NotFound desc = could not find container \"a0e16276af3c631c2899dd5a3ae576fe9dd638db8d1f6b71072f22d319371adb\": container with ID starting with a0e16276af3c631c2899dd5a3ae576fe9dd638db8d1f6b71072f22d319371adb not found: ID does not exist"
Sep 17 18:35:59.376911 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:35:59.376879    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41bf207e-2eea-4616-9c26-d20142d1c329" path="/var/lib/kubelet/pods/41bf207e-2eea-4616-9c26-d20142d1c329/volumes"
Sep 17 18:36:04.268683 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:36:04.268634    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-rt7hq" podUID="094068e2-76dd-41d0-8682-acddc8dd32f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 18:36:04.269086 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:36:04.268642    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-hw6dc" podUID="9290625f-2be6-4370-b08b-8955a51a8d4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 18:36:14.267874 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:36:14.267820    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-rt7hq" podUID="094068e2-76dd-41d0-8682-acddc8dd32f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 18:36:14.268284 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:36:14.268219    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-hw6dc" podUID="9290625f-2be6-4370-b08b-8955a51a8d4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 18:36:24.268430 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:36:24.268386    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-hw6dc" podUID="9290625f-2be6-4370-b08b-8955a51a8d4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 18:36:24.268960 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:36:24.268386    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-rt7hq" podUID="094068e2-76dd-41d0-8682-acddc8dd32f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 18:36:34.267881 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:36:34.267836    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-rt7hq" podUID="094068e2-76dd-41d0-8682-acddc8dd32f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 18:36:34.268297 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:36:34.268131    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-hw6dc" podUID="9290625f-2be6-4370-b08b-8955a51a8d4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 18:36:42.371472 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:36:42.371422    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-rt7hq" podUID="094068e2-76dd-41d0-8682-acddc8dd32f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 18:36:44.269831 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:36:44.269797    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-hw6dc"
Sep 17 18:36:47.460587 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:36:47.460554    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-btmqz_622ee5be-fb50-42b6-ae23-42900b7c2d98/ovn-acl-logging/0.log"
Sep 17 18:36:47.467661 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:36:47.467632    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-btmqz_622ee5be-fb50-42b6-ae23-42900b7c2d98/ovn-acl-logging/0.log"
Sep 17 18:36:52.372557 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:36:52.372520    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-rt7hq"
Sep 17 18:36:55.945244 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:36:55.945207    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-rt7hq"]
Sep 17 18:36:55.945801 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:36:55.945494    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-rt7hq" podUID="094068e2-76dd-41d0-8682-acddc8dd32f1" containerName="kserve-container" containerID="cri-o://0b10f5c7bb0a88d0d5234d308a8222addb138ab3c1be84fcb0c6b143b5a78eee" gracePeriod=30
Sep 17 18:36:55.945801 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:36:55.945560    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-rt7hq" podUID="094068e2-76dd-41d0-8682-acddc8dd32f1" containerName="kube-rbac-proxy" containerID="cri-o://12546091cb647b6a35c140a7a5a2e3177daed88f90a0be316d97f12f6f841b1d" gracePeriod=30
Sep 17 18:36:55.973994 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:36:55.973964    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-hw6dc"]
Sep 17 18:36:55.974287 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:36:55.974251    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-hw6dc" podUID="9290625f-2be6-4370-b08b-8955a51a8d4e" containerName="kserve-container" containerID="cri-o://55339e0e302b81c8683fbf5c775de625657bad6093868739a8d5c89ef030c178" gracePeriod=30
Sep 17 18:36:55.974408 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:36:55.974295    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-hw6dc" podUID="9290625f-2be6-4370-b08b-8955a51a8d4e" containerName="kube-rbac-proxy" containerID="cri-o://b86c0b56bcce07e0fefecd0430baba7a848c5140153704f02d0882cd5e92a0ee" gracePeriod=30
Sep 17 18:36:56.681704 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:36:56.681667    2553 generic.go:356] "Generic (PLEG): container finished" podID="9290625f-2be6-4370-b08b-8955a51a8d4e" containerID="b86c0b56bcce07e0fefecd0430baba7a848c5140153704f02d0882cd5e92a0ee" exitCode=2
Sep 17 18:36:56.681907 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:36:56.681742    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-hw6dc" event={"ID":"9290625f-2be6-4370-b08b-8955a51a8d4e","Type":"ContainerDied","Data":"b86c0b56bcce07e0fefecd0430baba7a848c5140153704f02d0882cd5e92a0ee"}
Sep 17 18:36:56.683882 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:36:56.683860    2553 generic.go:356] "Generic (PLEG): container finished" podID="094068e2-76dd-41d0-8682-acddc8dd32f1" containerID="12546091cb647b6a35c140a7a5a2e3177daed88f90a0be316d97f12f6f841b1d" exitCode=2
Sep 17 18:36:56.683990 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:36:56.683896    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-rt7hq" event={"ID":"094068e2-76dd-41d0-8682-acddc8dd32f1","Type":"ContainerDied","Data":"12546091cb647b6a35c140a7a5a2e3177daed88f90a0be316d97f12f6f841b1d"}
Sep 17 18:36:59.263317 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:36:59.263228    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-hw6dc" podUID="9290625f-2be6-4370-b08b-8955a51a8d4e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.49:8643/healthz\": dial tcp 10.134.0.49:8643: connect: connection refused"
Sep 17 18:36:59.263710 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:36:59.263228    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-rt7hq" podUID="094068e2-76dd-41d0-8682-acddc8dd32f1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.48:8643/healthz\": dial tcp 10.134.0.48:8643: connect: connection refused"
Sep 17 18:37:01.037670 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:01.037637    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-rt7hq"
Sep 17 18:37:01.041303 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:01.041281    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-hw6dc"
Sep 17 18:37:01.079244 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:01.079215    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/094068e2-76dd-41d0-8682-acddc8dd32f1-proxy-tls\") pod \"094068e2-76dd-41d0-8682-acddc8dd32f1\" (UID: \"094068e2-76dd-41d0-8682-acddc8dd32f1\") "
Sep 17 18:37:01.079244 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:01.079259    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9290625f-2be6-4370-b08b-8955a51a8d4e-cabundle-cert\") pod \"9290625f-2be6-4370-b08b-8955a51a8d4e\" (UID: \"9290625f-2be6-4370-b08b-8955a51a8d4e\") "
Sep 17 18:37:01.079498 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:01.079299    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9290625f-2be6-4370-b08b-8955a51a8d4e-kube-rbac-proxy-sar-config\") pod \"9290625f-2be6-4370-b08b-8955a51a8d4e\" (UID: \"9290625f-2be6-4370-b08b-8955a51a8d4e\") "
Sep 17 18:37:01.079498 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:01.079341    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/094068e2-76dd-41d0-8682-acddc8dd32f1-kube-rbac-proxy-sar-config\") pod \"094068e2-76dd-41d0-8682-acddc8dd32f1\" (UID: \"094068e2-76dd-41d0-8682-acddc8dd32f1\") "
Sep 17 18:37:01.079498 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:01.079369    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/094068e2-76dd-41d0-8682-acddc8dd32f1-cabundle-cert\") pod \"094068e2-76dd-41d0-8682-acddc8dd32f1\" (UID: \"094068e2-76dd-41d0-8682-acddc8dd32f1\") "
Sep 17 18:37:01.079498 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:01.079417    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9xjsd\" (UniqueName: \"kubernetes.io/projected/9290625f-2be6-4370-b08b-8955a51a8d4e-kube-api-access-9xjsd\") pod \"9290625f-2be6-4370-b08b-8955a51a8d4e\" (UID: \"9290625f-2be6-4370-b08b-8955a51a8d4e\") "
Sep 17 18:37:01.079498 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:01.079444    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/094068e2-76dd-41d0-8682-acddc8dd32f1-kserve-provision-location\") pod \"094068e2-76dd-41d0-8682-acddc8dd32f1\" (UID: \"094068e2-76dd-41d0-8682-acddc8dd32f1\") "
Sep 17 18:37:01.080237 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:01.079505    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vw7vt\" (UniqueName: \"kubernetes.io/projected/094068e2-76dd-41d0-8682-acddc8dd32f1-kube-api-access-vw7vt\") pod \"094068e2-76dd-41d0-8682-acddc8dd32f1\" (UID: \"094068e2-76dd-41d0-8682-acddc8dd32f1\") "
Sep 17 18:37:01.080237 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:01.079530    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9290625f-2be6-4370-b08b-8955a51a8d4e-kserve-provision-location\") pod \"9290625f-2be6-4370-b08b-8955a51a8d4e\" (UID: \"9290625f-2be6-4370-b08b-8955a51a8d4e\") "
Sep 17 18:37:01.080237 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:01.079555    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9290625f-2be6-4370-b08b-8955a51a8d4e-proxy-tls\") pod \"9290625f-2be6-4370-b08b-8955a51a8d4e\" (UID: \"9290625f-2be6-4370-b08b-8955a51a8d4e\") "
Sep 17 18:37:01.080237 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:01.079840    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/094068e2-76dd-41d0-8682-acddc8dd32f1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "094068e2-76dd-41d0-8682-acddc8dd32f1" (UID: "094068e2-76dd-41d0-8682-acddc8dd32f1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:37:01.080237 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:01.079901    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/094068e2-76dd-41d0-8682-acddc8dd32f1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "094068e2-76dd-41d0-8682-acddc8dd32f1" (UID: "094068e2-76dd-41d0-8682-acddc8dd32f1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:37:01.080237 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:01.079934    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9290625f-2be6-4370-b08b-8955a51a8d4e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9290625f-2be6-4370-b08b-8955a51a8d4e" (UID: "9290625f-2be6-4370-b08b-8955a51a8d4e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:37:01.080237 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:01.080035    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/094068e2-76dd-41d0-8682-acddc8dd32f1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "094068e2-76dd-41d0-8682-acddc8dd32f1" (UID: "094068e2-76dd-41d0-8682-acddc8dd32f1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:37:01.080237 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:01.080039    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9290625f-2be6-4370-b08b-8955a51a8d4e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9290625f-2be6-4370-b08b-8955a51a8d4e" (UID: "9290625f-2be6-4370-b08b-8955a51a8d4e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:37:01.080921 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:01.080898    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9290625f-2be6-4370-b08b-8955a51a8d4e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9290625f-2be6-4370-b08b-8955a51a8d4e" (UID: "9290625f-2be6-4370-b08b-8955a51a8d4e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:37:01.081904 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:01.081878    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/094068e2-76dd-41d0-8682-acddc8dd32f1-kube-api-access-vw7vt" (OuterVolumeSpecName: "kube-api-access-vw7vt") pod "094068e2-76dd-41d0-8682-acddc8dd32f1" (UID: "094068e2-76dd-41d0-8682-acddc8dd32f1"). InnerVolumeSpecName "kube-api-access-vw7vt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:37:01.082017 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:01.082000    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9290625f-2be6-4370-b08b-8955a51a8d4e-kube-api-access-9xjsd" (OuterVolumeSpecName: "kube-api-access-9xjsd") pod "9290625f-2be6-4370-b08b-8955a51a8d4e" (UID: "9290625f-2be6-4370-b08b-8955a51a8d4e"). InnerVolumeSpecName "kube-api-access-9xjsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:37:01.082084 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:01.082005    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/094068e2-76dd-41d0-8682-acddc8dd32f1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "094068e2-76dd-41d0-8682-acddc8dd32f1" (UID: "094068e2-76dd-41d0-8682-acddc8dd32f1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:37:01.082084 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:01.082032    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9290625f-2be6-4370-b08b-8955a51a8d4e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9290625f-2be6-4370-b08b-8955a51a8d4e" (UID: "9290625f-2be6-4370-b08b-8955a51a8d4e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:37:01.180710 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:01.180674    2553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/094068e2-76dd-41d0-8682-acddc8dd32f1-proxy-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:37:01.180710 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:01.180703    2553 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9290625f-2be6-4370-b08b-8955a51a8d4e-cabundle-cert\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:37:01.180710 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:01.180712    2553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9290625f-2be6-4370-b08b-8955a51a8d4e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:37:01.180939 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:01.180721    2553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/094068e2-76dd-41d0-8682-acddc8dd32f1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:37:01.180939 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:01.180730    2553 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/094068e2-76dd-41d0-8682-acddc8dd32f1-cabundle-cert\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:37:01.180939 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:01.180738    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xjsd\" (UniqueName: \"kubernetes.io/projected/9290625f-2be6-4370-b08b-8955a51a8d4e-kube-api-access-9xjsd\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:37:01.180939 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:01.180746    2553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/094068e2-76dd-41d0-8682-acddc8dd32f1-kserve-provision-location\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:37:01.180939 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:01.180754    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vw7vt\" (UniqueName: \"kubernetes.io/projected/094068e2-76dd-41d0-8682-acddc8dd32f1-kube-api-access-vw7vt\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:37:01.180939 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:01.180763    2553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9290625f-2be6-4370-b08b-8955a51a8d4e-kserve-provision-location\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:37:01.180939 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:01.180772    2553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9290625f-2be6-4370-b08b-8955a51a8d4e-proxy-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:37:01.707144 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:01.707112    2553 generic.go:356] "Generic (PLEG): container finished" podID="9290625f-2be6-4370-b08b-8955a51a8d4e" containerID="55339e0e302b81c8683fbf5c775de625657bad6093868739a8d5c89ef030c178" exitCode=0
Sep 17 18:37:01.707357 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:01.707177    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-hw6dc"
Sep 17 18:37:01.707357 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:01.707193    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-hw6dc" event={"ID":"9290625f-2be6-4370-b08b-8955a51a8d4e","Type":"ContainerDied","Data":"55339e0e302b81c8683fbf5c775de625657bad6093868739a8d5c89ef030c178"}
Sep 17 18:37:01.707357 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:01.707249    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-hw6dc" event={"ID":"9290625f-2be6-4370-b08b-8955a51a8d4e","Type":"ContainerDied","Data":"4cc3dee7318b94d83c2a3f06fdc047acb4637cd312b873e04d6a22568428fcf6"}
Sep 17 18:37:01.707357 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:01.707276    2553 scope.go:117] "RemoveContainer" containerID="b86c0b56bcce07e0fefecd0430baba7a848c5140153704f02d0882cd5e92a0ee"
Sep 17 18:37:01.709416 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:01.709366    2553 generic.go:356] "Generic (PLEG): container finished" podID="094068e2-76dd-41d0-8682-acddc8dd32f1" containerID="0b10f5c7bb0a88d0d5234d308a8222addb138ab3c1be84fcb0c6b143b5a78eee" exitCode=0
Sep 17 18:37:01.709538 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:01.709409    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-rt7hq" event={"ID":"094068e2-76dd-41d0-8682-acddc8dd32f1","Type":"ContainerDied","Data":"0b10f5c7bb0a88d0d5234d308a8222addb138ab3c1be84fcb0c6b143b5a78eee"}
Sep 17 18:37:01.709538 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:01.709421    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-rt7hq"
Sep 17 18:37:01.709538 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:01.709483    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-rt7hq" event={"ID":"094068e2-76dd-41d0-8682-acddc8dd32f1","Type":"ContainerDied","Data":"989ce651a7a0bd25ad5545ed78f9fd6a598a64046db870de76e6a8c84c9f095a"}
Sep 17 18:37:01.715783 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:01.715766    2553 scope.go:117] "RemoveContainer" containerID="55339e0e302b81c8683fbf5c775de625657bad6093868739a8d5c89ef030c178"
Sep 17 18:37:01.722994 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:01.722974    2553 scope.go:117] "RemoveContainer" containerID="08ed73f1197f10d2112d49ebe040eefa25788348390767c492ab302a346de551"
Sep 17 18:37:01.726114 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:01.726090    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-hw6dc"]
Sep 17 18:37:01.727898 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:01.727875    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-b84cdfc7-hw6dc"]
Sep 17 18:37:01.731339 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:01.731319    2553 scope.go:117] "RemoveContainer" containerID="b86c0b56bcce07e0fefecd0430baba7a848c5140153704f02d0882cd5e92a0ee"
Sep 17 18:37:01.731665 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:37:01.731645    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b86c0b56bcce07e0fefecd0430baba7a848c5140153704f02d0882cd5e92a0ee\": container with ID starting with b86c0b56bcce07e0fefecd0430baba7a848c5140153704f02d0882cd5e92a0ee not found: ID does not exist" containerID="b86c0b56bcce07e0fefecd0430baba7a848c5140153704f02d0882cd5e92a0ee"
Sep 17 18:37:01.731724 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:01.731675    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b86c0b56bcce07e0fefecd0430baba7a848c5140153704f02d0882cd5e92a0ee"} err="failed to get container status \"b86c0b56bcce07e0fefecd0430baba7a848c5140153704f02d0882cd5e92a0ee\": rpc error: code = NotFound desc = could not find container \"b86c0b56bcce07e0fefecd0430baba7a848c5140153704f02d0882cd5e92a0ee\": container with ID starting with b86c0b56bcce07e0fefecd0430baba7a848c5140153704f02d0882cd5e92a0ee not found: ID does not exist"
Sep 17 18:37:01.731724 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:01.731693    2553 scope.go:117] "RemoveContainer" containerID="55339e0e302b81c8683fbf5c775de625657bad6093868739a8d5c89ef030c178"
Sep 17 18:37:01.731957 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:37:01.731940    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55339e0e302b81c8683fbf5c775de625657bad6093868739a8d5c89ef030c178\": container with ID starting with 55339e0e302b81c8683fbf5c775de625657bad6093868739a8d5c89ef030c178 not found: ID does not exist" containerID="55339e0e302b81c8683fbf5c775de625657bad6093868739a8d5c89ef030c178"
Sep 17 18:37:01.732003 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:01.731963    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55339e0e302b81c8683fbf5c775de625657bad6093868739a8d5c89ef030c178"} err="failed to get container status \"55339e0e302b81c8683fbf5c775de625657bad6093868739a8d5c89ef030c178\": rpc error: code = NotFound desc = could not find container \"55339e0e302b81c8683fbf5c775de625657bad6093868739a8d5c89ef030c178\": container with ID starting with 55339e0e302b81c8683fbf5c775de625657bad6093868739a8d5c89ef030c178 not found: ID does not exist"
Sep 17 18:37:01.732003 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:01.731980    2553 scope.go:117] "RemoveContainer" containerID="08ed73f1197f10d2112d49ebe040eefa25788348390767c492ab302a346de551"
Sep 17 18:37:01.732204 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:37:01.732186    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08ed73f1197f10d2112d49ebe040eefa25788348390767c492ab302a346de551\": container with ID starting with 08ed73f1197f10d2112d49ebe040eefa25788348390767c492ab302a346de551 not found: ID does not exist" containerID="08ed73f1197f10d2112d49ebe040eefa25788348390767c492ab302a346de551"
Sep 17 18:37:01.732276 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:01.732212    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08ed73f1197f10d2112d49ebe040eefa25788348390767c492ab302a346de551"} err="failed to get container status \"08ed73f1197f10d2112d49ebe040eefa25788348390767c492ab302a346de551\": rpc error: code = NotFound desc = could not find container \"08ed73f1197f10d2112d49ebe040eefa25788348390767c492ab302a346de551\": container with ID starting with 08ed73f1197f10d2112d49ebe040eefa25788348390767c492ab302a346de551 not found: ID does not exist"
Sep 17 18:37:01.732276 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:01.732236    2553 scope.go:117] "RemoveContainer" containerID="12546091cb647b6a35c140a7a5a2e3177daed88f90a0be316d97f12f6f841b1d"
Sep 17 18:37:01.738962 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:01.738942    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-rt7hq"]
Sep 17 18:37:01.740000 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:01.739985    2553 scope.go:117] "RemoveContainer" containerID="0b10f5c7bb0a88d0d5234d308a8222addb138ab3c1be84fcb0c6b143b5a78eee"
Sep 17 18:37:01.742611 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:01.742593    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-867f674bf6-rt7hq"]
Sep 17 18:37:01.747483 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:01.747467    2553 scope.go:117] "RemoveContainer" containerID="75279d871f2effd7c0e91ab473a2e21c1c39a9646dd5b5fdfeeda30e964feaa2"
Sep 17 18:37:01.754766 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:01.754748    2553 scope.go:117] "RemoveContainer" containerID="12546091cb647b6a35c140a7a5a2e3177daed88f90a0be316d97f12f6f841b1d"
Sep 17 18:37:01.755029 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:37:01.755009    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12546091cb647b6a35c140a7a5a2e3177daed88f90a0be316d97f12f6f841b1d\": container with ID starting with 12546091cb647b6a35c140a7a5a2e3177daed88f90a0be316d97f12f6f841b1d not found: ID does not exist" containerID="12546091cb647b6a35c140a7a5a2e3177daed88f90a0be316d97f12f6f841b1d"
Sep 17 18:37:01.755102 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:01.755038    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12546091cb647b6a35c140a7a5a2e3177daed88f90a0be316d97f12f6f841b1d"} err="failed to get container status \"12546091cb647b6a35c140a7a5a2e3177daed88f90a0be316d97f12f6f841b1d\": rpc error: code = NotFound desc = could not find container \"12546091cb647b6a35c140a7a5a2e3177daed88f90a0be316d97f12f6f841b1d\": container with ID starting with 12546091cb647b6a35c140a7a5a2e3177daed88f90a0be316d97f12f6f841b1d not found: ID does not exist"
Sep 17 18:37:01.755102 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:01.755063    2553 scope.go:117] "RemoveContainer" containerID="0b10f5c7bb0a88d0d5234d308a8222addb138ab3c1be84fcb0c6b143b5a78eee"
Sep 17 18:37:01.755339 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:37:01.755322    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b10f5c7bb0a88d0d5234d308a8222addb138ab3c1be84fcb0c6b143b5a78eee\": container with ID starting with 0b10f5c7bb0a88d0d5234d308a8222addb138ab3c1be84fcb0c6b143b5a78eee not found: ID does not exist" containerID="0b10f5c7bb0a88d0d5234d308a8222addb138ab3c1be84fcb0c6b143b5a78eee"
Sep 17 18:37:01.755376 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:01.755346    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b10f5c7bb0a88d0d5234d308a8222addb138ab3c1be84fcb0c6b143b5a78eee"} err="failed to get container status \"0b10f5c7bb0a88d0d5234d308a8222addb138ab3c1be84fcb0c6b143b5a78eee\": rpc error: code = NotFound desc = could not find container \"0b10f5c7bb0a88d0d5234d308a8222addb138ab3c1be84fcb0c6b143b5a78eee\": container with ID starting with 0b10f5c7bb0a88d0d5234d308a8222addb138ab3c1be84fcb0c6b143b5a78eee not found: ID does not exist"
Sep 17 18:37:01.755376 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:01.755363    2553 scope.go:117] "RemoveContainer" containerID="75279d871f2effd7c0e91ab473a2e21c1c39a9646dd5b5fdfeeda30e964feaa2"
Sep 17 18:37:01.755558 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:37:01.755543    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75279d871f2effd7c0e91ab473a2e21c1c39a9646dd5b5fdfeeda30e964feaa2\": container with ID starting with 75279d871f2effd7c0e91ab473a2e21c1c39a9646dd5b5fdfeeda30e964feaa2 not found: ID does not exist" containerID="75279d871f2effd7c0e91ab473a2e21c1c39a9646dd5b5fdfeeda30e964feaa2"
Sep 17 18:37:01.755604 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:01.755560    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75279d871f2effd7c0e91ab473a2e21c1c39a9646dd5b5fdfeeda30e964feaa2"} err="failed to get container status \"75279d871f2effd7c0e91ab473a2e21c1c39a9646dd5b5fdfeeda30e964feaa2\": rpc error: code = NotFound desc = could not find container \"75279d871f2effd7c0e91ab473a2e21c1c39a9646dd5b5fdfeeda30e964feaa2\": container with ID starting with 75279d871f2effd7c0e91ab473a2e21c1c39a9646dd5b5fdfeeda30e964feaa2 not found: ID does not exist"
Sep 17 18:37:03.380013 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:03.377356    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="094068e2-76dd-41d0-8682-acddc8dd32f1" path="/var/lib/kubelet/pods/094068e2-76dd-41d0-8682-acddc8dd32f1/volumes"
Sep 17 18:37:03.380013 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:03.378158    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9290625f-2be6-4370-b08b-8955a51a8d4e" path="/var/lib/kubelet/pods/9290625f-2be6-4370-b08b-8955a51a8d4e/volumes"
Sep 17 18:37:06.216782 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:06.216745    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qbn6f"]
Sep 17 18:37:06.217160 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:06.217107    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="094068e2-76dd-41d0-8682-acddc8dd32f1" containerName="kserve-container"
Sep 17 18:37:06.217160 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:06.217118    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="094068e2-76dd-41d0-8682-acddc8dd32f1" containerName="kserve-container"
Sep 17 18:37:06.217160 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:06.217130    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41bf207e-2eea-4616-9c26-d20142d1c329" containerName="kserve-container"
Sep 17 18:37:06.217160 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:06.217136    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="41bf207e-2eea-4616-9c26-d20142d1c329" containerName="kserve-container"
Sep 17 18:37:06.217160 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:06.217143    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41bf207e-2eea-4616-9c26-d20142d1c329" containerName="kube-rbac-proxy"
Sep 17 18:37:06.217160 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:06.217148    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="41bf207e-2eea-4616-9c26-d20142d1c329" containerName="kube-rbac-proxy"
Sep 17 18:37:06.217160 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:06.217155    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41bf207e-2eea-4616-9c26-d20142d1c329" containerName="storage-initializer"
Sep 17 18:37:06.217160 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:06.217160    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="41bf207e-2eea-4616-9c26-d20142d1c329" containerName="storage-initializer"
Sep 17 18:37:06.217437 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:06.217166    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9290625f-2be6-4370-b08b-8955a51a8d4e" containerName="storage-initializer"
Sep 17 18:37:06.217437 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:06.217171    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="9290625f-2be6-4370-b08b-8955a51a8d4e" containerName="storage-initializer"
Sep 17 18:37:06.217437 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:06.217178    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9290625f-2be6-4370-b08b-8955a51a8d4e" containerName="kserve-container"
Sep 17 18:37:06.217437 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:06.217183    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="9290625f-2be6-4370-b08b-8955a51a8d4e" containerName="kserve-container"
Sep 17 18:37:06.217437 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:06.217189    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="094068e2-76dd-41d0-8682-acddc8dd32f1" containerName="storage-initializer"
Sep 17 18:37:06.217437 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:06.217194    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="094068e2-76dd-41d0-8682-acddc8dd32f1" containerName="storage-initializer"
Sep 17 18:37:06.217437 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:06.217202    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="094068e2-76dd-41d0-8682-acddc8dd32f1" containerName="kube-rbac-proxy"
Sep 17 18:37:06.217437 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:06.217207    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="094068e2-76dd-41d0-8682-acddc8dd32f1" containerName="kube-rbac-proxy"
Sep 17 18:37:06.217437 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:06.217214    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9290625f-2be6-4370-b08b-8955a51a8d4e" containerName="kube-rbac-proxy"
Sep 17 18:37:06.217437 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:06.217219    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="9290625f-2be6-4370-b08b-8955a51a8d4e" containerName="kube-rbac-proxy"
Sep 17 18:37:06.217437 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:06.217271    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="41bf207e-2eea-4616-9c26-d20142d1c329" containerName="kube-rbac-proxy"
Sep 17 18:37:06.217437 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:06.217280    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="41bf207e-2eea-4616-9c26-d20142d1c329" containerName="kserve-container"
Sep 17 18:37:06.217437 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:06.217287    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="094068e2-76dd-41d0-8682-acddc8dd32f1" containerName="kserve-container"
Sep 17 18:37:06.217437 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:06.217293    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="9290625f-2be6-4370-b08b-8955a51a8d4e" containerName="kube-rbac-proxy"
Sep 17 18:37:06.217437 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:06.217298    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="094068e2-76dd-41d0-8682-acddc8dd32f1" containerName="kube-rbac-proxy"
Sep 17 18:37:06.217437 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:06.217303    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="9290625f-2be6-4370-b08b-8955a51a8d4e" containerName="kserve-container"
Sep 17 18:37:06.220577 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:06.220558    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qbn6f"
Sep 17 18:37:06.223012 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:06.222980    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-predictor-serving-cert\""
Sep 17 18:37:06.223189 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:06.223166    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:37:06.223256 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:06.223192    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:37:06.223314 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:06.223293    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-7qcmd\""
Sep 17 18:37:06.223376 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:06.223325    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:37:06.223562 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:06.223544    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-kube-rbac-proxy-sar-config\""
Sep 17 18:37:06.224120 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:06.224101    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:37:06.230630 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:06.230590    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qbn6f"]
Sep 17 18:37:06.320289 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:06.320240    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/747ee8a1-a909-4850-a841-d6bbaf08203f-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-qbn6f\" (UID: \"747ee8a1-a909-4850-a841-d6bbaf08203f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qbn6f"
Sep 17 18:37:06.320289 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:06.320295    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/747ee8a1-a909-4850-a841-d6bbaf08203f-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-qbn6f\" (UID: \"747ee8a1-a909-4850-a841-d6bbaf08203f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qbn6f"
Sep 17 18:37:06.320509 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:06.320357    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/747ee8a1-a909-4850-a841-d6bbaf08203f-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-qbn6f\" (UID: \"747ee8a1-a909-4850-a841-d6bbaf08203f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qbn6f"
Sep 17 18:37:06.320509 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:06.320384    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd5jg\" (UniqueName: \"kubernetes.io/projected/747ee8a1-a909-4850-a841-d6bbaf08203f-kube-api-access-rd5jg\") pod \"isvc-lightgbm-predictor-54f85ddf88-qbn6f\" (UID: \"747ee8a1-a909-4850-a841-d6bbaf08203f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qbn6f"
Sep 17 18:37:06.320509 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:06.320406    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/747ee8a1-a909-4850-a841-d6bbaf08203f-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-qbn6f\" (UID: \"747ee8a1-a909-4850-a841-d6bbaf08203f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qbn6f"
Sep 17 18:37:06.421306 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:06.421252    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/747ee8a1-a909-4850-a841-d6bbaf08203f-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-qbn6f\" (UID: \"747ee8a1-a909-4850-a841-d6bbaf08203f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qbn6f"
Sep 17 18:37:06.421527 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:06.421326    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd5jg\" (UniqueName: \"kubernetes.io/projected/747ee8a1-a909-4850-a841-d6bbaf08203f-kube-api-access-rd5jg\") pod \"isvc-lightgbm-predictor-54f85ddf88-qbn6f\" (UID: \"747ee8a1-a909-4850-a841-d6bbaf08203f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qbn6f"
Sep 17 18:37:06.421527 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:06.421362    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/747ee8a1-a909-4850-a841-d6bbaf08203f-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-qbn6f\" (UID: \"747ee8a1-a909-4850-a841-d6bbaf08203f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qbn6f"
Sep 17 18:37:06.421527 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:06.421422    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/747ee8a1-a909-4850-a841-d6bbaf08203f-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-qbn6f\" (UID: \"747ee8a1-a909-4850-a841-d6bbaf08203f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qbn6f"
Sep 17 18:37:06.421527 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:06.421470    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/747ee8a1-a909-4850-a841-d6bbaf08203f-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-qbn6f\" (UID: \"747ee8a1-a909-4850-a841-d6bbaf08203f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qbn6f"
Sep 17 18:37:06.421806 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:37:06.421608    2553 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-predictor-serving-cert: secret "isvc-lightgbm-predictor-serving-cert" not found
Sep 17 18:37:06.421806 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:37:06.421705    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/747ee8a1-a909-4850-a841-d6bbaf08203f-proxy-tls podName:747ee8a1-a909-4850-a841-d6bbaf08203f nodeName:}" failed. No retries permitted until 2026-09-17 18:37:06.921685417 +0000 UTC m=+2421.617776094 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/747ee8a1-a909-4850-a841-d6bbaf08203f-proxy-tls") pod "isvc-lightgbm-predictor-54f85ddf88-qbn6f" (UID: "747ee8a1-a909-4850-a841-d6bbaf08203f") : secret "isvc-lightgbm-predictor-serving-cert" not found
Sep 17 18:37:06.421933 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:06.421839    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/747ee8a1-a909-4850-a841-d6bbaf08203f-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-qbn6f\" (UID: \"747ee8a1-a909-4850-a841-d6bbaf08203f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qbn6f"
Sep 17 18:37:06.422148 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:06.422126    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/747ee8a1-a909-4850-a841-d6bbaf08203f-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-qbn6f\" (UID: \"747ee8a1-a909-4850-a841-d6bbaf08203f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qbn6f"
Sep 17 18:37:06.422187 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:06.422148    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/747ee8a1-a909-4850-a841-d6bbaf08203f-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-qbn6f\" (UID: \"747ee8a1-a909-4850-a841-d6bbaf08203f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qbn6f"
Sep 17 18:37:06.430087 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:06.430062    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd5jg\" (UniqueName: \"kubernetes.io/projected/747ee8a1-a909-4850-a841-d6bbaf08203f-kube-api-access-rd5jg\") pod \"isvc-lightgbm-predictor-54f85ddf88-qbn6f\" (UID: \"747ee8a1-a909-4850-a841-d6bbaf08203f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qbn6f"
Sep 17 18:37:06.927315 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:06.927278    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/747ee8a1-a909-4850-a841-d6bbaf08203f-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-qbn6f\" (UID: \"747ee8a1-a909-4850-a841-d6bbaf08203f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qbn6f"
Sep 17 18:37:06.929776 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:06.929742    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/747ee8a1-a909-4850-a841-d6bbaf08203f-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-qbn6f\" (UID: \"747ee8a1-a909-4850-a841-d6bbaf08203f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qbn6f"
Sep 17 18:37:07.132429 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:07.132374    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qbn6f"
Sep 17 18:37:07.406350 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:07.406281    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qbn6f"]
Sep 17 18:37:07.409043 ip-10-0-137-95 kubenswrapper[2553]: W0917 18:37:07.409006    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod747ee8a1_a909_4850_a841_d6bbaf08203f.slice/crio-086b594b57edf35ffd33420c863033b96cdfee3b05894a48b7dde00b2f3ca054 WatchSource:0}: Error finding container 086b594b57edf35ffd33420c863033b96cdfee3b05894a48b7dde00b2f3ca054: Status 404 returned error can't find the container with id 086b594b57edf35ffd33420c863033b96cdfee3b05894a48b7dde00b2f3ca054
Sep 17 18:37:07.740227 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:07.740135    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qbn6f" event={"ID":"747ee8a1-a909-4850-a841-d6bbaf08203f","Type":"ContainerStarted","Data":"a815009ef1de85fcaa03bfa77753fccbdad6da374677797bbf285bbc0ecd4413"}
Sep 17 18:37:07.740227 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:07.740196    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qbn6f" event={"ID":"747ee8a1-a909-4850-a841-d6bbaf08203f","Type":"ContainerStarted","Data":"086b594b57edf35ffd33420c863033b96cdfee3b05894a48b7dde00b2f3ca054"}
Sep 17 18:37:11.760710 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:11.760601    2553 generic.go:356] "Generic (PLEG): container finished" podID="747ee8a1-a909-4850-a841-d6bbaf08203f" containerID="a815009ef1de85fcaa03bfa77753fccbdad6da374677797bbf285bbc0ecd4413" exitCode=0
Sep 17 18:37:11.760710 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:11.760674    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qbn6f" event={"ID":"747ee8a1-a909-4850-a841-d6bbaf08203f","Type":"ContainerDied","Data":"a815009ef1de85fcaa03bfa77753fccbdad6da374677797bbf285bbc0ecd4413"}
Sep 17 18:37:11.761970 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:11.761952    2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:37:18.799699 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:18.799661    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qbn6f" event={"ID":"747ee8a1-a909-4850-a841-d6bbaf08203f","Type":"ContainerStarted","Data":"57f16995fe08f4531e6b7cdfb35a7e77e13b62101065d95d9ce93e2aee072020"}
Sep 17 18:37:18.800079 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:18.799728    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qbn6f" event={"ID":"747ee8a1-a909-4850-a841-d6bbaf08203f","Type":"ContainerStarted","Data":"2ac844a65f6be323a4fca8e1a86f49f3e247dc116e0e1f00a0c713d62899f4c1"}
Sep 17 18:37:18.800079 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:18.800004    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qbn6f"
Sep 17 18:37:18.800168 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:18.800150    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qbn6f"
Sep 17 18:37:18.801358 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:18.801331    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qbn6f" podUID="747ee8a1-a909-4850-a841-d6bbaf08203f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 17 18:37:18.819663 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:18.819598    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qbn6f" podStartSLOduration=2.258958852 podStartE2EDuration="12.819582221s" totalImagesPullingTime="6.538261939s" totalInitContainerRuntime="4.02236143s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:37:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:37:18.818023403 +0000 UTC m=+2433.514114086" watchObservedRunningTime="2026-09-17 18:37:18.819582221 +0000 UTC m=+2433.515672904"
Sep 17 18:37:19.803948 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:19.803907    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qbn6f" podUID="747ee8a1-a909-4850-a841-d6bbaf08203f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 17 18:37:24.809225 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:24.809186    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qbn6f"
Sep 17 18:37:24.809784 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:24.809753    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qbn6f" podUID="747ee8a1-a909-4850-a841-d6bbaf08203f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 17 18:37:34.809874 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:34.809833    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qbn6f" podUID="747ee8a1-a909-4850-a841-d6bbaf08203f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 17 18:37:44.810012 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:44.809967    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qbn6f" podUID="747ee8a1-a909-4850-a841-d6bbaf08203f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 17 18:37:48.226278 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:48.226239    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gddlr"]
Sep 17 18:37:48.416470 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:48.416430    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gddlr"]
Sep 17 18:37:48.416672 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:48.416571    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gddlr"
Sep 17 18:37:48.421639 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:48.420069    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\""
Sep 17 18:37:48.421639 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:48.420336    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-predictor-serving-cert\""
Sep 17 18:37:48.503674 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:48.503539    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-gddlr\" (UID: \"93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gddlr"
Sep 17 18:37:48.503674 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:48.503641    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-gddlr\" (UID: \"93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gddlr"
Sep 17 18:37:48.503908 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:48.503714    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-gddlr\" (UID: \"93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gddlr"
Sep 17 18:37:48.503908 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:48.503782    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s28sd\" (UniqueName: \"kubernetes.io/projected/93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b-kube-api-access-s28sd\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-gddlr\" (UID: \"93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gddlr"
Sep 17 18:37:48.503908 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:48.503809    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-gddlr\" (UID: \"93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gddlr"
Sep 17 18:37:48.605043 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:48.605001    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-gddlr\" (UID: \"93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gddlr"
Sep 17 18:37:48.605234 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:48.605066    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-gddlr\" (UID: \"93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gddlr"
Sep 17 18:37:48.605234 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:48.605101    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-gddlr\" (UID: \"93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gddlr"
Sep 17 18:37:48.605234 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:48.605138    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-gddlr\" (UID: \"93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gddlr"
Sep 17 18:37:48.605234 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:48.605168    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s28sd\" (UniqueName: \"kubernetes.io/projected/93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b-kube-api-access-s28sd\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-gddlr\" (UID: \"93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gddlr"
Sep 17 18:37:48.605678 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:48.605656    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-gddlr\" (UID: \"93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gddlr"
Sep 17 18:37:48.605867 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:48.605847    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-gddlr\" (UID: \"93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gddlr"
Sep 17 18:37:48.605915 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:48.605871    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-gddlr\" (UID: \"93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gddlr"
Sep 17 18:37:48.607542 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:48.607516    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-gddlr\" (UID: \"93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gddlr"
Sep 17 18:37:48.612642 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:48.612610    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s28sd\" (UniqueName: \"kubernetes.io/projected/93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b-kube-api-access-s28sd\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-gddlr\" (UID: \"93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gddlr"
Sep 17 18:37:48.731780 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:48.731735    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gddlr"
Sep 17 18:37:49.013431 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:49.013354    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gddlr"]
Sep 17 18:37:49.015796 ip-10-0-137-95 kubenswrapper[2553]: W0917 18:37:49.015768    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93a8b7d3_d3bd_41c2_a4f3_fc404604ff0b.slice/crio-0a50ea656d33d379902e17364f3a44ea6d4ca673dc9ba9376989ceba12f1d874 WatchSource:0}: Error finding container 0a50ea656d33d379902e17364f3a44ea6d4ca673dc9ba9376989ceba12f1d874: Status 404 returned error can't find the container with id 0a50ea656d33d379902e17364f3a44ea6d4ca673dc9ba9376989ceba12f1d874
Sep 17 18:37:49.940147 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:49.940109    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gddlr" event={"ID":"93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b","Type":"ContainerStarted","Data":"3b9d63c400134f5122f3522e831acbb25ccf419c958569c0ff4cc6aacc2fb5aa"}
Sep 17 18:37:49.940656 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:49.940168    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gddlr" event={"ID":"93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b","Type":"ContainerStarted","Data":"0a50ea656d33d379902e17364f3a44ea6d4ca673dc9ba9376989ceba12f1d874"}
Sep 17 18:37:53.961356 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:53.961318    2553 generic.go:356] "Generic (PLEG): container finished" podID="93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b" containerID="3b9d63c400134f5122f3522e831acbb25ccf419c958569c0ff4cc6aacc2fb5aa" exitCode=0
Sep 17 18:37:53.961754 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:53.961395    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gddlr" event={"ID":"93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b","Type":"ContainerDied","Data":"3b9d63c400134f5122f3522e831acbb25ccf419c958569c0ff4cc6aacc2fb5aa"}
Sep 17 18:37:54.809710 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:54.809650    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qbn6f" podUID="747ee8a1-a909-4850-a841-d6bbaf08203f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 17 18:37:54.968540 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:54.968496    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gddlr" event={"ID":"93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b","Type":"ContainerStarted","Data":"7e341f30877400ef4e2b00deedd6a11fee4bec03c34bc13290bdb598a233c568"}
Sep 17 18:37:54.968959 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:54.968588    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gddlr" event={"ID":"93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b","Type":"ContainerStarted","Data":"9c3a9442e7268f04df895bb3b760418cc7b3f5a6caad17c6fd3a2d4cc336a734"}
Sep 17 18:37:54.968959 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:54.968807    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gddlr"
Sep 17 18:37:54.989692 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:54.989614    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gddlr" podStartSLOduration=3.003019049 podStartE2EDuration="6.989594865s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.986575816s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:37:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:37:54.988190683 +0000 UTC m=+2469.684281390" watchObservedRunningTime="2026-09-17 18:37:54.989594865 +0000 UTC m=+2469.685685552"
Sep 17 18:37:55.973822 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:55.973788    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gddlr"
Sep 17 18:37:55.975216 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:55.975185    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gddlr" podUID="93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 17 18:37:56.978853 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:37:56.978813    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gddlr" podUID="93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 17 18:38:01.983679 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:38:01.983642    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gddlr"
Sep 17 18:38:01.984211 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:38:01.984184    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gddlr" podUID="93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 17 18:38:04.810594 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:38:04.810551    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qbn6f" podUID="747ee8a1-a909-4850-a841-d6bbaf08203f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 17 18:38:11.984173 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:38:11.984133    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gddlr" podUID="93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 17 18:38:14.809789 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:38:14.809742    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qbn6f" podUID="747ee8a1-a909-4850-a841-d6bbaf08203f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 17 18:38:21.984561 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:38:21.984520    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gddlr" podUID="93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 17 18:38:24.810027 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:38:24.809978    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qbn6f" podUID="747ee8a1-a909-4850-a841-d6bbaf08203f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 17 18:38:31.371556 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:38:31.371459    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qbn6f" podUID="747ee8a1-a909-4850-a841-d6bbaf08203f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 17 18:38:31.984277 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:38:31.984234    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gddlr" podUID="93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 17 18:38:41.371486 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:38:41.371431    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qbn6f" podUID="747ee8a1-a909-4850-a841-d6bbaf08203f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 17 18:38:41.984852 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:38:41.984810    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gddlr" podUID="93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 17 18:38:51.371580 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:38:51.371536    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qbn6f" podUID="747ee8a1-a909-4850-a841-d6bbaf08203f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 17 18:38:51.984948 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:38:51.984903    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gddlr" podUID="93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 17 18:39:01.376844 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:01.376815    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qbn6f"
Sep 17 18:39:01.985167 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:01.985130    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gddlr" podUID="93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 17 18:39:07.072898 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:07.072864    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qbn6f"]
Sep 17 18:39:07.073437 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:07.073123    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qbn6f" podUID="747ee8a1-a909-4850-a841-d6bbaf08203f" containerName="kserve-container" containerID="cri-o://2ac844a65f6be323a4fca8e1a86f49f3e247dc116e0e1f00a0c713d62899f4c1" gracePeriod=30
Sep 17 18:39:07.073437 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:07.073168    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qbn6f" podUID="747ee8a1-a909-4850-a841-d6bbaf08203f" containerName="kube-rbac-proxy" containerID="cri-o://57f16995fe08f4531e6b7cdfb35a7e77e13b62101065d95d9ce93e2aee072020" gracePeriod=30
Sep 17 18:39:07.315567 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:07.315530    2553 generic.go:356] "Generic (PLEG): container finished" podID="747ee8a1-a909-4850-a841-d6bbaf08203f" containerID="57f16995fe08f4531e6b7cdfb35a7e77e13b62101065d95d9ce93e2aee072020" exitCode=2
Sep 17 18:39:07.315771 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:07.315579    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qbn6f" event={"ID":"747ee8a1-a909-4850-a841-d6bbaf08203f","Type":"ContainerDied","Data":"57f16995fe08f4531e6b7cdfb35a7e77e13b62101065d95d9ce93e2aee072020"}
Sep 17 18:39:09.805216 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:09.805170    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qbn6f" podUID="747ee8a1-a909-4850-a841-d6bbaf08203f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.50:8643/healthz\": dial tcp 10.134.0.50:8643: connect: connection refused"
Sep 17 18:39:11.371806 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:11.371763    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qbn6f" podUID="747ee8a1-a909-4850-a841-d6bbaf08203f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 17 18:39:11.985722 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:11.985690    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gddlr"
Sep 17 18:39:12.341581 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:12.341539    2553 generic.go:356] "Generic (PLEG): container finished" podID="747ee8a1-a909-4850-a841-d6bbaf08203f" containerID="2ac844a65f6be323a4fca8e1a86f49f3e247dc116e0e1f00a0c713d62899f4c1" exitCode=0
Sep 17 18:39:12.341910 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:12.341612    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qbn6f" event={"ID":"747ee8a1-a909-4850-a841-d6bbaf08203f","Type":"ContainerDied","Data":"2ac844a65f6be323a4fca8e1a86f49f3e247dc116e0e1f00a0c713d62899f4c1"}
Sep 17 18:39:12.418577 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:12.418552    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qbn6f"
Sep 17 18:39:12.536511 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:12.536423    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/747ee8a1-a909-4850-a841-d6bbaf08203f-proxy-tls\") pod \"747ee8a1-a909-4850-a841-d6bbaf08203f\" (UID: \"747ee8a1-a909-4850-a841-d6bbaf08203f\") "
Sep 17 18:39:12.536511 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:12.536492    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/747ee8a1-a909-4850-a841-d6bbaf08203f-kserve-provision-location\") pod \"747ee8a1-a909-4850-a841-d6bbaf08203f\" (UID: \"747ee8a1-a909-4850-a841-d6bbaf08203f\") "
Sep 17 18:39:12.536786 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:12.536523    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/747ee8a1-a909-4850-a841-d6bbaf08203f-cabundle-cert\") pod \"747ee8a1-a909-4850-a841-d6bbaf08203f\" (UID: \"747ee8a1-a909-4850-a841-d6bbaf08203f\") "
Sep 17 18:39:12.536786 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:12.536555    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/747ee8a1-a909-4850-a841-d6bbaf08203f-kube-rbac-proxy-sar-config\") pod \"747ee8a1-a909-4850-a841-d6bbaf08203f\" (UID: \"747ee8a1-a909-4850-a841-d6bbaf08203f\") "
Sep 17 18:39:12.536786 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:12.536594    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rd5jg\" (UniqueName: \"kubernetes.io/projected/747ee8a1-a909-4850-a841-d6bbaf08203f-kube-api-access-rd5jg\") pod \"747ee8a1-a909-4850-a841-d6bbaf08203f\" (UID: \"747ee8a1-a909-4850-a841-d6bbaf08203f\") "
Sep 17 18:39:12.536957 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:12.536896    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/747ee8a1-a909-4850-a841-d6bbaf08203f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "747ee8a1-a909-4850-a841-d6bbaf08203f" (UID: "747ee8a1-a909-4850-a841-d6bbaf08203f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:39:12.537016 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:12.536971    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/747ee8a1-a909-4850-a841-d6bbaf08203f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "747ee8a1-a909-4850-a841-d6bbaf08203f" (UID: "747ee8a1-a909-4850-a841-d6bbaf08203f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:39:12.537016 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:12.536986    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/747ee8a1-a909-4850-a841-d6bbaf08203f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "747ee8a1-a909-4850-a841-d6bbaf08203f" (UID: "747ee8a1-a909-4850-a841-d6bbaf08203f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:39:12.538762 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:12.538737    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/747ee8a1-a909-4850-a841-d6bbaf08203f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "747ee8a1-a909-4850-a841-d6bbaf08203f" (UID: "747ee8a1-a909-4850-a841-d6bbaf08203f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:39:12.538867 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:12.538844    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/747ee8a1-a909-4850-a841-d6bbaf08203f-kube-api-access-rd5jg" (OuterVolumeSpecName: "kube-api-access-rd5jg") pod "747ee8a1-a909-4850-a841-d6bbaf08203f" (UID: "747ee8a1-a909-4850-a841-d6bbaf08203f"). InnerVolumeSpecName "kube-api-access-rd5jg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:39:12.637328 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:12.637294    2553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/747ee8a1-a909-4850-a841-d6bbaf08203f-kserve-provision-location\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:39:12.637328 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:12.637323    2553 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/747ee8a1-a909-4850-a841-d6bbaf08203f-cabundle-cert\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:39:12.637328 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:12.637332    2553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/747ee8a1-a909-4850-a841-d6bbaf08203f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:39:12.637570 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:12.637341    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rd5jg\" (UniqueName: \"kubernetes.io/projected/747ee8a1-a909-4850-a841-d6bbaf08203f-kube-api-access-rd5jg\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:39:12.637570 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:12.637349    2553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/747ee8a1-a909-4850-a841-d6bbaf08203f-proxy-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:39:13.347979 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:13.347948    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qbn6f"
Sep 17 18:39:13.348150 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:13.347956    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qbn6f" event={"ID":"747ee8a1-a909-4850-a841-d6bbaf08203f","Type":"ContainerDied","Data":"086b594b57edf35ffd33420c863033b96cdfee3b05894a48b7dde00b2f3ca054"}
Sep 17 18:39:13.348150 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:13.348090    2553 scope.go:117] "RemoveContainer" containerID="57f16995fe08f4531e6b7cdfb35a7e77e13b62101065d95d9ce93e2aee072020"
Sep 17 18:39:13.363051 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:13.363030    2553 scope.go:117] "RemoveContainer" containerID="2ac844a65f6be323a4fca8e1a86f49f3e247dc116e0e1f00a0c713d62899f4c1"
Sep 17 18:39:13.371296 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:13.371093    2553 scope.go:117] "RemoveContainer" containerID="a815009ef1de85fcaa03bfa77753fccbdad6da374677797bbf285bbc0ecd4413"
Sep 17 18:39:13.378603 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:13.378576    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qbn6f"]
Sep 17 18:39:13.378734 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:13.378641    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-qbn6f"]
Sep 17 18:39:15.376264 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:15.376227    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="747ee8a1-a909-4850-a841-d6bbaf08203f" path="/var/lib/kubelet/pods/747ee8a1-a909-4850-a841-d6bbaf08203f/volumes"
Sep 17 18:39:18.842834 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:18.842796    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gddlr"]
Sep 17 18:39:18.843273 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:18.843029    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gddlr" podUID="93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b" containerName="kserve-container" containerID="cri-o://9c3a9442e7268f04df895bb3b760418cc7b3f5a6caad17c6fd3a2d4cc336a734" gracePeriod=30
Sep 17 18:39:18.843273 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:18.843071    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gddlr" podUID="93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b" containerName="kube-rbac-proxy" containerID="cri-o://7e341f30877400ef4e2b00deedd6a11fee4bec03c34bc13290bdb598a233c568" gracePeriod=30
Sep 17 18:39:19.378057 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:19.378028    2553 generic.go:356] "Generic (PLEG): container finished" podID="93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b" containerID="7e341f30877400ef4e2b00deedd6a11fee4bec03c34bc13290bdb598a233c568" exitCode=2
Sep 17 18:39:19.378231 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:19.378096    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gddlr" event={"ID":"93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b","Type":"ContainerDied","Data":"7e341f30877400ef4e2b00deedd6a11fee4bec03c34bc13290bdb598a233c568"}
Sep 17 18:39:21.979273 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:21.979225    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gddlr" podUID="93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.51:8643/healthz\": dial tcp 10.134.0.51:8643: connect: connection refused"
Sep 17 18:39:21.984583 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:21.984551    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gddlr" podUID="93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 17 18:39:22.980176 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:22.980150    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gddlr"
Sep 17 18:39:23.135351 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:23.135323    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s28sd\" (UniqueName: \"kubernetes.io/projected/93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b-kube-api-access-s28sd\") pod \"93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b\" (UID: \"93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b\") "
Sep 17 18:39:23.135528 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:23.135395    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b-kube-rbac-proxy-sar-config\") pod \"93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b\" (UID: \"93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b\") "
Sep 17 18:39:23.135528 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:23.135415    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b-proxy-tls\") pod \"93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b\" (UID: \"93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b\") "
Sep 17 18:39:23.135528 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:23.135463    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b-kserve-provision-location\") pod \"93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b\" (UID: \"93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b\") "
Sep 17 18:39:23.135528 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:23.135495    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b-cabundle-cert\") pod \"93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b\" (UID: \"93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b\") "
Sep 17 18:39:23.135870 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:23.135838    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b" (UID: "93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:39:23.135977 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:23.135886    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b" (UID: "93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:39:23.135977 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:23.135922    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b" (UID: "93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:39:23.137790 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:23.137764    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b" (UID: "93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:39:23.137883 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:23.137791    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b-kube-api-access-s28sd" (OuterVolumeSpecName: "kube-api-access-s28sd") pod "93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b" (UID: "93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b"). InnerVolumeSpecName "kube-api-access-s28sd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:39:23.236108 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:23.236073    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s28sd\" (UniqueName: \"kubernetes.io/projected/93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b-kube-api-access-s28sd\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:39:23.236108 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:23.236102    2553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:39:23.236108 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:23.236113    2553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b-proxy-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:39:23.236331 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:23.236124    2553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b-kserve-provision-location\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:39:23.236331 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:23.236132    2553 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b-cabundle-cert\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:39:23.398290 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:23.398209    2553 generic.go:356] "Generic (PLEG): container finished" podID="93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b" containerID="9c3a9442e7268f04df895bb3b760418cc7b3f5a6caad17c6fd3a2d4cc336a734" exitCode=0
Sep 17 18:39:23.398290 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:23.398260    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gddlr"
Sep 17 18:39:23.398290 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:23.398282    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gddlr" event={"ID":"93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b","Type":"ContainerDied","Data":"9c3a9442e7268f04df895bb3b760418cc7b3f5a6caad17c6fd3a2d4cc336a734"}
Sep 17 18:39:23.398570 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:23.398333    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gddlr" event={"ID":"93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b","Type":"ContainerDied","Data":"0a50ea656d33d379902e17364f3a44ea6d4ca673dc9ba9376989ceba12f1d874"}
Sep 17 18:39:23.398570 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:23.398361    2553 scope.go:117] "RemoveContainer" containerID="7e341f30877400ef4e2b00deedd6a11fee4bec03c34bc13290bdb598a233c568"
Sep 17 18:39:23.407106 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:23.407081    2553 scope.go:117] "RemoveContainer" containerID="9c3a9442e7268f04df895bb3b760418cc7b3f5a6caad17c6fd3a2d4cc336a734"
Sep 17 18:39:23.415716 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:23.415690    2553 scope.go:117] "RemoveContainer" containerID="3b9d63c400134f5122f3522e831acbb25ccf419c958569c0ff4cc6aacc2fb5aa"
Sep 17 18:39:23.416715 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:23.416667    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gddlr"]
Sep 17 18:39:23.418390 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:23.418369    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gddlr"]
Sep 17 18:39:23.423700 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:23.423679    2553 scope.go:117] "RemoveContainer" containerID="7e341f30877400ef4e2b00deedd6a11fee4bec03c34bc13290bdb598a233c568"
Sep 17 18:39:23.423967 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:39:23.423951    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e341f30877400ef4e2b00deedd6a11fee4bec03c34bc13290bdb598a233c568\": container with ID starting with 7e341f30877400ef4e2b00deedd6a11fee4bec03c34bc13290bdb598a233c568 not found: ID does not exist" containerID="7e341f30877400ef4e2b00deedd6a11fee4bec03c34bc13290bdb598a233c568"
Sep 17 18:39:23.424031 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:23.423971    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e341f30877400ef4e2b00deedd6a11fee4bec03c34bc13290bdb598a233c568"} err="failed to get container status \"7e341f30877400ef4e2b00deedd6a11fee4bec03c34bc13290bdb598a233c568\": rpc error: code = NotFound desc = could not find container \"7e341f30877400ef4e2b00deedd6a11fee4bec03c34bc13290bdb598a233c568\": container with ID starting with 7e341f30877400ef4e2b00deedd6a11fee4bec03c34bc13290bdb598a233c568 not found: ID does not exist"
Sep 17 18:39:23.424031 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:23.423988    2553 scope.go:117] "RemoveContainer" containerID="9c3a9442e7268f04df895bb3b760418cc7b3f5a6caad17c6fd3a2d4cc336a734"
Sep 17 18:39:23.424223 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:39:23.424198    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c3a9442e7268f04df895bb3b760418cc7b3f5a6caad17c6fd3a2d4cc336a734\": container with ID starting with 9c3a9442e7268f04df895bb3b760418cc7b3f5a6caad17c6fd3a2d4cc336a734 not found: ID does not exist" containerID="9c3a9442e7268f04df895bb3b760418cc7b3f5a6caad17c6fd3a2d4cc336a734"
Sep 17 18:39:23.424281 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:23.424219    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c3a9442e7268f04df895bb3b760418cc7b3f5a6caad17c6fd3a2d4cc336a734"} err="failed to get container status \"9c3a9442e7268f04df895bb3b760418cc7b3f5a6caad17c6fd3a2d4cc336a734\": rpc error: code = NotFound desc = could not find container \"9c3a9442e7268f04df895bb3b760418cc7b3f5a6caad17c6fd3a2d4cc336a734\": container with ID starting with 9c3a9442e7268f04df895bb3b760418cc7b3f5a6caad17c6fd3a2d4cc336a734 not found: ID does not exist"
Sep 17 18:39:23.424281 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:23.424235    2553 scope.go:117] "RemoveContainer" containerID="3b9d63c400134f5122f3522e831acbb25ccf419c958569c0ff4cc6aacc2fb5aa"
Sep 17 18:39:23.424477 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:39:23.424463    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b9d63c400134f5122f3522e831acbb25ccf419c958569c0ff4cc6aacc2fb5aa\": container with ID starting with 3b9d63c400134f5122f3522e831acbb25ccf419c958569c0ff4cc6aacc2fb5aa not found: ID does not exist" containerID="3b9d63c400134f5122f3522e831acbb25ccf419c958569c0ff4cc6aacc2fb5aa"
Sep 17 18:39:23.424521 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:23.424479    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b9d63c400134f5122f3522e831acbb25ccf419c958569c0ff4cc6aacc2fb5aa"} err="failed to get container status \"3b9d63c400134f5122f3522e831acbb25ccf419c958569c0ff4cc6aacc2fb5aa\": rpc error: code = NotFound desc = could not find container \"3b9d63c400134f5122f3522e831acbb25ccf419c958569c0ff4cc6aacc2fb5aa\": container with ID starting with 3b9d63c400134f5122f3522e831acbb25ccf419c958569c0ff4cc6aacc2fb5aa not found: ID does not exist"
Sep 17 18:39:25.382471 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:39:25.382428    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b" path="/var/lib/kubelet/pods/93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b/volumes"
Sep 17 18:40:29.187953 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:40:29.187865    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ncg4v"]
Sep 17 18:40:29.188411 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:40:29.188186    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="747ee8a1-a909-4850-a841-d6bbaf08203f" containerName="kube-rbac-proxy"
Sep 17 18:40:29.188411 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:40:29.188199    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="747ee8a1-a909-4850-a841-d6bbaf08203f" containerName="kube-rbac-proxy"
Sep 17 18:40:29.188411 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:40:29.188208    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b" containerName="kube-rbac-proxy"
Sep 17 18:40:29.188411 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:40:29.188214    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b" containerName="kube-rbac-proxy"
Sep 17 18:40:29.188411 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:40:29.188229    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b" containerName="storage-initializer"
Sep 17 18:40:29.188411 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:40:29.188235    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b" containerName="storage-initializer"
Sep 17 18:40:29.188411 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:40:29.188244    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="747ee8a1-a909-4850-a841-d6bbaf08203f" containerName="storage-initializer"
Sep 17 18:40:29.188411 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:40:29.188249    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="747ee8a1-a909-4850-a841-d6bbaf08203f" containerName="storage-initializer"
Sep 17 18:40:29.188411 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:40:29.188255    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="747ee8a1-a909-4850-a841-d6bbaf08203f" containerName="kserve-container"
Sep 17 18:40:29.188411 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:40:29.188260    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="747ee8a1-a909-4850-a841-d6bbaf08203f" containerName="kserve-container"
Sep 17 18:40:29.188411 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:40:29.188266    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b" containerName="kserve-container"
Sep 17 18:40:29.188411 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:40:29.188271    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b" containerName="kserve-container"
Sep 17 18:40:29.188411 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:40:29.188320    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="747ee8a1-a909-4850-a841-d6bbaf08203f" containerName="kube-rbac-proxy"
Sep 17 18:40:29.188411 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:40:29.188329    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b" containerName="kserve-container"
Sep 17 18:40:29.188411 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:40:29.188336    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="747ee8a1-a909-4850-a841-d6bbaf08203f" containerName="kserve-container"
Sep 17 18:40:29.188411 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:40:29.188341    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="93a8b7d3-d3bd-41c2-a4f3-fc404604ff0b" containerName="kube-rbac-proxy"
Sep 17 18:40:29.190356 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:40:29.190336    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ncg4v"
Sep 17 18:40:29.192986 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:40:29.192944    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:40:29.192986 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:40:29.192947    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-kube-rbac-proxy-sar-config\""
Sep 17 18:40:29.193189 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:40:29.193060    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:40:29.193189 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:40:29.193139    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-predictor-serving-cert\""
Sep 17 18:40:29.193599 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:40:29.193582    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:40:29.194083 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:40:29.194068    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-7qcmd\""
Sep 17 18:40:29.194149 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:40:29.194105    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:40:29.206883 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:40:29.203101    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ncg4v"]
Sep 17 18:40:29.283368 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:40:29.283321    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69433246-8de1-49f3-9d4e-75edda00a31b-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-ncg4v\" (UID: \"69433246-8de1-49f3-9d4e-75edda00a31b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ncg4v"
Sep 17 18:40:29.283554 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:40:29.283394    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69433246-8de1-49f3-9d4e-75edda00a31b-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-ncg4v\" (UID: \"69433246-8de1-49f3-9d4e-75edda00a31b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ncg4v"
Sep 17 18:40:29.283554 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:40:29.283428    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69433246-8de1-49f3-9d4e-75edda00a31b-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-ncg4v\" (UID: \"69433246-8de1-49f3-9d4e-75edda00a31b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ncg4v"
Sep 17 18:40:29.283554 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:40:29.283444    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69433246-8de1-49f3-9d4e-75edda00a31b-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-ncg4v\" (UID: \"69433246-8de1-49f3-9d4e-75edda00a31b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ncg4v"
Sep 17 18:40:29.283554 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:40:29.283460    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2frbs\" (UniqueName: \"kubernetes.io/projected/69433246-8de1-49f3-9d4e-75edda00a31b-kube-api-access-2frbs\") pod \"isvc-xgboost-v2-predictor-fcc668bc-ncg4v\" (UID: \"69433246-8de1-49f3-9d4e-75edda00a31b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ncg4v"
Sep 17 18:40:29.384384 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:40:29.384346    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69433246-8de1-49f3-9d4e-75edda00a31b-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-ncg4v\" (UID: \"69433246-8de1-49f3-9d4e-75edda00a31b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ncg4v"
Sep 17 18:40:29.384568 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:40:29.384409    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69433246-8de1-49f3-9d4e-75edda00a31b-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-ncg4v\" (UID: \"69433246-8de1-49f3-9d4e-75edda00a31b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ncg4v"
Sep 17 18:40:29.384568 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:40:29.384434    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69433246-8de1-49f3-9d4e-75edda00a31b-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-ncg4v\" (UID: \"69433246-8de1-49f3-9d4e-75edda00a31b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ncg4v"
Sep 17 18:40:29.384568 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:40:29.384448    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69433246-8de1-49f3-9d4e-75edda00a31b-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-ncg4v\" (UID: \"69433246-8de1-49f3-9d4e-75edda00a31b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ncg4v"
Sep 17 18:40:29.384568 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:40:29.384464    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2frbs\" (UniqueName: \"kubernetes.io/projected/69433246-8de1-49f3-9d4e-75edda00a31b-kube-api-access-2frbs\") pod \"isvc-xgboost-v2-predictor-fcc668bc-ncg4v\" (UID: \"69433246-8de1-49f3-9d4e-75edda00a31b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ncg4v"
Sep 17 18:40:29.384568 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:40:29.384521    2553 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-predictor-serving-cert: secret "isvc-xgboost-v2-predictor-serving-cert" not found
Sep 17 18:40:29.384824 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:40:29.384613    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69433246-8de1-49f3-9d4e-75edda00a31b-proxy-tls podName:69433246-8de1-49f3-9d4e-75edda00a31b nodeName:}" failed. No retries permitted until 2026-09-17 18:40:29.884591755 +0000 UTC m=+2624.580682438 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/69433246-8de1-49f3-9d4e-75edda00a31b-proxy-tls") pod "isvc-xgboost-v2-predictor-fcc668bc-ncg4v" (UID: "69433246-8de1-49f3-9d4e-75edda00a31b") : secret "isvc-xgboost-v2-predictor-serving-cert" not found
Sep 17 18:40:29.384888 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:40:29.384840    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69433246-8de1-49f3-9d4e-75edda00a31b-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-ncg4v\" (UID: \"69433246-8de1-49f3-9d4e-75edda00a31b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ncg4v"
Sep 17 18:40:29.385157 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:40:29.385136    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69433246-8de1-49f3-9d4e-75edda00a31b-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-ncg4v\" (UID: \"69433246-8de1-49f3-9d4e-75edda00a31b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ncg4v"
Sep 17 18:40:29.385204 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:40:29.385152    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69433246-8de1-49f3-9d4e-75edda00a31b-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-ncg4v\" (UID: \"69433246-8de1-49f3-9d4e-75edda00a31b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ncg4v"
Sep 17 18:40:29.393000 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:40:29.392965    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2frbs\" (UniqueName: \"kubernetes.io/projected/69433246-8de1-49f3-9d4e-75edda00a31b-kube-api-access-2frbs\") pod \"isvc-xgboost-v2-predictor-fcc668bc-ncg4v\" (UID: \"69433246-8de1-49f3-9d4e-75edda00a31b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ncg4v"
Sep 17 18:40:29.888076 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:40:29.888035    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69433246-8de1-49f3-9d4e-75edda00a31b-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-ncg4v\" (UID: \"69433246-8de1-49f3-9d4e-75edda00a31b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ncg4v"
Sep 17 18:40:29.890469 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:40:29.890438    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69433246-8de1-49f3-9d4e-75edda00a31b-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-ncg4v\" (UID: \"69433246-8de1-49f3-9d4e-75edda00a31b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ncg4v"
Sep 17 18:40:30.101042 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:40:30.100999    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ncg4v"
Sep 17 18:40:30.375071 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:40:30.374949    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ncg4v"]
Sep 17 18:40:30.377646 ip-10-0-137-95 kubenswrapper[2553]: W0917 18:40:30.377585    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69433246_8de1_49f3_9d4e_75edda00a31b.slice/crio-f42fbd010d7fc66c609dc0f48af060dc56be81449fe55dbe16af32f8a2274f2f WatchSource:0}: Error finding container f42fbd010d7fc66c609dc0f48af060dc56be81449fe55dbe16af32f8a2274f2f: Status 404 returned error can't find the container with id f42fbd010d7fc66c609dc0f48af060dc56be81449fe55dbe16af32f8a2274f2f
Sep 17 18:40:30.696652 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:40:30.696527    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ncg4v" event={"ID":"69433246-8de1-49f3-9d4e-75edda00a31b","Type":"ContainerStarted","Data":"a5ae0e4e37ae6dc3fdb954ec391060e7d4d8c8db92b6f781f9e4d5c4c940e0c3"}
Sep 17 18:40:30.696652 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:40:30.696588    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ncg4v" event={"ID":"69433246-8de1-49f3-9d4e-75edda00a31b","Type":"ContainerStarted","Data":"f42fbd010d7fc66c609dc0f48af060dc56be81449fe55dbe16af32f8a2274f2f"}
Sep 17 18:40:34.715201 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:40:34.715168    2553 generic.go:356] "Generic (PLEG): container finished" podID="69433246-8de1-49f3-9d4e-75edda00a31b" containerID="a5ae0e4e37ae6dc3fdb954ec391060e7d4d8c8db92b6f781f9e4d5c4c940e0c3" exitCode=0
Sep 17 18:40:34.715697 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:40:34.715244    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ncg4v" event={"ID":"69433246-8de1-49f3-9d4e-75edda00a31b","Type":"ContainerDied","Data":"a5ae0e4e37ae6dc3fdb954ec391060e7d4d8c8db92b6f781f9e4d5c4c940e0c3"}
Sep 17 18:40:35.721505 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:40:35.721465    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ncg4v" event={"ID":"69433246-8de1-49f3-9d4e-75edda00a31b","Type":"ContainerStarted","Data":"8d59a5e28e16b516a1d15b1c3d7b760b8a0c0a3234462ef950a5e25aea39d231"}
Sep 17 18:40:35.721912 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:40:35.721532    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ncg4v" event={"ID":"69433246-8de1-49f3-9d4e-75edda00a31b","Type":"ContainerStarted","Data":"c225536668fb6be07f96a40115148b05b4c00a71e02beaef2886a1448348a322"}
Sep 17 18:40:35.721912 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:40:35.721696    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ncg4v"
Sep 17 18:40:35.740700 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:40:35.740644    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ncg4v" podStartSLOduration=2.8044791140000003 podStartE2EDuration="6.740603391s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.936124277s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:40:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:35.738875416 +0000 UTC m=+2630.434966114" watchObservedRunningTime="2026-09-17 18:40:35.740603391 +0000 UTC m=+2630.436694073"
Sep 17 18:40:36.727004 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:40:36.726972    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ncg4v"
Sep 17 18:40:36.728436 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:40:36.728405    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ncg4v" podUID="69433246-8de1-49f3-9d4e-75edda00a31b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 17 18:40:37.731386 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:40:37.731347    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ncg4v" podUID="69433246-8de1-49f3-9d4e-75edda00a31b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 17 18:40:42.736306 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:40:42.736277    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ncg4v"
Sep 17 18:40:42.736805 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:40:42.736777    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ncg4v" podUID="69433246-8de1-49f3-9d4e-75edda00a31b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 17 18:40:52.737018 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:40:52.736973    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ncg4v" podUID="69433246-8de1-49f3-9d4e-75edda00a31b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 17 18:41:02.737547 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:02.737506    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ncg4v" podUID="69433246-8de1-49f3-9d4e-75edda00a31b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 17 18:41:12.737746 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:12.737702    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ncg4v" podUID="69433246-8de1-49f3-9d4e-75edda00a31b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 17 18:41:22.737355 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:22.737313    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ncg4v" podUID="69433246-8de1-49f3-9d4e-75edda00a31b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 17 18:41:32.737489 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:32.737393    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ncg4v" podUID="69433246-8de1-49f3-9d4e-75edda00a31b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 17 18:41:42.737777 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:42.737744    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ncg4v"
Sep 17 18:41:47.487848 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:47.487812    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-btmqz_622ee5be-fb50-42b6-ae23-42900b7c2d98/ovn-acl-logging/0.log"
Sep 17 18:41:47.496067 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:47.496043    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-btmqz_622ee5be-fb50-42b6-ae23-42900b7c2d98/ovn-acl-logging/0.log"
Sep 17 18:41:49.860908 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:49.860874    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ncg4v"]
Sep 17 18:41:49.861334 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:49.861167    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ncg4v" podUID="69433246-8de1-49f3-9d4e-75edda00a31b" containerName="kserve-container" containerID="cri-o://c225536668fb6be07f96a40115148b05b4c00a71e02beaef2886a1448348a322" gracePeriod=30
Sep 17 18:41:49.861334 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:49.861251    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ncg4v" podUID="69433246-8de1-49f3-9d4e-75edda00a31b" containerName="kube-rbac-proxy" containerID="cri-o://8d59a5e28e16b516a1d15b1c3d7b760b8a0c0a3234462ef950a5e25aea39d231" gracePeriod=30
Sep 17 18:41:50.064709 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:50.064677    2553 generic.go:356] "Generic (PLEG): container finished" podID="69433246-8de1-49f3-9d4e-75edda00a31b" containerID="8d59a5e28e16b516a1d15b1c3d7b760b8a0c0a3234462ef950a5e25aea39d231" exitCode=2
Sep 17 18:41:50.064882 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:50.064735    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ncg4v" event={"ID":"69433246-8de1-49f3-9d4e-75edda00a31b","Type":"ContainerDied","Data":"8d59a5e28e16b516a1d15b1c3d7b760b8a0c0a3234462ef950a5e25aea39d231"}
Sep 17 18:41:52.731958 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:52.731911    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ncg4v" podUID="69433246-8de1-49f3-9d4e-75edda00a31b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.52:8643/healthz\": dial tcp 10.134.0.52:8643: connect: connection refused"
Sep 17 18:41:52.737301 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:52.737263    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ncg4v" podUID="69433246-8de1-49f3-9d4e-75edda00a31b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 17 18:41:54.086447 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:54.086413    2553 generic.go:356] "Generic (PLEG): container finished" podID="69433246-8de1-49f3-9d4e-75edda00a31b" containerID="c225536668fb6be07f96a40115148b05b4c00a71e02beaef2886a1448348a322" exitCode=0
Sep 17 18:41:54.086863 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:54.086484    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ncg4v" event={"ID":"69433246-8de1-49f3-9d4e-75edda00a31b","Type":"ContainerDied","Data":"c225536668fb6be07f96a40115148b05b4c00a71e02beaef2886a1448348a322"}
Sep 17 18:41:54.209903 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:54.209876    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ncg4v"
Sep 17 18:41:54.317773 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:54.317681    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69433246-8de1-49f3-9d4e-75edda00a31b-proxy-tls\") pod \"69433246-8de1-49f3-9d4e-75edda00a31b\" (UID: \"69433246-8de1-49f3-9d4e-75edda00a31b\") "
Sep 17 18:41:54.317773 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:54.317721    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2frbs\" (UniqueName: \"kubernetes.io/projected/69433246-8de1-49f3-9d4e-75edda00a31b-kube-api-access-2frbs\") pod \"69433246-8de1-49f3-9d4e-75edda00a31b\" (UID: \"69433246-8de1-49f3-9d4e-75edda00a31b\") "
Sep 17 18:41:54.317773 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:54.317762    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69433246-8de1-49f3-9d4e-75edda00a31b-kserve-provision-location\") pod \"69433246-8de1-49f3-9d4e-75edda00a31b\" (UID: \"69433246-8de1-49f3-9d4e-75edda00a31b\") "
Sep 17 18:41:54.318068 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:54.317801    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69433246-8de1-49f3-9d4e-75edda00a31b-cabundle-cert\") pod \"69433246-8de1-49f3-9d4e-75edda00a31b\" (UID: \"69433246-8de1-49f3-9d4e-75edda00a31b\") "
Sep 17 18:41:54.318068 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:54.317844    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69433246-8de1-49f3-9d4e-75edda00a31b-kube-rbac-proxy-sar-config\") pod \"69433246-8de1-49f3-9d4e-75edda00a31b\" (UID: \"69433246-8de1-49f3-9d4e-75edda00a31b\") "
Sep 17 18:41:54.318204 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:54.318179    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69433246-8de1-49f3-9d4e-75edda00a31b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "69433246-8de1-49f3-9d4e-75edda00a31b" (UID: "69433246-8de1-49f3-9d4e-75edda00a31b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:41:54.318277 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:54.318197    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69433246-8de1-49f3-9d4e-75edda00a31b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "69433246-8de1-49f3-9d4e-75edda00a31b" (UID: "69433246-8de1-49f3-9d4e-75edda00a31b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:41:54.318277 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:54.318234    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69433246-8de1-49f3-9d4e-75edda00a31b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "69433246-8de1-49f3-9d4e-75edda00a31b" (UID: "69433246-8de1-49f3-9d4e-75edda00a31b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:41:54.319900 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:54.319874    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69433246-8de1-49f3-9d4e-75edda00a31b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "69433246-8de1-49f3-9d4e-75edda00a31b" (UID: "69433246-8de1-49f3-9d4e-75edda00a31b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:41:54.320227 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:54.320206    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69433246-8de1-49f3-9d4e-75edda00a31b-kube-api-access-2frbs" (OuterVolumeSpecName: "kube-api-access-2frbs") pod "69433246-8de1-49f3-9d4e-75edda00a31b" (UID: "69433246-8de1-49f3-9d4e-75edda00a31b"). InnerVolumeSpecName "kube-api-access-2frbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:41:54.418490 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:54.418445    2553 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69433246-8de1-49f3-9d4e-75edda00a31b-cabundle-cert\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:41:54.418490 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:54.418483    2553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69433246-8de1-49f3-9d4e-75edda00a31b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:41:54.418490 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:54.418498    2553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69433246-8de1-49f3-9d4e-75edda00a31b-proxy-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:41:54.418775 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:54.418513    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2frbs\" (UniqueName: \"kubernetes.io/projected/69433246-8de1-49f3-9d4e-75edda00a31b-kube-api-access-2frbs\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:41:54.418775 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:54.418525    2553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69433246-8de1-49f3-9d4e-75edda00a31b-kserve-provision-location\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:41:55.093300 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:55.093260    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ncg4v"
Sep 17 18:41:55.093826 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:55.093426    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ncg4v" event={"ID":"69433246-8de1-49f3-9d4e-75edda00a31b","Type":"ContainerDied","Data":"f42fbd010d7fc66c609dc0f48af060dc56be81449fe55dbe16af32f8a2274f2f"}
Sep 17 18:41:55.093826 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:55.093540    2553 scope.go:117] "RemoveContainer" containerID="8d59a5e28e16b516a1d15b1c3d7b760b8a0c0a3234462ef950a5e25aea39d231"
Sep 17 18:41:55.106766 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:55.106738    2553 scope.go:117] "RemoveContainer" containerID="c225536668fb6be07f96a40115148b05b4c00a71e02beaef2886a1448348a322"
Sep 17 18:41:55.116006 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:55.115980    2553 scope.go:117] "RemoveContainer" containerID="a5ae0e4e37ae6dc3fdb954ec391060e7d4d8c8db92b6f781f9e4d5c4c940e0c3"
Sep 17 18:41:55.120882 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:55.120853    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ncg4v"]
Sep 17 18:41:55.128168 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:55.128138    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-ncg4v"]
Sep 17 18:41:55.172548 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:55.172512    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8b7476b77-skpn5"]
Sep 17 18:41:55.172857 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:55.172844    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69433246-8de1-49f3-9d4e-75edda00a31b" containerName="kserve-container"
Sep 17 18:41:55.172934 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:55.172858    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="69433246-8de1-49f3-9d4e-75edda00a31b" containerName="kserve-container"
Sep 17 18:41:55.172934 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:55.172872    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69433246-8de1-49f3-9d4e-75edda00a31b" containerName="kube-rbac-proxy"
Sep 17 18:41:55.172934 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:55.172878    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="69433246-8de1-49f3-9d4e-75edda00a31b" containerName="kube-rbac-proxy"
Sep 17 18:41:55.172934 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:55.172891    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69433246-8de1-49f3-9d4e-75edda00a31b" containerName="storage-initializer"
Sep 17 18:41:55.172934 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:55.172897    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="69433246-8de1-49f3-9d4e-75edda00a31b" containerName="storage-initializer"
Sep 17 18:41:55.173091 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:55.172945    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="69433246-8de1-49f3-9d4e-75edda00a31b" containerName="kserve-container"
Sep 17 18:41:55.173091 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:55.172954    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="69433246-8de1-49f3-9d4e-75edda00a31b" containerName="kube-rbac-proxy"
Sep 17 18:41:55.177484 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:55.177457    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8b7476b77-skpn5"
Sep 17 18:41:55.180235 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:55.180204    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-7qcmd\""
Sep 17 18:41:55.180235 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:55.180207    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 17 18:41:55.180450 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:55.180269    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-predictor-serving-cert\""
Sep 17 18:41:55.180450 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:55.180346    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:41:55.180450 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:55.180349    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-kube-rbac-proxy-sar-config\""
Sep 17 18:41:55.180450 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:55.180411    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:41:55.186878 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:55.186849    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8b7476b77-skpn5"]
Sep 17 18:41:55.225609 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:55.225569    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19eb4850-b062-413d-baa6-877d343bcebe-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-8b7476b77-skpn5\" (UID: \"19eb4850-b062-413d-baa6-877d343bcebe\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8b7476b77-skpn5"
Sep 17 18:41:55.225806 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:55.225674    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19eb4850-b062-413d-baa6-877d343bcebe-proxy-tls\") pod \"isvc-sklearn-s3-predictor-8b7476b77-skpn5\" (UID: \"19eb4850-b062-413d-baa6-877d343bcebe\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8b7476b77-skpn5"
Sep 17 18:41:55.225806 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:55.225696    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19eb4850-b062-413d-baa6-877d343bcebe-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-8b7476b77-skpn5\" (UID: \"19eb4850-b062-413d-baa6-877d343bcebe\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8b7476b77-skpn5"
Sep 17 18:41:55.225806 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:55.225749    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2rz4\" (UniqueName: \"kubernetes.io/projected/19eb4850-b062-413d-baa6-877d343bcebe-kube-api-access-v2rz4\") pod \"isvc-sklearn-s3-predictor-8b7476b77-skpn5\" (UID: \"19eb4850-b062-413d-baa6-877d343bcebe\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8b7476b77-skpn5"
Sep 17 18:41:55.326703 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:55.326667    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2rz4\" (UniqueName: \"kubernetes.io/projected/19eb4850-b062-413d-baa6-877d343bcebe-kube-api-access-v2rz4\") pod \"isvc-sklearn-s3-predictor-8b7476b77-skpn5\" (UID: \"19eb4850-b062-413d-baa6-877d343bcebe\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8b7476b77-skpn5"
Sep 17 18:41:55.326894 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:55.326723    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19eb4850-b062-413d-baa6-877d343bcebe-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-8b7476b77-skpn5\" (UID: \"19eb4850-b062-413d-baa6-877d343bcebe\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8b7476b77-skpn5"
Sep 17 18:41:55.326894 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:55.326773    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19eb4850-b062-413d-baa6-877d343bcebe-proxy-tls\") pod \"isvc-sklearn-s3-predictor-8b7476b77-skpn5\" (UID: \"19eb4850-b062-413d-baa6-877d343bcebe\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8b7476b77-skpn5"
Sep 17 18:41:55.326894 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:55.326793    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19eb4850-b062-413d-baa6-877d343bcebe-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-8b7476b77-skpn5\" (UID: \"19eb4850-b062-413d-baa6-877d343bcebe\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8b7476b77-skpn5"
Sep 17 18:41:55.327245 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:55.327222    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19eb4850-b062-413d-baa6-877d343bcebe-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-8b7476b77-skpn5\" (UID: \"19eb4850-b062-413d-baa6-877d343bcebe\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8b7476b77-skpn5"
Sep 17 18:41:55.327515 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:55.327496    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19eb4850-b062-413d-baa6-877d343bcebe-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-8b7476b77-skpn5\" (UID: \"19eb4850-b062-413d-baa6-877d343bcebe\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8b7476b77-skpn5"
Sep 17 18:41:55.329405 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:55.329381    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19eb4850-b062-413d-baa6-877d343bcebe-proxy-tls\") pod \"isvc-sklearn-s3-predictor-8b7476b77-skpn5\" (UID: \"19eb4850-b062-413d-baa6-877d343bcebe\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8b7476b77-skpn5"
Sep 17 18:41:55.334914 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:55.334884    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2rz4\" (UniqueName: \"kubernetes.io/projected/19eb4850-b062-413d-baa6-877d343bcebe-kube-api-access-v2rz4\") pod \"isvc-sklearn-s3-predictor-8b7476b77-skpn5\" (UID: \"19eb4850-b062-413d-baa6-877d343bcebe\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8b7476b77-skpn5"
Sep 17 18:41:55.377343 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:55.377257    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69433246-8de1-49f3-9d4e-75edda00a31b" path="/var/lib/kubelet/pods/69433246-8de1-49f3-9d4e-75edda00a31b/volumes"
Sep 17 18:41:55.489361 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:55.489311    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8b7476b77-skpn5"
Sep 17 18:41:55.764912 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:55.764883    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8b7476b77-skpn5"]
Sep 17 18:41:55.767452 ip-10-0-137-95 kubenswrapper[2553]: W0917 18:41:55.767422    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19eb4850_b062_413d_baa6_877d343bcebe.slice/crio-31006f9d174d7ec38789d13b1d7087316f9c5dec43e0f33bb5aca23b68795e38 WatchSource:0}: Error finding container 31006f9d174d7ec38789d13b1d7087316f9c5dec43e0f33bb5aca23b68795e38: Status 404 returned error can't find the container with id 31006f9d174d7ec38789d13b1d7087316f9c5dec43e0f33bb5aca23b68795e38
Sep 17 18:41:56.100880 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:56.100779    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8b7476b77-skpn5" event={"ID":"19eb4850-b062-413d-baa6-877d343bcebe","Type":"ContainerStarted","Data":"bde36bea3c857603d5d891c302412c4687ed57fc27e472fe29dc6559d08d63c8"}
Sep 17 18:41:56.101324 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:56.100880    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8b7476b77-skpn5" event={"ID":"19eb4850-b062-413d-baa6-877d343bcebe","Type":"ContainerStarted","Data":"31006f9d174d7ec38789d13b1d7087316f9c5dec43e0f33bb5aca23b68795e38"}
Sep 17 18:41:58.112138 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:58.112104    2553 generic.go:356] "Generic (PLEG): container finished" podID="19eb4850-b062-413d-baa6-877d343bcebe" containerID="bde36bea3c857603d5d891c302412c4687ed57fc27e472fe29dc6559d08d63c8" exitCode=0
Sep 17 18:41:58.112555 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:58.112181    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8b7476b77-skpn5" event={"ID":"19eb4850-b062-413d-baa6-877d343bcebe","Type":"ContainerDied","Data":"bde36bea3c857603d5d891c302412c4687ed57fc27e472fe29dc6559d08d63c8"}
Sep 17 18:41:59.119357 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:59.119321    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8b7476b77-skpn5" event={"ID":"19eb4850-b062-413d-baa6-877d343bcebe","Type":"ContainerStarted","Data":"8cf1d59ecb487b073744b1f835f7a4ba3463c21a41d5eb89771884033eb4adef"}
Sep 17 18:41:59.119792 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:59.119384    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8b7476b77-skpn5" event={"ID":"19eb4850-b062-413d-baa6-877d343bcebe","Type":"ContainerStarted","Data":"45b15233e796ee08a148e1b44f9c4770dfd580daa4961523ab34eb5089d755b5"}
Sep 17 18:41:59.119792 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:59.119588    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8b7476b77-skpn5"
Sep 17 18:41:59.149817 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:41:59.149760    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8b7476b77-skpn5" podStartSLOduration=2.708414875 podStartE2EDuration="4.149739329s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.441324454s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:41:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:41:59.147942693 +0000 UTC m=+2713.844033390" watchObservedRunningTime="2026-09-17 18:41:59.149739329 +0000 UTC m=+2713.845830012"
Sep 17 18:42:00.124161 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:42:00.124121    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8b7476b77-skpn5"
Sep 17 18:42:00.125517 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:42:00.125486    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8b7476b77-skpn5" podUID="19eb4850-b062-413d-baa6-877d343bcebe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 17 18:42:01.128652 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:42:01.128592    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8b7476b77-skpn5" podUID="19eb4850-b062-413d-baa6-877d343bcebe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 17 18:42:06.132995 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:42:06.132963    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8b7476b77-skpn5"
Sep 17 18:42:06.133491 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:42:06.133405    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8b7476b77-skpn5" podUID="19eb4850-b062-413d-baa6-877d343bcebe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 17 18:42:16.134440 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:42:16.134400    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8b7476b77-skpn5" podUID="19eb4850-b062-413d-baa6-877d343bcebe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 17 18:42:26.134361 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:42:26.134317    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8b7476b77-skpn5" podUID="19eb4850-b062-413d-baa6-877d343bcebe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 17 18:42:36.133588 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:42:36.133543    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8b7476b77-skpn5" podUID="19eb4850-b062-413d-baa6-877d343bcebe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 17 18:42:46.134231 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:42:46.134190    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8b7476b77-skpn5" podUID="19eb4850-b062-413d-baa6-877d343bcebe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 17 18:42:56.133582 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:42:56.133532    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8b7476b77-skpn5" podUID="19eb4850-b062-413d-baa6-877d343bcebe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 17 18:43:06.133600 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:06.133503    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8b7476b77-skpn5" podUID="19eb4850-b062-413d-baa6-877d343bcebe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 17 18:43:16.133791 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:16.133755    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8b7476b77-skpn5"
Sep 17 18:43:25.244036 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:25.244005    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8b7476b77-skpn5"]
Sep 17 18:43:25.244446 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:25.244241    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8b7476b77-skpn5" podUID="19eb4850-b062-413d-baa6-877d343bcebe" containerName="kserve-container" containerID="cri-o://45b15233e796ee08a148e1b44f9c4770dfd580daa4961523ab34eb5089d755b5" gracePeriod=30
Sep 17 18:43:25.244446 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:25.244284    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8b7476b77-skpn5" podUID="19eb4850-b062-413d-baa6-877d343bcebe" containerName="kube-rbac-proxy" containerID="cri-o://8cf1d59ecb487b073744b1f835f7a4ba3463c21a41d5eb89771884033eb4adef" gracePeriod=30
Sep 17 18:43:25.432372 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:25.432340    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64484c89d5-r5v86"]
Sep 17 18:43:25.435850 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:25.435832    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64484c89d5-r5v86"
Sep 17 18:43:25.438246 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:25.438215    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-predictor-serving-cert\""
Sep 17 18:43:25.438463 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:25.438441    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\""
Sep 17 18:43:25.438533 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:25.438443    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:43:25.445649 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:25.445605    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64484c89d5-r5v86"]
Sep 17 18:43:25.499201 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:25.499095    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3ac5d326-3303-4184-8741-4b7d67843fb6-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-64484c89d5-r5v86\" (UID: \"3ac5d326-3303-4184-8741-4b7d67843fb6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64484c89d5-r5v86"
Sep 17 18:43:25.499201 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:25.499150    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ac5d326-3303-4184-8741-4b7d67843fb6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-64484c89d5-r5v86\" (UID: \"3ac5d326-3303-4184-8741-4b7d67843fb6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64484c89d5-r5v86"
Sep 17 18:43:25.499201 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:25.499170    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ac5d326-3303-4184-8741-4b7d67843fb6-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-64484c89d5-r5v86\" (UID: \"3ac5d326-3303-4184-8741-4b7d67843fb6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64484c89d5-r5v86"
Sep 17 18:43:25.499485 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:25.499229    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ac5d326-3303-4184-8741-4b7d67843fb6-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-64484c89d5-r5v86\" (UID: \"3ac5d326-3303-4184-8741-4b7d67843fb6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64484c89d5-r5v86"
Sep 17 18:43:25.499485 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:25.499346    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgxds\" (UniqueName: \"kubernetes.io/projected/3ac5d326-3303-4184-8741-4b7d67843fb6-kube-api-access-vgxds\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-64484c89d5-r5v86\" (UID: \"3ac5d326-3303-4184-8741-4b7d67843fb6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64484c89d5-r5v86"
Sep 17 18:43:25.509026 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:25.508994    2553 generic.go:356] "Generic (PLEG): container finished" podID="19eb4850-b062-413d-baa6-877d343bcebe" containerID="8cf1d59ecb487b073744b1f835f7a4ba3463c21a41d5eb89771884033eb4adef" exitCode=2
Sep 17 18:43:25.509163 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:25.509070    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8b7476b77-skpn5" event={"ID":"19eb4850-b062-413d-baa6-877d343bcebe","Type":"ContainerDied","Data":"8cf1d59ecb487b073744b1f835f7a4ba3463c21a41d5eb89771884033eb4adef"}
Sep 17 18:43:25.600232 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:25.600190    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgxds\" (UniqueName: \"kubernetes.io/projected/3ac5d326-3303-4184-8741-4b7d67843fb6-kube-api-access-vgxds\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-64484c89d5-r5v86\" (UID: \"3ac5d326-3303-4184-8741-4b7d67843fb6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64484c89d5-r5v86"
Sep 17 18:43:25.600444 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:25.600259    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3ac5d326-3303-4184-8741-4b7d67843fb6-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-64484c89d5-r5v86\" (UID: \"3ac5d326-3303-4184-8741-4b7d67843fb6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64484c89d5-r5v86"
Sep 17 18:43:25.600444 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:25.600311    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ac5d326-3303-4184-8741-4b7d67843fb6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-64484c89d5-r5v86\" (UID: \"3ac5d326-3303-4184-8741-4b7d67843fb6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64484c89d5-r5v86"
Sep 17 18:43:25.600444 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:25.600335    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ac5d326-3303-4184-8741-4b7d67843fb6-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-64484c89d5-r5v86\" (UID: \"3ac5d326-3303-4184-8741-4b7d67843fb6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64484c89d5-r5v86"
Sep 17 18:43:25.600444 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:25.600366    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ac5d326-3303-4184-8741-4b7d67843fb6-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-64484c89d5-r5v86\" (UID: \"3ac5d326-3303-4184-8741-4b7d67843fb6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64484c89d5-r5v86"
Sep 17 18:43:25.600900 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:25.600869    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ac5d326-3303-4184-8741-4b7d67843fb6-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-64484c89d5-r5v86\" (UID: \"3ac5d326-3303-4184-8741-4b7d67843fb6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64484c89d5-r5v86"
Sep 17 18:43:25.601097 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:25.601077    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ac5d326-3303-4184-8741-4b7d67843fb6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-64484c89d5-r5v86\" (UID: \"3ac5d326-3303-4184-8741-4b7d67843fb6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64484c89d5-r5v86"
Sep 17 18:43:25.601141 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:25.601105    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3ac5d326-3303-4184-8741-4b7d67843fb6-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-64484c89d5-r5v86\" (UID: \"3ac5d326-3303-4184-8741-4b7d67843fb6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64484c89d5-r5v86"
Sep 17 18:43:25.602925 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:25.602901    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ac5d326-3303-4184-8741-4b7d67843fb6-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-64484c89d5-r5v86\" (UID: \"3ac5d326-3303-4184-8741-4b7d67843fb6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64484c89d5-r5v86"
Sep 17 18:43:25.608231 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:25.608204    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgxds\" (UniqueName: \"kubernetes.io/projected/3ac5d326-3303-4184-8741-4b7d67843fb6-kube-api-access-vgxds\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-64484c89d5-r5v86\" (UID: \"3ac5d326-3303-4184-8741-4b7d67843fb6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64484c89d5-r5v86"
Sep 17 18:43:25.747889 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:25.747840    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64484c89d5-r5v86"
Sep 17 18:43:26.020804 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:26.020559    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64484c89d5-r5v86"]
Sep 17 18:43:26.023613 ip-10-0-137-95 kubenswrapper[2553]: W0917 18:43:26.023585    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ac5d326_3303_4184_8741_4b7d67843fb6.slice/crio-632854c65d10c65d39a2d703040e6aad1d9270e72069e2a0f2fa85e0425bcec6 WatchSource:0}: Error finding container 632854c65d10c65d39a2d703040e6aad1d9270e72069e2a0f2fa85e0425bcec6: Status 404 returned error can't find the container with id 632854c65d10c65d39a2d703040e6aad1d9270e72069e2a0f2fa85e0425bcec6
Sep 17 18:43:26.129280 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:26.129232    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8b7476b77-skpn5" podUID="19eb4850-b062-413d-baa6-877d343bcebe" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.53:8643/healthz\": dial tcp 10.134.0.53:8643: connect: connection refused"
Sep 17 18:43:26.133589 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:26.133544    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8b7476b77-skpn5" podUID="19eb4850-b062-413d-baa6-877d343bcebe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 17 18:43:26.515873 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:26.515827    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64484c89d5-r5v86" event={"ID":"3ac5d326-3303-4184-8741-4b7d67843fb6","Type":"ContainerStarted","Data":"3f174833efe65043a0357c7c5ef007f397a4e7c82de0f985f458759dd4d4fb43"}
Sep 17 18:43:26.516359 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:26.515899    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64484c89d5-r5v86" event={"ID":"3ac5d326-3303-4184-8741-4b7d67843fb6","Type":"ContainerStarted","Data":"632854c65d10c65d39a2d703040e6aad1d9270e72069e2a0f2fa85e0425bcec6"}
Sep 17 18:43:27.521763 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:27.521669    2553 generic.go:356] "Generic (PLEG): container finished" podID="3ac5d326-3303-4184-8741-4b7d67843fb6" containerID="3f174833efe65043a0357c7c5ef007f397a4e7c82de0f985f458759dd4d4fb43" exitCode=0
Sep 17 18:43:27.521763 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:27.521720    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64484c89d5-r5v86" event={"ID":"3ac5d326-3303-4184-8741-4b7d67843fb6","Type":"ContainerDied","Data":"3f174833efe65043a0357c7c5ef007f397a4e7c82de0f985f458759dd4d4fb43"}
Sep 17 18:43:28.528378 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:28.528339    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64484c89d5-r5v86" event={"ID":"3ac5d326-3303-4184-8741-4b7d67843fb6","Type":"ContainerStarted","Data":"4241634f2b8887beac481abcb50f8caeaa77e4e69a27f47ffdad6a8765bd80b9"}
Sep 17 18:43:28.528917 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:28.528399    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64484c89d5-r5v86" event={"ID":"3ac5d326-3303-4184-8741-4b7d67843fb6","Type":"ContainerStarted","Data":"2051fcca997eb7ab1aaa90dce70f24aa5e6df1701722afb0dc0fd4ac74277f95"}
Sep 17 18:43:28.528917 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:28.528497    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64484c89d5-r5v86"
Sep 17 18:43:28.548282 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:28.548212    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64484c89d5-r5v86" podStartSLOduration=2.400378475 podStartE2EDuration="3.548190065s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.14781159s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:43:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:43:28.547530444 +0000 UTC m=+2803.243621128" watchObservedRunningTime="2026-09-17 18:43:28.548190065 +0000 UTC m=+2803.244280750"
Sep 17 18:43:29.533468 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:29.533435    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64484c89d5-r5v86"
Sep 17 18:43:29.534952 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:29.534918    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64484c89d5-r5v86" podUID="3ac5d326-3303-4184-8741-4b7d67843fb6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 17 18:43:30.407016 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:30.406986    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8b7476b77-skpn5"
Sep 17 18:43:30.436899 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:30.436864    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19eb4850-b062-413d-baa6-877d343bcebe-kserve-provision-location\") pod \"19eb4850-b062-413d-baa6-877d343bcebe\" (UID: \"19eb4850-b062-413d-baa6-877d343bcebe\") "
Sep 17 18:43:30.437078 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:30.436931    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19eb4850-b062-413d-baa6-877d343bcebe-proxy-tls\") pod \"19eb4850-b062-413d-baa6-877d343bcebe\" (UID: \"19eb4850-b062-413d-baa6-877d343bcebe\") "
Sep 17 18:43:30.437078 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:30.436978    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v2rz4\" (UniqueName: \"kubernetes.io/projected/19eb4850-b062-413d-baa6-877d343bcebe-kube-api-access-v2rz4\") pod \"19eb4850-b062-413d-baa6-877d343bcebe\" (UID: \"19eb4850-b062-413d-baa6-877d343bcebe\") "
Sep 17 18:43:30.437078 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:30.437028    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19eb4850-b062-413d-baa6-877d343bcebe-kube-rbac-proxy-sar-config\") pod \"19eb4850-b062-413d-baa6-877d343bcebe\" (UID: \"19eb4850-b062-413d-baa6-877d343bcebe\") "
Sep 17 18:43:30.437279 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:30.437254    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19eb4850-b062-413d-baa6-877d343bcebe-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "19eb4850-b062-413d-baa6-877d343bcebe" (UID: "19eb4850-b062-413d-baa6-877d343bcebe"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:43:30.437390 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:30.437366    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19eb4850-b062-413d-baa6-877d343bcebe-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "19eb4850-b062-413d-baa6-877d343bcebe" (UID: "19eb4850-b062-413d-baa6-877d343bcebe"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:43:30.439209 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:30.439175    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19eb4850-b062-413d-baa6-877d343bcebe-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "19eb4850-b062-413d-baa6-877d343bcebe" (UID: "19eb4850-b062-413d-baa6-877d343bcebe"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:43:30.439209 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:30.439192    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19eb4850-b062-413d-baa6-877d343bcebe-kube-api-access-v2rz4" (OuterVolumeSpecName: "kube-api-access-v2rz4") pod "19eb4850-b062-413d-baa6-877d343bcebe" (UID: "19eb4850-b062-413d-baa6-877d343bcebe"). InnerVolumeSpecName "kube-api-access-v2rz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:43:30.537868 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:30.537771    2553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19eb4850-b062-413d-baa6-877d343bcebe-proxy-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:43:30.537868 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:30.537802    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2rz4\" (UniqueName: \"kubernetes.io/projected/19eb4850-b062-413d-baa6-877d343bcebe-kube-api-access-v2rz4\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:43:30.537868 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:30.537815    2553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19eb4850-b062-413d-baa6-877d343bcebe-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:43:30.537868 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:30.537828    2553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19eb4850-b062-413d-baa6-877d343bcebe-kserve-provision-location\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:43:30.539554 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:30.539519    2553 generic.go:356] "Generic (PLEG): container finished" podID="19eb4850-b062-413d-baa6-877d343bcebe" containerID="45b15233e796ee08a148e1b44f9c4770dfd580daa4961523ab34eb5089d755b5" exitCode=0
Sep 17 18:43:30.539652 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:30.539586    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8b7476b77-skpn5" event={"ID":"19eb4850-b062-413d-baa6-877d343bcebe","Type":"ContainerDied","Data":"45b15233e796ee08a148e1b44f9c4770dfd580daa4961523ab34eb5089d755b5"}
Sep 17 18:43:30.539708 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:30.539658    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8b7476b77-skpn5" event={"ID":"19eb4850-b062-413d-baa6-877d343bcebe","Type":"ContainerDied","Data":"31006f9d174d7ec38789d13b1d7087316f9c5dec43e0f33bb5aca23b68795e38"}
Sep 17 18:43:30.539708 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:30.539675    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8b7476b77-skpn5"
Sep 17 18:43:30.539708 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:30.539689    2553 scope.go:117] "RemoveContainer" containerID="8cf1d59ecb487b073744b1f835f7a4ba3463c21a41d5eb89771884033eb4adef"
Sep 17 18:43:30.539955 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:30.539930    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64484c89d5-r5v86" podUID="3ac5d326-3303-4184-8741-4b7d67843fb6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 17 18:43:30.549215 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:30.549191    2553 scope.go:117] "RemoveContainer" containerID="45b15233e796ee08a148e1b44f9c4770dfd580daa4961523ab34eb5089d755b5"
Sep 17 18:43:30.557452 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:30.557430    2553 scope.go:117] "RemoveContainer" containerID="bde36bea3c857603d5d891c302412c4687ed57fc27e472fe29dc6559d08d63c8"
Sep 17 18:43:30.564769 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:30.564740    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8b7476b77-skpn5"]
Sep 17 18:43:30.565582 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:30.565566    2553 scope.go:117] "RemoveContainer" containerID="8cf1d59ecb487b073744b1f835f7a4ba3463c21a41d5eb89771884033eb4adef"
Sep 17 18:43:30.566010 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:43:30.565988    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cf1d59ecb487b073744b1f835f7a4ba3463c21a41d5eb89771884033eb4adef\": container with ID starting with 8cf1d59ecb487b073744b1f835f7a4ba3463c21a41d5eb89771884033eb4adef not found: ID does not exist" containerID="8cf1d59ecb487b073744b1f835f7a4ba3463c21a41d5eb89771884033eb4adef"
Sep 17 18:43:30.566086 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:30.566019    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cf1d59ecb487b073744b1f835f7a4ba3463c21a41d5eb89771884033eb4adef"} err="failed to get container status \"8cf1d59ecb487b073744b1f835f7a4ba3463c21a41d5eb89771884033eb4adef\": rpc error: code = NotFound desc = could not find container \"8cf1d59ecb487b073744b1f835f7a4ba3463c21a41d5eb89771884033eb4adef\": container with ID starting with 8cf1d59ecb487b073744b1f835f7a4ba3463c21a41d5eb89771884033eb4adef not found: ID does not exist"
Sep 17 18:43:30.566086 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:30.566036    2553 scope.go:117] "RemoveContainer" containerID="45b15233e796ee08a148e1b44f9c4770dfd580daa4961523ab34eb5089d755b5"
Sep 17 18:43:30.566305 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:43:30.566292    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45b15233e796ee08a148e1b44f9c4770dfd580daa4961523ab34eb5089d755b5\": container with ID starting with 45b15233e796ee08a148e1b44f9c4770dfd580daa4961523ab34eb5089d755b5 not found: ID does not exist" containerID="45b15233e796ee08a148e1b44f9c4770dfd580daa4961523ab34eb5089d755b5"
Sep 17 18:43:30.566355 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:30.566308    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45b15233e796ee08a148e1b44f9c4770dfd580daa4961523ab34eb5089d755b5"} err="failed to get container status \"45b15233e796ee08a148e1b44f9c4770dfd580daa4961523ab34eb5089d755b5\": rpc error: code = NotFound desc = could not find container \"45b15233e796ee08a148e1b44f9c4770dfd580daa4961523ab34eb5089d755b5\": container with ID starting with 45b15233e796ee08a148e1b44f9c4770dfd580daa4961523ab34eb5089d755b5 not found: ID does not exist"
Sep 17 18:43:30.566355 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:30.566319    2553 scope.go:117] "RemoveContainer" containerID="bde36bea3c857603d5d891c302412c4687ed57fc27e472fe29dc6559d08d63c8"
Sep 17 18:43:30.566597 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:43:30.566545    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bde36bea3c857603d5d891c302412c4687ed57fc27e472fe29dc6559d08d63c8\": container with ID starting with bde36bea3c857603d5d891c302412c4687ed57fc27e472fe29dc6559d08d63c8 not found: ID does not exist" containerID="bde36bea3c857603d5d891c302412c4687ed57fc27e472fe29dc6559d08d63c8"
Sep 17 18:43:30.566687 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:30.566606    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bde36bea3c857603d5d891c302412c4687ed57fc27e472fe29dc6559d08d63c8"} err="failed to get container status \"bde36bea3c857603d5d891c302412c4687ed57fc27e472fe29dc6559d08d63c8\": rpc error: code = NotFound desc = could not find container \"bde36bea3c857603d5d891c302412c4687ed57fc27e472fe29dc6559d08d63c8\": container with ID starting with bde36bea3c857603d5d891c302412c4687ed57fc27e472fe29dc6559d08d63c8 not found: ID does not exist"
Sep 17 18:43:30.568855 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:30.568835    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8b7476b77-skpn5"]
Sep 17 18:43:31.376228 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:31.376193    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19eb4850-b062-413d-baa6-877d343bcebe" path="/var/lib/kubelet/pods/19eb4850-b062-413d-baa6-877d343bcebe/volumes"
Sep 17 18:43:35.544333 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:35.544299    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64484c89d5-r5v86"
Sep 17 18:43:35.544906 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:35.544877    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64484c89d5-r5v86" podUID="3ac5d326-3303-4184-8741-4b7d67843fb6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 17 18:43:45.545228 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:45.545184    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64484c89d5-r5v86" podUID="3ac5d326-3303-4184-8741-4b7d67843fb6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 17 18:43:55.545036 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:43:55.544990    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64484c89d5-r5v86" podUID="3ac5d326-3303-4184-8741-4b7d67843fb6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 17 18:44:05.544816 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:44:05.544774    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64484c89d5-r5v86" podUID="3ac5d326-3303-4184-8741-4b7d67843fb6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 17 18:44:15.545266 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:44:15.545225    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64484c89d5-r5v86" podUID="3ac5d326-3303-4184-8741-4b7d67843fb6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 17 18:44:25.545034 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:44:25.544985    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64484c89d5-r5v86" podUID="3ac5d326-3303-4184-8741-4b7d67843fb6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 17 18:44:35.545829 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:44:35.545743    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64484c89d5-r5v86" podUID="3ac5d326-3303-4184-8741-4b7d67843fb6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 17 18:44:45.545782 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:44:45.545746    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64484c89d5-r5v86"
Sep 17 18:44:55.459241 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:44:55.459206    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64484c89d5-r5v86"]
Sep 17 18:44:55.459768 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:44:55.459461    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64484c89d5-r5v86" podUID="3ac5d326-3303-4184-8741-4b7d67843fb6" containerName="kserve-container" containerID="cri-o://2051fcca997eb7ab1aaa90dce70f24aa5e6df1701722afb0dc0fd4ac74277f95" gracePeriod=30
Sep 17 18:44:55.459768 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:44:55.459508    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64484c89d5-r5v86" podUID="3ac5d326-3303-4184-8741-4b7d67843fb6" containerName="kube-rbac-proxy" containerID="cri-o://4241634f2b8887beac481abcb50f8caeaa77e4e69a27f47ffdad6a8765bd80b9" gracePeriod=30
Sep 17 18:44:55.540514 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:44:55.540465    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64484c89d5-r5v86" podUID="3ac5d326-3303-4184-8741-4b7d67843fb6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.54:8643/healthz\": dial tcp 10.134.0.54:8643: connect: connection refused"
Sep 17 18:44:55.544790 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:44:55.544761    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64484c89d5-r5v86" podUID="3ac5d326-3303-4184-8741-4b7d67843fb6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 17 18:44:55.938789 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:44:55.938755    2553 generic.go:356] "Generic (PLEG): container finished" podID="3ac5d326-3303-4184-8741-4b7d67843fb6" containerID="4241634f2b8887beac481abcb50f8caeaa77e4e69a27f47ffdad6a8765bd80b9" exitCode=2
Sep 17 18:44:55.938968 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:44:55.938805    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64484c89d5-r5v86" event={"ID":"3ac5d326-3303-4184-8741-4b7d67843fb6","Type":"ContainerDied","Data":"4241634f2b8887beac481abcb50f8caeaa77e4e69a27f47ffdad6a8765bd80b9"}
Sep 17 18:45:00.402528 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:00.402496    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64484c89d5-r5v86"
Sep 17 18:45:00.513224 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:00.513123    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ac5d326-3303-4184-8741-4b7d67843fb6-kube-rbac-proxy-sar-config\") pod \"3ac5d326-3303-4184-8741-4b7d67843fb6\" (UID: \"3ac5d326-3303-4184-8741-4b7d67843fb6\") "
Sep 17 18:45:00.513224 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:00.513184    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vgxds\" (UniqueName: \"kubernetes.io/projected/3ac5d326-3303-4184-8741-4b7d67843fb6-kube-api-access-vgxds\") pod \"3ac5d326-3303-4184-8741-4b7d67843fb6\" (UID: \"3ac5d326-3303-4184-8741-4b7d67843fb6\") "
Sep 17 18:45:00.513224 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:00.513217    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ac5d326-3303-4184-8741-4b7d67843fb6-proxy-tls\") pod \"3ac5d326-3303-4184-8741-4b7d67843fb6\" (UID: \"3ac5d326-3303-4184-8741-4b7d67843fb6\") "
Sep 17 18:45:00.513531 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:00.513235    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3ac5d326-3303-4184-8741-4b7d67843fb6-cabundle-cert\") pod \"3ac5d326-3303-4184-8741-4b7d67843fb6\" (UID: \"3ac5d326-3303-4184-8741-4b7d67843fb6\") "
Sep 17 18:45:00.513531 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:00.513253    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ac5d326-3303-4184-8741-4b7d67843fb6-kserve-provision-location\") pod \"3ac5d326-3303-4184-8741-4b7d67843fb6\" (UID: \"3ac5d326-3303-4184-8741-4b7d67843fb6\") "
Sep 17 18:45:00.513664 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:00.513555    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ac5d326-3303-4184-8741-4b7d67843fb6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3ac5d326-3303-4184-8741-4b7d67843fb6" (UID: "3ac5d326-3303-4184-8741-4b7d67843fb6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:45:00.513710 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:00.513671    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ac5d326-3303-4184-8741-4b7d67843fb6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3ac5d326-3303-4184-8741-4b7d67843fb6" (UID: "3ac5d326-3303-4184-8741-4b7d67843fb6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:45:00.513710 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:00.513675    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ac5d326-3303-4184-8741-4b7d67843fb6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3ac5d326-3303-4184-8741-4b7d67843fb6" (UID: "3ac5d326-3303-4184-8741-4b7d67843fb6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:45:00.515391 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:00.515368    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ac5d326-3303-4184-8741-4b7d67843fb6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3ac5d326-3303-4184-8741-4b7d67843fb6" (UID: "3ac5d326-3303-4184-8741-4b7d67843fb6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:45:00.515470 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:00.515422    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ac5d326-3303-4184-8741-4b7d67843fb6-kube-api-access-vgxds" (OuterVolumeSpecName: "kube-api-access-vgxds") pod "3ac5d326-3303-4184-8741-4b7d67843fb6" (UID: "3ac5d326-3303-4184-8741-4b7d67843fb6"). InnerVolumeSpecName "kube-api-access-vgxds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:45:00.614208 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:00.614172    2553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ac5d326-3303-4184-8741-4b7d67843fb6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:45:00.614208 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:00.614202    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgxds\" (UniqueName: \"kubernetes.io/projected/3ac5d326-3303-4184-8741-4b7d67843fb6-kube-api-access-vgxds\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:45:00.614208 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:00.614212    2553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ac5d326-3303-4184-8741-4b7d67843fb6-proxy-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:45:00.614208 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:00.614222    2553 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3ac5d326-3303-4184-8741-4b7d67843fb6-cabundle-cert\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:45:00.614486 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:00.614230    2553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ac5d326-3303-4184-8741-4b7d67843fb6-kserve-provision-location\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:45:00.962766 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:00.962729    2553 generic.go:356] "Generic (PLEG): container finished" podID="3ac5d326-3303-4184-8741-4b7d67843fb6" containerID="2051fcca997eb7ab1aaa90dce70f24aa5e6df1701722afb0dc0fd4ac74277f95" exitCode=0
Sep 17 18:45:00.962975 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:00.962782    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64484c89d5-r5v86"
Sep 17 18:45:00.962975 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:00.962790    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64484c89d5-r5v86" event={"ID":"3ac5d326-3303-4184-8741-4b7d67843fb6","Type":"ContainerDied","Data":"2051fcca997eb7ab1aaa90dce70f24aa5e6df1701722afb0dc0fd4ac74277f95"}
Sep 17 18:45:00.962975 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:00.962853    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64484c89d5-r5v86" event={"ID":"3ac5d326-3303-4184-8741-4b7d67843fb6","Type":"ContainerDied","Data":"632854c65d10c65d39a2d703040e6aad1d9270e72069e2a0f2fa85e0425bcec6"}
Sep 17 18:45:00.962975 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:00.962888    2553 scope.go:117] "RemoveContainer" containerID="4241634f2b8887beac481abcb50f8caeaa77e4e69a27f47ffdad6a8765bd80b9"
Sep 17 18:45:00.971514 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:00.971490    2553 scope.go:117] "RemoveContainer" containerID="2051fcca997eb7ab1aaa90dce70f24aa5e6df1701722afb0dc0fd4ac74277f95"
Sep 17 18:45:00.981046 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:00.981025    2553 scope.go:117] "RemoveContainer" containerID="3f174833efe65043a0357c7c5ef007f397a4e7c82de0f985f458759dd4d4fb43"
Sep 17 18:45:00.985558 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:00.985533    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64484c89d5-r5v86"]
Sep 17 18:45:00.988553 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:00.988526    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-64484c89d5-r5v86"]
Sep 17 18:45:00.989906 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:00.989888    2553 scope.go:117] "RemoveContainer" containerID="4241634f2b8887beac481abcb50f8caeaa77e4e69a27f47ffdad6a8765bd80b9"
Sep 17 18:45:00.990218 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:45:00.990201    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4241634f2b8887beac481abcb50f8caeaa77e4e69a27f47ffdad6a8765bd80b9\": container with ID starting with 4241634f2b8887beac481abcb50f8caeaa77e4e69a27f47ffdad6a8765bd80b9 not found: ID does not exist" containerID="4241634f2b8887beac481abcb50f8caeaa77e4e69a27f47ffdad6a8765bd80b9"
Sep 17 18:45:00.990262 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:00.990227    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4241634f2b8887beac481abcb50f8caeaa77e4e69a27f47ffdad6a8765bd80b9"} err="failed to get container status \"4241634f2b8887beac481abcb50f8caeaa77e4e69a27f47ffdad6a8765bd80b9\": rpc error: code = NotFound desc = could not find container \"4241634f2b8887beac481abcb50f8caeaa77e4e69a27f47ffdad6a8765bd80b9\": container with ID starting with 4241634f2b8887beac481abcb50f8caeaa77e4e69a27f47ffdad6a8765bd80b9 not found: ID does not exist"
Sep 17 18:45:00.990262 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:00.990248    2553 scope.go:117] "RemoveContainer" containerID="2051fcca997eb7ab1aaa90dce70f24aa5e6df1701722afb0dc0fd4ac74277f95"
Sep 17 18:45:00.990485 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:45:00.990470    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2051fcca997eb7ab1aaa90dce70f24aa5e6df1701722afb0dc0fd4ac74277f95\": container with ID starting with 2051fcca997eb7ab1aaa90dce70f24aa5e6df1701722afb0dc0fd4ac74277f95 not found: ID does not exist" containerID="2051fcca997eb7ab1aaa90dce70f24aa5e6df1701722afb0dc0fd4ac74277f95"
Sep 17 18:45:00.990541 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:00.990487    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2051fcca997eb7ab1aaa90dce70f24aa5e6df1701722afb0dc0fd4ac74277f95"} err="failed to get container status \"2051fcca997eb7ab1aaa90dce70f24aa5e6df1701722afb0dc0fd4ac74277f95\": rpc error: code = NotFound desc = could not find container \"2051fcca997eb7ab1aaa90dce70f24aa5e6df1701722afb0dc0fd4ac74277f95\": container with ID starting with 2051fcca997eb7ab1aaa90dce70f24aa5e6df1701722afb0dc0fd4ac74277f95 not found: ID does not exist"
Sep 17 18:45:00.990541 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:00.990499    2553 scope.go:117] "RemoveContainer" containerID="3f174833efe65043a0357c7c5ef007f397a4e7c82de0f985f458759dd4d4fb43"
Sep 17 18:45:00.990775 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:45:00.990755    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f174833efe65043a0357c7c5ef007f397a4e7c82de0f985f458759dd4d4fb43\": container with ID starting with 3f174833efe65043a0357c7c5ef007f397a4e7c82de0f985f458759dd4d4fb43 not found: ID does not exist" containerID="3f174833efe65043a0357c7c5ef007f397a4e7c82de0f985f458759dd4d4fb43"
Sep 17 18:45:00.990823 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:00.990783    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f174833efe65043a0357c7c5ef007f397a4e7c82de0f985f458759dd4d4fb43"} err="failed to get container status \"3f174833efe65043a0357c7c5ef007f397a4e7c82de0f985f458759dd4d4fb43\": rpc error: code = NotFound desc = could not find container \"3f174833efe65043a0357c7c5ef007f397a4e7c82de0f985f458759dd4d4fb43\": container with ID starting with 3f174833efe65043a0357c7c5ef007f397a4e7c82de0f985f458759dd4d4fb43 not found: ID does not exist"
Sep 17 18:45:01.377317 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:01.377226    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ac5d326-3303-4184-8741-4b7d67843fb6" path="/var/lib/kubelet/pods/3ac5d326-3303-4184-8741-4b7d67843fb6/volumes"
Sep 17 18:45:07.680814 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:07.680774    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85b45f5f77-62tpw"]
Sep 17 18:45:07.681204 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:07.681083    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19eb4850-b062-413d-baa6-877d343bcebe" containerName="storage-initializer"
Sep 17 18:45:07.681204 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:07.681095    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="19eb4850-b062-413d-baa6-877d343bcebe" containerName="storage-initializer"
Sep 17 18:45:07.681204 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:07.681109    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19eb4850-b062-413d-baa6-877d343bcebe" containerName="kube-rbac-proxy"
Sep 17 18:45:07.681204 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:07.681115    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="19eb4850-b062-413d-baa6-877d343bcebe" containerName="kube-rbac-proxy"
Sep 17 18:45:07.681204 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:07.681121    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ac5d326-3303-4184-8741-4b7d67843fb6" containerName="kserve-container"
Sep 17 18:45:07.681204 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:07.681129    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ac5d326-3303-4184-8741-4b7d67843fb6" containerName="kserve-container"
Sep 17 18:45:07.681204 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:07.681137    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19eb4850-b062-413d-baa6-877d343bcebe" containerName="kserve-container"
Sep 17 18:45:07.681204 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:07.681143    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="19eb4850-b062-413d-baa6-877d343bcebe" containerName="kserve-container"
Sep 17 18:45:07.681204 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:07.681151    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ac5d326-3303-4184-8741-4b7d67843fb6" containerName="kube-rbac-proxy"
Sep 17 18:45:07.681204 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:07.681156    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ac5d326-3303-4184-8741-4b7d67843fb6" containerName="kube-rbac-proxy"
Sep 17 18:45:07.681204 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:07.681166    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ac5d326-3303-4184-8741-4b7d67843fb6" containerName="storage-initializer"
Sep 17 18:45:07.681204 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:07.681171    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ac5d326-3303-4184-8741-4b7d67843fb6" containerName="storage-initializer"
Sep 17 18:45:07.681574 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:07.681216    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="19eb4850-b062-413d-baa6-877d343bcebe" containerName="kserve-container"
Sep 17 18:45:07.681574 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:07.681228    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ac5d326-3303-4184-8741-4b7d67843fb6" containerName="kserve-container"
Sep 17 18:45:07.681574 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:07.681234    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="19eb4850-b062-413d-baa6-877d343bcebe" containerName="kube-rbac-proxy"
Sep 17 18:45:07.681574 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:07.681240    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ac5d326-3303-4184-8741-4b7d67843fb6" containerName="kube-rbac-proxy"
Sep 17 18:45:07.685756 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:07.685727    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85b45f5f77-62tpw"
Sep 17 18:45:07.688340 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:07.688309    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert\""
Sep 17 18:45:07.688539 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:07.688341    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-7qcmd\""
Sep 17 18:45:07.688539 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:07.688377    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:45:07.689170 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:07.689147    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:45:07.689299 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:07.689287    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:45:07.689371 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:07.689317    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 17 18:45:07.689433 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:07.689367    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\""
Sep 17 18:45:07.694174 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:07.694151    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85b45f5f77-62tpw"]
Sep 17 18:45:07.784057 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:07.784009    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/856e4110-093f-4c84-a24e-996bf76118a5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-85b45f5f77-62tpw\" (UID: \"856e4110-093f-4c84-a24e-996bf76118a5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85b45f5f77-62tpw"
Sep 17 18:45:07.784057 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:07.784064    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpbk5\" (UniqueName: \"kubernetes.io/projected/856e4110-093f-4c84-a24e-996bf76118a5-kube-api-access-zpbk5\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-85b45f5f77-62tpw\" (UID: \"856e4110-093f-4c84-a24e-996bf76118a5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85b45f5f77-62tpw"
Sep 17 18:45:07.784275 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:07.784105    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/856e4110-093f-4c84-a24e-996bf76118a5-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-85b45f5f77-62tpw\" (UID: \"856e4110-093f-4c84-a24e-996bf76118a5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85b45f5f77-62tpw"
Sep 17 18:45:07.784275 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:07.784134    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/856e4110-093f-4c84-a24e-996bf76118a5-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-85b45f5f77-62tpw\" (UID: \"856e4110-093f-4c84-a24e-996bf76118a5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85b45f5f77-62tpw"
Sep 17 18:45:07.784275 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:07.784164    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/856e4110-093f-4c84-a24e-996bf76118a5-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-85b45f5f77-62tpw\" (UID: \"856e4110-093f-4c84-a24e-996bf76118a5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85b45f5f77-62tpw"
Sep 17 18:45:07.884922 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:07.884877    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/856e4110-093f-4c84-a24e-996bf76118a5-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-85b45f5f77-62tpw\" (UID: \"856e4110-093f-4c84-a24e-996bf76118a5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85b45f5f77-62tpw"
Sep 17 18:45:07.885101 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:07.885027    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/856e4110-093f-4c84-a24e-996bf76118a5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-85b45f5f77-62tpw\" (UID: \"856e4110-093f-4c84-a24e-996bf76118a5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85b45f5f77-62tpw"
Sep 17 18:45:07.885101 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:07.885078    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpbk5\" (UniqueName: \"kubernetes.io/projected/856e4110-093f-4c84-a24e-996bf76118a5-kube-api-access-zpbk5\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-85b45f5f77-62tpw\" (UID: \"856e4110-093f-4c84-a24e-996bf76118a5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85b45f5f77-62tpw"
Sep 17 18:45:07.885234 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:07.885111    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/856e4110-093f-4c84-a24e-996bf76118a5-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-85b45f5f77-62tpw\" (UID: \"856e4110-093f-4c84-a24e-996bf76118a5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85b45f5f77-62tpw"
Sep 17 18:45:07.885234 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:07.885159    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/856e4110-093f-4c84-a24e-996bf76118a5-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-85b45f5f77-62tpw\" (UID: \"856e4110-093f-4c84-a24e-996bf76118a5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85b45f5f77-62tpw"
Sep 17 18:45:07.885354 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:45:07.885264    2553 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert" not found
Sep 17 18:45:07.885354 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:07.885299    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/856e4110-093f-4c84-a24e-996bf76118a5-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-85b45f5f77-62tpw\" (UID: \"856e4110-093f-4c84-a24e-996bf76118a5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85b45f5f77-62tpw"
Sep 17 18:45:07.885469 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:45:07.885358    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/856e4110-093f-4c84-a24e-996bf76118a5-proxy-tls podName:856e4110-093f-4c84-a24e-996bf76118a5 nodeName:}" failed. No retries permitted until 2026-09-17 18:45:08.385336048 +0000 UTC m=+2903.081426727 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/856e4110-093f-4c84-a24e-996bf76118a5-proxy-tls") pod "isvc-sklearn-s3-tls-custom-pass-predictor-85b45f5f77-62tpw" (UID: "856e4110-093f-4c84-a24e-996bf76118a5") : secret "isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert" not found
Sep 17 18:45:07.885792 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:07.885771    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/856e4110-093f-4c84-a24e-996bf76118a5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-85b45f5f77-62tpw\" (UID: \"856e4110-093f-4c84-a24e-996bf76118a5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85b45f5f77-62tpw"
Sep 17 18:45:07.885875 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:07.885800    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/856e4110-093f-4c84-a24e-996bf76118a5-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-85b45f5f77-62tpw\" (UID: \"856e4110-093f-4c84-a24e-996bf76118a5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85b45f5f77-62tpw"
Sep 17 18:45:07.893632 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:07.893591    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpbk5\" (UniqueName: \"kubernetes.io/projected/856e4110-093f-4c84-a24e-996bf76118a5-kube-api-access-zpbk5\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-85b45f5f77-62tpw\" (UID: \"856e4110-093f-4c84-a24e-996bf76118a5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85b45f5f77-62tpw"
Sep 17 18:45:08.389190 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:08.389147    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/856e4110-093f-4c84-a24e-996bf76118a5-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-85b45f5f77-62tpw\" (UID: \"856e4110-093f-4c84-a24e-996bf76118a5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85b45f5f77-62tpw"
Sep 17 18:45:08.391601 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:08.391574    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/856e4110-093f-4c84-a24e-996bf76118a5-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-85b45f5f77-62tpw\" (UID: \"856e4110-093f-4c84-a24e-996bf76118a5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85b45f5f77-62tpw"
Sep 17 18:45:08.597722 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:08.597679    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85b45f5f77-62tpw"
Sep 17 18:45:08.872132 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:08.872107    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85b45f5f77-62tpw"]
Sep 17 18:45:08.874768 ip-10-0-137-95 kubenswrapper[2553]: W0917 18:45:08.874736    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod856e4110_093f_4c84_a24e_996bf76118a5.slice/crio-860115a8fe23a64282fcaaf32cc1f3143bcd06e3dc84a6fd855f65ee621a0a9b WatchSource:0}: Error finding container 860115a8fe23a64282fcaaf32cc1f3143bcd06e3dc84a6fd855f65ee621a0a9b: Status 404 returned error can't find the container with id 860115a8fe23a64282fcaaf32cc1f3143bcd06e3dc84a6fd855f65ee621a0a9b
Sep 17 18:45:08.998013 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:08.997975    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85b45f5f77-62tpw" event={"ID":"856e4110-093f-4c84-a24e-996bf76118a5","Type":"ContainerStarted","Data":"4d621bcfb5a4d2fae6866d2d22c054844ee9b559b5cea56a2a3d17666fd3308b"}
Sep 17 18:45:08.998184 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:08.998034    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85b45f5f77-62tpw" event={"ID":"856e4110-093f-4c84-a24e-996bf76118a5","Type":"ContainerStarted","Data":"860115a8fe23a64282fcaaf32cc1f3143bcd06e3dc84a6fd855f65ee621a0a9b"}
Sep 17 18:45:11.008781 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:11.008746    2553 generic.go:356] "Generic (PLEG): container finished" podID="856e4110-093f-4c84-a24e-996bf76118a5" containerID="4d621bcfb5a4d2fae6866d2d22c054844ee9b559b5cea56a2a3d17666fd3308b" exitCode=0
Sep 17 18:45:11.009289 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:11.008795    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85b45f5f77-62tpw" event={"ID":"856e4110-093f-4c84-a24e-996bf76118a5","Type":"ContainerDied","Data":"4d621bcfb5a4d2fae6866d2d22c054844ee9b559b5cea56a2a3d17666fd3308b"}
Sep 17 18:45:12.015427 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:12.015389    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85b45f5f77-62tpw" event={"ID":"856e4110-093f-4c84-a24e-996bf76118a5","Type":"ContainerStarted","Data":"0e8427f326647c58c6fcf5cc76c064061e4baca28ddf6442059f7c5dce2a723c"}
Sep 17 18:45:12.015882 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:12.015456    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85b45f5f77-62tpw" event={"ID":"856e4110-093f-4c84-a24e-996bf76118a5","Type":"ContainerStarted","Data":"6ee38f0122490490237449dd8f21311df1bc02d79a326a6135fe389fab30bcf1"}
Sep 17 18:45:12.015882 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:12.015658    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85b45f5f77-62tpw"
Sep 17 18:45:12.036954 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:12.036899    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85b45f5f77-62tpw" podStartSLOduration=3.903064354 podStartE2EDuration="5.036882858s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.133818504s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:45:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:45:12.03578409 +0000 UTC m=+2906.731874776" watchObservedRunningTime="2026-09-17 18:45:12.036882858 +0000 UTC m=+2906.732973541"
Sep 17 18:45:13.020513 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:13.020478    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85b45f5f77-62tpw"
Sep 17 18:45:13.021762 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:13.021735    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85b45f5f77-62tpw" podUID="856e4110-093f-4c84-a24e-996bf76118a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 17 18:45:14.024467 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:14.024426    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85b45f5f77-62tpw" podUID="856e4110-093f-4c84-a24e-996bf76118a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 17 18:45:19.029123 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:19.029093    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85b45f5f77-62tpw"
Sep 17 18:45:19.029738 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:19.029581    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85b45f5f77-62tpw" podUID="856e4110-093f-4c84-a24e-996bf76118a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 17 18:45:29.030066 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:29.030026    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85b45f5f77-62tpw" podUID="856e4110-093f-4c84-a24e-996bf76118a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 17 18:45:39.030541 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:39.030496    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85b45f5f77-62tpw" podUID="856e4110-093f-4c84-a24e-996bf76118a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 17 18:45:49.029637 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:49.029574    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85b45f5f77-62tpw" podUID="856e4110-093f-4c84-a24e-996bf76118a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 17 18:45:59.030294 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:45:59.030246    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85b45f5f77-62tpw" podUID="856e4110-093f-4c84-a24e-996bf76118a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 17 18:46:09.029673 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:09.029602    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85b45f5f77-62tpw" podUID="856e4110-093f-4c84-a24e-996bf76118a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 17 18:46:19.030276 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:19.030227    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85b45f5f77-62tpw" podUID="856e4110-093f-4c84-a24e-996bf76118a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 17 18:46:29.029780 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:29.029741    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85b45f5f77-62tpw"
Sep 17 18:46:37.704424 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:37.704386    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85b45f5f77-62tpw"]
Sep 17 18:46:37.705009 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:37.704651    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85b45f5f77-62tpw" podUID="856e4110-093f-4c84-a24e-996bf76118a5" containerName="kserve-container" containerID="cri-o://6ee38f0122490490237449dd8f21311df1bc02d79a326a6135fe389fab30bcf1" gracePeriod=30
Sep 17 18:46:37.705009 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:37.704689    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85b45f5f77-62tpw" podUID="856e4110-093f-4c84-a24e-996bf76118a5" containerName="kube-rbac-proxy" containerID="cri-o://0e8427f326647c58c6fcf5cc76c064061e4baca28ddf6442059f7c5dce2a723c" gracePeriod=30
Sep 17 18:46:38.405095 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:38.405045    2553 generic.go:356] "Generic (PLEG): container finished" podID="856e4110-093f-4c84-a24e-996bf76118a5" containerID="0e8427f326647c58c6fcf5cc76c064061e4baca28ddf6442059f7c5dce2a723c" exitCode=2
Sep 17 18:46:38.405319 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:38.405203    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85b45f5f77-62tpw" event={"ID":"856e4110-093f-4c84-a24e-996bf76118a5","Type":"ContainerDied","Data":"0e8427f326647c58c6fcf5cc76c064061e4baca28ddf6442059f7c5dce2a723c"}
Sep 17 18:46:39.025553 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:39.025504    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85b45f5f77-62tpw" podUID="856e4110-093f-4c84-a24e-996bf76118a5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.55:8643/healthz\": dial tcp 10.134.0.55:8643: connect: connection refused"
Sep 17 18:46:39.029930 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:39.029894    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85b45f5f77-62tpw" podUID="856e4110-093f-4c84-a24e-996bf76118a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 17 18:46:42.856086 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:42.856059    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85b45f5f77-62tpw"
Sep 17 18:46:43.022895 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:43.022796    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zpbk5\" (UniqueName: \"kubernetes.io/projected/856e4110-093f-4c84-a24e-996bf76118a5-kube-api-access-zpbk5\") pod \"856e4110-093f-4c84-a24e-996bf76118a5\" (UID: \"856e4110-093f-4c84-a24e-996bf76118a5\") "
Sep 17 18:46:43.022895 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:43.022866    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/856e4110-093f-4c84-a24e-996bf76118a5-cabundle-cert\") pod \"856e4110-093f-4c84-a24e-996bf76118a5\" (UID: \"856e4110-093f-4c84-a24e-996bf76118a5\") "
Sep 17 18:46:43.023144 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:43.022917    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/856e4110-093f-4c84-a24e-996bf76118a5-proxy-tls\") pod \"856e4110-093f-4c84-a24e-996bf76118a5\" (UID: \"856e4110-093f-4c84-a24e-996bf76118a5\") "
Sep 17 18:46:43.023144 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:43.022943    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/856e4110-093f-4c84-a24e-996bf76118a5-kserve-provision-location\") pod \"856e4110-093f-4c84-a24e-996bf76118a5\" (UID: \"856e4110-093f-4c84-a24e-996bf76118a5\") "
Sep 17 18:46:43.023144 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:43.022982    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/856e4110-093f-4c84-a24e-996bf76118a5-kube-rbac-proxy-sar-config\") pod \"856e4110-093f-4c84-a24e-996bf76118a5\" (UID: \"856e4110-093f-4c84-a24e-996bf76118a5\") "
Sep 17 18:46:43.023323 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:43.023291    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/856e4110-093f-4c84-a24e-996bf76118a5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "856e4110-093f-4c84-a24e-996bf76118a5" (UID: "856e4110-093f-4c84-a24e-996bf76118a5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:46:43.023396 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:43.023357    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/856e4110-093f-4c84-a24e-996bf76118a5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "856e4110-093f-4c84-a24e-996bf76118a5" (UID: "856e4110-093f-4c84-a24e-996bf76118a5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:46:43.023396 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:43.023367    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/856e4110-093f-4c84-a24e-996bf76118a5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "856e4110-093f-4c84-a24e-996bf76118a5" (UID: "856e4110-093f-4c84-a24e-996bf76118a5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:46:43.025142 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:43.025113    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/856e4110-093f-4c84-a24e-996bf76118a5-kube-api-access-zpbk5" (OuterVolumeSpecName: "kube-api-access-zpbk5") pod "856e4110-093f-4c84-a24e-996bf76118a5" (UID: "856e4110-093f-4c84-a24e-996bf76118a5"). InnerVolumeSpecName "kube-api-access-zpbk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:46:43.025253 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:43.025175    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/856e4110-093f-4c84-a24e-996bf76118a5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "856e4110-093f-4c84-a24e-996bf76118a5" (UID: "856e4110-093f-4c84-a24e-996bf76118a5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:46:43.123611 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:43.123569    2553 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/856e4110-093f-4c84-a24e-996bf76118a5-cabundle-cert\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:46:43.123611 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:43.123605    2553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/856e4110-093f-4c84-a24e-996bf76118a5-proxy-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:46:43.123611 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:43.123615    2553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/856e4110-093f-4c84-a24e-996bf76118a5-kserve-provision-location\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:46:43.123903 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:43.123649    2553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/856e4110-093f-4c84-a24e-996bf76118a5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:46:43.123903 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:43.123661    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpbk5\" (UniqueName: \"kubernetes.io/projected/856e4110-093f-4c84-a24e-996bf76118a5-kube-api-access-zpbk5\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:46:43.430311 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:43.430274    2553 generic.go:356] "Generic (PLEG): container finished" podID="856e4110-093f-4c84-a24e-996bf76118a5" containerID="6ee38f0122490490237449dd8f21311df1bc02d79a326a6135fe389fab30bcf1" exitCode=0
Sep 17 18:46:43.430460 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:43.430337    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85b45f5f77-62tpw"
Sep 17 18:46:43.430460 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:43.430353    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85b45f5f77-62tpw" event={"ID":"856e4110-093f-4c84-a24e-996bf76118a5","Type":"ContainerDied","Data":"6ee38f0122490490237449dd8f21311df1bc02d79a326a6135fe389fab30bcf1"}
Sep 17 18:46:43.430460 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:43.430430    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85b45f5f77-62tpw" event={"ID":"856e4110-093f-4c84-a24e-996bf76118a5","Type":"ContainerDied","Data":"860115a8fe23a64282fcaaf32cc1f3143bcd06e3dc84a6fd855f65ee621a0a9b"}
Sep 17 18:46:43.430591 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:43.430469    2553 scope.go:117] "RemoveContainer" containerID="0e8427f326647c58c6fcf5cc76c064061e4baca28ddf6442059f7c5dce2a723c"
Sep 17 18:46:43.439250 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:43.439226    2553 scope.go:117] "RemoveContainer" containerID="6ee38f0122490490237449dd8f21311df1bc02d79a326a6135fe389fab30bcf1"
Sep 17 18:46:43.447364 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:43.447345    2553 scope.go:117] "RemoveContainer" containerID="4d621bcfb5a4d2fae6866d2d22c054844ee9b559b5cea56a2a3d17666fd3308b"
Sep 17 18:46:43.448085 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:43.448061    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85b45f5f77-62tpw"]
Sep 17 18:46:43.453533 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:43.453508    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85b45f5f77-62tpw"]
Sep 17 18:46:43.455786 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:43.455766    2553 scope.go:117] "RemoveContainer" containerID="0e8427f326647c58c6fcf5cc76c064061e4baca28ddf6442059f7c5dce2a723c"
Sep 17 18:46:43.456125 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:46:43.456100    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e8427f326647c58c6fcf5cc76c064061e4baca28ddf6442059f7c5dce2a723c\": container with ID starting with 0e8427f326647c58c6fcf5cc76c064061e4baca28ddf6442059f7c5dce2a723c not found: ID does not exist" containerID="0e8427f326647c58c6fcf5cc76c064061e4baca28ddf6442059f7c5dce2a723c"
Sep 17 18:46:43.456174 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:43.456137    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e8427f326647c58c6fcf5cc76c064061e4baca28ddf6442059f7c5dce2a723c"} err="failed to get container status \"0e8427f326647c58c6fcf5cc76c064061e4baca28ddf6442059f7c5dce2a723c\": rpc error: code = NotFound desc = could not find container \"0e8427f326647c58c6fcf5cc76c064061e4baca28ddf6442059f7c5dce2a723c\": container with ID starting with 0e8427f326647c58c6fcf5cc76c064061e4baca28ddf6442059f7c5dce2a723c not found: ID does not exist"
Sep 17 18:46:43.456174 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:43.456159    2553 scope.go:117] "RemoveContainer" containerID="6ee38f0122490490237449dd8f21311df1bc02d79a326a6135fe389fab30bcf1"
Sep 17 18:46:43.456441 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:46:43.456421    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ee38f0122490490237449dd8f21311df1bc02d79a326a6135fe389fab30bcf1\": container with ID starting with 6ee38f0122490490237449dd8f21311df1bc02d79a326a6135fe389fab30bcf1 not found: ID does not exist" containerID="6ee38f0122490490237449dd8f21311df1bc02d79a326a6135fe389fab30bcf1"
Sep 17 18:46:43.456488 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:43.456446    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ee38f0122490490237449dd8f21311df1bc02d79a326a6135fe389fab30bcf1"} err="failed to get container status \"6ee38f0122490490237449dd8f21311df1bc02d79a326a6135fe389fab30bcf1\": rpc error: code = NotFound desc = could not find container \"6ee38f0122490490237449dd8f21311df1bc02d79a326a6135fe389fab30bcf1\": container with ID starting with 6ee38f0122490490237449dd8f21311df1bc02d79a326a6135fe389fab30bcf1 not found: ID does not exist"
Sep 17 18:46:43.456488 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:43.456463    2553 scope.go:117] "RemoveContainer" containerID="4d621bcfb5a4d2fae6866d2d22c054844ee9b559b5cea56a2a3d17666fd3308b"
Sep 17 18:46:43.456732 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:46:43.456712    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d621bcfb5a4d2fae6866d2d22c054844ee9b559b5cea56a2a3d17666fd3308b\": container with ID starting with 4d621bcfb5a4d2fae6866d2d22c054844ee9b559b5cea56a2a3d17666fd3308b not found: ID does not exist" containerID="4d621bcfb5a4d2fae6866d2d22c054844ee9b559b5cea56a2a3d17666fd3308b"
Sep 17 18:46:43.456788 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:43.456739    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d621bcfb5a4d2fae6866d2d22c054844ee9b559b5cea56a2a3d17666fd3308b"} err="failed to get container status \"4d621bcfb5a4d2fae6866d2d22c054844ee9b559b5cea56a2a3d17666fd3308b\": rpc error: code = NotFound desc = could not find container \"4d621bcfb5a4d2fae6866d2d22c054844ee9b559b5cea56a2a3d17666fd3308b\": container with ID starting with 4d621bcfb5a4d2fae6866d2d22c054844ee9b559b5cea56a2a3d17666fd3308b not found: ID does not exist"
Sep 17 18:46:45.376902 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:45.376869    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="856e4110-093f-4c84-a24e-996bf76118a5" path="/var/lib/kubelet/pods/856e4110-093f-4c84-a24e-996bf76118a5/volumes"
Sep 17 18:46:47.513810 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:47.513778    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-btmqz_622ee5be-fb50-42b6-ae23-42900b7c2d98/ovn-acl-logging/0.log"
Sep 17 18:46:47.522943 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:47.522919    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-btmqz_622ee5be-fb50-42b6-ae23-42900b7c2d98/ovn-acl-logging/0.log"
Sep 17 18:46:49.929061 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:49.929028    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7f49b89cc4-vb9zc"]
Sep 17 18:46:49.929451 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:49.929338    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="856e4110-093f-4c84-a24e-996bf76118a5" containerName="storage-initializer"
Sep 17 18:46:49.929451 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:49.929349    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="856e4110-093f-4c84-a24e-996bf76118a5" containerName="storage-initializer"
Sep 17 18:46:49.929451 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:49.929366    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="856e4110-093f-4c84-a24e-996bf76118a5" containerName="kube-rbac-proxy"
Sep 17 18:46:49.929451 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:49.929372    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="856e4110-093f-4c84-a24e-996bf76118a5" containerName="kube-rbac-proxy"
Sep 17 18:46:49.929451 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:49.929378    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="856e4110-093f-4c84-a24e-996bf76118a5" containerName="kserve-container"
Sep 17 18:46:49.929451 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:49.929384    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="856e4110-093f-4c84-a24e-996bf76118a5" containerName="kserve-container"
Sep 17 18:46:49.929451 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:49.929436    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="856e4110-093f-4c84-a24e-996bf76118a5" containerName="kserve-container"
Sep 17 18:46:49.929451 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:49.929444    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="856e4110-093f-4c84-a24e-996bf76118a5" containerName="kube-rbac-proxy"
Sep 17 18:46:49.932505 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:49.932487    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7f49b89cc4-vb9zc"
Sep 17 18:46:49.935246 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:49.935219    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 17 18:46:49.935419 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:49.935227    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert\""
Sep 17 18:46:49.935419 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:49.935357    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\""
Sep 17 18:46:49.935569 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:49.935429    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:46:49.935569 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:49.935490    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:46:49.941785 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:49.941758    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:46:49.942035 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:49.941845    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-7qcmd\""
Sep 17 18:46:49.944635 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:49.944590    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7f49b89cc4-vb9zc"]
Sep 17 18:46:50.082228 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:50.082191    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61097d86-fb07-44d8-be5b-315d49463702-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7f49b89cc4-vb9zc\" (UID: \"61097d86-fb07-44d8-be5b-315d49463702\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7f49b89cc4-vb9zc"
Sep 17 18:46:50.082408 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:50.082264    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61097d86-fb07-44d8-be5b-315d49463702-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7f49b89cc4-vb9zc\" (UID: \"61097d86-fb07-44d8-be5b-315d49463702\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7f49b89cc4-vb9zc"
Sep 17 18:46:50.082408 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:50.082292    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/61097d86-fb07-44d8-be5b-315d49463702-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7f49b89cc4-vb9zc\" (UID: \"61097d86-fb07-44d8-be5b-315d49463702\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7f49b89cc4-vb9zc"
Sep 17 18:46:50.082408 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:50.082319    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g2bv\" (UniqueName: \"kubernetes.io/projected/61097d86-fb07-44d8-be5b-315d49463702-kube-api-access-4g2bv\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7f49b89cc4-vb9zc\" (UID: \"61097d86-fb07-44d8-be5b-315d49463702\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7f49b89cc4-vb9zc"
Sep 17 18:46:50.082408 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:50.082334    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/61097d86-fb07-44d8-be5b-315d49463702-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7f49b89cc4-vb9zc\" (UID: \"61097d86-fb07-44d8-be5b-315d49463702\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7f49b89cc4-vb9zc"
Sep 17 18:46:50.183787 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:50.183696    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61097d86-fb07-44d8-be5b-315d49463702-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7f49b89cc4-vb9zc\" (UID: \"61097d86-fb07-44d8-be5b-315d49463702\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7f49b89cc4-vb9zc"
Sep 17 18:46:50.183787 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:50.183746    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/61097d86-fb07-44d8-be5b-315d49463702-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7f49b89cc4-vb9zc\" (UID: \"61097d86-fb07-44d8-be5b-315d49463702\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7f49b89cc4-vb9zc"
Sep 17 18:46:50.183787 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:50.183772    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g2bv\" (UniqueName: \"kubernetes.io/projected/61097d86-fb07-44d8-be5b-315d49463702-kube-api-access-4g2bv\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7f49b89cc4-vb9zc\" (UID: \"61097d86-fb07-44d8-be5b-315d49463702\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7f49b89cc4-vb9zc"
Sep 17 18:46:50.184050 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:50.183793    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/61097d86-fb07-44d8-be5b-315d49463702-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7f49b89cc4-vb9zc\" (UID: \"61097d86-fb07-44d8-be5b-315d49463702\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7f49b89cc4-vb9zc"
Sep 17 18:46:50.184050 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:50.183820    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61097d86-fb07-44d8-be5b-315d49463702-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7f49b89cc4-vb9zc\" (UID: \"61097d86-fb07-44d8-be5b-315d49463702\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7f49b89cc4-vb9zc"
Sep 17 18:46:50.184239 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:50.184218    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61097d86-fb07-44d8-be5b-315d49463702-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7f49b89cc4-vb9zc\" (UID: \"61097d86-fb07-44d8-be5b-315d49463702\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7f49b89cc4-vb9zc"
Sep 17 18:46:50.184510 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:50.184485    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/61097d86-fb07-44d8-be5b-315d49463702-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7f49b89cc4-vb9zc\" (UID: \"61097d86-fb07-44d8-be5b-315d49463702\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7f49b89cc4-vb9zc"
Sep 17 18:46:50.184592 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:50.184548    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/61097d86-fb07-44d8-be5b-315d49463702-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7f49b89cc4-vb9zc\" (UID: \"61097d86-fb07-44d8-be5b-315d49463702\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7f49b89cc4-vb9zc"
Sep 17 18:46:50.186172 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:50.186152    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61097d86-fb07-44d8-be5b-315d49463702-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7f49b89cc4-vb9zc\" (UID: \"61097d86-fb07-44d8-be5b-315d49463702\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7f49b89cc4-vb9zc"
Sep 17 18:46:50.191489 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:50.191461    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g2bv\" (UniqueName: \"kubernetes.io/projected/61097d86-fb07-44d8-be5b-315d49463702-kube-api-access-4g2bv\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7f49b89cc4-vb9zc\" (UID: \"61097d86-fb07-44d8-be5b-315d49463702\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7f49b89cc4-vb9zc"
Sep 17 18:46:50.249694 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:50.249656    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7f49b89cc4-vb9zc"
Sep 17 18:46:50.522498 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:50.522471    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7f49b89cc4-vb9zc"]
Sep 17 18:46:50.524991 ip-10-0-137-95 kubenswrapper[2553]: W0917 18:46:50.524962    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61097d86_fb07_44d8_be5b_315d49463702.slice/crio-fdcb6125aea21c55a5882c3e9ddcf1928b8f05a007f381fc61f16ea507262dae WatchSource:0}: Error finding container fdcb6125aea21c55a5882c3e9ddcf1928b8f05a007f381fc61f16ea507262dae: Status 404 returned error can't find the container with id fdcb6125aea21c55a5882c3e9ddcf1928b8f05a007f381fc61f16ea507262dae
Sep 17 18:46:51.468557 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:51.468519    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7f49b89cc4-vb9zc" event={"ID":"61097d86-fb07-44d8-be5b-315d49463702","Type":"ContainerStarted","Data":"3b050089bf82af63d35e0f1b21107b9bc911401a18fbee0a9bbe0d0789bdbd84"}
Sep 17 18:46:51.468974 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:51.468591    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7f49b89cc4-vb9zc" event={"ID":"61097d86-fb07-44d8-be5b-315d49463702","Type":"ContainerStarted","Data":"fdcb6125aea21c55a5882c3e9ddcf1928b8f05a007f381fc61f16ea507262dae"}
Sep 17 18:46:52.474710 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:52.474676    2553 generic.go:356] "Generic (PLEG): container finished" podID="61097d86-fb07-44d8-be5b-315d49463702" containerID="3b050089bf82af63d35e0f1b21107b9bc911401a18fbee0a9bbe0d0789bdbd84" exitCode=0
Sep 17 18:46:52.475099 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:52.474739    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7f49b89cc4-vb9zc" event={"ID":"61097d86-fb07-44d8-be5b-315d49463702","Type":"ContainerDied","Data":"3b050089bf82af63d35e0f1b21107b9bc911401a18fbee0a9bbe0d0789bdbd84"}
Sep 17 18:46:53.481397 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:53.481358    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7f49b89cc4-vb9zc" event={"ID":"61097d86-fb07-44d8-be5b-315d49463702","Type":"ContainerStarted","Data":"6cda594d313999550bb8cd87f7048014ca0b6297e38f737470a9e13b6cf3fb86"}
Sep 17 18:46:53.481913 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:53.481423    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7f49b89cc4-vb9zc" event={"ID":"61097d86-fb07-44d8-be5b-315d49463702","Type":"ContainerStarted","Data":"845995d1ac838f5d44d0d75cb3742596d774f30f4dd1702f2ddbfebe63f42025"}
Sep 17 18:46:53.481913 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:53.481459    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7f49b89cc4-vb9zc"
Sep 17 18:46:53.481913 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:53.481489    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7f49b89cc4-vb9zc"
Sep 17 18:46:53.482996 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:53.482967    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7f49b89cc4-vb9zc" podUID="61097d86-fb07-44d8-be5b-315d49463702" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 17 18:46:53.502546 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:53.502488    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7f49b89cc4-vb9zc" podStartSLOduration=3.388799 podStartE2EDuration="4.502474561s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.113675561s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:46:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:46:53.500552159 +0000 UTC m=+3008.196642844" watchObservedRunningTime="2026-09-17 18:46:53.502474561 +0000 UTC m=+3008.198565244"
Sep 17 18:46:54.485523 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:54.485484    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7f49b89cc4-vb9zc" podUID="61097d86-fb07-44d8-be5b-315d49463702" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 17 18:46:59.490530 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:59.490489    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7f49b89cc4-vb9zc"
Sep 17 18:46:59.491189 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:46:59.491153    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7f49b89cc4-vb9zc" podUID="61097d86-fb07-44d8-be5b-315d49463702" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 17 18:47:09.491917 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:47:09.491874    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7f49b89cc4-vb9zc" podUID="61097d86-fb07-44d8-be5b-315d49463702" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 17 18:47:19.491309 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:47:19.491270    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7f49b89cc4-vb9zc" podUID="61097d86-fb07-44d8-be5b-315d49463702" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 17 18:47:29.492011 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:47:29.491910    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7f49b89cc4-vb9zc" podUID="61097d86-fb07-44d8-be5b-315d49463702" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 17 18:47:39.491209 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:47:39.491164    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7f49b89cc4-vb9zc" podUID="61097d86-fb07-44d8-be5b-315d49463702" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 17 18:47:49.491831 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:47:49.491790    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7f49b89cc4-vb9zc" podUID="61097d86-fb07-44d8-be5b-315d49463702" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 17 18:47:59.491547 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:47:59.491502    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7f49b89cc4-vb9zc" podUID="61097d86-fb07-44d8-be5b-315d49463702" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 17 18:48:09.492123 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:09.492088    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7f49b89cc4-vb9zc"
Sep 17 18:48:09.928260 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:09.928229    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7f49b89cc4-vb9zc"]
Sep 17 18:48:09.928496 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:09.928466    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7f49b89cc4-vb9zc" podUID="61097d86-fb07-44d8-be5b-315d49463702" containerName="kserve-container" containerID="cri-o://845995d1ac838f5d44d0d75cb3742596d774f30f4dd1702f2ddbfebe63f42025" gracePeriod=30
Sep 17 18:48:09.928582 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:09.928500    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7f49b89cc4-vb9zc" podUID="61097d86-fb07-44d8-be5b-315d49463702" containerName="kube-rbac-proxy" containerID="cri-o://6cda594d313999550bb8cd87f7048014ca0b6297e38f737470a9e13b6cf3fb86" gracePeriod=30
Sep 17 18:48:10.824280 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:10.824245    2553 generic.go:356] "Generic (PLEG): container finished" podID="61097d86-fb07-44d8-be5b-315d49463702" containerID="6cda594d313999550bb8cd87f7048014ca0b6297e38f737470a9e13b6cf3fb86" exitCode=2
Sep 17 18:48:10.824698 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:10.824310    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7f49b89cc4-vb9zc" event={"ID":"61097d86-fb07-44d8-be5b-315d49463702","Type":"ContainerDied","Data":"6cda594d313999550bb8cd87f7048014ca0b6297e38f737470a9e13b6cf3fb86"}
Sep 17 18:48:14.486067 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:14.486019    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7f49b89cc4-vb9zc" podUID="61097d86-fb07-44d8-be5b-315d49463702" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.56:8643/healthz\": dial tcp 10.134.0.56:8643: connect: connection refused"
Sep 17 18:48:14.845556 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:14.845524    2553 generic.go:356] "Generic (PLEG): container finished" podID="61097d86-fb07-44d8-be5b-315d49463702" containerID="845995d1ac838f5d44d0d75cb3742596d774f30f4dd1702f2ddbfebe63f42025" exitCode=0
Sep 17 18:48:14.845748 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:14.845599    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7f49b89cc4-vb9zc" event={"ID":"61097d86-fb07-44d8-be5b-315d49463702","Type":"ContainerDied","Data":"845995d1ac838f5d44d0d75cb3742596d774f30f4dd1702f2ddbfebe63f42025"}
Sep 17 18:48:14.982881 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:14.982856    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7f49b89cc4-vb9zc"
Sep 17 18:48:15.070823 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:15.070728    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61097d86-fb07-44d8-be5b-315d49463702-kserve-provision-location\") pod \"61097d86-fb07-44d8-be5b-315d49463702\" (UID: \"61097d86-fb07-44d8-be5b-315d49463702\") "
Sep 17 18:48:15.070823 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:15.070768    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4g2bv\" (UniqueName: \"kubernetes.io/projected/61097d86-fb07-44d8-be5b-315d49463702-kube-api-access-4g2bv\") pod \"61097d86-fb07-44d8-be5b-315d49463702\" (UID: \"61097d86-fb07-44d8-be5b-315d49463702\") "
Sep 17 18:48:15.070823 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:15.070788    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/61097d86-fb07-44d8-be5b-315d49463702-cabundle-cert\") pod \"61097d86-fb07-44d8-be5b-315d49463702\" (UID: \"61097d86-fb07-44d8-be5b-315d49463702\") "
Sep 17 18:48:15.071082 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:15.070827    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/61097d86-fb07-44d8-be5b-315d49463702-kube-rbac-proxy-sar-config\") pod \"61097d86-fb07-44d8-be5b-315d49463702\" (UID: \"61097d86-fb07-44d8-be5b-315d49463702\") "
Sep 17 18:48:15.071082 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:15.070908    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61097d86-fb07-44d8-be5b-315d49463702-proxy-tls\") pod \"61097d86-fb07-44d8-be5b-315d49463702\" (UID: \"61097d86-fb07-44d8-be5b-315d49463702\") "
Sep 17 18:48:15.071166 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:15.071089    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61097d86-fb07-44d8-be5b-315d49463702-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "61097d86-fb07-44d8-be5b-315d49463702" (UID: "61097d86-fb07-44d8-be5b-315d49463702"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:48:15.071214 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:15.071171    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61097d86-fb07-44d8-be5b-315d49463702-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "61097d86-fb07-44d8-be5b-315d49463702" (UID: "61097d86-fb07-44d8-be5b-315d49463702"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:48:15.071214 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:15.071182    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61097d86-fb07-44d8-be5b-315d49463702-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "61097d86-fb07-44d8-be5b-315d49463702" (UID: "61097d86-fb07-44d8-be5b-315d49463702"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:48:15.071214 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:15.071196    2553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61097d86-fb07-44d8-be5b-315d49463702-kserve-provision-location\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:48:15.073000 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:15.072971    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61097d86-fb07-44d8-be5b-315d49463702-kube-api-access-4g2bv" (OuterVolumeSpecName: "kube-api-access-4g2bv") pod "61097d86-fb07-44d8-be5b-315d49463702" (UID: "61097d86-fb07-44d8-be5b-315d49463702"). InnerVolumeSpecName "kube-api-access-4g2bv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:48:15.073112 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:15.072997    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61097d86-fb07-44d8-be5b-315d49463702-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "61097d86-fb07-44d8-be5b-315d49463702" (UID: "61097d86-fb07-44d8-be5b-315d49463702"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:48:15.172327 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:15.172280    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4g2bv\" (UniqueName: \"kubernetes.io/projected/61097d86-fb07-44d8-be5b-315d49463702-kube-api-access-4g2bv\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:48:15.172327 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:15.172322    2553 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/61097d86-fb07-44d8-be5b-315d49463702-cabundle-cert\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:48:15.172327 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:15.172333    2553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/61097d86-fb07-44d8-be5b-315d49463702-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:48:15.172572 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:15.172342    2553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61097d86-fb07-44d8-be5b-315d49463702-proxy-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:48:15.852023 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:15.851993    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7f49b89cc4-vb9zc"
Sep 17 18:48:15.852435 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:15.851997    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7f49b89cc4-vb9zc" event={"ID":"61097d86-fb07-44d8-be5b-315d49463702","Type":"ContainerDied","Data":"fdcb6125aea21c55a5882c3e9ddcf1928b8f05a007f381fc61f16ea507262dae"}
Sep 17 18:48:15.852435 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:15.852135    2553 scope.go:117] "RemoveContainer" containerID="6cda594d313999550bb8cd87f7048014ca0b6297e38f737470a9e13b6cf3fb86"
Sep 17 18:48:15.860786 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:15.860770    2553 scope.go:117] "RemoveContainer" containerID="845995d1ac838f5d44d0d75cb3742596d774f30f4dd1702f2ddbfebe63f42025"
Sep 17 18:48:15.868274 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:15.868254    2553 scope.go:117] "RemoveContainer" containerID="3b050089bf82af63d35e0f1b21107b9bc911401a18fbee0a9bbe0d0789bdbd84"
Sep 17 18:48:15.870198 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:15.870175    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7f49b89cc4-vb9zc"]
Sep 17 18:48:15.873848 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:15.873825    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7f49b89cc4-vb9zc"]
Sep 17 18:48:17.376362 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:17.376326    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61097d86-fb07-44d8-be5b-315d49463702" path="/var/lib/kubelet/pods/61097d86-fb07-44d8-be5b-315d49463702/volumes"
Sep 17 18:48:22.157079 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:22.157039    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zl8dq"]
Sep 17 18:48:22.157584 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:22.157382    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61097d86-fb07-44d8-be5b-315d49463702" containerName="kserve-container"
Sep 17 18:48:22.157584 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:22.157394    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="61097d86-fb07-44d8-be5b-315d49463702" containerName="kserve-container"
Sep 17 18:48:22.157584 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:22.157438    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61097d86-fb07-44d8-be5b-315d49463702" containerName="kube-rbac-proxy"
Sep 17 18:48:22.157584 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:22.157446    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="61097d86-fb07-44d8-be5b-315d49463702" containerName="kube-rbac-proxy"
Sep 17 18:48:22.157584 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:22.157456    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61097d86-fb07-44d8-be5b-315d49463702" containerName="storage-initializer"
Sep 17 18:48:22.157584 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:22.157462    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="61097d86-fb07-44d8-be5b-315d49463702" containerName="storage-initializer"
Sep 17 18:48:22.157584 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:22.157528    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="61097d86-fb07-44d8-be5b-315d49463702" containerName="kube-rbac-proxy"
Sep 17 18:48:22.157584 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:22.157539    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="61097d86-fb07-44d8-be5b-315d49463702" containerName="kserve-container"
Sep 17 18:48:22.160461 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:22.160441    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zl8dq"
Sep 17 18:48:22.162932 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:22.162898    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-kube-rbac-proxy-sar-config\""
Sep 17 18:48:22.163068 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:22.162899    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-predictor-serving-cert\""
Sep 17 18:48:22.163068 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:22.162907    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:48:22.163199 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:22.163083    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:48:22.163836 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:22.163812    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-7qcmd\""
Sep 17 18:48:22.163955 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:22.163818    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:48:22.163955 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:22.163822    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:48:22.171813 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:22.171792    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zl8dq"]
Sep 17 18:48:22.233774 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:22.233737    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cfd1dd23-96d5-42d8-911a-72d7d3959901-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-zl8dq\" (UID: \"cfd1dd23-96d5-42d8-911a-72d7d3959901\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zl8dq"
Sep 17 18:48:22.233943 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:22.233792    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cfd1dd23-96d5-42d8-911a-72d7d3959901-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-zl8dq\" (UID: \"cfd1dd23-96d5-42d8-911a-72d7d3959901\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zl8dq"
Sep 17 18:48:22.233943 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:22.233853    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzgnv\" (UniqueName: \"kubernetes.io/projected/cfd1dd23-96d5-42d8-911a-72d7d3959901-kube-api-access-gzgnv\") pod \"isvc-pmml-runtime-predictor-885fc8777-zl8dq\" (UID: \"cfd1dd23-96d5-42d8-911a-72d7d3959901\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zl8dq"
Sep 17 18:48:22.233943 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:22.233880    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cfd1dd23-96d5-42d8-911a-72d7d3959901-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-zl8dq\" (UID: \"cfd1dd23-96d5-42d8-911a-72d7d3959901\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zl8dq"
Sep 17 18:48:22.233943 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:22.233906    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cfd1dd23-96d5-42d8-911a-72d7d3959901-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-zl8dq\" (UID: \"cfd1dd23-96d5-42d8-911a-72d7d3959901\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zl8dq"
Sep 17 18:48:22.334286 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:22.334242    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cfd1dd23-96d5-42d8-911a-72d7d3959901-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-zl8dq\" (UID: \"cfd1dd23-96d5-42d8-911a-72d7d3959901\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zl8dq"
Sep 17 18:48:22.334490 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:22.334300    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cfd1dd23-96d5-42d8-911a-72d7d3959901-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-zl8dq\" (UID: \"cfd1dd23-96d5-42d8-911a-72d7d3959901\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zl8dq"
Sep 17 18:48:22.334490 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:22.334359    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzgnv\" (UniqueName: \"kubernetes.io/projected/cfd1dd23-96d5-42d8-911a-72d7d3959901-kube-api-access-gzgnv\") pod \"isvc-pmml-runtime-predictor-885fc8777-zl8dq\" (UID: \"cfd1dd23-96d5-42d8-911a-72d7d3959901\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zl8dq"
Sep 17 18:48:22.334490 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:22.334479    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cfd1dd23-96d5-42d8-911a-72d7d3959901-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-zl8dq\" (UID: \"cfd1dd23-96d5-42d8-911a-72d7d3959901\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zl8dq"
Sep 17 18:48:22.334704 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:22.334526    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cfd1dd23-96d5-42d8-911a-72d7d3959901-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-zl8dq\" (UID: \"cfd1dd23-96d5-42d8-911a-72d7d3959901\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zl8dq"
Sep 17 18:48:22.334827 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:22.334771    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cfd1dd23-96d5-42d8-911a-72d7d3959901-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-zl8dq\" (UID: \"cfd1dd23-96d5-42d8-911a-72d7d3959901\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zl8dq"
Sep 17 18:48:22.335303 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:22.335281    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cfd1dd23-96d5-42d8-911a-72d7d3959901-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-zl8dq\" (UID: \"cfd1dd23-96d5-42d8-911a-72d7d3959901\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zl8dq"
Sep 17 18:48:22.335346 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:22.335281    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cfd1dd23-96d5-42d8-911a-72d7d3959901-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-zl8dq\" (UID: \"cfd1dd23-96d5-42d8-911a-72d7d3959901\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zl8dq"
Sep 17 18:48:22.337066 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:22.337045    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cfd1dd23-96d5-42d8-911a-72d7d3959901-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-zl8dq\" (UID: \"cfd1dd23-96d5-42d8-911a-72d7d3959901\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zl8dq"
Sep 17 18:48:22.343019 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:22.342994    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzgnv\" (UniqueName: \"kubernetes.io/projected/cfd1dd23-96d5-42d8-911a-72d7d3959901-kube-api-access-gzgnv\") pod \"isvc-pmml-runtime-predictor-885fc8777-zl8dq\" (UID: \"cfd1dd23-96d5-42d8-911a-72d7d3959901\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zl8dq"
Sep 17 18:48:22.472013 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:22.471916    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zl8dq"
Sep 17 18:48:22.744511 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:22.744483    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zl8dq"]
Sep 17 18:48:22.747110 ip-10-0-137-95 kubenswrapper[2553]: W0917 18:48:22.747076    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfd1dd23_96d5_42d8_911a_72d7d3959901.slice/crio-c78ad14529b3a87f27af21a3f82a9cc6ef78166cea9bd7378606c58695528641 WatchSource:0}: Error finding container c78ad14529b3a87f27af21a3f82a9cc6ef78166cea9bd7378606c58695528641: Status 404 returned error can't find the container with id c78ad14529b3a87f27af21a3f82a9cc6ef78166cea9bd7378606c58695528641
Sep 17 18:48:22.885197 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:22.885157    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zl8dq" event={"ID":"cfd1dd23-96d5-42d8-911a-72d7d3959901","Type":"ContainerStarted","Data":"66b35d744b951569d6a4c2fcb3c53357cd06d978485a0044d2d4dbebcbc0c788"}
Sep 17 18:48:22.885377 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:22.885226    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zl8dq" event={"ID":"cfd1dd23-96d5-42d8-911a-72d7d3959901","Type":"ContainerStarted","Data":"c78ad14529b3a87f27af21a3f82a9cc6ef78166cea9bd7378606c58695528641"}
Sep 17 18:48:26.904280 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:26.904245    2553 generic.go:356] "Generic (PLEG): container finished" podID="cfd1dd23-96d5-42d8-911a-72d7d3959901" containerID="66b35d744b951569d6a4c2fcb3c53357cd06d978485a0044d2d4dbebcbc0c788" exitCode=0
Sep 17 18:48:26.904280 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:26.904286    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zl8dq" event={"ID":"cfd1dd23-96d5-42d8-911a-72d7d3959901","Type":"ContainerDied","Data":"66b35d744b951569d6a4c2fcb3c53357cd06d978485a0044d2d4dbebcbc0c788"}
Sep 17 18:48:26.905509 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:26.905489    2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:48:29.897514 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:29.897474    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5httc"]
Sep 17 18:48:29.901780 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:29.901746    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5httc"
Sep 17 18:48:29.904282 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:29.904069    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-predictor-serving-cert\""
Sep 17 18:48:29.904282 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:29.904122    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-kube-rbac-proxy-sar-config\""
Sep 17 18:48:29.912832 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:29.912799    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5httc"]
Sep 17 18:48:30.004573 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:30.004536    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f9700e5-a902-4e1e-8d61-16c90f84cc8d-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-5httc\" (UID: \"8f9700e5-a902-4e1e-8d61-16c90f84cc8d\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5httc"
Sep 17 18:48:30.004573 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:30.004582    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f9700e5-a902-4e1e-8d61-16c90f84cc8d-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-5httc\" (UID: \"8f9700e5-a902-4e1e-8d61-16c90f84cc8d\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5httc"
Sep 17 18:48:30.004832 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:30.004607    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52bsd\" (UniqueName: \"kubernetes.io/projected/8f9700e5-a902-4e1e-8d61-16c90f84cc8d-kube-api-access-52bsd\") pod \"isvc-pmml-predictor-8674795f59-5httc\" (UID: \"8f9700e5-a902-4e1e-8d61-16c90f84cc8d\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5httc"
Sep 17 18:48:30.004832 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:30.004724    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f9700e5-a902-4e1e-8d61-16c90f84cc8d-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-5httc\" (UID: \"8f9700e5-a902-4e1e-8d61-16c90f84cc8d\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5httc"
Sep 17 18:48:30.004832 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:30.004775    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f9700e5-a902-4e1e-8d61-16c90f84cc8d-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-5httc\" (UID: \"8f9700e5-a902-4e1e-8d61-16c90f84cc8d\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5httc"
Sep 17 18:48:30.106016 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:30.105974    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f9700e5-a902-4e1e-8d61-16c90f84cc8d-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-5httc\" (UID: \"8f9700e5-a902-4e1e-8d61-16c90f84cc8d\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5httc"
Sep 17 18:48:30.106216 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:30.106035    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f9700e5-a902-4e1e-8d61-16c90f84cc8d-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-5httc\" (UID: \"8f9700e5-a902-4e1e-8d61-16c90f84cc8d\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5httc"
Sep 17 18:48:30.106216 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:30.106116    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52bsd\" (UniqueName: \"kubernetes.io/projected/8f9700e5-a902-4e1e-8d61-16c90f84cc8d-kube-api-access-52bsd\") pod \"isvc-pmml-predictor-8674795f59-5httc\" (UID: \"8f9700e5-a902-4e1e-8d61-16c90f84cc8d\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5httc"
Sep 17 18:48:30.106216 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:30.106186    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f9700e5-a902-4e1e-8d61-16c90f84cc8d-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-5httc\" (UID: \"8f9700e5-a902-4e1e-8d61-16c90f84cc8d\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5httc"
Sep 17 18:48:30.106393 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:30.106225    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f9700e5-a902-4e1e-8d61-16c90f84cc8d-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-5httc\" (UID: \"8f9700e5-a902-4e1e-8d61-16c90f84cc8d\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5httc"
Sep 17 18:48:30.106767 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:30.106714    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f9700e5-a902-4e1e-8d61-16c90f84cc8d-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-5httc\" (UID: \"8f9700e5-a902-4e1e-8d61-16c90f84cc8d\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5httc"
Sep 17 18:48:30.106900 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:30.106836    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f9700e5-a902-4e1e-8d61-16c90f84cc8d-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-5httc\" (UID: \"8f9700e5-a902-4e1e-8d61-16c90f84cc8d\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5httc"
Sep 17 18:48:30.106966 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:30.106914    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f9700e5-a902-4e1e-8d61-16c90f84cc8d-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-5httc\" (UID: \"8f9700e5-a902-4e1e-8d61-16c90f84cc8d\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5httc"
Sep 17 18:48:30.109265 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:30.109243    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f9700e5-a902-4e1e-8d61-16c90f84cc8d-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-5httc\" (UID: \"8f9700e5-a902-4e1e-8d61-16c90f84cc8d\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5httc"
Sep 17 18:48:30.115263 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:30.115235    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52bsd\" (UniqueName: \"kubernetes.io/projected/8f9700e5-a902-4e1e-8d61-16c90f84cc8d-kube-api-access-52bsd\") pod \"isvc-pmml-predictor-8674795f59-5httc\" (UID: \"8f9700e5-a902-4e1e-8d61-16c90f84cc8d\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5httc"
Sep 17 18:48:30.216311 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:30.216220    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5httc"
Sep 17 18:48:34.596902 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:34.596869    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5httc"]
Sep 17 18:48:34.599048 ip-10-0-137-95 kubenswrapper[2553]: W0917 18:48:34.599021    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f9700e5_a902_4e1e_8d61_16c90f84cc8d.slice/crio-eb952dca1471d5fa5ad98a3f1bb9494f890fd11ec9952b913b1bb264f4469e44 WatchSource:0}: Error finding container eb952dca1471d5fa5ad98a3f1bb9494f890fd11ec9952b913b1bb264f4469e44: Status 404 returned error can't find the container with id eb952dca1471d5fa5ad98a3f1bb9494f890fd11ec9952b913b1bb264f4469e44
Sep 17 18:48:34.957237 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:34.957193    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zl8dq" event={"ID":"cfd1dd23-96d5-42d8-911a-72d7d3959901","Type":"ContainerStarted","Data":"e7ca45841e0a00fbfa228331896b12f7f1c33d458133d7c34f29062c21fa9524"}
Sep 17 18:48:34.957455 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:34.957287    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zl8dq" event={"ID":"cfd1dd23-96d5-42d8-911a-72d7d3959901","Type":"ContainerStarted","Data":"08478890cfe56357483e272df9f1bdd801a73be9edd40067b1e28e0b16ead73e"}
Sep 17 18:48:34.957818 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:34.957558    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zl8dq"
Sep 17 18:48:34.957818 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:34.957646    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zl8dq"
Sep 17 18:48:34.958922 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:34.958897    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5httc" event={"ID":"8f9700e5-a902-4e1e-8d61-16c90f84cc8d","Type":"ContainerStarted","Data":"3cc0f40c67fa130b25ce04f03794d004f074f72503c27671ba306a6a7bd2e075"}
Sep 17 18:48:34.959154 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:34.958944    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5httc" event={"ID":"8f9700e5-a902-4e1e-8d61-16c90f84cc8d","Type":"ContainerStarted","Data":"eb952dca1471d5fa5ad98a3f1bb9494f890fd11ec9952b913b1bb264f4469e44"}
Sep 17 18:48:34.959486 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:34.959461    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zl8dq" podUID="cfd1dd23-96d5-42d8-911a-72d7d3959901" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 17 18:48:34.976449 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:34.976391    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zl8dq" podStartSLOduration=1.808707946 podStartE2EDuration="12.976371s" totalImagesPullingTime="7.510550537s" totalInitContainerRuntime="3.657112517s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:48:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:48:34.975003603 +0000 UTC m=+3109.671094287" watchObservedRunningTime="2026-09-17 18:48:34.976371 +0000 UTC m=+3109.672461683"
Sep 17 18:48:35.964077 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:35.964033    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zl8dq" podUID="cfd1dd23-96d5-42d8-911a-72d7d3959901" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 17 18:48:38.977399 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:38.977367    2553 generic.go:356] "Generic (PLEG): container finished" podID="8f9700e5-a902-4e1e-8d61-16c90f84cc8d" containerID="3cc0f40c67fa130b25ce04f03794d004f074f72503c27671ba306a6a7bd2e075" exitCode=0
Sep 17 18:48:38.977899 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:38.977450    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5httc" event={"ID":"8f9700e5-a902-4e1e-8d61-16c90f84cc8d","Type":"ContainerDied","Data":"3cc0f40c67fa130b25ce04f03794d004f074f72503c27671ba306a6a7bd2e075"}
Sep 17 18:48:39.983547 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:39.983510    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5httc" event={"ID":"8f9700e5-a902-4e1e-8d61-16c90f84cc8d","Type":"ContainerStarted","Data":"a58568fb8509b4e24903d512d221fb72ee23ec418d70798910c6418ddb5f083b"}
Sep 17 18:48:39.984007 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:39.983573    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5httc" event={"ID":"8f9700e5-a902-4e1e-8d61-16c90f84cc8d","Type":"ContainerStarted","Data":"a3da48ad01d4c6eaf71dd4b3d4aefaa6c2f77de64e34e67b49f7ee3ade38df50"}
Sep 17 18:48:39.984007 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:39.983797    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5httc"
Sep 17 18:48:40.002711 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:40.002650    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5httc" podStartSLOduration=7.13326825 podStartE2EDuration="11.002611631s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.869343381s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:48:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:48:40.00187284 +0000 UTC m=+3114.697963525" watchObservedRunningTime="2026-09-17 18:48:40.002611631 +0000 UTC m=+3114.698702314"
Sep 17 18:48:40.969387 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:40.969356    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zl8dq"
Sep 17 18:48:40.969866 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:40.969837    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zl8dq" podUID="cfd1dd23-96d5-42d8-911a-72d7d3959901" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 17 18:48:40.989187 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:40.989157    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5httc"
Sep 17 18:48:40.990484 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:40.990459    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5httc" podUID="8f9700e5-a902-4e1e-8d61-16c90f84cc8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 17 18:48:41.993432 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:41.993395    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5httc" podUID="8f9700e5-a902-4e1e-8d61-16c90f84cc8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 17 18:48:46.998332 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:46.998299    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5httc"
Sep 17 18:48:46.998847 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:46.998817    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5httc" podUID="8f9700e5-a902-4e1e-8d61-16c90f84cc8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 17 18:48:50.969917 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:50.969874    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zl8dq" podUID="cfd1dd23-96d5-42d8-911a-72d7d3959901" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 17 18:48:56.999775 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:48:56.999733    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5httc" podUID="8f9700e5-a902-4e1e-8d61-16c90f84cc8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 17 18:49:00.970475 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:49:00.970376    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zl8dq" podUID="cfd1dd23-96d5-42d8-911a-72d7d3959901" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 17 18:49:06.998919 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:49:06.998877    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5httc" podUID="8f9700e5-a902-4e1e-8d61-16c90f84cc8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 17 18:49:10.970282 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:49:10.970242    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zl8dq" podUID="cfd1dd23-96d5-42d8-911a-72d7d3959901" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 17 18:49:16.998901 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:49:16.998856    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5httc" podUID="8f9700e5-a902-4e1e-8d61-16c90f84cc8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 17 18:49:20.969924 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:49:20.969878    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zl8dq" podUID="cfd1dd23-96d5-42d8-911a-72d7d3959901" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 17 18:49:26.999579 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:49:26.999538    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5httc" podUID="8f9700e5-a902-4e1e-8d61-16c90f84cc8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 17 18:49:30.970271 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:49:30.970228    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zl8dq" podUID="cfd1dd23-96d5-42d8-911a-72d7d3959901" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 17 18:49:36.999023 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:49:36.998980    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5httc" podUID="8f9700e5-a902-4e1e-8d61-16c90f84cc8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 17 18:49:40.970407 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:49:40.970361    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zl8dq" podUID="cfd1dd23-96d5-42d8-911a-72d7d3959901" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 17 18:49:46.999454 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:49:46.999410    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5httc" podUID="8f9700e5-a902-4e1e-8d61-16c90f84cc8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 17 18:49:50.371805 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:49:50.371762    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5httc" podUID="8f9700e5-a902-4e1e-8d61-16c90f84cc8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 17 18:49:50.970338 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:49:50.970297    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zl8dq" podUID="cfd1dd23-96d5-42d8-911a-72d7d3959901" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 17 18:50:00.372664 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:00.372589    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5httc" podUID="8f9700e5-a902-4e1e-8d61-16c90f84cc8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 17 18:50:00.970068 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:00.970022    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zl8dq" podUID="cfd1dd23-96d5-42d8-911a-72d7d3959901" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 17 18:50:07.383128 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:07.383097    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zl8dq"
Sep 17 18:50:10.372457 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:10.372423    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5httc"
Sep 17 18:50:13.900838 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:13.900803    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zl8dq"]
Sep 17 18:50:13.901338 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:13.901157    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zl8dq" podUID="cfd1dd23-96d5-42d8-911a-72d7d3959901" containerName="kserve-container" containerID="cri-o://08478890cfe56357483e272df9f1bdd801a73be9edd40067b1e28e0b16ead73e" gracePeriod=30
Sep 17 18:50:13.901338 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:13.901161    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zl8dq" podUID="cfd1dd23-96d5-42d8-911a-72d7d3959901" containerName="kube-rbac-proxy" containerID="cri-o://e7ca45841e0a00fbfa228331896b12f7f1c33d458133d7c34f29062c21fa9524" gracePeriod=30
Sep 17 18:50:14.422433 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:14.422401    2553 generic.go:356] "Generic (PLEG): container finished" podID="cfd1dd23-96d5-42d8-911a-72d7d3959901" containerID="e7ca45841e0a00fbfa228331896b12f7f1c33d458133d7c34f29062c21fa9524" exitCode=2
Sep 17 18:50:14.422604 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:14.422473    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zl8dq" event={"ID":"cfd1dd23-96d5-42d8-911a-72d7d3959901","Type":"ContainerDied","Data":"e7ca45841e0a00fbfa228331896b12f7f1c33d458133d7c34f29062c21fa9524"}
Sep 17 18:50:15.965041 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:15.964998    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zl8dq" podUID="cfd1dd23-96d5-42d8-911a-72d7d3959901" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.57:8643/healthz\": dial tcp 10.134.0.57:8643: connect: connection refused"
Sep 17 18:50:17.375298 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:17.375262    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zl8dq" podUID="cfd1dd23-96d5-42d8-911a-72d7d3959901" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 17 18:50:18.250828 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:18.250799    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zl8dq"
Sep 17 18:50:18.359260 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:18.359165    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cfd1dd23-96d5-42d8-911a-72d7d3959901-cabundle-cert\") pod \"cfd1dd23-96d5-42d8-911a-72d7d3959901\" (UID: \"cfd1dd23-96d5-42d8-911a-72d7d3959901\") "
Sep 17 18:50:18.359260 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:18.359214    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gzgnv\" (UniqueName: \"kubernetes.io/projected/cfd1dd23-96d5-42d8-911a-72d7d3959901-kube-api-access-gzgnv\") pod \"cfd1dd23-96d5-42d8-911a-72d7d3959901\" (UID: \"cfd1dd23-96d5-42d8-911a-72d7d3959901\") "
Sep 17 18:50:18.359260 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:18.359244    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cfd1dd23-96d5-42d8-911a-72d7d3959901-kube-rbac-proxy-sar-config\") pod \"cfd1dd23-96d5-42d8-911a-72d7d3959901\" (UID: \"cfd1dd23-96d5-42d8-911a-72d7d3959901\") "
Sep 17 18:50:18.359505 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:18.359299    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cfd1dd23-96d5-42d8-911a-72d7d3959901-proxy-tls\") pod \"cfd1dd23-96d5-42d8-911a-72d7d3959901\" (UID: \"cfd1dd23-96d5-42d8-911a-72d7d3959901\") "
Sep 17 18:50:18.359505 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:18.359325    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cfd1dd23-96d5-42d8-911a-72d7d3959901-kserve-provision-location\") pod \"cfd1dd23-96d5-42d8-911a-72d7d3959901\" (UID: \"cfd1dd23-96d5-42d8-911a-72d7d3959901\") "
Sep 17 18:50:18.359684 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:18.359654    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfd1dd23-96d5-42d8-911a-72d7d3959901-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cfd1dd23-96d5-42d8-911a-72d7d3959901" (UID: "cfd1dd23-96d5-42d8-911a-72d7d3959901"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:50:18.359841 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:18.359809    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfd1dd23-96d5-42d8-911a-72d7d3959901-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cfd1dd23-96d5-42d8-911a-72d7d3959901" (UID: "cfd1dd23-96d5-42d8-911a-72d7d3959901"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:50:18.359841 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:18.359817    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfd1dd23-96d5-42d8-911a-72d7d3959901-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cfd1dd23-96d5-42d8-911a-72d7d3959901" (UID: "cfd1dd23-96d5-42d8-911a-72d7d3959901"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:50:18.361548 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:18.361526    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfd1dd23-96d5-42d8-911a-72d7d3959901-kube-api-access-gzgnv" (OuterVolumeSpecName: "kube-api-access-gzgnv") pod "cfd1dd23-96d5-42d8-911a-72d7d3959901" (UID: "cfd1dd23-96d5-42d8-911a-72d7d3959901"). InnerVolumeSpecName "kube-api-access-gzgnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:50:18.361633 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:18.361543    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfd1dd23-96d5-42d8-911a-72d7d3959901-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cfd1dd23-96d5-42d8-911a-72d7d3959901" (UID: "cfd1dd23-96d5-42d8-911a-72d7d3959901"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:50:18.443114 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:18.443076    2553 generic.go:356] "Generic (PLEG): container finished" podID="cfd1dd23-96d5-42d8-911a-72d7d3959901" containerID="08478890cfe56357483e272df9f1bdd801a73be9edd40067b1e28e0b16ead73e" exitCode=0
Sep 17 18:50:18.443560 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:18.443130    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zl8dq"
Sep 17 18:50:18.443560 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:18.443159    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zl8dq" event={"ID":"cfd1dd23-96d5-42d8-911a-72d7d3959901","Type":"ContainerDied","Data":"08478890cfe56357483e272df9f1bdd801a73be9edd40067b1e28e0b16ead73e"}
Sep 17 18:50:18.443560 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:18.443218    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zl8dq" event={"ID":"cfd1dd23-96d5-42d8-911a-72d7d3959901","Type":"ContainerDied","Data":"c78ad14529b3a87f27af21a3f82a9cc6ef78166cea9bd7378606c58695528641"}
Sep 17 18:50:18.443560 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:18.443244    2553 scope.go:117] "RemoveContainer" containerID="e7ca45841e0a00fbfa228331896b12f7f1c33d458133d7c34f29062c21fa9524"
Sep 17 18:50:18.452208 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:18.452191    2553 scope.go:117] "RemoveContainer" containerID="08478890cfe56357483e272df9f1bdd801a73be9edd40067b1e28e0b16ead73e"
Sep 17 18:50:18.459961 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:18.459943    2553 scope.go:117] "RemoveContainer" containerID="66b35d744b951569d6a4c2fcb3c53357cd06d978485a0044d2d4dbebcbc0c788"
Sep 17 18:50:18.460334 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:18.460311    2553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cfd1dd23-96d5-42d8-911a-72d7d3959901-proxy-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:50:18.460414 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:18.460337    2553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cfd1dd23-96d5-42d8-911a-72d7d3959901-kserve-provision-location\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:50:18.460414 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:18.460351    2553 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cfd1dd23-96d5-42d8-911a-72d7d3959901-cabundle-cert\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:50:18.460414 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:18.460365    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzgnv\" (UniqueName: \"kubernetes.io/projected/cfd1dd23-96d5-42d8-911a-72d7d3959901-kube-api-access-gzgnv\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:50:18.460414 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:18.460375    2553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cfd1dd23-96d5-42d8-911a-72d7d3959901-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:50:18.466555 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:18.466533    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zl8dq"]
Sep 17 18:50:18.467338 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:18.467320    2553 scope.go:117] "RemoveContainer" containerID="e7ca45841e0a00fbfa228331896b12f7f1c33d458133d7c34f29062c21fa9524"
Sep 17 18:50:18.467779 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:50:18.467755    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7ca45841e0a00fbfa228331896b12f7f1c33d458133d7c34f29062c21fa9524\": container with ID starting with e7ca45841e0a00fbfa228331896b12f7f1c33d458133d7c34f29062c21fa9524 not found: ID does not exist" containerID="e7ca45841e0a00fbfa228331896b12f7f1c33d458133d7c34f29062c21fa9524"
Sep 17 18:50:18.467849 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:18.467794    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7ca45841e0a00fbfa228331896b12f7f1c33d458133d7c34f29062c21fa9524"} err="failed to get container status \"e7ca45841e0a00fbfa228331896b12f7f1c33d458133d7c34f29062c21fa9524\": rpc error: code = NotFound desc = could not find container \"e7ca45841e0a00fbfa228331896b12f7f1c33d458133d7c34f29062c21fa9524\": container with ID starting with e7ca45841e0a00fbfa228331896b12f7f1c33d458133d7c34f29062c21fa9524 not found: ID does not exist"
Sep 17 18:50:18.467849 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:18.467820    2553 scope.go:117] "RemoveContainer" containerID="08478890cfe56357483e272df9f1bdd801a73be9edd40067b1e28e0b16ead73e"
Sep 17 18:50:18.468128 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:50:18.468107    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08478890cfe56357483e272df9f1bdd801a73be9edd40067b1e28e0b16ead73e\": container with ID starting with 08478890cfe56357483e272df9f1bdd801a73be9edd40067b1e28e0b16ead73e not found: ID does not exist" containerID="08478890cfe56357483e272df9f1bdd801a73be9edd40067b1e28e0b16ead73e"
Sep 17 18:50:18.468200 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:18.468134    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08478890cfe56357483e272df9f1bdd801a73be9edd40067b1e28e0b16ead73e"} err="failed to get container status \"08478890cfe56357483e272df9f1bdd801a73be9edd40067b1e28e0b16ead73e\": rpc error: code = NotFound desc = could not find container \"08478890cfe56357483e272df9f1bdd801a73be9edd40067b1e28e0b16ead73e\": container with ID starting with 08478890cfe56357483e272df9f1bdd801a73be9edd40067b1e28e0b16ead73e not found: ID does not exist"
Sep 17 18:50:18.468200 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:18.468155    2553 scope.go:117] "RemoveContainer" containerID="66b35d744b951569d6a4c2fcb3c53357cd06d978485a0044d2d4dbebcbc0c788"
Sep 17 18:50:18.468635 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:50:18.468579    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66b35d744b951569d6a4c2fcb3c53357cd06d978485a0044d2d4dbebcbc0c788\": container with ID starting with 66b35d744b951569d6a4c2fcb3c53357cd06d978485a0044d2d4dbebcbc0c788 not found: ID does not exist" containerID="66b35d744b951569d6a4c2fcb3c53357cd06d978485a0044d2d4dbebcbc0c788"
Sep 17 18:50:18.468724 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:18.468645    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66b35d744b951569d6a4c2fcb3c53357cd06d978485a0044d2d4dbebcbc0c788"} err="failed to get container status \"66b35d744b951569d6a4c2fcb3c53357cd06d978485a0044d2d4dbebcbc0c788\": rpc error: code = NotFound desc = could not find container \"66b35d744b951569d6a4c2fcb3c53357cd06d978485a0044d2d4dbebcbc0c788\": container with ID starting with 66b35d744b951569d6a4c2fcb3c53357cd06d978485a0044d2d4dbebcbc0c788 not found: ID does not exist"
Sep 17 18:50:18.469877 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:18.469859    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zl8dq"]
Sep 17 18:50:19.079503 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:19.079467    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n"]
Sep 17 18:50:19.079852 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:19.079839    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfd1dd23-96d5-42d8-911a-72d7d3959901" containerName="kube-rbac-proxy"
Sep 17 18:50:19.079917 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:19.079854    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfd1dd23-96d5-42d8-911a-72d7d3959901" containerName="kube-rbac-proxy"
Sep 17 18:50:19.079917 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:19.079862    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfd1dd23-96d5-42d8-911a-72d7d3959901" containerName="storage-initializer"
Sep 17 18:50:19.079917 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:19.079869    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfd1dd23-96d5-42d8-911a-72d7d3959901" containerName="storage-initializer"
Sep 17 18:50:19.079917 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:19.079882    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfd1dd23-96d5-42d8-911a-72d7d3959901" containerName="kserve-container"
Sep 17 18:50:19.079917 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:19.079888    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfd1dd23-96d5-42d8-911a-72d7d3959901" containerName="kserve-container"
Sep 17 18:50:19.080075 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:19.079936    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfd1dd23-96d5-42d8-911a-72d7d3959901" containerName="kserve-container"
Sep 17 18:50:19.080075 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:19.079947    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfd1dd23-96d5-42d8-911a-72d7d3959901" containerName="kube-rbac-proxy"
Sep 17 18:50:19.082009 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:19.081992    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n"
Sep 17 18:50:19.084708 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:19.084689    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-predictor-serving-cert\""
Sep 17 18:50:19.084708 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:19.084698    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 17 18:50:19.092904 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:19.092880    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n"]
Sep 17 18:50:19.165844 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:19.165806    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34432538-1018-4fec-9991-4e4fa4b8a3ab-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n\" (UID: \"34432538-1018-4fec-9991-4e4fa4b8a3ab\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n"
Sep 17 18:50:19.166023 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:19.165918    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34432538-1018-4fec-9991-4e4fa4b8a3ab-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n\" (UID: \"34432538-1018-4fec-9991-4e4fa4b8a3ab\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n"
Sep 17 18:50:19.166023 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:19.165958    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww2qv\" (UniqueName: \"kubernetes.io/projected/34432538-1018-4fec-9991-4e4fa4b8a3ab-kube-api-access-ww2qv\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n\" (UID: \"34432538-1018-4fec-9991-4e4fa4b8a3ab\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n"
Sep 17 18:50:19.166023 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:19.165979    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34432538-1018-4fec-9991-4e4fa4b8a3ab-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n\" (UID: \"34432538-1018-4fec-9991-4e4fa4b8a3ab\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n"
Sep 17 18:50:19.166023 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:19.165997    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34432538-1018-4fec-9991-4e4fa4b8a3ab-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n\" (UID: \"34432538-1018-4fec-9991-4e4fa4b8a3ab\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n"
Sep 17 18:50:19.267388 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:19.267349    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34432538-1018-4fec-9991-4e4fa4b8a3ab-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n\" (UID: \"34432538-1018-4fec-9991-4e4fa4b8a3ab\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n"
Sep 17 18:50:19.267388 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:19.267394    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww2qv\" (UniqueName: \"kubernetes.io/projected/34432538-1018-4fec-9991-4e4fa4b8a3ab-kube-api-access-ww2qv\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n\" (UID: \"34432538-1018-4fec-9991-4e4fa4b8a3ab\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n"
Sep 17 18:50:19.267639 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:19.267418    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34432538-1018-4fec-9991-4e4fa4b8a3ab-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n\" (UID: \"34432538-1018-4fec-9991-4e4fa4b8a3ab\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n"
Sep 17 18:50:19.267639 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:19.267442    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34432538-1018-4fec-9991-4e4fa4b8a3ab-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n\" (UID: \"34432538-1018-4fec-9991-4e4fa4b8a3ab\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n"
Sep 17 18:50:19.267639 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:19.267479    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34432538-1018-4fec-9991-4e4fa4b8a3ab-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n\" (UID: \"34432538-1018-4fec-9991-4e4fa4b8a3ab\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n"
Sep 17 18:50:19.267639 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:50:19.267530    2553 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-serving-cert: secret "isvc-pmml-v2-kserve-predictor-serving-cert" not found
Sep 17 18:50:19.267837 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:50:19.267648    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34432538-1018-4fec-9991-4e4fa4b8a3ab-proxy-tls podName:34432538-1018-4fec-9991-4e4fa4b8a3ab nodeName:}" failed. No retries permitted until 2026-09-17 18:50:19.767599697 +0000 UTC m=+3214.463690373 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/34432538-1018-4fec-9991-4e4fa4b8a3ab-proxy-tls") pod "isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n" (UID: "34432538-1018-4fec-9991-4e4fa4b8a3ab") : secret "isvc-pmml-v2-kserve-predictor-serving-cert" not found
Sep 17 18:50:19.267837 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:19.267775    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34432538-1018-4fec-9991-4e4fa4b8a3ab-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n\" (UID: \"34432538-1018-4fec-9991-4e4fa4b8a3ab\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n"
Sep 17 18:50:19.268178 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:19.268160    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34432538-1018-4fec-9991-4e4fa4b8a3ab-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n\" (UID: \"34432538-1018-4fec-9991-4e4fa4b8a3ab\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n"
Sep 17 18:50:19.268216 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:19.268203    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34432538-1018-4fec-9991-4e4fa4b8a3ab-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n\" (UID: \"34432538-1018-4fec-9991-4e4fa4b8a3ab\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n"
Sep 17 18:50:19.278031 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:19.278000    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww2qv\" (UniqueName: \"kubernetes.io/projected/34432538-1018-4fec-9991-4e4fa4b8a3ab-kube-api-access-ww2qv\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n\" (UID: \"34432538-1018-4fec-9991-4e4fa4b8a3ab\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n"
Sep 17 18:50:19.377596 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:19.377513    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfd1dd23-96d5-42d8-911a-72d7d3959901" path="/var/lib/kubelet/pods/cfd1dd23-96d5-42d8-911a-72d7d3959901/volumes"
Sep 17 18:50:19.772751 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:19.772708    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34432538-1018-4fec-9991-4e4fa4b8a3ab-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n\" (UID: \"34432538-1018-4fec-9991-4e4fa4b8a3ab\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n"
Sep 17 18:50:19.775191 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:19.775160    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34432538-1018-4fec-9991-4e4fa4b8a3ab-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n\" (UID: \"34432538-1018-4fec-9991-4e4fa4b8a3ab\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n"
Sep 17 18:50:19.992768 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:19.992728    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n"
Sep 17 18:50:20.265126 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:20.265066    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n"]
Sep 17 18:50:20.267415 ip-10-0-137-95 kubenswrapper[2553]: W0917 18:50:20.267381    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34432538_1018_4fec_9991_4e4fa4b8a3ab.slice/crio-92765ba7c31b1f3529ac5745441c60445cd2fb40922fd79c19066d7eac1c7b6f WatchSource:0}: Error finding container 92765ba7c31b1f3529ac5745441c60445cd2fb40922fd79c19066d7eac1c7b6f: Status 404 returned error can't find the container with id 92765ba7c31b1f3529ac5745441c60445cd2fb40922fd79c19066d7eac1c7b6f
Sep 17 18:50:20.455095 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:20.455056    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n" event={"ID":"34432538-1018-4fec-9991-4e4fa4b8a3ab","Type":"ContainerStarted","Data":"606a67dd6159b580253887a476d9ebf62f96475f23cb69a89d2e85a02fcf07ac"}
Sep 17 18:50:20.455288 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:20.455121    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n" event={"ID":"34432538-1018-4fec-9991-4e4fa4b8a3ab","Type":"ContainerStarted","Data":"92765ba7c31b1f3529ac5745441c60445cd2fb40922fd79c19066d7eac1c7b6f"}
Sep 17 18:50:21.553299 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:21.553258    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5httc"]
Sep 17 18:50:21.553798 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:21.553502    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5httc" podUID="8f9700e5-a902-4e1e-8d61-16c90f84cc8d" containerName="kserve-container" containerID="cri-o://a3da48ad01d4c6eaf71dd4b3d4aefaa6c2f77de64e34e67b49f7ee3ade38df50" gracePeriod=30
Sep 17 18:50:21.553798 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:21.553551    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5httc" podUID="8f9700e5-a902-4e1e-8d61-16c90f84cc8d" containerName="kube-rbac-proxy" containerID="cri-o://a58568fb8509b4e24903d512d221fb72ee23ec418d70798910c6418ddb5f083b" gracePeriod=30
Sep 17 18:50:21.994361 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:21.994316    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5httc" podUID="8f9700e5-a902-4e1e-8d61-16c90f84cc8d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.58:8643/healthz\": dial tcp 10.134.0.58:8643: connect: connection refused"
Sep 17 18:50:22.466458 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:22.466421    2553 generic.go:356] "Generic (PLEG): container finished" podID="8f9700e5-a902-4e1e-8d61-16c90f84cc8d" containerID="a58568fb8509b4e24903d512d221fb72ee23ec418d70798910c6418ddb5f083b" exitCode=2
Sep 17 18:50:22.466656 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:22.466493    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5httc" event={"ID":"8f9700e5-a902-4e1e-8d61-16c90f84cc8d","Type":"ContainerDied","Data":"a58568fb8509b4e24903d512d221fb72ee23ec418d70798910c6418ddb5f083b"}
Sep 17 18:50:24.476771 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:24.476735    2553 generic.go:356] "Generic (PLEG): container finished" podID="34432538-1018-4fec-9991-4e4fa4b8a3ab" containerID="606a67dd6159b580253887a476d9ebf62f96475f23cb69a89d2e85a02fcf07ac" exitCode=0
Sep 17 18:50:24.477238 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:24.476812    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n" event={"ID":"34432538-1018-4fec-9991-4e4fa4b8a3ab","Type":"ContainerDied","Data":"606a67dd6159b580253887a476d9ebf62f96475f23cb69a89d2e85a02fcf07ac"}
Sep 17 18:50:25.483071 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:25.483038    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n" event={"ID":"34432538-1018-4fec-9991-4e4fa4b8a3ab","Type":"ContainerStarted","Data":"94ef2efbd9be23837688c4c44442d13ff8285576026fe09a5b8473589c39f407"}
Sep 17 18:50:25.483454 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:25.483099    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n" event={"ID":"34432538-1018-4fec-9991-4e4fa4b8a3ab","Type":"ContainerStarted","Data":"8b28cffe18563c67dd99e21f0a0fe6c7f602501544e604d97678c519ad1676d3"}
Sep 17 18:50:25.483454 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:25.483433    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n"
Sep 17 18:50:25.483532 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:25.483495    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n"
Sep 17 18:50:25.484889 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:25.484860    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n" podUID="34432538-1018-4fec-9991-4e4fa4b8a3ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 17 18:50:25.502800 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:25.502752    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n" podStartSLOduration=2.585374029 podStartE2EDuration="6.502738104s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.917364075s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:50:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:50:25.501849429 +0000 UTC m=+3220.197940113" watchObservedRunningTime="2026-09-17 18:50:25.502738104 +0000 UTC m=+3220.198828787"
Sep 17 18:50:26.002436 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:26.002404    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5httc"
Sep 17 18:50:26.127042 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:26.126965    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f9700e5-a902-4e1e-8d61-16c90f84cc8d-kube-rbac-proxy-sar-config\") pod \"8f9700e5-a902-4e1e-8d61-16c90f84cc8d\" (UID: \"8f9700e5-a902-4e1e-8d61-16c90f84cc8d\") "
Sep 17 18:50:26.127219 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:26.127059    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f9700e5-a902-4e1e-8d61-16c90f84cc8d-kserve-provision-location\") pod \"8f9700e5-a902-4e1e-8d61-16c90f84cc8d\" (UID: \"8f9700e5-a902-4e1e-8d61-16c90f84cc8d\") "
Sep 17 18:50:26.127219 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:26.127083    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-52bsd\" (UniqueName: \"kubernetes.io/projected/8f9700e5-a902-4e1e-8d61-16c90f84cc8d-kube-api-access-52bsd\") pod \"8f9700e5-a902-4e1e-8d61-16c90f84cc8d\" (UID: \"8f9700e5-a902-4e1e-8d61-16c90f84cc8d\") "
Sep 17 18:50:26.127219 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:26.127127    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f9700e5-a902-4e1e-8d61-16c90f84cc8d-proxy-tls\") pod \"8f9700e5-a902-4e1e-8d61-16c90f84cc8d\" (UID: \"8f9700e5-a902-4e1e-8d61-16c90f84cc8d\") "
Sep 17 18:50:26.127219 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:26.127154    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f9700e5-a902-4e1e-8d61-16c90f84cc8d-cabundle-cert\") pod \"8f9700e5-a902-4e1e-8d61-16c90f84cc8d\" (UID: \"8f9700e5-a902-4e1e-8d61-16c90f84cc8d\") "
Sep 17 18:50:26.127433 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:26.127410    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f9700e5-a902-4e1e-8d61-16c90f84cc8d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8f9700e5-a902-4e1e-8d61-16c90f84cc8d" (UID: "8f9700e5-a902-4e1e-8d61-16c90f84cc8d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:50:26.127496 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:26.127409    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f9700e5-a902-4e1e-8d61-16c90f84cc8d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8f9700e5-a902-4e1e-8d61-16c90f84cc8d" (UID: "8f9700e5-a902-4e1e-8d61-16c90f84cc8d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:50:26.127540 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:26.127523    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f9700e5-a902-4e1e-8d61-16c90f84cc8d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8f9700e5-a902-4e1e-8d61-16c90f84cc8d" (UID: "8f9700e5-a902-4e1e-8d61-16c90f84cc8d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:50:26.129250 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:26.129226    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f9700e5-a902-4e1e-8d61-16c90f84cc8d-kube-api-access-52bsd" (OuterVolumeSpecName: "kube-api-access-52bsd") pod "8f9700e5-a902-4e1e-8d61-16c90f84cc8d" (UID: "8f9700e5-a902-4e1e-8d61-16c90f84cc8d"). InnerVolumeSpecName "kube-api-access-52bsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:50:26.129349 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:26.129319    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f9700e5-a902-4e1e-8d61-16c90f84cc8d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8f9700e5-a902-4e1e-8d61-16c90f84cc8d" (UID: "8f9700e5-a902-4e1e-8d61-16c90f84cc8d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:50:26.228237 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:26.228191    2553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f9700e5-a902-4e1e-8d61-16c90f84cc8d-kserve-provision-location\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:50:26.228237 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:26.228226    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52bsd\" (UniqueName: \"kubernetes.io/projected/8f9700e5-a902-4e1e-8d61-16c90f84cc8d-kube-api-access-52bsd\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:50:26.228237 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:26.228236    2553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f9700e5-a902-4e1e-8d61-16c90f84cc8d-proxy-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:50:26.228237 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:26.228250    2553 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f9700e5-a902-4e1e-8d61-16c90f84cc8d-cabundle-cert\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:50:26.228515 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:26.228258    2553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f9700e5-a902-4e1e-8d61-16c90f84cc8d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:50:26.490447 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:26.490408    2553 generic.go:356] "Generic (PLEG): container finished" podID="8f9700e5-a902-4e1e-8d61-16c90f84cc8d" containerID="a3da48ad01d4c6eaf71dd4b3d4aefaa6c2f77de64e34e67b49f7ee3ade38df50" exitCode=0
Sep 17 18:50:26.490865 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:26.490473    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5httc"
Sep 17 18:50:26.490865 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:26.490494    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5httc" event={"ID":"8f9700e5-a902-4e1e-8d61-16c90f84cc8d","Type":"ContainerDied","Data":"a3da48ad01d4c6eaf71dd4b3d4aefaa6c2f77de64e34e67b49f7ee3ade38df50"}
Sep 17 18:50:26.490865 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:26.490556    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5httc" event={"ID":"8f9700e5-a902-4e1e-8d61-16c90f84cc8d","Type":"ContainerDied","Data":"eb952dca1471d5fa5ad98a3f1bb9494f890fd11ec9952b913b1bb264f4469e44"}
Sep 17 18:50:26.490865 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:26.490582    2553 scope.go:117] "RemoveContainer" containerID="a58568fb8509b4e24903d512d221fb72ee23ec418d70798910c6418ddb5f083b"
Sep 17 18:50:26.491161 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:26.491135    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n" podUID="34432538-1018-4fec-9991-4e4fa4b8a3ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 17 18:50:26.499441 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:26.499422    2553 scope.go:117] "RemoveContainer" containerID="a3da48ad01d4c6eaf71dd4b3d4aefaa6c2f77de64e34e67b49f7ee3ade38df50"
Sep 17 18:50:26.507151 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:26.507134    2553 scope.go:117] "RemoveContainer" containerID="3cc0f40c67fa130b25ce04f03794d004f074f72503c27671ba306a6a7bd2e075"
Sep 17 18:50:26.512929 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:26.512903    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5httc"]
Sep 17 18:50:26.514940 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:26.514920    2553 scope.go:117] "RemoveContainer" containerID="a58568fb8509b4e24903d512d221fb72ee23ec418d70798910c6418ddb5f083b"
Sep 17 18:50:26.515251 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:50:26.515226    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a58568fb8509b4e24903d512d221fb72ee23ec418d70798910c6418ddb5f083b\": container with ID starting with a58568fb8509b4e24903d512d221fb72ee23ec418d70798910c6418ddb5f083b not found: ID does not exist" containerID="a58568fb8509b4e24903d512d221fb72ee23ec418d70798910c6418ddb5f083b"
Sep 17 18:50:26.515307 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:26.515261    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a58568fb8509b4e24903d512d221fb72ee23ec418d70798910c6418ddb5f083b"} err="failed to get container status \"a58568fb8509b4e24903d512d221fb72ee23ec418d70798910c6418ddb5f083b\": rpc error: code = NotFound desc = could not find container \"a58568fb8509b4e24903d512d221fb72ee23ec418d70798910c6418ddb5f083b\": container with ID starting with a58568fb8509b4e24903d512d221fb72ee23ec418d70798910c6418ddb5f083b not found: ID does not exist"
Sep 17 18:50:26.515307 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:26.515280    2553 scope.go:117] "RemoveContainer" containerID="a3da48ad01d4c6eaf71dd4b3d4aefaa6c2f77de64e34e67b49f7ee3ade38df50"
Sep 17 18:50:26.515552 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:50:26.515530    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3da48ad01d4c6eaf71dd4b3d4aefaa6c2f77de64e34e67b49f7ee3ade38df50\": container with ID starting with a3da48ad01d4c6eaf71dd4b3d4aefaa6c2f77de64e34e67b49f7ee3ade38df50 not found: ID does not exist" containerID="a3da48ad01d4c6eaf71dd4b3d4aefaa6c2f77de64e34e67b49f7ee3ade38df50"
Sep 17 18:50:26.515655 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:26.515582    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3da48ad01d4c6eaf71dd4b3d4aefaa6c2f77de64e34e67b49f7ee3ade38df50"} err="failed to get container status \"a3da48ad01d4c6eaf71dd4b3d4aefaa6c2f77de64e34e67b49f7ee3ade38df50\": rpc error: code = NotFound desc = could not find container \"a3da48ad01d4c6eaf71dd4b3d4aefaa6c2f77de64e34e67b49f7ee3ade38df50\": container with ID starting with a3da48ad01d4c6eaf71dd4b3d4aefaa6c2f77de64e34e67b49f7ee3ade38df50 not found: ID does not exist"
Sep 17 18:50:26.515655 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:26.515608    2553 scope.go:117] "RemoveContainer" containerID="3cc0f40c67fa130b25ce04f03794d004f074f72503c27671ba306a6a7bd2e075"
Sep 17 18:50:26.516008 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:50:26.515981    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cc0f40c67fa130b25ce04f03794d004f074f72503c27671ba306a6a7bd2e075\": container with ID starting with 3cc0f40c67fa130b25ce04f03794d004f074f72503c27671ba306a6a7bd2e075 not found: ID does not exist" containerID="3cc0f40c67fa130b25ce04f03794d004f074f72503c27671ba306a6a7bd2e075"
Sep 17 18:50:26.516118 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:26.516075    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cc0f40c67fa130b25ce04f03794d004f074f72503c27671ba306a6a7bd2e075"} err="failed to get container status \"3cc0f40c67fa130b25ce04f03794d004f074f72503c27671ba306a6a7bd2e075\": rpc error: code = NotFound desc = could not find container \"3cc0f40c67fa130b25ce04f03794d004f074f72503c27671ba306a6a7bd2e075\": container with ID starting with 3cc0f40c67fa130b25ce04f03794d004f074f72503c27671ba306a6a7bd2e075 not found: ID does not exist"
Sep 17 18:50:26.517960 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:26.517940    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5httc"]
Sep 17 18:50:27.381898 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:27.381861    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f9700e5-a902-4e1e-8d61-16c90f84cc8d" path="/var/lib/kubelet/pods/8f9700e5-a902-4e1e-8d61-16c90f84cc8d/volumes"
Sep 17 18:50:31.496206 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:31.496132    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n"
Sep 17 18:50:31.496705 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:31.496677    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n" podUID="34432538-1018-4fec-9991-4e4fa4b8a3ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 17 18:50:41.497030 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:41.496990    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n" podUID="34432538-1018-4fec-9991-4e4fa4b8a3ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 17 18:50:51.496909 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:50:51.496866    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n" podUID="34432538-1018-4fec-9991-4e4fa4b8a3ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 17 18:51:01.497641 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:51:01.497577    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n" podUID="34432538-1018-4fec-9991-4e4fa4b8a3ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 17 18:51:11.497140 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:51:11.497099    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n" podUID="34432538-1018-4fec-9991-4e4fa4b8a3ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 17 18:51:21.496759 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:51:21.496716    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n" podUID="34432538-1018-4fec-9991-4e4fa4b8a3ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 17 18:51:31.497046 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:51:31.497003    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n" podUID="34432538-1018-4fec-9991-4e4fa4b8a3ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 17 18:51:41.496777 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:51:41.496735    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n" podUID="34432538-1018-4fec-9991-4e4fa4b8a3ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 17 18:51:47.541973 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:51:47.541936    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-btmqz_622ee5be-fb50-42b6-ae23-42900b7c2d98/ovn-acl-logging/0.log"
Sep 17 18:51:47.551899 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:51:47.551873    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-btmqz_622ee5be-fb50-42b6-ae23-42900b7c2d98/ovn-acl-logging/0.log"
Sep 17 18:51:51.496581 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:51:51.496544    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n" podUID="34432538-1018-4fec-9991-4e4fa4b8a3ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 17 18:51:53.372040 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:51:53.371990    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n" podUID="34432538-1018-4fec-9991-4e4fa4b8a3ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 17 18:52:03.377589 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:03.377513    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n"
Sep 17 18:52:10.824825 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:10.824787    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n"]
Sep 17 18:52:10.825338 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:10.825042    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n" podUID="34432538-1018-4fec-9991-4e4fa4b8a3ab" containerName="kserve-container" containerID="cri-o://8b28cffe18563c67dd99e21f0a0fe6c7f602501544e604d97678c519ad1676d3" gracePeriod=30
Sep 17 18:52:10.825338 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:10.825080    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n" podUID="34432538-1018-4fec-9991-4e4fa4b8a3ab" containerName="kube-rbac-proxy" containerID="cri-o://94ef2efbd9be23837688c4c44442d13ff8285576026fe09a5b8473589c39f407" gracePeriod=30
Sep 17 18:52:10.963954 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:10.963922    2553 generic.go:356] "Generic (PLEG): container finished" podID="34432538-1018-4fec-9991-4e4fa4b8a3ab" containerID="94ef2efbd9be23837688c4c44442d13ff8285576026fe09a5b8473589c39f407" exitCode=2
Sep 17 18:52:10.964126 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:10.964006    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n" event={"ID":"34432538-1018-4fec-9991-4e4fa4b8a3ab","Type":"ContainerDied","Data":"94ef2efbd9be23837688c4c44442d13ff8285576026fe09a5b8473589c39f407"}
Sep 17 18:52:11.492076 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:11.492029    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n" podUID="34432538-1018-4fec-9991-4e4fa4b8a3ab" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.59:8643/healthz\": dial tcp 10.134.0.59:8643: connect: connection refused"
Sep 17 18:52:13.372002 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:13.371948    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n" podUID="34432538-1018-4fec-9991-4e4fa4b8a3ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 17 18:52:15.275761 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:15.275732    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n"
Sep 17 18:52:15.339936 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:15.339836    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34432538-1018-4fec-9991-4e4fa4b8a3ab-cabundle-cert\") pod \"34432538-1018-4fec-9991-4e4fa4b8a3ab\" (UID: \"34432538-1018-4fec-9991-4e4fa4b8a3ab\") "
Sep 17 18:52:15.339936 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:15.339907    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34432538-1018-4fec-9991-4e4fa4b8a3ab-proxy-tls\") pod \"34432538-1018-4fec-9991-4e4fa4b8a3ab\" (UID: \"34432538-1018-4fec-9991-4e4fa4b8a3ab\") "
Sep 17 18:52:15.340179 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:15.340024    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34432538-1018-4fec-9991-4e4fa4b8a3ab-kserve-provision-location\") pod \"34432538-1018-4fec-9991-4e4fa4b8a3ab\" (UID: \"34432538-1018-4fec-9991-4e4fa4b8a3ab\") "
Sep 17 18:52:15.340179 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:15.340083    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34432538-1018-4fec-9991-4e4fa4b8a3ab-kube-rbac-proxy-sar-config\") pod \"34432538-1018-4fec-9991-4e4fa4b8a3ab\" (UID: \"34432538-1018-4fec-9991-4e4fa4b8a3ab\") "
Sep 17 18:52:15.340179 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:15.340118    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ww2qv\" (UniqueName: \"kubernetes.io/projected/34432538-1018-4fec-9991-4e4fa4b8a3ab-kube-api-access-ww2qv\") pod \"34432538-1018-4fec-9991-4e4fa4b8a3ab\" (UID: \"34432538-1018-4fec-9991-4e4fa4b8a3ab\") "
Sep 17 18:52:15.340338 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:15.340248    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34432538-1018-4fec-9991-4e4fa4b8a3ab-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "34432538-1018-4fec-9991-4e4fa4b8a3ab" (UID: "34432538-1018-4fec-9991-4e4fa4b8a3ab"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:52:15.340396 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:15.340333    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34432538-1018-4fec-9991-4e4fa4b8a3ab-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "34432538-1018-4fec-9991-4e4fa4b8a3ab" (UID: "34432538-1018-4fec-9991-4e4fa4b8a3ab"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:52:15.340396 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:15.340383    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34432538-1018-4fec-9991-4e4fa4b8a3ab-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "34432538-1018-4fec-9991-4e4fa4b8a3ab" (UID: "34432538-1018-4fec-9991-4e4fa4b8a3ab"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:52:15.340396 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:15.340389    2553 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34432538-1018-4fec-9991-4e4fa4b8a3ab-cabundle-cert\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:52:15.342255 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:15.342223    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34432538-1018-4fec-9991-4e4fa4b8a3ab-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "34432538-1018-4fec-9991-4e4fa4b8a3ab" (UID: "34432538-1018-4fec-9991-4e4fa4b8a3ab"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:52:15.342255 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:15.342233    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34432538-1018-4fec-9991-4e4fa4b8a3ab-kube-api-access-ww2qv" (OuterVolumeSpecName: "kube-api-access-ww2qv") pod "34432538-1018-4fec-9991-4e4fa4b8a3ab" (UID: "34432538-1018-4fec-9991-4e4fa4b8a3ab"). InnerVolumeSpecName "kube-api-access-ww2qv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:52:15.441423 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:15.441380    2553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34432538-1018-4fec-9991-4e4fa4b8a3ab-kserve-provision-location\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:52:15.441423 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:15.441414    2553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34432538-1018-4fec-9991-4e4fa4b8a3ab-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:52:15.441423 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:15.441424    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ww2qv\" (UniqueName: \"kubernetes.io/projected/34432538-1018-4fec-9991-4e4fa4b8a3ab-kube-api-access-ww2qv\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:52:15.441708 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:15.441433    2553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34432538-1018-4fec-9991-4e4fa4b8a3ab-proxy-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:52:15.990346 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:15.990303    2553 generic.go:356] "Generic (PLEG): container finished" podID="34432538-1018-4fec-9991-4e4fa4b8a3ab" containerID="8b28cffe18563c67dd99e21f0a0fe6c7f602501544e604d97678c519ad1676d3" exitCode=0
Sep 17 18:52:15.990541 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:15.990363    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n"
Sep 17 18:52:15.990541 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:15.990377    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n" event={"ID":"34432538-1018-4fec-9991-4e4fa4b8a3ab","Type":"ContainerDied","Data":"8b28cffe18563c67dd99e21f0a0fe6c7f602501544e604d97678c519ad1676d3"}
Sep 17 18:52:15.990541 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:15.990436    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n" event={"ID":"34432538-1018-4fec-9991-4e4fa4b8a3ab","Type":"ContainerDied","Data":"92765ba7c31b1f3529ac5745441c60445cd2fb40922fd79c19066d7eac1c7b6f"}
Sep 17 18:52:15.990541 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:15.990463    2553 scope.go:117] "RemoveContainer" containerID="94ef2efbd9be23837688c4c44442d13ff8285576026fe09a5b8473589c39f407"
Sep 17 18:52:15.998718 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:15.998695    2553 scope.go:117] "RemoveContainer" containerID="8b28cffe18563c67dd99e21f0a0fe6c7f602501544e604d97678c519ad1676d3"
Sep 17 18:52:16.006608 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:16.006586    2553 scope.go:117] "RemoveContainer" containerID="606a67dd6159b580253887a476d9ebf62f96475f23cb69a89d2e85a02fcf07ac"
Sep 17 18:52:16.006880 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:16.006857    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-bc87e7-predictor-565d888469-mkfsr"]
Sep 17 18:52:16.007201 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:16.007187    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34432538-1018-4fec-9991-4e4fa4b8a3ab" containerName="kserve-container"
Sep 17 18:52:16.007271 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:16.007203    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="34432538-1018-4fec-9991-4e4fa4b8a3ab" containerName="kserve-container"
Sep 17 18:52:16.007271 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:16.007214    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f9700e5-a902-4e1e-8d61-16c90f84cc8d" containerName="kube-rbac-proxy"
Sep 17 18:52:16.007271 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:16.007219    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f9700e5-a902-4e1e-8d61-16c90f84cc8d" containerName="kube-rbac-proxy"
Sep 17 18:52:16.007271 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:16.007227    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34432538-1018-4fec-9991-4e4fa4b8a3ab" containerName="storage-initializer"
Sep 17 18:52:16.007271 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:16.007233    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="34432538-1018-4fec-9991-4e4fa4b8a3ab" containerName="storage-initializer"
Sep 17 18:52:16.007271 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:16.007241    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34432538-1018-4fec-9991-4e4fa4b8a3ab" containerName="kube-rbac-proxy"
Sep 17 18:52:16.007271 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:16.007248    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="34432538-1018-4fec-9991-4e4fa4b8a3ab" containerName="kube-rbac-proxy"
Sep 17 18:52:16.007271 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:16.007262    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f9700e5-a902-4e1e-8d61-16c90f84cc8d" containerName="storage-initializer"
Sep 17 18:52:16.007271 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:16.007268    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f9700e5-a902-4e1e-8d61-16c90f84cc8d" containerName="storage-initializer"
Sep 17 18:52:16.007599 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:16.007275    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f9700e5-a902-4e1e-8d61-16c90f84cc8d" containerName="kserve-container"
Sep 17 18:52:16.007599 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:16.007283    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f9700e5-a902-4e1e-8d61-16c90f84cc8d" containerName="kserve-container"
Sep 17 18:52:16.007599 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:16.007368    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="34432538-1018-4fec-9991-4e4fa4b8a3ab" containerName="kserve-container"
Sep 17 18:52:16.007599 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:16.007376    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f9700e5-a902-4e1e-8d61-16c90f84cc8d" containerName="kube-rbac-proxy"
Sep 17 18:52:16.007599 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:16.007384    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="34432538-1018-4fec-9991-4e4fa4b8a3ab" containerName="kube-rbac-proxy"
Sep 17 18:52:16.007599 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:16.007391    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f9700e5-a902-4e1e-8d61-16c90f84cc8d" containerName="kserve-container"
Sep 17 18:52:16.011932 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:16.011910    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-bc87e7-predictor-565d888469-mkfsr"
Sep 17 18:52:16.015457 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:16.015433    2553 scope.go:117] "RemoveContainer" containerID="94ef2efbd9be23837688c4c44442d13ff8285576026fe09a5b8473589c39f407"
Sep 17 18:52:16.015771 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:52:16.015750    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94ef2efbd9be23837688c4c44442d13ff8285576026fe09a5b8473589c39f407\": container with ID starting with 94ef2efbd9be23837688c4c44442d13ff8285576026fe09a5b8473589c39f407 not found: ID does not exist" containerID="94ef2efbd9be23837688c4c44442d13ff8285576026fe09a5b8473589c39f407"
Sep 17 18:52:16.015869 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:16.015779    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94ef2efbd9be23837688c4c44442d13ff8285576026fe09a5b8473589c39f407"} err="failed to get container status \"94ef2efbd9be23837688c4c44442d13ff8285576026fe09a5b8473589c39f407\": rpc error: code = NotFound desc = could not find container \"94ef2efbd9be23837688c4c44442d13ff8285576026fe09a5b8473589c39f407\": container with ID starting with 94ef2efbd9be23837688c4c44442d13ff8285576026fe09a5b8473589c39f407 not found: ID does not exist"
Sep 17 18:52:16.015869 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:16.015797    2553 scope.go:117] "RemoveContainer" containerID="8b28cffe18563c67dd99e21f0a0fe6c7f602501544e604d97678c519ad1676d3"
Sep 17 18:52:16.016079 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:52:16.016058    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b28cffe18563c67dd99e21f0a0fe6c7f602501544e604d97678c519ad1676d3\": container with ID starting with 8b28cffe18563c67dd99e21f0a0fe6c7f602501544e604d97678c519ad1676d3 not found: ID does not exist" containerID="8b28cffe18563c67dd99e21f0a0fe6c7f602501544e604d97678c519ad1676d3"
Sep 17 18:52:16.016158 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:16.016086    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b28cffe18563c67dd99e21f0a0fe6c7f602501544e604d97678c519ad1676d3"} err="failed to get container status \"8b28cffe18563c67dd99e21f0a0fe6c7f602501544e604d97678c519ad1676d3\": rpc error: code = NotFound desc = could not find container \"8b28cffe18563c67dd99e21f0a0fe6c7f602501544e604d97678c519ad1676d3\": container with ID starting with 8b28cffe18563c67dd99e21f0a0fe6c7f602501544e604d97678c519ad1676d3 not found: ID does not exist"
Sep 17 18:52:16.016158 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:16.016105    2553 scope.go:117] "RemoveContainer" containerID="606a67dd6159b580253887a476d9ebf62f96475f23cb69a89d2e85a02fcf07ac"
Sep 17 18:52:16.016338 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:52:16.016320    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"606a67dd6159b580253887a476d9ebf62f96475f23cb69a89d2e85a02fcf07ac\": container with ID starting with 606a67dd6159b580253887a476d9ebf62f96475f23cb69a89d2e85a02fcf07ac not found: ID does not exist" containerID="606a67dd6159b580253887a476d9ebf62f96475f23cb69a89d2e85a02fcf07ac"
Sep 17 18:52:16.016406 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:16.016340    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"606a67dd6159b580253887a476d9ebf62f96475f23cb69a89d2e85a02fcf07ac"} err="failed to get container status \"606a67dd6159b580253887a476d9ebf62f96475f23cb69a89d2e85a02fcf07ac\": rpc error: code = NotFound desc = could not find container \"606a67dd6159b580253887a476d9ebf62f96475f23cb69a89d2e85a02fcf07ac\": container with ID starting with 606a67dd6159b580253887a476d9ebf62f96475f23cb69a89d2e85a02fcf07ac not found: ID does not exist"
Sep 17 18:52:16.016560 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:16.016544    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-bc87e7-kube-rbac-proxy-sar-config\""
Sep 17 18:52:16.016753 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:16.016552    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:52:16.016869 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:16.016840    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-bc87e7-predictor-serving-cert\""
Sep 17 18:52:16.017020 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:16.017004    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-7qcmd\""
Sep 17 18:52:16.017614 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:16.017595    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:52:16.018073 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:16.018052    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:52:16.018145 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:16.018059    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:52:16.039860 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:16.039824    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-bc87e7-predictor-565d888469-mkfsr"]
Sep 17 18:52:16.066481 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:16.066448    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n"]
Sep 17 18:52:16.072092 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:16.072063    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-ff99n"]
Sep 17 18:52:16.146867 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:16.146829    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a9b0df6e-ae3d-46c1-b890-b0df7edf5701-kserve-provision-location\") pod \"isvc-primary-bc87e7-predictor-565d888469-mkfsr\" (UID: \"a9b0df6e-ae3d-46c1-b890-b0df7edf5701\") " pod="kserve-ci-e2e-test/isvc-primary-bc87e7-predictor-565d888469-mkfsr"
Sep 17 18:52:16.147060 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:16.146887    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9b0df6e-ae3d-46c1-b890-b0df7edf5701-kube-rbac-proxy-sar-config\") pod \"isvc-primary-bc87e7-predictor-565d888469-mkfsr\" (UID: \"a9b0df6e-ae3d-46c1-b890-b0df7edf5701\") " pod="kserve-ci-e2e-test/isvc-primary-bc87e7-predictor-565d888469-mkfsr"
Sep 17 18:52:16.147060 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:16.146925    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g46rl\" (UniqueName: \"kubernetes.io/projected/a9b0df6e-ae3d-46c1-b890-b0df7edf5701-kube-api-access-g46rl\") pod \"isvc-primary-bc87e7-predictor-565d888469-mkfsr\" (UID: \"a9b0df6e-ae3d-46c1-b890-b0df7edf5701\") " pod="kserve-ci-e2e-test/isvc-primary-bc87e7-predictor-565d888469-mkfsr"
Sep 17 18:52:16.147060 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:16.146991    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9b0df6e-ae3d-46c1-b890-b0df7edf5701-proxy-tls\") pod \"isvc-primary-bc87e7-predictor-565d888469-mkfsr\" (UID: \"a9b0df6e-ae3d-46c1-b890-b0df7edf5701\") " pod="kserve-ci-e2e-test/isvc-primary-bc87e7-predictor-565d888469-mkfsr"
Sep 17 18:52:16.147060 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:16.147021    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a9b0df6e-ae3d-46c1-b890-b0df7edf5701-cabundle-cert\") pod \"isvc-primary-bc87e7-predictor-565d888469-mkfsr\" (UID: \"a9b0df6e-ae3d-46c1-b890-b0df7edf5701\") " pod="kserve-ci-e2e-test/isvc-primary-bc87e7-predictor-565d888469-mkfsr"
Sep 17 18:52:16.248429 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:16.248331    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9b0df6e-ae3d-46c1-b890-b0df7edf5701-proxy-tls\") pod \"isvc-primary-bc87e7-predictor-565d888469-mkfsr\" (UID: \"a9b0df6e-ae3d-46c1-b890-b0df7edf5701\") " pod="kserve-ci-e2e-test/isvc-primary-bc87e7-predictor-565d888469-mkfsr"
Sep 17 18:52:16.248429 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:16.248370    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a9b0df6e-ae3d-46c1-b890-b0df7edf5701-cabundle-cert\") pod \"isvc-primary-bc87e7-predictor-565d888469-mkfsr\" (UID: \"a9b0df6e-ae3d-46c1-b890-b0df7edf5701\") " pod="kserve-ci-e2e-test/isvc-primary-bc87e7-predictor-565d888469-mkfsr"
Sep 17 18:52:16.248429 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:16.248427    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a9b0df6e-ae3d-46c1-b890-b0df7edf5701-kserve-provision-location\") pod \"isvc-primary-bc87e7-predictor-565d888469-mkfsr\" (UID: \"a9b0df6e-ae3d-46c1-b890-b0df7edf5701\") " pod="kserve-ci-e2e-test/isvc-primary-bc87e7-predictor-565d888469-mkfsr"
Sep 17 18:52:16.248786 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:16.248500    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9b0df6e-ae3d-46c1-b890-b0df7edf5701-kube-rbac-proxy-sar-config\") pod \"isvc-primary-bc87e7-predictor-565d888469-mkfsr\" (UID: \"a9b0df6e-ae3d-46c1-b890-b0df7edf5701\") " pod="kserve-ci-e2e-test/isvc-primary-bc87e7-predictor-565d888469-mkfsr"
Sep 17 18:52:16.248786 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:16.248531    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g46rl\" (UniqueName: \"kubernetes.io/projected/a9b0df6e-ae3d-46c1-b890-b0df7edf5701-kube-api-access-g46rl\") pod \"isvc-primary-bc87e7-predictor-565d888469-mkfsr\" (UID: \"a9b0df6e-ae3d-46c1-b890-b0df7edf5701\") " pod="kserve-ci-e2e-test/isvc-primary-bc87e7-predictor-565d888469-mkfsr"
Sep 17 18:52:16.248979 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:16.248958    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a9b0df6e-ae3d-46c1-b890-b0df7edf5701-kserve-provision-location\") pod \"isvc-primary-bc87e7-predictor-565d888469-mkfsr\" (UID: \"a9b0df6e-ae3d-46c1-b890-b0df7edf5701\") " pod="kserve-ci-e2e-test/isvc-primary-bc87e7-predictor-565d888469-mkfsr"
Sep 17 18:52:16.249186 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:16.249168    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9b0df6e-ae3d-46c1-b890-b0df7edf5701-kube-rbac-proxy-sar-config\") pod \"isvc-primary-bc87e7-predictor-565d888469-mkfsr\" (UID: \"a9b0df6e-ae3d-46c1-b890-b0df7edf5701\") " pod="kserve-ci-e2e-test/isvc-primary-bc87e7-predictor-565d888469-mkfsr"
Sep 17 18:52:16.249265 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:16.249242    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a9b0df6e-ae3d-46c1-b890-b0df7edf5701-cabundle-cert\") pod \"isvc-primary-bc87e7-predictor-565d888469-mkfsr\" (UID: \"a9b0df6e-ae3d-46c1-b890-b0df7edf5701\") " pod="kserve-ci-e2e-test/isvc-primary-bc87e7-predictor-565d888469-mkfsr"
Sep 17 18:52:16.251000 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:16.250982    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9b0df6e-ae3d-46c1-b890-b0df7edf5701-proxy-tls\") pod \"isvc-primary-bc87e7-predictor-565d888469-mkfsr\" (UID: \"a9b0df6e-ae3d-46c1-b890-b0df7edf5701\") " pod="kserve-ci-e2e-test/isvc-primary-bc87e7-predictor-565d888469-mkfsr"
Sep 17 18:52:16.256749 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:16.256726    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g46rl\" (UniqueName: \"kubernetes.io/projected/a9b0df6e-ae3d-46c1-b890-b0df7edf5701-kube-api-access-g46rl\") pod \"isvc-primary-bc87e7-predictor-565d888469-mkfsr\" (UID: \"a9b0df6e-ae3d-46c1-b890-b0df7edf5701\") " pod="kserve-ci-e2e-test/isvc-primary-bc87e7-predictor-565d888469-mkfsr"
Sep 17 18:52:16.322782 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:16.322746    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-bc87e7-predictor-565d888469-mkfsr"
Sep 17 18:52:16.597958 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:16.597780    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-bc87e7-predictor-565d888469-mkfsr"]
Sep 17 18:52:16.600168 ip-10-0-137-95 kubenswrapper[2553]: W0917 18:52:16.600140    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9b0df6e_ae3d_46c1_b890_b0df7edf5701.slice/crio-836026fc41b65c7fed9af601b4d14994da8e1b6b3bc0dee10bb2ba0f1ed7a6be WatchSource:0}: Error finding container 836026fc41b65c7fed9af601b4d14994da8e1b6b3bc0dee10bb2ba0f1ed7a6be: Status 404 returned error can't find the container with id 836026fc41b65c7fed9af601b4d14994da8e1b6b3bc0dee10bb2ba0f1ed7a6be
Sep 17 18:52:16.997903 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:16.997855    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-bc87e7-predictor-565d888469-mkfsr" event={"ID":"a9b0df6e-ae3d-46c1-b890-b0df7edf5701","Type":"ContainerStarted","Data":"9aadee85c84b06e011fd1f48fbd54f131644a98b6ce8ab181ac178db28af1f13"}
Sep 17 18:52:16.998113 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:16.997914    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-bc87e7-predictor-565d888469-mkfsr" event={"ID":"a9b0df6e-ae3d-46c1-b890-b0df7edf5701","Type":"ContainerStarted","Data":"836026fc41b65c7fed9af601b4d14994da8e1b6b3bc0dee10bb2ba0f1ed7a6be"}
Sep 17 18:52:17.376589 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:17.376507    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34432538-1018-4fec-9991-4e4fa4b8a3ab" path="/var/lib/kubelet/pods/34432538-1018-4fec-9991-4e4fa4b8a3ab/volumes"
Sep 17 18:52:21.017175 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:21.017138    2553 generic.go:356] "Generic (PLEG): container finished" podID="a9b0df6e-ae3d-46c1-b890-b0df7edf5701" containerID="9aadee85c84b06e011fd1f48fbd54f131644a98b6ce8ab181ac178db28af1f13" exitCode=0
Sep 17 18:52:21.017613 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:21.017211    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-bc87e7-predictor-565d888469-mkfsr" event={"ID":"a9b0df6e-ae3d-46c1-b890-b0df7edf5701","Type":"ContainerDied","Data":"9aadee85c84b06e011fd1f48fbd54f131644a98b6ce8ab181ac178db28af1f13"}
Sep 17 18:52:22.024413 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:22.024374    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-bc87e7-predictor-565d888469-mkfsr" event={"ID":"a9b0df6e-ae3d-46c1-b890-b0df7edf5701","Type":"ContainerStarted","Data":"0c0df1774f35faf4fab4f10a7828e62b8b493534dae6577ba32a662526369769"}
Sep 17 18:52:22.024847 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:22.024433    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-bc87e7-predictor-565d888469-mkfsr" event={"ID":"a9b0df6e-ae3d-46c1-b890-b0df7edf5701","Type":"ContainerStarted","Data":"f7224f0c5510b9ca27d9de6afd60bce09379e4d74946bae8ef3a04c5f9e3258f"}
Sep 17 18:52:22.024847 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:22.024757    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-bc87e7-predictor-565d888469-mkfsr"
Sep 17 18:52:22.024847 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:22.024813    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-bc87e7-predictor-565d888469-mkfsr"
Sep 17 18:52:22.026274 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:22.026247    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-bc87e7-predictor-565d888469-mkfsr" podUID="a9b0df6e-ae3d-46c1-b890-b0df7edf5701" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 17 18:52:22.046919 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:22.046861    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-bc87e7-predictor-565d888469-mkfsr" podStartSLOduration=3.374098985 podStartE2EDuration="7.046844669s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.672745684s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:52:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:52:22.045380793 +0000 UTC m=+3336.741471490" watchObservedRunningTime="2026-09-17 18:52:22.046844669 +0000 UTC m=+3336.742935352"
Sep 17 18:52:23.030290 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:23.030243    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-bc87e7-predictor-565d888469-mkfsr" podUID="a9b0df6e-ae3d-46c1-b890-b0df7edf5701" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 17 18:52:28.035115 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:28.035084    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-bc87e7-predictor-565d888469-mkfsr"
Sep 17 18:52:28.035591 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:28.035561    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-bc87e7-predictor-565d888469-mkfsr" podUID="a9b0df6e-ae3d-46c1-b890-b0df7edf5701" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 17 18:52:38.035573 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:38.035526    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-bc87e7-predictor-565d888469-mkfsr" podUID="a9b0df6e-ae3d-46c1-b890-b0df7edf5701" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 17 18:52:48.035881 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:48.035834    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-bc87e7-predictor-565d888469-mkfsr" podUID="a9b0df6e-ae3d-46c1-b890-b0df7edf5701" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 17 18:52:58.036437 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:52:58.036391    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-bc87e7-predictor-565d888469-mkfsr" podUID="a9b0df6e-ae3d-46c1-b890-b0df7edf5701" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 17 18:53:08.035636 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:53:08.035588    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-bc87e7-predictor-565d888469-mkfsr" podUID="a9b0df6e-ae3d-46c1-b890-b0df7edf5701" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 17 18:53:18.035557 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:53:18.035506    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-bc87e7-predictor-565d888469-mkfsr" podUID="a9b0df6e-ae3d-46c1-b890-b0df7edf5701" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 17 18:53:28.036024 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:53:28.035981    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-bc87e7-predictor-565d888469-mkfsr" podUID="a9b0df6e-ae3d-46c1-b890-b0df7edf5701" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 17 18:53:30.371311 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:53:30.371212    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-bc87e7-predictor-565d888469-mkfsr" podUID="a9b0df6e-ae3d-46c1-b890-b0df7edf5701" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 17 18:53:40.372763 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:53:40.372729    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-bc87e7-predictor-565d888469-mkfsr"
Sep 17 18:54:01.204147 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:01.204108    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-bc87e7-predictor-5c66fb9558-j7lx8"]
Sep 17 18:54:01.207584 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:01.207564    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-bc87e7-predictor-5c66fb9558-j7lx8"
Sep 17 18:54:01.210611 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:01.210571    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-bc87e7-predictor-serving-cert\""
Sep 17 18:54:01.210764 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:01.210667    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-bc87e7-kube-rbac-proxy-sar-config\""
Sep 17 18:54:01.210921 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:01.210897    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-bc87e7\""
Sep 17 18:54:01.211586 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:01.211561    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-bc87e7-dockercfg-vr5gv\""
Sep 17 18:54:01.220914 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:01.220886    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-bc87e7-predictor-5c66fb9558-j7lx8"]
Sep 17 18:54:01.381396 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:01.381364    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db982b76-c378-4a17-9fbf-f9a8d36160bb-proxy-tls\") pod \"isvc-secondary-bc87e7-predictor-5c66fb9558-j7lx8\" (UID: \"db982b76-c378-4a17-9fbf-f9a8d36160bb\") " pod="kserve-ci-e2e-test/isvc-secondary-bc87e7-predictor-5c66fb9558-j7lx8"
Sep 17 18:54:01.381396 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:01.381400    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5krw4\" (UniqueName: \"kubernetes.io/projected/db982b76-c378-4a17-9fbf-f9a8d36160bb-kube-api-access-5krw4\") pod \"isvc-secondary-bc87e7-predictor-5c66fb9558-j7lx8\" (UID: \"db982b76-c378-4a17-9fbf-f9a8d36160bb\") " pod="kserve-ci-e2e-test/isvc-secondary-bc87e7-predictor-5c66fb9558-j7lx8"
Sep 17 18:54:01.381659 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:01.381431    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db982b76-c378-4a17-9fbf-f9a8d36160bb-kserve-provision-location\") pod \"isvc-secondary-bc87e7-predictor-5c66fb9558-j7lx8\" (UID: \"db982b76-c378-4a17-9fbf-f9a8d36160bb\") " pod="kserve-ci-e2e-test/isvc-secondary-bc87e7-predictor-5c66fb9558-j7lx8"
Sep 17 18:54:01.381659 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:01.381509    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db982b76-c378-4a17-9fbf-f9a8d36160bb-cabundle-cert\") pod \"isvc-secondary-bc87e7-predictor-5c66fb9558-j7lx8\" (UID: \"db982b76-c378-4a17-9fbf-f9a8d36160bb\") " pod="kserve-ci-e2e-test/isvc-secondary-bc87e7-predictor-5c66fb9558-j7lx8"
Sep 17 18:54:01.381659 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:01.381564    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db982b76-c378-4a17-9fbf-f9a8d36160bb-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-bc87e7-predictor-5c66fb9558-j7lx8\" (UID: \"db982b76-c378-4a17-9fbf-f9a8d36160bb\") " pod="kserve-ci-e2e-test/isvc-secondary-bc87e7-predictor-5c66fb9558-j7lx8"
Sep 17 18:54:01.482717 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:01.482585    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db982b76-c378-4a17-9fbf-f9a8d36160bb-proxy-tls\") pod \"isvc-secondary-bc87e7-predictor-5c66fb9558-j7lx8\" (UID: \"db982b76-c378-4a17-9fbf-f9a8d36160bb\") " pod="kserve-ci-e2e-test/isvc-secondary-bc87e7-predictor-5c66fb9558-j7lx8"
Sep 17 18:54:01.482717 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:01.482663    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5krw4\" (UniqueName: \"kubernetes.io/projected/db982b76-c378-4a17-9fbf-f9a8d36160bb-kube-api-access-5krw4\") pod \"isvc-secondary-bc87e7-predictor-5c66fb9558-j7lx8\" (UID: \"db982b76-c378-4a17-9fbf-f9a8d36160bb\") " pod="kserve-ci-e2e-test/isvc-secondary-bc87e7-predictor-5c66fb9558-j7lx8"
Sep 17 18:54:01.482717 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:01.482690    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db982b76-c378-4a17-9fbf-f9a8d36160bb-kserve-provision-location\") pod \"isvc-secondary-bc87e7-predictor-5c66fb9558-j7lx8\" (UID: \"db982b76-c378-4a17-9fbf-f9a8d36160bb\") " pod="kserve-ci-e2e-test/isvc-secondary-bc87e7-predictor-5c66fb9558-j7lx8"
Sep 17 18:54:01.483018 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:01.482729    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db982b76-c378-4a17-9fbf-f9a8d36160bb-cabundle-cert\") pod \"isvc-secondary-bc87e7-predictor-5c66fb9558-j7lx8\" (UID: \"db982b76-c378-4a17-9fbf-f9a8d36160bb\") " pod="kserve-ci-e2e-test/isvc-secondary-bc87e7-predictor-5c66fb9558-j7lx8"
Sep 17 18:54:01.483018 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:54:01.482749    2553 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-secondary-bc87e7-predictor-serving-cert: secret "isvc-secondary-bc87e7-predictor-serving-cert" not found
Sep 17 18:54:01.483018 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:01.482798    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db982b76-c378-4a17-9fbf-f9a8d36160bb-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-bc87e7-predictor-5c66fb9558-j7lx8\" (UID: \"db982b76-c378-4a17-9fbf-f9a8d36160bb\") " pod="kserve-ci-e2e-test/isvc-secondary-bc87e7-predictor-5c66fb9558-j7lx8"
Sep 17 18:54:01.483018 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:54:01.482828    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db982b76-c378-4a17-9fbf-f9a8d36160bb-proxy-tls podName:db982b76-c378-4a17-9fbf-f9a8d36160bb nodeName:}" failed. No retries permitted until 2026-09-17 18:54:01.982808146 +0000 UTC m=+3436.678898812 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/db982b76-c378-4a17-9fbf-f9a8d36160bb-proxy-tls") pod "isvc-secondary-bc87e7-predictor-5c66fb9558-j7lx8" (UID: "db982b76-c378-4a17-9fbf-f9a8d36160bb") : secret "isvc-secondary-bc87e7-predictor-serving-cert" not found
Sep 17 18:54:01.483244 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:01.483169    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db982b76-c378-4a17-9fbf-f9a8d36160bb-kserve-provision-location\") pod \"isvc-secondary-bc87e7-predictor-5c66fb9558-j7lx8\" (UID: \"db982b76-c378-4a17-9fbf-f9a8d36160bb\") " pod="kserve-ci-e2e-test/isvc-secondary-bc87e7-predictor-5c66fb9558-j7lx8"
Sep 17 18:54:01.483581 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:01.483560    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db982b76-c378-4a17-9fbf-f9a8d36160bb-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-bc87e7-predictor-5c66fb9558-j7lx8\" (UID: \"db982b76-c378-4a17-9fbf-f9a8d36160bb\") " pod="kserve-ci-e2e-test/isvc-secondary-bc87e7-predictor-5c66fb9558-j7lx8"
Sep 17 18:54:01.483690 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:01.483562    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db982b76-c378-4a17-9fbf-f9a8d36160bb-cabundle-cert\") pod \"isvc-secondary-bc87e7-predictor-5c66fb9558-j7lx8\" (UID: \"db982b76-c378-4a17-9fbf-f9a8d36160bb\") " pod="kserve-ci-e2e-test/isvc-secondary-bc87e7-predictor-5c66fb9558-j7lx8"
Sep 17 18:54:01.491273 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:01.491240    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5krw4\" (UniqueName: \"kubernetes.io/projected/db982b76-c378-4a17-9fbf-f9a8d36160bb-kube-api-access-5krw4\") pod \"isvc-secondary-bc87e7-predictor-5c66fb9558-j7lx8\" (UID: \"db982b76-c378-4a17-9fbf-f9a8d36160bb\") " pod="kserve-ci-e2e-test/isvc-secondary-bc87e7-predictor-5c66fb9558-j7lx8"
Sep 17 18:54:01.987302 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:01.987258    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db982b76-c378-4a17-9fbf-f9a8d36160bb-proxy-tls\") pod \"isvc-secondary-bc87e7-predictor-5c66fb9558-j7lx8\" (UID: \"db982b76-c378-4a17-9fbf-f9a8d36160bb\") " pod="kserve-ci-e2e-test/isvc-secondary-bc87e7-predictor-5c66fb9558-j7lx8"
Sep 17 18:54:01.989736 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:01.989707    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db982b76-c378-4a17-9fbf-f9a8d36160bb-proxy-tls\") pod \"isvc-secondary-bc87e7-predictor-5c66fb9558-j7lx8\" (UID: \"db982b76-c378-4a17-9fbf-f9a8d36160bb\") " pod="kserve-ci-e2e-test/isvc-secondary-bc87e7-predictor-5c66fb9558-j7lx8"
Sep 17 18:54:02.122381 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:02.122338    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-bc87e7-predictor-5c66fb9558-j7lx8"
Sep 17 18:54:02.397493 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:02.397470    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-bc87e7-predictor-5c66fb9558-j7lx8"]
Sep 17 18:54:02.399553 ip-10-0-137-95 kubenswrapper[2553]: W0917 18:54:02.399526    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb982b76_c378_4a17_9fbf_f9a8d36160bb.slice/crio-2ddcfa33c892096b12dadae201edcd2b6524c783eb1e5489aee7f09e423faae4 WatchSource:0}: Error finding container 2ddcfa33c892096b12dadae201edcd2b6524c783eb1e5489aee7f09e423faae4: Status 404 returned error can't find the container with id 2ddcfa33c892096b12dadae201edcd2b6524c783eb1e5489aee7f09e423faae4
Sep 17 18:54:02.469134 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:02.469092    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-bc87e7-predictor-5c66fb9558-j7lx8" event={"ID":"db982b76-c378-4a17-9fbf-f9a8d36160bb","Type":"ContainerStarted","Data":"d27c3b1207bed9da06329889b5fb29d15e3d04e8365eb6895ee5f08639b54552"}
Sep 17 18:54:02.469304 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:02.469167    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-bc87e7-predictor-5c66fb9558-j7lx8" event={"ID":"db982b76-c378-4a17-9fbf-f9a8d36160bb","Type":"ContainerStarted","Data":"2ddcfa33c892096b12dadae201edcd2b6524c783eb1e5489aee7f09e423faae4"}
Sep 17 18:54:07.492529 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:07.492500    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-bc87e7-predictor-5c66fb9558-j7lx8_db982b76-c378-4a17-9fbf-f9a8d36160bb/storage-initializer/0.log"
Sep 17 18:54:07.492917 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:07.492536    2553 generic.go:356] "Generic (PLEG): container finished" podID="db982b76-c378-4a17-9fbf-f9a8d36160bb" containerID="d27c3b1207bed9da06329889b5fb29d15e3d04e8365eb6895ee5f08639b54552" exitCode=1
Sep 17 18:54:07.492917 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:07.492564    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-bc87e7-predictor-5c66fb9558-j7lx8" event={"ID":"db982b76-c378-4a17-9fbf-f9a8d36160bb","Type":"ContainerDied","Data":"d27c3b1207bed9da06329889b5fb29d15e3d04e8365eb6895ee5f08639b54552"}
Sep 17 18:54:08.499917 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:08.499889    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-bc87e7-predictor-5c66fb9558-j7lx8_db982b76-c378-4a17-9fbf-f9a8d36160bb/storage-initializer/0.log"
Sep 17 18:54:08.500354 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:08.499998    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-bc87e7-predictor-5c66fb9558-j7lx8" event={"ID":"db982b76-c378-4a17-9fbf-f9a8d36160bb","Type":"ContainerStarted","Data":"f5d607512bc63b9bbc5e6dcadc37c4d31958dd090d89306526fa757e735409b2"}
Sep 17 18:54:13.523669 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:13.523571    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-bc87e7-predictor-5c66fb9558-j7lx8_db982b76-c378-4a17-9fbf-f9a8d36160bb/storage-initializer/1.log"
Sep 17 18:54:13.524058 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:13.523990    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-bc87e7-predictor-5c66fb9558-j7lx8_db982b76-c378-4a17-9fbf-f9a8d36160bb/storage-initializer/0.log"
Sep 17 18:54:13.524058 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:13.524022    2553 generic.go:356] "Generic (PLEG): container finished" podID="db982b76-c378-4a17-9fbf-f9a8d36160bb" containerID="f5d607512bc63b9bbc5e6dcadc37c4d31958dd090d89306526fa757e735409b2" exitCode=1
Sep 17 18:54:13.524134 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:13.524071    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-bc87e7-predictor-5c66fb9558-j7lx8" event={"ID":"db982b76-c378-4a17-9fbf-f9a8d36160bb","Type":"ContainerDied","Data":"f5d607512bc63b9bbc5e6dcadc37c4d31958dd090d89306526fa757e735409b2"}
Sep 17 18:54:13.524166 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:13.524143    2553 scope.go:117] "RemoveContainer" containerID="d27c3b1207bed9da06329889b5fb29d15e3d04e8365eb6895ee5f08639b54552"
Sep 17 18:54:13.524468 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:13.524449    2553 scope.go:117] "RemoveContainer" containerID="d27c3b1207bed9da06329889b5fb29d15e3d04e8365eb6895ee5f08639b54552"
Sep 17 18:54:13.535170 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:54:13.535143    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-bc87e7-predictor-5c66fb9558-j7lx8_kserve-ci-e2e-test(db982b76-c378-4a17-9fbf-f9a8d36160bb)\"" pod="kserve-ci-e2e-test/isvc-secondary-bc87e7-predictor-5c66fb9558-j7lx8" podUID="db982b76-c378-4a17-9fbf-f9a8d36160bb"
Sep 17 18:54:13.535375 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:54:13.535348    2553 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-bc87e7-predictor-5c66fb9558-j7lx8_kserve-ci-e2e-test_db982b76-c378-4a17-9fbf-f9a8d36160bb_0 in pod sandbox 2ddcfa33c892096b12dadae201edcd2b6524c783eb1e5489aee7f09e423faae4 from index: no such id: 'd27c3b1207bed9da06329889b5fb29d15e3d04e8365eb6895ee5f08639b54552'" containerID="d27c3b1207bed9da06329889b5fb29d15e3d04e8365eb6895ee5f08639b54552"
Sep 17 18:54:13.535458 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:13.535388    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d27c3b1207bed9da06329889b5fb29d15e3d04e8365eb6895ee5f08639b54552"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-bc87e7-predictor-5c66fb9558-j7lx8_kserve-ci-e2e-test_db982b76-c378-4a17-9fbf-f9a8d36160bb_0 in pod sandbox 2ddcfa33c892096b12dadae201edcd2b6524c783eb1e5489aee7f09e423faae4 from index: no such id: 'd27c3b1207bed9da06329889b5fb29d15e3d04e8365eb6895ee5f08639b54552'"
Sep 17 18:54:14.530287 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:14.530261    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-bc87e7-predictor-5c66fb9558-j7lx8_db982b76-c378-4a17-9fbf-f9a8d36160bb/storage-initializer/1.log"
Sep 17 18:54:19.199015 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:19.198979    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-bc87e7-predictor-5c66fb9558-j7lx8"]
Sep 17 18:54:19.233333 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:19.233287    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-bc87e7-predictor-565d888469-mkfsr"]
Sep 17 18:54:19.233990 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:19.233954    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-bc87e7-predictor-565d888469-mkfsr" podUID="a9b0df6e-ae3d-46c1-b890-b0df7edf5701" containerName="kube-rbac-proxy" containerID="cri-o://0c0df1774f35faf4fab4f10a7828e62b8b493534dae6577ba32a662526369769" gracePeriod=30
Sep 17 18:54:19.233990 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:19.233958    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-bc87e7-predictor-565d888469-mkfsr" podUID="a9b0df6e-ae3d-46c1-b890-b0df7edf5701" containerName="kserve-container" containerID="cri-o://f7224f0c5510b9ca27d9de6afd60bce09379e4d74946bae8ef3a04c5f9e3258f" gracePeriod=30
Sep 17 18:54:19.356469 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:19.356445    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-bc87e7-predictor-5c66fb9558-j7lx8_db982b76-c378-4a17-9fbf-f9a8d36160bb/storage-initializer/1.log"
Sep 17 18:54:19.358658 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:19.358595    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-bc87e7-predictor-5c66fb9558-j7lx8"
Sep 17 18:54:19.436011 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:19.435977    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5krw4\" (UniqueName: \"kubernetes.io/projected/db982b76-c378-4a17-9fbf-f9a8d36160bb-kube-api-access-5krw4\") pod \"db982b76-c378-4a17-9fbf-f9a8d36160bb\" (UID: \"db982b76-c378-4a17-9fbf-f9a8d36160bb\") "
Sep 17 18:54:19.436011 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:19.436013    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db982b76-c378-4a17-9fbf-f9a8d36160bb-kube-rbac-proxy-sar-config\") pod \"db982b76-c378-4a17-9fbf-f9a8d36160bb\" (UID: \"db982b76-c378-4a17-9fbf-f9a8d36160bb\") "
Sep 17 18:54:19.436281 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:19.436041    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db982b76-c378-4a17-9fbf-f9a8d36160bb-cabundle-cert\") pod \"db982b76-c378-4a17-9fbf-f9a8d36160bb\" (UID: \"db982b76-c378-4a17-9fbf-f9a8d36160bb\") "
Sep 17 18:54:19.436281 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:19.436164    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db982b76-c378-4a17-9fbf-f9a8d36160bb-kserve-provision-location\") pod \"db982b76-c378-4a17-9fbf-f9a8d36160bb\" (UID: \"db982b76-c378-4a17-9fbf-f9a8d36160bb\") "
Sep 17 18:54:19.436281 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:19.436211    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db982b76-c378-4a17-9fbf-f9a8d36160bb-proxy-tls\") pod \"db982b76-c378-4a17-9fbf-f9a8d36160bb\" (UID: \"db982b76-c378-4a17-9fbf-f9a8d36160bb\") "
Sep 17 18:54:19.436479 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:19.436452    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db982b76-c378-4a17-9fbf-f9a8d36160bb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "db982b76-c378-4a17-9fbf-f9a8d36160bb" (UID: "db982b76-c378-4a17-9fbf-f9a8d36160bb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:54:19.436535 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:19.436460    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db982b76-c378-4a17-9fbf-f9a8d36160bb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "db982b76-c378-4a17-9fbf-f9a8d36160bb" (UID: "db982b76-c378-4a17-9fbf-f9a8d36160bb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:54:19.436535 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:19.436455    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db982b76-c378-4a17-9fbf-f9a8d36160bb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "db982b76-c378-4a17-9fbf-f9a8d36160bb" (UID: "db982b76-c378-4a17-9fbf-f9a8d36160bb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:54:19.438165 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:19.438137    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db982b76-c378-4a17-9fbf-f9a8d36160bb-kube-api-access-5krw4" (OuterVolumeSpecName: "kube-api-access-5krw4") pod "db982b76-c378-4a17-9fbf-f9a8d36160bb" (UID: "db982b76-c378-4a17-9fbf-f9a8d36160bb"). InnerVolumeSpecName "kube-api-access-5krw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:54:19.438322 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:19.438298    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db982b76-c378-4a17-9fbf-f9a8d36160bb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "db982b76-c378-4a17-9fbf-f9a8d36160bb" (UID: "db982b76-c378-4a17-9fbf-f9a8d36160bb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:54:19.537500 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:19.537412    2553 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db982b76-c378-4a17-9fbf-f9a8d36160bb-cabundle-cert\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:54:19.537500 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:19.537443    2553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db982b76-c378-4a17-9fbf-f9a8d36160bb-kserve-provision-location\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:54:19.537500 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:19.537454    2553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db982b76-c378-4a17-9fbf-f9a8d36160bb-proxy-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:54:19.537500 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:19.537465    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5krw4\" (UniqueName: \"kubernetes.io/projected/db982b76-c378-4a17-9fbf-f9a8d36160bb-kube-api-access-5krw4\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:54:19.537500 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:19.537474    2553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db982b76-c378-4a17-9fbf-f9a8d36160bb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:54:19.557801 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:19.557766    2553 generic.go:356] "Generic (PLEG): container finished" podID="a9b0df6e-ae3d-46c1-b890-b0df7edf5701" containerID="0c0df1774f35faf4fab4f10a7828e62b8b493534dae6577ba32a662526369769" exitCode=2
Sep 17 18:54:19.557965 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:19.557841    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-bc87e7-predictor-565d888469-mkfsr" event={"ID":"a9b0df6e-ae3d-46c1-b890-b0df7edf5701","Type":"ContainerDied","Data":"0c0df1774f35faf4fab4f10a7828e62b8b493534dae6577ba32a662526369769"}
Sep 17 18:54:19.559071 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:19.559051    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-bc87e7-predictor-5c66fb9558-j7lx8_db982b76-c378-4a17-9fbf-f9a8d36160bb/storage-initializer/1.log"
Sep 17 18:54:19.559179 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:19.559143    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-bc87e7-predictor-5c66fb9558-j7lx8"
Sep 17 18:54:19.559179 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:19.559152    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-bc87e7-predictor-5c66fb9558-j7lx8" event={"ID":"db982b76-c378-4a17-9fbf-f9a8d36160bb","Type":"ContainerDied","Data":"2ddcfa33c892096b12dadae201edcd2b6524c783eb1e5489aee7f09e423faae4"}
Sep 17 18:54:19.559268 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:19.559211    2553 scope.go:117] "RemoveContainer" containerID="f5d607512bc63b9bbc5e6dcadc37c4d31958dd090d89306526fa757e735409b2"
Sep 17 18:54:19.601359 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:19.601323    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-bc87e7-predictor-5c66fb9558-j7lx8"]
Sep 17 18:54:19.605436 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:19.605410    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-bc87e7-predictor-5c66fb9558-j7lx8"]
Sep 17 18:54:20.371370 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:20.371327    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-bc87e7-predictor-565d888469-mkfsr" podUID="a9b0df6e-ae3d-46c1-b890-b0df7edf5701" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 17 18:54:21.375802 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:21.375766    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db982b76-c378-4a17-9fbf-f9a8d36160bb" path="/var/lib/kubelet/pods/db982b76-c378-4a17-9fbf-f9a8d36160bb/volumes"
Sep 17 18:54:23.030514 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:23.030470    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-bc87e7-predictor-565d888469-mkfsr" podUID="a9b0df6e-ae3d-46c1-b890-b0df7edf5701" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.60:8643/healthz\": dial tcp 10.134.0.60:8643: connect: connection refused"
Sep 17 18:54:24.187837 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:24.187814    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-bc87e7-predictor-565d888469-mkfsr"
Sep 17 18:54:24.281140 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:24.281041    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a9b0df6e-ae3d-46c1-b890-b0df7edf5701-kserve-provision-location\") pod \"a9b0df6e-ae3d-46c1-b890-b0df7edf5701\" (UID: \"a9b0df6e-ae3d-46c1-b890-b0df7edf5701\") "
Sep 17 18:54:24.281140 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:24.281082    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9b0df6e-ae3d-46c1-b890-b0df7edf5701-proxy-tls\") pod \"a9b0df6e-ae3d-46c1-b890-b0df7edf5701\" (UID: \"a9b0df6e-ae3d-46c1-b890-b0df7edf5701\") "
Sep 17 18:54:24.281140 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:24.281110    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9b0df6e-ae3d-46c1-b890-b0df7edf5701-kube-rbac-proxy-sar-config\") pod \"a9b0df6e-ae3d-46c1-b890-b0df7edf5701\" (UID: \"a9b0df6e-ae3d-46c1-b890-b0df7edf5701\") "
Sep 17 18:54:24.281442 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:24.281164    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g46rl\" (UniqueName: \"kubernetes.io/projected/a9b0df6e-ae3d-46c1-b890-b0df7edf5701-kube-api-access-g46rl\") pod \"a9b0df6e-ae3d-46c1-b890-b0df7edf5701\" (UID: \"a9b0df6e-ae3d-46c1-b890-b0df7edf5701\") "
Sep 17 18:54:24.281442 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:24.281188    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a9b0df6e-ae3d-46c1-b890-b0df7edf5701-cabundle-cert\") pod \"a9b0df6e-ae3d-46c1-b890-b0df7edf5701\" (UID: \"a9b0df6e-ae3d-46c1-b890-b0df7edf5701\") "
Sep 17 18:54:24.281547 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:24.281439    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9b0df6e-ae3d-46c1-b890-b0df7edf5701-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a9b0df6e-ae3d-46c1-b890-b0df7edf5701" (UID: "a9b0df6e-ae3d-46c1-b890-b0df7edf5701"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:54:24.281608 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:24.281548    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9b0df6e-ae3d-46c1-b890-b0df7edf5701-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a9b0df6e-ae3d-46c1-b890-b0df7edf5701" (UID: "a9b0df6e-ae3d-46c1-b890-b0df7edf5701"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:54:24.281608 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:24.281589    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9b0df6e-ae3d-46c1-b890-b0df7edf5701-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a9b0df6e-ae3d-46c1-b890-b0df7edf5701" (UID: "a9b0df6e-ae3d-46c1-b890-b0df7edf5701"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:54:24.283281 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:24.283261    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9b0df6e-ae3d-46c1-b890-b0df7edf5701-kube-api-access-g46rl" (OuterVolumeSpecName: "kube-api-access-g46rl") pod "a9b0df6e-ae3d-46c1-b890-b0df7edf5701" (UID: "a9b0df6e-ae3d-46c1-b890-b0df7edf5701"). InnerVolumeSpecName "kube-api-access-g46rl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:54:24.283351 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:24.283334    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9b0df6e-ae3d-46c1-b890-b0df7edf5701-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a9b0df6e-ae3d-46c1-b890-b0df7edf5701" (UID: "a9b0df6e-ae3d-46c1-b890-b0df7edf5701"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:54:24.381918 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:24.381884    2553 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a9b0df6e-ae3d-46c1-b890-b0df7edf5701-cabundle-cert\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:54:24.381918 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:24.381912    2553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a9b0df6e-ae3d-46c1-b890-b0df7edf5701-kserve-provision-location\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:54:24.381918 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:24.381922    2553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9b0df6e-ae3d-46c1-b890-b0df7edf5701-proxy-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:54:24.382152 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:24.381933    2553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9b0df6e-ae3d-46c1-b890-b0df7edf5701-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:54:24.382152 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:24.381941    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g46rl\" (UniqueName: \"kubernetes.io/projected/a9b0df6e-ae3d-46c1-b890-b0df7edf5701-kube-api-access-g46rl\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\""
Sep 17 18:54:24.585244 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:24.585164    2553 generic.go:356] "Generic (PLEG): container finished" podID="a9b0df6e-ae3d-46c1-b890-b0df7edf5701" containerID="f7224f0c5510b9ca27d9de6afd60bce09379e4d74946bae8ef3a04c5f9e3258f" exitCode=0
Sep 17 18:54:24.585244 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:24.585218    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-bc87e7-predictor-565d888469-mkfsr"
Sep 17 18:54:24.585438 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:24.585221    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-bc87e7-predictor-565d888469-mkfsr" event={"ID":"a9b0df6e-ae3d-46c1-b890-b0df7edf5701","Type":"ContainerDied","Data":"f7224f0c5510b9ca27d9de6afd60bce09379e4d74946bae8ef3a04c5f9e3258f"}
Sep 17 18:54:24.585438 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:24.585365    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-bc87e7-predictor-565d888469-mkfsr" event={"ID":"a9b0df6e-ae3d-46c1-b890-b0df7edf5701","Type":"ContainerDied","Data":"836026fc41b65c7fed9af601b4d14994da8e1b6b3bc0dee10bb2ba0f1ed7a6be"}
Sep 17 18:54:24.585438 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:24.585402    2553 scope.go:117] "RemoveContainer" containerID="0c0df1774f35faf4fab4f10a7828e62b8b493534dae6577ba32a662526369769"
Sep 17 18:54:24.594182 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:24.594164    2553 scope.go:117] "RemoveContainer" containerID="f7224f0c5510b9ca27d9de6afd60bce09379e4d74946bae8ef3a04c5f9e3258f"
Sep 17 18:54:24.601845 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:24.601825    2553 scope.go:117] "RemoveContainer" containerID="9aadee85c84b06e011fd1f48fbd54f131644a98b6ce8ab181ac178db28af1f13"
Sep 17 18:54:24.606326 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:24.606301    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-bc87e7-predictor-565d888469-mkfsr"]
Sep 17 18:54:24.609436 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:24.609408    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-bc87e7-predictor-565d888469-mkfsr"]
Sep 17 18:54:24.610298 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:24.610282    2553 scope.go:117] "RemoveContainer" containerID="0c0df1774f35faf4fab4f10a7828e62b8b493534dae6577ba32a662526369769"
Sep 17 18:54:24.610614 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:54:24.610596    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c0df1774f35faf4fab4f10a7828e62b8b493534dae6577ba32a662526369769\": container with ID starting with 0c0df1774f35faf4fab4f10a7828e62b8b493534dae6577ba32a662526369769 not found: ID does not exist" containerID="0c0df1774f35faf4fab4f10a7828e62b8b493534dae6577ba32a662526369769"
Sep 17 18:54:24.610721 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:24.610641    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c0df1774f35faf4fab4f10a7828e62b8b493534dae6577ba32a662526369769"} err="failed to get container status \"0c0df1774f35faf4fab4f10a7828e62b8b493534dae6577ba32a662526369769\": rpc error: code = NotFound desc = could not find container \"0c0df1774f35faf4fab4f10a7828e62b8b493534dae6577ba32a662526369769\": container with ID starting with 0c0df1774f35faf4fab4f10a7828e62b8b493534dae6577ba32a662526369769 not found: ID does not exist"
Sep 17 18:54:24.610721 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:24.610665    2553 scope.go:117] "RemoveContainer" containerID="f7224f0c5510b9ca27d9de6afd60bce09379e4d74946bae8ef3a04c5f9e3258f"
Sep 17 18:54:24.610953 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:54:24.610935    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7224f0c5510b9ca27d9de6afd60bce09379e4d74946bae8ef3a04c5f9e3258f\": container with ID starting with f7224f0c5510b9ca27d9de6afd60bce09379e4d74946bae8ef3a04c5f9e3258f not found: ID does not exist" containerID="f7224f0c5510b9ca27d9de6afd60bce09379e4d74946bae8ef3a04c5f9e3258f"
Sep 17 18:54:24.610993 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:24.610959    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7224f0c5510b9ca27d9de6afd60bce09379e4d74946bae8ef3a04c5f9e3258f"} err="failed to get container status \"f7224f0c5510b9ca27d9de6afd60bce09379e4d74946bae8ef3a04c5f9e3258f\": rpc error: code = NotFound desc = could not find container \"f7224f0c5510b9ca27d9de6afd60bce09379e4d74946bae8ef3a04c5f9e3258f\": container with ID starting with f7224f0c5510b9ca27d9de6afd60bce09379e4d74946bae8ef3a04c5f9e3258f not found: ID does not exist"
Sep 17 18:54:24.610993 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:24.610974    2553 scope.go:117] "RemoveContainer" containerID="9aadee85c84b06e011fd1f48fbd54f131644a98b6ce8ab181ac178db28af1f13"
Sep 17 18:54:24.611212 ip-10-0-137-95 kubenswrapper[2553]: E0917 18:54:24.611193    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9aadee85c84b06e011fd1f48fbd54f131644a98b6ce8ab181ac178db28af1f13\": container with ID starting with 9aadee85c84b06e011fd1f48fbd54f131644a98b6ce8ab181ac178db28af1f13 not found: ID does not exist" containerID="9aadee85c84b06e011fd1f48fbd54f131644a98b6ce8ab181ac178db28af1f13"
Sep 17 18:54:24.611262 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:24.611220    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9aadee85c84b06e011fd1f48fbd54f131644a98b6ce8ab181ac178db28af1f13"} err="failed to get container status \"9aadee85c84b06e011fd1f48fbd54f131644a98b6ce8ab181ac178db28af1f13\": rpc error: code = NotFound desc = could not find container \"9aadee85c84b06e011fd1f48fbd54f131644a98b6ce8ab181ac178db28af1f13\": container with ID starting with 9aadee85c84b06e011fd1f48fbd54f131644a98b6ce8ab181ac178db28af1f13 not found: ID does not exist"
Sep 17 18:54:25.376395 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:54:25.376358    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9b0df6e-ae3d-46c1-b890-b0df7edf5701" path="/var/lib/kubelet/pods/a9b0df6e-ae3d-46c1-b890-b0df7edf5701/volumes"
Sep 17 18:55:04.560567 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:04.560492    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-xr8hx_45254d28-1b05-4ddd-8ea9-22f6fb0eb3eb/global-pull-secret-syncer/0.log"
Sep 17 18:55:04.658011 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:04.657978    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-jz66f_b281f20e-0314-4794-98c1-50e5cba38212/konnectivity-agent/0.log"
Sep 17 18:55:04.782104 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:04.782066    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-95.ec2.internal_01e1e8be5fa1c8721ccc54011f518b1a/haproxy/0.log"
Sep 17 18:55:09.393246 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:09.393215    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-847b8bd8d7-c9mxg_8433dbe1-addc-4e5e-acdb-381db8334300/cluster-monitoring-operator/0.log"
Sep 17 18:55:09.422204 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:09.422167    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-fbfzt_9eb07b72-ea20-4e7b-b92f-7bbf3a882223/kube-state-metrics/0.log"
Sep 17 18:55:09.445952 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:09.445925    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-fbfzt_9eb07b72-ea20-4e7b-b92f-7bbf3a882223/kube-rbac-proxy-main/0.log"
Sep 17 18:55:09.476948 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:09.476920    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-fbfzt_9eb07b72-ea20-4e7b-b92f-7bbf3a882223/kube-rbac-proxy-self/0.log"
Sep 17 18:55:09.540334 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:09.540307    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5984459ff8-b5zt5_e1a66bb3-53a8-4e8d-be66-ecf547963837/monitoring-plugin/0.log"
Sep 17 18:55:09.574545 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:09.574520    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6w2lq_9592822d-5079-442d-94ec-242a4f10ee82/node-exporter/0.log"
Sep 17 18:55:09.606650 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:09.606611    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6w2lq_9592822d-5079-442d-94ec-242a4f10ee82/kube-rbac-proxy/0.log"
Sep 17 18:55:09.635638 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:09.635606    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6w2lq_9592822d-5079-442d-94ec-242a4f10ee82/init-textfile/0.log"
Sep 17 18:55:10.101022 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:10.100992    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-gbkmm_df8d85da-2c74-414a-9aa9-98270323c887/prometheus-operator/0.log"
Sep 17 18:55:10.121877 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:10.121854    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-gbkmm_df8d85da-2c74-414a-9aa9-98270323c887/kube-rbac-proxy/0.log"
Sep 17 18:55:12.575579 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:12.575554    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6d7695cc68-g67w9_3717632b-23a0-4096-98a3-f9ee8352293b/console/0.log"
Sep 17 18:55:13.018339 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:13.018303    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5mfgc/perf-node-gather-daemonset-9cdkp"]
Sep 17 18:55:13.018660 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:13.018644    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9b0df6e-ae3d-46c1-b890-b0df7edf5701" containerName="kube-rbac-proxy"
Sep 17 18:55:13.018715 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:13.018661    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9b0df6e-ae3d-46c1-b890-b0df7edf5701" containerName="kube-rbac-proxy"
Sep 17 18:55:13.018715 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:13.018668    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db982b76-c378-4a17-9fbf-f9a8d36160bb" containerName="storage-initializer"
Sep 17 18:55:13.018715 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:13.018674    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="db982b76-c378-4a17-9fbf-f9a8d36160bb" containerName="storage-initializer"
Sep 17 18:55:13.018715 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:13.018689    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9b0df6e-ae3d-46c1-b890-b0df7edf5701" containerName="storage-initializer"
Sep 17 18:55:13.018715 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:13.018694    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9b0df6e-ae3d-46c1-b890-b0df7edf5701" containerName="storage-initializer"
Sep 17 18:55:13.018715 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:13.018705    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9b0df6e-ae3d-46c1-b890-b0df7edf5701" containerName="kserve-container"
Sep 17 18:55:13.018715 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:13.018710    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9b0df6e-ae3d-46c1-b890-b0df7edf5701" containerName="kserve-container"
Sep 17 18:55:13.018931 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:13.018758    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="db982b76-c378-4a17-9fbf-f9a8d36160bb" containerName="storage-initializer"
Sep 17 18:55:13.018931 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:13.018768    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9b0df6e-ae3d-46c1-b890-b0df7edf5701" containerName="kube-rbac-proxy"
Sep 17 18:55:13.018931 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:13.018775    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9b0df6e-ae3d-46c1-b890-b0df7edf5701" containerName="kserve-container"
Sep 17 18:55:13.021590 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:13.021568    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5mfgc/perf-node-gather-daemonset-9cdkp"
Sep 17 18:55:13.024145 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:13.024122    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5mfgc\"/\"kube-root-ca.crt\""
Sep 17 18:55:13.025241 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:13.025223    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5mfgc\"/\"default-dockercfg-mm674\""
Sep 17 18:55:13.025354 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:13.025254    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5mfgc\"/\"openshift-service-ca.crt\""
Sep 17 18:55:13.034299 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:13.034270    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5mfgc/perf-node-gather-daemonset-9cdkp"]
Sep 17 18:55:13.111376 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:13.111338    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f9c74ffc-a36e-497f-98ad-542becd0c9e3-sys\") pod \"perf-node-gather-daemonset-9cdkp\" (UID: \"f9c74ffc-a36e-497f-98ad-542becd0c9e3\") " pod="openshift-must-gather-5mfgc/perf-node-gather-daemonset-9cdkp"
Sep 17 18:55:13.111376 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:13.111377    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f9c74ffc-a36e-497f-98ad-542becd0c9e3-podres\") pod \"perf-node-gather-daemonset-9cdkp\" (UID: \"f9c74ffc-a36e-497f-98ad-542becd0c9e3\") " pod="openshift-must-gather-5mfgc/perf-node-gather-daemonset-9cdkp"
Sep 17 18:55:13.111581 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:13.111399    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f9c74ffc-a36e-497f-98ad-542becd0c9e3-proc\") pod \"perf-node-gather-daemonset-9cdkp\" (UID: \"f9c74ffc-a36e-497f-98ad-542becd0c9e3\") " pod="openshift-must-gather-5mfgc/perf-node-gather-daemonset-9cdkp"
Sep 17 18:55:13.111581 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:13.111416    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbnc7\" (UniqueName: \"kubernetes.io/projected/f9c74ffc-a36e-497f-98ad-542becd0c9e3-kube-api-access-xbnc7\") pod \"perf-node-gather-daemonset-9cdkp\" (UID: \"f9c74ffc-a36e-497f-98ad-542becd0c9e3\") " pod="openshift-must-gather-5mfgc/perf-node-gather-daemonset-9cdkp"
Sep 17 18:55:13.111581 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:13.111488    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f9c74ffc-a36e-497f-98ad-542becd0c9e3-lib-modules\") pod \"perf-node-gather-daemonset-9cdkp\" (UID: \"f9c74ffc-a36e-497f-98ad-542becd0c9e3\") " pod="openshift-must-gather-5mfgc/perf-node-gather-daemonset-9cdkp"
Sep 17 18:55:13.212246 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:13.212206    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f9c74ffc-a36e-497f-98ad-542becd0c9e3-sys\") pod \"perf-node-gather-daemonset-9cdkp\" (UID: \"f9c74ffc-a36e-497f-98ad-542becd0c9e3\") " pod="openshift-must-gather-5mfgc/perf-node-gather-daemonset-9cdkp"
Sep 17 18:55:13.212246 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:13.212245    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f9c74ffc-a36e-497f-98ad-542becd0c9e3-podres\") pod \"perf-node-gather-daemonset-9cdkp\" (UID: \"f9c74ffc-a36e-497f-98ad-542becd0c9e3\") " pod="openshift-must-gather-5mfgc/perf-node-gather-daemonset-9cdkp"
Sep 17 18:55:13.212493 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:13.212268    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f9c74ffc-a36e-497f-98ad-542becd0c9e3-proc\") pod \"perf-node-gather-daemonset-9cdkp\" (UID: \"f9c74ffc-a36e-497f-98ad-542becd0c9e3\") " pod="openshift-must-gather-5mfgc/perf-node-gather-daemonset-9cdkp"
Sep 17 18:55:13.212493 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:13.212283    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbnc7\" (UniqueName: \"kubernetes.io/projected/f9c74ffc-a36e-497f-98ad-542becd0c9e3-kube-api-access-xbnc7\") pod \"perf-node-gather-daemonset-9cdkp\" (UID: \"f9c74ffc-a36e-497f-98ad-542becd0c9e3\") " pod="openshift-must-gather-5mfgc/perf-node-gather-daemonset-9cdkp"
Sep 17 18:55:13.212493 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:13.212342    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f9c74ffc-a36e-497f-98ad-542becd0c9e3-sys\") pod \"perf-node-gather-daemonset-9cdkp\" (UID: \"f9c74ffc-a36e-497f-98ad-542becd0c9e3\") " pod="openshift-must-gather-5mfgc/perf-node-gather-daemonset-9cdkp"
Sep 17 18:55:13.212493 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:13.212359    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f9c74ffc-a36e-497f-98ad-542becd0c9e3-proc\") pod \"perf-node-gather-daemonset-9cdkp\" (UID: \"f9c74ffc-a36e-497f-98ad-542becd0c9e3\") " pod="openshift-must-gather-5mfgc/perf-node-gather-daemonset-9cdkp"
Sep 17 18:55:13.212493 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:13.212383    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f9c74ffc-a36e-497f-98ad-542becd0c9e3-podres\") pod \"perf-node-gather-daemonset-9cdkp\" (UID: \"f9c74ffc-a36e-497f-98ad-542becd0c9e3\") " pod="openshift-must-gather-5mfgc/perf-node-gather-daemonset-9cdkp"
Sep 17 18:55:13.212493 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:13.212404    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f9c74ffc-a36e-497f-98ad-542becd0c9e3-lib-modules\") pod \"perf-node-gather-daemonset-9cdkp\" (UID: \"f9c74ffc-a36e-497f-98ad-542becd0c9e3\") " pod="openshift-must-gather-5mfgc/perf-node-gather-daemonset-9cdkp"
Sep 17 18:55:13.212727 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:13.212542    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f9c74ffc-a36e-497f-98ad-542becd0c9e3-lib-modules\") pod \"perf-node-gather-daemonset-9cdkp\" (UID: \"f9c74ffc-a36e-497f-98ad-542becd0c9e3\") " pod="openshift-must-gather-5mfgc/perf-node-gather-daemonset-9cdkp"
Sep 17 18:55:13.219499 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:13.219472    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbnc7\" (UniqueName: \"kubernetes.io/projected/f9c74ffc-a36e-497f-98ad-542becd0c9e3-kube-api-access-xbnc7\") pod \"perf-node-gather-daemonset-9cdkp\" (UID: \"f9c74ffc-a36e-497f-98ad-542becd0c9e3\") " pod="openshift-must-gather-5mfgc/perf-node-gather-daemonset-9cdkp"
Sep 17 18:55:13.331656 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:13.331548    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5mfgc/perf-node-gather-daemonset-9cdkp"
Sep 17 18:55:13.601777 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:13.601751    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5mfgc/perf-node-gather-daemonset-9cdkp"]
Sep 17 18:55:13.604108 ip-10-0-137-95 kubenswrapper[2553]: W0917 18:55:13.604072    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf9c74ffc_a36e_497f_98ad_542becd0c9e3.slice/crio-0506c060b43e7d46d5c09dc3d68b1ddbf5aa1a2c7fc2bf8ec0d52065e1e2652a WatchSource:0}: Error finding container 0506c060b43e7d46d5c09dc3d68b1ddbf5aa1a2c7fc2bf8ec0d52065e1e2652a: Status 404 returned error can't find the container with id 0506c060b43e7d46d5c09dc3d68b1ddbf5aa1a2c7fc2bf8ec0d52065e1e2652a
Sep 17 18:55:13.804684 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:13.804647    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5mfgc/perf-node-gather-daemonset-9cdkp" event={"ID":"f9c74ffc-a36e-497f-98ad-542becd0c9e3","Type":"ContainerStarted","Data":"e9a0666a01d33da6ef385f3a65962e183efbf6bbd633ceb4c3d1ab747ab0aa46"}
Sep 17 18:55:13.804854 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:13.804742    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5mfgc/perf-node-gather-daemonset-9cdkp" event={"ID":"f9c74ffc-a36e-497f-98ad-542becd0c9e3","Type":"ContainerStarted","Data":"0506c060b43e7d46d5c09dc3d68b1ddbf5aa1a2c7fc2bf8ec0d52065e1e2652a"}
Sep 17 18:55:13.804854 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:13.804769    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5mfgc/perf-node-gather-daemonset-9cdkp"
Sep 17 18:55:13.821102 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:13.821043    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5mfgc/perf-node-gather-daemonset-9cdkp" podStartSLOduration=0.821026902 podStartE2EDuration="821.026902ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:55:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:55:13.81996524 +0000 UTC m=+3508.516055925" watchObservedRunningTime="2026-09-17 18:55:13.821026902 +0000 UTC m=+3508.517117656"
Sep 17 18:55:14.076500 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:14.076471    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dm5l8_05707133-87f8-4036-89b3-f783c8f5e563/dns/0.log"
Sep 17 18:55:14.103400 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:14.103373    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dm5l8_05707133-87f8-4036-89b3-f783c8f5e563/kube-rbac-proxy/0.log"
Sep 17 18:55:14.334146 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:14.334069    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4fdj7_56b9ff80-4b68-4449-bd38-e2e4d2de6aa2/dns-node-resolver/0.log"
Sep 17 18:55:14.855341 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:14.855311    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5b5767bf9b-cckrb_f467fe4f-0b64-4a65-9cf1-d1aebdb1baa1/registry/0.log"
Sep 17 18:55:14.877131 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:14.877106    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6pxfw_c5297750-e3a6-4690-9f70-4be7eac03a96/node-ca/0.log"
Sep 17 18:55:16.148881 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:16.148842    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7pqps_94bf369d-1ce4-4ed5-bf52-c45425223e50/serve-healthcheck-canary/0.log"
Sep 17 18:55:16.583693 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:16.583597    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-wrv4g_bb30f947-1c4d-478f-95b1-939d82415f95/insights-operator/0.log"
Sep 17 18:55:16.583946 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:16.583925    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-wrv4g_bb30f947-1c4d-478f-95b1-939d82415f95/insights-operator/1.log"
Sep 17 18:55:16.745065 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:16.745037    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-s782r_8c882b94-d6bf-422b-b04f-13232d9828f3/kube-rbac-proxy/0.log"
Sep 17 18:55:16.769194 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:16.769169    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-s782r_8c882b94-d6bf-422b-b04f-13232d9828f3/exporter/0.log"
Sep 17 18:55:16.792883 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:16.792857    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-s782r_8c882b94-d6bf-422b-b04f-13232d9828f3/extractor/0.log"
Sep 17 18:55:19.521022 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:19.520995    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-645f9c8fd6-bt5l6_82cc79ef-12c8-4304-934c-0528003d61e2/manager/0.log"
Sep 17 18:55:19.543488 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:19.543461    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-p7rj6_a72ece66-1bea-49e9-95c4-791546aab450/server/0.log"
Sep 17 18:55:19.819937 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:19.819843    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5mfgc/perf-node-gather-daemonset-9cdkp"
Sep 17 18:55:20.340132 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:20.340103    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-serving-67b7477c6d-vs9tv_97400e98-445c-42f0-8b53-b98a8e9ddfd2/seaweedfs-tls-serving/0.log"
Sep 17 18:55:25.094038 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:25.094001    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-sxqrz_1ec26660-d938-4072-b5da-eaa607ea5ba5/kube-storage-version-migrator-operator/1.log"
Sep 17 18:55:25.095366 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:25.095339    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-sxqrz_1ec26660-d938-4072-b5da-eaa607ea5ba5/kube-storage-version-migrator-operator/0.log"
Sep 17 18:55:25.940463 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:25.940431    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bmjrp_4e137835-2bdc-4b32-aa6b-64635bdc26d7/kube-multus-additional-cni-plugins/0.log"
Sep 17 18:55:25.963764 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:25.963737    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bmjrp_4e137835-2bdc-4b32-aa6b-64635bdc26d7/egress-router-binary-copy/0.log"
Sep 17 18:55:25.986684 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:25.986658    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bmjrp_4e137835-2bdc-4b32-aa6b-64635bdc26d7/cni-plugins/0.log"
Sep 17 18:55:26.007758 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:26.007723    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bmjrp_4e137835-2bdc-4b32-aa6b-64635bdc26d7/bond-cni-plugin/0.log"
Sep 17 18:55:26.028272 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:26.028234    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bmjrp_4e137835-2bdc-4b32-aa6b-64635bdc26d7/routeoverride-cni/0.log"
Sep 17 18:55:26.052645 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:26.052607    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bmjrp_4e137835-2bdc-4b32-aa6b-64635bdc26d7/whereabouts-cni-bincopy/0.log"
Sep 17 18:55:26.073030 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:26.072999    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bmjrp_4e137835-2bdc-4b32-aa6b-64635bdc26d7/whereabouts-cni/0.log"
Sep 17 18:55:26.509730 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:26.509700    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-tjrts_855a7f82-274d-4b6d-bcde-0c6f4c19016b/kube-multus/0.log"
Sep 17 18:55:26.531697 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:26.531666    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-5gdn2_dfc1cb73-d5de-4902-b16c-5744271855c5/network-metrics-daemon/0.log"
Sep 17 18:55:26.552674 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:26.552648    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-5gdn2_dfc1cb73-d5de-4902-b16c-5744271855c5/kube-rbac-proxy/0.log"
Sep 17 18:55:27.379151 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:27.379117    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-btmqz_622ee5be-fb50-42b6-ae23-42900b7c2d98/ovn-controller/0.log"
Sep 17 18:55:27.397501 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:27.397468    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-btmqz_622ee5be-fb50-42b6-ae23-42900b7c2d98/ovn-acl-logging/1.log"
Sep 17 18:55:27.397501 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:27.397517    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-btmqz_622ee5be-fb50-42b6-ae23-42900b7c2d98/ovn-acl-logging/0.log"
Sep 17 18:55:27.416349 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:27.416310    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-btmqz_622ee5be-fb50-42b6-ae23-42900b7c2d98/kube-rbac-proxy-node/0.log"
Sep 17 18:55:27.438563 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:27.438532    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-btmqz_622ee5be-fb50-42b6-ae23-42900b7c2d98/kube-rbac-proxy-ovn-metrics/0.log"
Sep 17 18:55:27.461138 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:27.461111    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-btmqz_622ee5be-fb50-42b6-ae23-42900b7c2d98/northd/0.log"
Sep 17 18:55:27.485506 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:27.485475    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-btmqz_622ee5be-fb50-42b6-ae23-42900b7c2d98/nbdb/0.log"
Sep 17 18:55:27.509947 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:27.509913    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-btmqz_622ee5be-fb50-42b6-ae23-42900b7c2d98/sbdb/0.log"
Sep 17 18:55:27.556579 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:27.556547    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-btmqz_622ee5be-fb50-42b6-ae23-42900b7c2d98/ovnkube-controller/0.log"
Sep 17 18:55:28.996634 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:28.996598    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-lkn4v_cb953471-4308-48e7-9316-6b0bdf2b8738/network-check-target-container/0.log"
Sep 17 18:55:30.014479 ip-10-0-137-95 kubenswrapper[2553]: I0917 18:55:30.014449    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-p6z75_4468ff09-5965-493f-ab71-8ccfb33fbb61/iptables-alerter/0.log"