Sep 17 21:58:07.355645 ip-10-0-138-45 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 21:58:10.714542 ip-10-0-138-45 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 21:58:10.714542 ip-10-0-138-45 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 21:58:10.714542 ip-10-0-138-45 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 21:58:10.714542 ip-10-0-138-45 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 21:58:10.714542 ip-10-0-138-45 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 21:58:10.715373 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.714617    2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 21:58:10.718640 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718618    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 21:58:10.718640 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718634    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 21:58:10.718640 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718638    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 21:58:10.718640 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718641    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 21:58:10.718640 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718644    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 21:58:10.718640 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718647    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 21:58:10.718855 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718650    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 21:58:10.718855 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718653    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 21:58:10.718855 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718655    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 21:58:10.718855 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718659    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 21:58:10.718855 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718662    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 21:58:10.718855 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718665    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 21:58:10.718855 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718668    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 21:58:10.718855 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718671    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 21:58:10.718855 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718674    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 21:58:10.718855 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718676    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 21:58:10.718855 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718679    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 21:58:10.718855 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718682    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 21:58:10.718855 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718685    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 21:58:10.718855 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718687    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 21:58:10.718855 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718690    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 21:58:10.718855 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718693    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 21:58:10.718855 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718695    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 21:58:10.718855 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718698    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 21:58:10.718855 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718701    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 21:58:10.718855 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718703    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 21:58:10.719401 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718706    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 21:58:10.719401 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718708    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 21:58:10.719401 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718711    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 21:58:10.719401 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718714    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 21:58:10.719401 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718716    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 21:58:10.719401 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718724    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 21:58:10.719401 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718727    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 21:58:10.719401 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718730    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 21:58:10.719401 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718732    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 21:58:10.719401 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718735    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 21:58:10.719401 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718738    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 21:58:10.719401 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718740    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 21:58:10.719401 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718743    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 21:58:10.719401 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718746    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 21:58:10.719401 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718749    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 21:58:10.719401 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718752    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 21:58:10.719401 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718754    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 21:58:10.719401 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718757    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 21:58:10.719401 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718760    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 21:58:10.719401 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718762    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 21:58:10.719894 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718765    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 21:58:10.719894 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718768    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 21:58:10.719894 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718771    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 21:58:10.719894 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718773    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 21:58:10.719894 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718776    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 21:58:10.719894 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718778    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 21:58:10.719894 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718782    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 21:58:10.719894 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718784    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 21:58:10.719894 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718787    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 21:58:10.719894 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718790    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 21:58:10.719894 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718792    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 21:58:10.719894 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718795    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 21:58:10.719894 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718798    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 21:58:10.719894 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718801    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 21:58:10.719894 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718805    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 21:58:10.719894 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718809    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 21:58:10.719894 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718812    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 21:58:10.719894 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718815    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 21:58:10.719894 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718818    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 21:58:10.719894 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718821    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 21:58:10.720393 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718825    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 21:58:10.720393 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718830    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 21:58:10.720393 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718833    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 21:58:10.720393 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718836    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 21:58:10.720393 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718839    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 21:58:10.720393 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718843    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 21:58:10.720393 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718846    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 21:58:10.720393 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718849    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 21:58:10.720393 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718852    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 21:58:10.720393 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718855    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 21:58:10.720393 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718858    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 21:58:10.720393 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718861    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 21:58:10.720393 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718864    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 21:58:10.720393 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718867    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 21:58:10.720393 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718870    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 21:58:10.720393 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718873    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 21:58:10.720393 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718875    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 21:58:10.720393 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718885    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 21:58:10.720393 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718888    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 21:58:10.720859 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718890    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 21:58:10.720859 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718894    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 21:58:10.720859 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718898    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 21:58:10.720859 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718900    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 21:58:10.720859 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718903    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 21:58:10.720859 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718906    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 21:58:10.720859 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718909    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 21:58:10.720859 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718911    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 21:58:10.720859 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718914    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 21:58:10.720859 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718916    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 21:58:10.720859 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718919    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 21:58:10.720859 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718922    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 21:58:10.720859 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718925    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 21:58:10.720859 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.718928    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 21:58:10.720859 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719664    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 21:58:10.720859 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719671    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 21:58:10.720859 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719674    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 21:58:10.720859 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719677    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 21:58:10.720859 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719680    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 21:58:10.721379 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719683    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 21:58:10.721379 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719686    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 21:58:10.721379 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719688    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 21:58:10.721379 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719691    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 21:58:10.721379 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719694    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 21:58:10.721379 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719697    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 21:58:10.721379 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719699    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 21:58:10.721379 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719702    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 21:58:10.721379 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719705    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 21:58:10.721379 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719708    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 21:58:10.721379 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719710    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 21:58:10.721379 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719713    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 21:58:10.721379 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719723    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 21:58:10.721379 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719726    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 21:58:10.721379 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719729    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 21:58:10.721379 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719732    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 21:58:10.721379 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719734    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 21:58:10.721379 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719737    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 21:58:10.721379 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719739    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 21:58:10.721859 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719742    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 21:58:10.721859 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719745    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 21:58:10.721859 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719747    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 21:58:10.721859 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719750    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 21:58:10.721859 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719752    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 21:58:10.721859 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719755    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 21:58:10.721859 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719757    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 21:58:10.721859 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719760    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 21:58:10.721859 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719763    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 21:58:10.721859 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719766    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 21:58:10.721859 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719769    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 21:58:10.721859 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719771    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 21:58:10.721859 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719775    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 21:58:10.721859 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719778    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 21:58:10.721859 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719780    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 21:58:10.721859 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719783    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 21:58:10.721859 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719786    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 21:58:10.721859 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719788    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 21:58:10.721859 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719791    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 21:58:10.721859 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719794    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 21:58:10.722378 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719796    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 21:58:10.722378 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719799    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 21:58:10.722378 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719801    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 21:58:10.722378 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719804    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 21:58:10.722378 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719806    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 21:58:10.722378 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719809    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 21:58:10.722378 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719813    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 21:58:10.722378 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719815    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 21:58:10.722378 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719818    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 21:58:10.722378 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719822    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 21:58:10.722378 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719826    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 21:58:10.722378 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719829    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 21:58:10.722378 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719832    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 21:58:10.722378 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719835    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 21:58:10.722378 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719838    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 21:58:10.722378 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719841    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 21:58:10.722378 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719844    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 21:58:10.722378 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719847    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 21:58:10.722378 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719849    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 21:58:10.722378 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719852    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 21:58:10.722868 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719855    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 21:58:10.722868 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719858    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 21:58:10.722868 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719862    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 21:58:10.722868 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719864    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 21:58:10.722868 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719868    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 21:58:10.722868 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719871    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 21:58:10.722868 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719873    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 21:58:10.722868 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719877    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 21:58:10.722868 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719880    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 21:58:10.722868 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719883    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 21:58:10.722868 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719885    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 21:58:10.722868 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719888    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 21:58:10.722868 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719891    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 21:58:10.722868 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719893    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 21:58:10.722868 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719896    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 21:58:10.722868 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719899    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 21:58:10.722868 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719902    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 21:58:10.722868 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719904    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 21:58:10.722868 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719907    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 21:58:10.722868 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719911    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 21:58:10.723374 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719913    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 21:58:10.723374 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719916    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 21:58:10.723374 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719918    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 21:58:10.723374 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719921    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 21:58:10.723374 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719923    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 21:58:10.723374 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719926    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 21:58:10.723374 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719929    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 21:58:10.723374 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719931    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 21:58:10.723374 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719935    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 21:58:10.723374 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719938    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 21:58:10.723374 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719940    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 21:58:10.723374 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719943    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 21:58:10.723374 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719946    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 21:58:10.723374 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719949    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 21:58:10.723374 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.719953    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 21:58:10.723374 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720331    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 21:58:10.723374 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720337    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 21:58:10.723374 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720341    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 21:58:10.723374 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720344    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 21:58:10.723374 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720347    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 21:58:10.723865 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720351    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 21:58:10.723865 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720354    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 21:58:10.723865 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720357    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 21:58:10.723865 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720360    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 21:58:10.723865 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720362    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 21:58:10.723865 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720365    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 21:58:10.723865 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720368    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 21:58:10.723865 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720370    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 21:58:10.723865 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720373    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 21:58:10.723865 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720376    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 21:58:10.723865 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720379    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 21:58:10.723865 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720382    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 21:58:10.723865 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720384    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 21:58:10.723865 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720386    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 21:58:10.723865 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720389    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 21:58:10.723865 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720392    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 21:58:10.723865 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720394    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 21:58:10.723865 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720397    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 21:58:10.723865 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720408    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 21:58:10.724341 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720411    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 21:58:10.724341 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720415    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 21:58:10.724341 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720417    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 21:58:10.724341 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720420    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 21:58:10.724341 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720423    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 21:58:10.724341 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720426    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 21:58:10.724341 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720428    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 21:58:10.724341 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720431    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 21:58:10.724341 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720434    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 21:58:10.724341 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720437    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 21:58:10.724341 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720440    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 21:58:10.724341 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720443    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 21:58:10.724341 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720445    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 21:58:10.724341 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720448    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 21:58:10.724341 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720450    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 21:58:10.724341 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720453    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 21:58:10.724341 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720455    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 21:58:10.724341 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720458    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 21:58:10.724341 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720460    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 21:58:10.724341 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720463    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 21:58:10.724840 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720465    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 21:58:10.724840 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720468    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 21:58:10.724840 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720470    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 21:58:10.724840 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720472    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 21:58:10.724840 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720475    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 21:58:10.724840 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720479    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 21:58:10.724840 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720481    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 21:58:10.724840 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720484    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 21:58:10.724840 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720486    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 21:58:10.724840 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720489    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 21:58:10.724840 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720491    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 21:58:10.724840 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720494    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 21:58:10.724840 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720497    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 21:58:10.724840 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720499    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 21:58:10.724840 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720502    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 21:58:10.724840 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720505    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 21:58:10.724840 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720507    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 21:58:10.724840 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720516    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 21:58:10.724840 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720518    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 21:58:10.724840 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720521    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 21:58:10.725350 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720523    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 21:58:10.725350 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720526    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 21:58:10.725350 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720528    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 21:58:10.725350 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720531    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 21:58:10.725350 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720533    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 21:58:10.725350 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720536    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 21:58:10.725350 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720538    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 21:58:10.725350 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720541    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 21:58:10.725350 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720543    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 21:58:10.725350 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720546    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 21:58:10.725350 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720548    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 21:58:10.725350 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720551    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 21:58:10.725350 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720553    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 21:58:10.725350 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720556    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 21:58:10.725350 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720558    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 21:58:10.725350 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720561    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 21:58:10.725350 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720563    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 21:58:10.725350 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720568    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 21:58:10.725350 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720573    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 21:58:10.725806 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720576    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 21:58:10.725806 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720579    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 21:58:10.725806 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720582    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 21:58:10.725806 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720584    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 21:58:10.725806 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720587    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 21:58:10.725806 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720590    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 21:58:10.725806 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720593    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 21:58:10.725806 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720595    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 21:58:10.725806 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720600    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 21:58:10.725806 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720603    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 21:58:10.725806 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720606    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 21:58:10.725806 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720609    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 21:58:10.725806 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720611    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 21:58:10.725806 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720614    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 21:58:10.725806 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720616    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 21:58:10.725806 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.720619    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 21:58:10.725806 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720691    2563 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 21:58:10.725806 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720734    2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 21:58:10.725806 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720741    2563 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 21:58:10.725806 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720745    2563 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 21:58:10.725806 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720750    2563 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 21:58:10.726352 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720753    2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 21:58:10.726352 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720757    2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 21:58:10.726352 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720761    2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 21:58:10.726352 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720764    2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 21:58:10.726352 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720767    2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 21:58:10.726352 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720771    2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 21:58:10.726352 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720774    2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 21:58:10.726352 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720778    2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 21:58:10.726352 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720781    2563 flags.go:64] FLAG: --cgroup-root=""
Sep 17 21:58:10.726352 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720785    2563 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 21:58:10.726352 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720789    2563 flags.go:64] FLAG: --client-ca-file=""
Sep 17 21:58:10.726352 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720796    2563 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 21:58:10.726352 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720799    2563 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 21:58:10.726352 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720804    2563 flags.go:64] FLAG: --cluster-domain=""
Sep 17 21:58:10.726352 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720807    2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 21:58:10.726352 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720810    2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 21:58:10.726352 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720813    2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 21:58:10.726352 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720816    2563 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 21:58:10.726352 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720820    2563 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 21:58:10.726352 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720823    2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 21:58:10.726352 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720826    2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 21:58:10.726352 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720829    2563 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 21:58:10.726352 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720832    2563 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 21:58:10.726352 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720835    2563 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 21:58:10.726938 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720838    2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 21:58:10.726938 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720841    2563 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 21:58:10.726938 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720843    2563 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 21:58:10.726938 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720847    2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 21:58:10.726938 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720850    2563 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 21:58:10.726938 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720853    2563 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 21:58:10.726938 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720856    2563 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 21:58:10.726938 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720859    2563 flags.go:64] FLAG: --enable-server="true"
Sep 17 21:58:10.726938 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720862    2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 21:58:10.726938 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720866    2563 flags.go:64] FLAG: --event-burst="100"
Sep 17 21:58:10.726938 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720869    2563 flags.go:64] FLAG: --event-qps="50"
Sep 17 21:58:10.726938 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720872    2563 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 21:58:10.726938 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720875    2563 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 21:58:10.726938 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720877    2563 flags.go:64] FLAG: --eviction-hard=""
Sep 17 21:58:10.726938 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720881    2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 21:58:10.726938 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720884    2563 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 21:58:10.726938 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720887    2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 21:58:10.726938 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720891    2563 flags.go:64] FLAG: --eviction-soft=""
Sep 17 21:58:10.726938 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720894    2563 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 21:58:10.726938 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720897    2563 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 21:58:10.726938 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720900    2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 21:58:10.726938 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720903    2563 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 21:58:10.726938 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720906    2563 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 21:58:10.726938 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720912    2563 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 21:58:10.726938 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720915    2563 flags.go:64] FLAG: --feature-gates=""
Sep 17 21:58:10.727577 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720918    2563 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 21:58:10.727577 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720921    2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 21:58:10.727577 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720924    2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 21:58:10.727577 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720927    2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 21:58:10.727577 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720930    2563 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 21:58:10.727577 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720934    2563 flags.go:64] FLAG: --help="false"
Sep 17 21:58:10.727577 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720937    2563 flags.go:64] FLAG: --hostname-override="ip-10-0-138-45.ec2.internal"
Sep 17 21:58:10.727577 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720940    2563 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 21:58:10.727577 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720943    2563 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 21:58:10.727577 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720945    2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 21:58:10.727577 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720949    2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 21:58:10.727577 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720952    2563 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 21:58:10.727577 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720955    2563 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 21:58:10.727577 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720958    2563 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 21:58:10.727577 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720961    2563 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 21:58:10.727577 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720964    2563 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 21:58:10.727577 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720967    2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 21:58:10.727577 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720970    2563 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 21:58:10.727577 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720973    2563 flags.go:64] FLAG: --kube-reserved=""
Sep 17 21:58:10.727577 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720976    2563 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 21:58:10.727577 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720979    2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 21:58:10.727577 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720982    2563 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 21:58:10.727577 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720985    2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 21:58:10.727577 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720988    2563 flags.go:64] FLAG: --lock-file=""
Sep 17 21:58:10.728187 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720992    2563 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 21:58:10.728187 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720995    2563 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 21:58:10.728187 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.720997    2563 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 21:58:10.728187 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.721003    2563 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 21:58:10.728187 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.721007    2563 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 21:58:10.728187 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.721010    2563 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 21:58:10.728187 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.721014    2563 flags.go:64] FLAG: --logging-format="text"
Sep 17 21:58:10.728187 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.721017    2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 21:58:10.728187 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.721020    2563 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 21:58:10.728187 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.721023    2563 flags.go:64] FLAG: --manifest-url=""
Sep 17 21:58:10.728187 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.721026    2563 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 21:58:10.728187 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.721031    2563 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 21:58:10.728187 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.721034    2563 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 21:58:10.728187 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.721042    2563 flags.go:64] FLAG: --max-pods="110"
Sep 17 21:58:10.728187 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.721045    2563 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 21:58:10.728187 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.721048    2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 21:58:10.728187 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.721050    2563 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 21:58:10.728187 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.721053    2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 21:58:10.728187 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.721057    2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 21:58:10.728187 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.721060    2563 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 21:58:10.728187 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.721063    2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 21:58:10.728187 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.721069    2563 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 21:58:10.728187 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.721073    2563 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 21:58:10.728187 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.721075    2563 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 21:58:10.728187 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.721078    2563 flags.go:64] FLAG: --pod-cidr=""
Sep 17 21:58:10.728804 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.721081    2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 21:58:10.728804 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.721087    2563 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 21:58:10.728804 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.721090    2563 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 21:58:10.728804 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.721093    2563 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 21:58:10.728804 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.721096    2563 flags.go:64] FLAG: --port="10250"
Sep 17 21:58:10.728804 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.721099    2563 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 21:58:10.728804 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.721102    2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-09f2038275fd5409e"
Sep 17 21:58:10.728804 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.721106    2563 flags.go:64] FLAG: --qos-reserved=""
Sep 17 21:58:10.728804 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.721110    2563 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 21:58:10.728804 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.721113    2563 flags.go:64] FLAG: --register-node="true"
Sep 17 21:58:10.728804 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.721116    2563 flags.go:64] FLAG: --register-with-taints=""
Sep 17 21:58:10.728804 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.721119    2563 flags.go:64] FLAG: --registry-burst="10"
Sep 17 21:58:10.728804 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.721126    2563 flags.go:64] FLAG: --registry-qps="5"
Sep 17 21:58:10.728804 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.721129    2563 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 21:58:10.728804 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.721133    2563 flags.go:64] FLAG: --reserved-memory=""
Sep 17 21:58:10.728804 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.721148    2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 21:58:10.728804 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.721152    2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 21:58:10.728804 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.721155    2563 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 21:58:10.728804 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.721158    2563 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 21:58:10.728804 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.721161    2563 flags.go:64] FLAG: --runonce="false"
Sep 17 21:58:10.728804 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.721164    2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 21:58:10.728804 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.721167    2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 21:58:10.728804 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.721170    2563 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 21:58:10.728804 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.721173    2563 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 21:58:10.728804 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.721175    2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 21:58:10.729432 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.721179    2563 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 21:58:10.729432 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.721182    2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 21:58:10.729432 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.721185    2563 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 21:58:10.729432 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.721187    2563 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 21:58:10.729432 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.721190    2563 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 21:58:10.729432 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.721193    2563 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 21:58:10.729432 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.721196    2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 21:58:10.729432 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.721199    2563 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 21:58:10.729432 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.721202    2563 flags.go:64] FLAG: --system-cgroups=""
Sep 17 21:58:10.729432 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.721204    2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 21:58:10.729432 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.721210    2563 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 21:58:10.729432 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.721213    2563 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 21:58:10.729432 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.721216    2563 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 21:58:10.729432 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.721225    2563 flags.go:64] FLAG: --tls-min-version=""
Sep 17 21:58:10.729432 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.721233    2563 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 21:58:10.729432 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.721236    2563 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 21:58:10.729432 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.721239    2563 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 21:58:10.729432 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.721242    2563 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 21:58:10.729432 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.721245    2563 flags.go:64] FLAG: --v="2"
Sep 17 21:58:10.729432 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.721249    2563 flags.go:64] FLAG: --version="false"
Sep 17 21:58:10.729432 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.721254    2563 flags.go:64] FLAG: --vmodule=""
Sep 17 21:58:10.729432 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.721261    2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 21:58:10.729432 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.721264    2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 21:58:10.729432 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721372    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 21:58:10.729432 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721375    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 21:58:10.730072 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721379    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 21:58:10.730072 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721381    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 21:58:10.730072 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721385    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 21:58:10.730072 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721388    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 21:58:10.730072 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721390    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 21:58:10.730072 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721393    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 21:58:10.730072 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721396    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 21:58:10.730072 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721398    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 21:58:10.730072 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721401    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 21:58:10.730072 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721404    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 21:58:10.730072 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721406    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 21:58:10.730072 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721409    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 21:58:10.730072 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721411    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 21:58:10.730072 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721414    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 21:58:10.730072 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721416    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 21:58:10.730072 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721419    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 21:58:10.730072 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721422    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 21:58:10.730072 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721424    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 21:58:10.730072 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721427    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 21:58:10.730072 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721430    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 21:58:10.730579 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721432    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 21:58:10.730579 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721437    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 21:58:10.730579 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721440    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 21:58:10.730579 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721442    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 21:58:10.730579 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721445    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 21:58:10.730579 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721447    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 21:58:10.730579 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721450    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 21:58:10.730579 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721454    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 21:58:10.730579 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721459    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 21:58:10.730579 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721462    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 21:58:10.730579 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721465    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 21:58:10.730579 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721468    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 21:58:10.730579 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721471    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 21:58:10.730579 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721474    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 21:58:10.730579 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721476    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 21:58:10.730579 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721479    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 21:58:10.730579 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721482    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 21:58:10.730579 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721484    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 21:58:10.730579 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721487    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 21:58:10.730579 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721491    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 21:58:10.731074 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721494    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 21:58:10.731074 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721497    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 21:58:10.731074 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721499    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 21:58:10.731074 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721502    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 21:58:10.731074 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721504    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 21:58:10.731074 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721507    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 21:58:10.731074 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721510    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 21:58:10.731074 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721513    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 21:58:10.731074 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721515    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 21:58:10.731074 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721518    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 21:58:10.731074 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721520    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 21:58:10.731074 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721523    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 21:58:10.731074 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721525    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 21:58:10.731074 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721529    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 21:58:10.731074 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721533    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 21:58:10.731074 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721535    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 21:58:10.731074 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721538    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 21:58:10.731074 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721540    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 21:58:10.731074 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721543    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 21:58:10.731074 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721545    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 21:58:10.731575 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721549    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 21:58:10.731575 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721553    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 21:58:10.731575 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721555    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 21:58:10.731575 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721558    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 21:58:10.731575 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721561    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 21:58:10.731575 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721564    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 21:58:10.731575 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721566    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 21:58:10.731575 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721569    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 21:58:10.731575 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721572    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 21:58:10.731575 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721574    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 21:58:10.731575 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721577    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 21:58:10.731575 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721579    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 21:58:10.731575 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721582    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 21:58:10.731575 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721585    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 21:58:10.731575 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721587    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 21:58:10.731575 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721590    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 21:58:10.731575 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721593    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 21:58:10.731575 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721595    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 21:58:10.731575 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721598    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 21:58:10.731575 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721600    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 21:58:10.732063 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721603    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 21:58:10.732063 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721605    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 21:58:10.732063 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721608    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 21:58:10.732063 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721611    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 21:58:10.732063 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721613    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 21:58:10.732063 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721617    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 21:58:10.732063 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721620    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 21:58:10.732063 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721624    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 21:58:10.732063 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721627    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 21:58:10.732063 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721631    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 21:58:10.732063 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721633    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 21:58:10.732063 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721636    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 21:58:10.732063 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721640    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 21:58:10.732063 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721643    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 21:58:10.732063 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721646    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 21:58:10.732063 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721648    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 21:58:10.732063 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.721651    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 21:58:10.732503 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.722119    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 21:58:10.732503 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.729378    2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 21:58:10.732503 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.729506    2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 21:58:10.732503 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.729526    2563 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 21:58:10.732503 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.729532    2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 21:58:10.732503 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729583    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 21:58:10.732503 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729588    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 21:58:10.732503 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729592    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 21:58:10.732503 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729594    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 21:58:10.732503 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729598    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 21:58:10.732503 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729601    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 21:58:10.732503 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729603    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 21:58:10.732503 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729607    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 21:58:10.732503 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729611    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 21:58:10.732861 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729614    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 21:58:10.732861 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729617    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 21:58:10.732861 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729620    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 21:58:10.732861 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729622    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 21:58:10.732861 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729625    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 21:58:10.732861 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729628    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 21:58:10.732861 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729630    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 21:58:10.732861 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729633    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 21:58:10.732861 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729636    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 21:58:10.732861 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729639    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 21:58:10.732861 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729641    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 21:58:10.732861 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729644    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 21:58:10.732861 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729647    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 21:58:10.732861 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729650    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 21:58:10.732861 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729652    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 21:58:10.732861 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729656    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 21:58:10.732861 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729659    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 21:58:10.732861 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729662    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 21:58:10.732861 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729665    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 21:58:10.732861 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729667    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 21:58:10.733397 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729672    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 21:58:10.733397 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729676    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 21:58:10.733397 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729679    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 21:58:10.733397 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729682    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 21:58:10.733397 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729686    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 21:58:10.733397 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729688    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 21:58:10.733397 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729691    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 21:58:10.733397 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729694    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 21:58:10.733397 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729697    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 21:58:10.733397 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729699    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 21:58:10.733397 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729702    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 21:58:10.733397 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729704    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 21:58:10.733397 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729707    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 21:58:10.733397 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729710    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 21:58:10.733397 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729712    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 21:58:10.733397 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729715    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 21:58:10.733397 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729718    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 21:58:10.733397 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729720    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 21:58:10.733397 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729723    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 21:58:10.733397 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729726    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 21:58:10.733884 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729728    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 21:58:10.733884 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729731    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 21:58:10.733884 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729733    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 21:58:10.733884 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729736    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 21:58:10.733884 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729739    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 21:58:10.733884 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729741    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 21:58:10.733884 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729744    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 21:58:10.733884 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729747    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 21:58:10.733884 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729750    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 21:58:10.733884 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729752    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 21:58:10.733884 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729755    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 21:58:10.733884 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729757    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 21:58:10.733884 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729761    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 21:58:10.733884 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729764    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 21:58:10.733884 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729766    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 21:58:10.733884 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729769    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 21:58:10.733884 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729772    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 21:58:10.733884 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729774    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 21:58:10.733884 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729777    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 21:58:10.733884 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729780    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 21:58:10.734384 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729782    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 21:58:10.734384 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729785    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 21:58:10.734384 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729788    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 21:58:10.734384 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729790    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 21:58:10.734384 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729793    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 21:58:10.734384 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729795    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 21:58:10.734384 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729798    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 21:58:10.734384 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729801    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 21:58:10.734384 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729803    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 21:58:10.734384 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729806    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 21:58:10.734384 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729809    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 21:58:10.734384 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729811    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 21:58:10.734384 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729814    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 21:58:10.734384 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729816    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 21:58:10.734384 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729819    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 21:58:10.734384 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729822    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 21:58:10.734384 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729825    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 21:58:10.734384 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729827    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 21:58:10.734384 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729830    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 21:58:10.734384 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729833    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 21:58:10.734875 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729836    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 21:58:10.734875 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729840    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 21:58:10.734875 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729843    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 21:58:10.734875 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729846    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 21:58:10.734875 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729850    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 21:58:10.734875 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729853    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 21:58:10.734875 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729855    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 21:58:10.734875 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729858    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 21:58:10.734875 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729861    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 21:58:10.734875 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729864    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 21:58:10.734875 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.729869    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 21:58:10.734875 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729974    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 21:58:10.734875 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729978    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 21:58:10.734875 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729981    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 21:58:10.735239 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729984    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 21:58:10.735239 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729987    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 21:58:10.735239 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729990    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 21:58:10.735239 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729994    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 21:58:10.735239 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.729997    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 21:58:10.735239 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730000    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 21:58:10.735239 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730003    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 21:58:10.735239 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730005    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 21:58:10.735239 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730008    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 21:58:10.735239 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730011    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 21:58:10.735239 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730013    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 21:58:10.735239 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730016    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 21:58:10.735239 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730019    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 21:58:10.735239 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730021    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 21:58:10.735239 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730024    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 21:58:10.735239 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730027    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 21:58:10.735239 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730029    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 21:58:10.735239 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730032    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 21:58:10.735239 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730035    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 21:58:10.735696 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730038    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 21:58:10.735696 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730040    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 21:58:10.735696 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730043    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 21:58:10.735696 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730046    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 21:58:10.735696 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730049    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 21:58:10.735696 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730053    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 21:58:10.735696 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730056    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 21:58:10.735696 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730059    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 21:58:10.735696 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730062    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 21:58:10.735696 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730064    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 21:58:10.735696 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730067    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 21:58:10.735696 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730069    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 21:58:10.735696 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730072    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 21:58:10.735696 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730074    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 21:58:10.735696 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730077    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 21:58:10.735696 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730080    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 21:58:10.735696 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730082    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 21:58:10.735696 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730085    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 21:58:10.735696 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730088    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 21:58:10.735696 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730090    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 21:58:10.736190 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730094    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 21:58:10.736190 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730096    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 21:58:10.736190 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730099    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 21:58:10.736190 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730102    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 21:58:10.736190 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730104    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 21:58:10.736190 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730107    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 21:58:10.736190 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730109    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 21:58:10.736190 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730112    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 21:58:10.736190 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730115    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 21:58:10.736190 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730117    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 21:58:10.736190 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730120    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 21:58:10.736190 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730122    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 21:58:10.736190 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730125    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 21:58:10.736190 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730127    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 21:58:10.736190 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730130    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 21:58:10.736190 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730133    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 21:58:10.736190 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730135    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 21:58:10.736190 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730151    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 21:58:10.736190 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730154    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 21:58:10.736190 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730157    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 21:58:10.736737 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730159    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 21:58:10.736737 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730162    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 21:58:10.736737 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730164    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 21:58:10.736737 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730167    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 21:58:10.736737 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730170    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 21:58:10.736737 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730172    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 21:58:10.736737 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730175    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 21:58:10.736737 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730177    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 21:58:10.736737 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730180    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 21:58:10.736737 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730183    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 21:58:10.736737 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730185    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 21:58:10.736737 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730188    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 21:58:10.736737 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730190    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 21:58:10.736737 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730193    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 21:58:10.736737 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730195    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 21:58:10.736737 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730198    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 21:58:10.736737 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730200    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 21:58:10.736737 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730203    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 21:58:10.736737 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730205    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 21:58:10.737229 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730208    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 21:58:10.737229 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730210    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 21:58:10.737229 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730213    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 21:58:10.737229 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730215    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 21:58:10.737229 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730218    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 21:58:10.737229 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730220    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 21:58:10.737229 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730223    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 21:58:10.737229 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730227    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 21:58:10.737229 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730230    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 21:58:10.737229 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730234    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 21:58:10.737229 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730237    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 21:58:10.737229 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730240    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 21:58:10.737229 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730243    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 21:58:10.737229 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730246    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 21:58:10.737229 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730249    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 21:58:10.737229 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730251    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 21:58:10.737229 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730254    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 21:58:10.737229 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:10.730257    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 21:58:10.737670 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.730262    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 21:58:10.737670 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.730362    2563 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 21:58:10.737670 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.733209    2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 21:58:10.737670 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.734002    2563 server.go:1019] "Starting client certificate rotation"
Sep 17 21:58:10.737670 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.734101    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 21:58:10.737670 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.734130    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 21:58:10.737670 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.736268    2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 21:58:10.738105 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.738076    2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 21:58:10.746333 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.746315    2563 log.go:25] "Validated CRI v1 runtime API"
Sep 17 21:58:10.748633 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.748608    2563 log.go:25] "Validated CRI v1 image API"
Sep 17 21:58:10.749678 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.749655    2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 21:58:10.751833 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.751808    2563 fs.go:135] Filesystem UUIDs: map[13e69a73-5fa6-48bb-90c8-6d7c83396d22:/dev/nvme0n1p3 17dfac9e-9a4f-4dfd-b297-918b57973f6a:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Sep 17 21:58:10.751888 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.751835    2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 21:58:10.756344 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.756233    2563 manager.go:216] Machine: {Timestamp:2026-09-17 21:58:10.755736686 +0000 UTC m=+1.923689660 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096991 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec251aa91a72ce9d15cfd06fa46fd6bf SystemUUID:ec251aa9-1a72-ce9d-15cf-d06fa46fd6bf BootID:3236412c-e145-4005-8050-6d9c44238f23 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d3:b6:34:43 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d3:b6:34:43 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:aa:19:59:5e:8d:a7 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 21:58:10.756344 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.756335    2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 21:58:10.756485 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.756448    2563 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 21:58:10.757878 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.757849    2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 21:58:10.758018 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.757881    2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-45.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 21:58:10.759048 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.759038    2563 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 21:58:10.759083 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.759051    2563 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 21:58:10.759083 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.759059    2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 21:58:10.759083 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.759071    2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 21:58:10.759178 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.759155    2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 21:58:10.762991 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.762977    2563 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 21:58:10.763273 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.763263    2563 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 21:58:10.763316 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.763294    2563 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 21:58:10.763316 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.763304    2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 21:58:10.763316 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.763315    2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 21:58:10.763391 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.763323    2563 kubelet.go:398] "Adding apiserver pod source"
Sep 17 21:58:10.763391 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.763333    2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 21:58:10.763656 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.763643    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 21:58:10.763704 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.763667    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 21:58:10.768947 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.768923    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 21:58:10.770781 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.770761    2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 21:58:10.773223 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.773207    2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 21:58:10.773307 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.773238    2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 21:58:10.774341 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.774329    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 21:58:10.774402 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.774346    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 21:58:10.774402 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.774353    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 21:58:10.774402 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.774358    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 21:58:10.774402 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.774367    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 21:58:10.774402 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.774372    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 21:58:10.774402 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.774385    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 21:58:10.774402 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.774391    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 21:58:10.774402 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.774401    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 21:58:10.774636 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.774407    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 21:58:10.774636 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.774417    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 21:58:10.774636 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.774422    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 21:58:10.774636 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.774473    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 21:58:10.774636 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.774480    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 21:58:10.778167 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.778152    2563 server.go:1268] "Started kubelet"
Sep 17 21:58:10.778251 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.778210    2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 21:58:10.778368 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.778320    2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 21:58:10.778421 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.778392    2563 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 21:58:10.778996 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:10.778948    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 21:58:10.778996 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.778986    2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-45.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 21:58:10.779096 ip-10-0-138-45 systemd[1]: Started Kubernetes Kubelet.
Sep 17 21:58:10.779422 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:10.779243    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-45.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 21:58:10.779484 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.779469    2563 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 21:58:10.779532 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.779484    2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 21:58:10.807751 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.807722    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9vkvg"
Sep 17 21:58:10.807751 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.807750    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9vkvg"
Sep 17 21:58:10.809157 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:10.809114    2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 21:58:10.813516 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:10.812622    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-45.ec2.internal.18d63a8fb2a30154  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-45.ec2.internal,UID:ip-10-0-138-45.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-45.ec2.internal,},FirstTimestamp:2026-09-17 21:58:10.77811234 +0000 UTC m=+1.946065314,LastTimestamp:2026-09-17 21:58:10.77811234 +0000 UTC m=+1.946065314,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-45.ec2.internal,}"
Sep 17 21:58:10.815037 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.815022    2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 21:58:10.815111 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.815033    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 21:58:10.815229 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:10.815213    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-45.ec2.internal\" not found"
Sep 17 21:58:10.815700 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.815678    2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 21:58:10.815700 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.815689    2563 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 21:58:10.815831 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.815711    2563 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 21:58:10.815831 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.815768    2563 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 21:58:10.815831 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.815778    2563 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 21:58:10.815971 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.815913    2563 factory.go:55] Registering systemd factory
Sep 17 21:58:10.815971 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.815926    2563 factory.go:223] Registration of the systemd container factory successfully
Sep 17 21:58:10.816215 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.816203    2563 factory.go:153] Registering CRI-O factory
Sep 17 21:58:10.816215 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.816215    2563 factory.go:223] Registration of the crio container factory successfully
Sep 17 21:58:10.816333 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.816265    2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 21:58:10.816333 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.816285    2563 factory.go:103] Registering Raw factory
Sep 17 21:58:10.816333 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.816300    2563 manager.go:1196] Started watching for new ooms in manager
Sep 17 21:58:10.817282 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.817267    2563 manager.go:349] Starting recovery of all containers
Sep 17 21:58:10.821595 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.821576    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 21:58:10.824752 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:10.824729    2563 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-138-45.ec2.internal\" not found" node="ip-10-0-138-45.ec2.internal"
Sep 17 21:58:10.828234 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.828217    2563 manager.go:354] Recovery completed
Sep 17 21:58:10.832031 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.832018    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 21:58:10.834732 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.834718    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-45.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 21:58:10.834790 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.834745    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-45.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 21:58:10.834790 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.834756    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-45.ec2.internal" event="NodeHasSufficientPID"
Sep 17 21:58:10.835325 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.835311    2563 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 21:58:10.835325 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.835321    2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 21:58:10.835430 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.835339    2563 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 21:58:10.835430 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.835339    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 21:58:10.836059 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.836046    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-45.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 21:58:10.836123 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.836069    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-45.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 21:58:10.836123 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.836081    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-45.ec2.internal" event="NodeHasSufficientPID"
Sep 17 21:58:10.837448 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.837437    2563 policy_none.go:49] "None policy: Start"
Sep 17 21:58:10.837482 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.837452    2563 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 21:58:10.837482 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.837463    2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 21:58:10.839387 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.839375    2563 policy_none.go:47] "Start"
Sep 17 21:58:10.884334 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.883712    2563 manager.go:337] "Starting Device Plugin manager"
Sep 17 21:58:10.884334 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:10.883746    2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 21:58:10.884334 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.883756    2563 server.go:87] "Starting device plugin registration server"
Sep 17 21:58:10.884334 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.884025    2563 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 21:58:10.884334 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.884054    2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 21:58:10.884334 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.884179    2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 21:58:10.884334 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.884266    2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 21:58:10.884334 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.884277    2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 21:58:10.884795 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:10.884778    2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 21:58:10.884848 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:10.884821    2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-45.ec2.internal\" not found"
Sep 17 21:58:10.907063 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.907041    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 21:58:10.908226 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.908207    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 21:58:10.908327 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.908258    2563 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 21:58:10.908327 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.908277    2563 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 21:58:10.908327 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:10.908313    2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 21:58:10.910495 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.910475    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 21:58:10.984303 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.984165    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 21:58:10.985351 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.985333    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-45.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 21:58:10.985458 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.985367    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-45.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 21:58:10.985458 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.985382    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-45.ec2.internal" event="NodeHasSufficientPID"
Sep 17 21:58:10.985458 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.985410    2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-45.ec2.internal"
Sep 17 21:58:10.993116 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:10.993100    2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-45.ec2.internal"
Sep 17 21:58:10.993200 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:10.993124    2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-45.ec2.internal\": node \"ip-10-0-138-45.ec2.internal\" not found"
Sep 17 21:58:11.004732 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:11.004707    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-45.ec2.internal\" not found"
Sep 17 21:58:11.008778 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:11.008758    2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-45.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-45.ec2.internal"]
Sep 17 21:58:11.008836 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:11.008823    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 21:58:11.010124 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:11.010104    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-45.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 21:58:11.010226 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:11.010130    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-45.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 21:58:11.010226 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:11.010156    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-45.ec2.internal" event="NodeHasSufficientPID"
Sep 17 21:58:11.013738 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:11.013726    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 21:58:11.013863 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:11.013839    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-45.ec2.internal"
Sep 17 21:58:11.013903 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:11.013866    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 21:58:11.014579 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:11.014564    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-45.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 21:58:11.014579 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:11.014578    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-45.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 21:58:11.014709 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:11.014589    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-45.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 21:58:11.014709 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:11.014597    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-45.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 21:58:11.014709 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:11.014599    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-45.ec2.internal" event="NodeHasSufficientPID"
Sep 17 21:58:11.014709 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:11.014608    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-45.ec2.internal" event="NodeHasSufficientPID"
Sep 17 21:58:11.016880 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:11.016865    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-45.ec2.internal"
Sep 17 21:58:11.016946 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:11.016905    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 21:58:11.020028 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:11.020007    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-45.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 21:58:11.020110 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:11.020041    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-45.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 21:58:11.020110 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:11.020057    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-45.ec2.internal" event="NodeHasSufficientPID"
Sep 17 21:58:11.028708 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:11.028689    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-45.ec2.internal\" not found" node="ip-10-0-138-45.ec2.internal"
Sep 17 21:58:11.034412 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:11.034396    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-45.ec2.internal\" not found" node="ip-10-0-138-45.ec2.internal"
Sep 17 21:58:11.105076 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:11.105047    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-45.ec2.internal\" not found"
Sep 17 21:58:11.117326 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:11.117304    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/26382571c0e2f6db815799fbfabf8da2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-45.ec2.internal\" (UID: \"26382571c0e2f6db815799fbfabf8da2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-45.ec2.internal"
Sep 17 21:58:11.117413 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:11.117329    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26382571c0e2f6db815799fbfabf8da2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-45.ec2.internal\" (UID: \"26382571c0e2f6db815799fbfabf8da2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-45.ec2.internal"
Sep 17 21:58:11.117413 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:11.117349    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2e7e38f229d83af38da836b565cddb01-config\") pod \"kube-apiserver-proxy-ip-10-0-138-45.ec2.internal\" (UID: \"2e7e38f229d83af38da836b565cddb01\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-45.ec2.internal"
Sep 17 21:58:11.206154 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:11.206092    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-45.ec2.internal\" not found"
Sep 17 21:58:11.218457 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:11.218433    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/26382571c0e2f6db815799fbfabf8da2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-45.ec2.internal\" (UID: \"26382571c0e2f6db815799fbfabf8da2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-45.ec2.internal"
Sep 17 21:58:11.218525 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:11.218462    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26382571c0e2f6db815799fbfabf8da2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-45.ec2.internal\" (UID: \"26382571c0e2f6db815799fbfabf8da2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-45.ec2.internal"
Sep 17 21:58:11.218525 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:11.218478    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2e7e38f229d83af38da836b565cddb01-config\") pod \"kube-apiserver-proxy-ip-10-0-138-45.ec2.internal\" (UID: \"2e7e38f229d83af38da836b565cddb01\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-45.ec2.internal"
Sep 17 21:58:11.218525 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:11.218519    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2e7e38f229d83af38da836b565cddb01-config\") pod \"kube-apiserver-proxy-ip-10-0-138-45.ec2.internal\" (UID: \"2e7e38f229d83af38da836b565cddb01\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-45.ec2.internal"
Sep 17 21:58:11.218617 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:11.218548    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/26382571c0e2f6db815799fbfabf8da2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-45.ec2.internal\" (UID: \"26382571c0e2f6db815799fbfabf8da2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-45.ec2.internal"
Sep 17 21:58:11.218617 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:11.218548    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26382571c0e2f6db815799fbfabf8da2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-45.ec2.internal\" (UID: \"26382571c0e2f6db815799fbfabf8da2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-45.ec2.internal"
Sep 17 21:58:11.306897 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:11.306820    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-45.ec2.internal\" not found"
Sep 17 21:58:11.332275 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:11.332250    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-45.ec2.internal"
Sep 17 21:58:11.338917 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:11.338900    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-45.ec2.internal"
Sep 17 21:58:11.407490 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:11.407452    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-45.ec2.internal\" not found"
Sep 17 21:58:11.508046 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:11.507994    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-45.ec2.internal\" not found"
Sep 17 21:58:11.608569 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:11.608490    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-45.ec2.internal\" not found"
Sep 17 21:58:11.709068 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:11.709035    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-45.ec2.internal\" not found"
Sep 17 21:58:11.734497 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:11.734469    2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 21:58:11.734949 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:11.734607    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 21:58:11.734949 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:11.734649    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 21:58:11.785109 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:11.785081    2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 21:58:11.814155 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:11.814112    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 21:53:10 +0000 UTC" deadline="2028-06-05 01:59:14.682664464 +0000 UTC"
Sep 17 21:58:11.814155 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:11.814148    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15028h1m2.868528633s"
Sep 17 21:58:11.815209 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:11.815196    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 21:58:11.815256 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:11.815198    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-45.ec2.internal"
Sep 17 21:58:11.826155 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:11.826117    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 21:58:11.827650 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:11.827635    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-45.ec2.internal"
Sep 17 21:58:11.827734 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:11.827662    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 21:58:11.833408 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:11.833393    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 21:58:11.849025 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:11.849005    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-wdncf"
Sep 17 21:58:11.854790 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:11.854773    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-wdncf"
Sep 17 21:58:12.020100 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:12.019947    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e7e38f229d83af38da836b565cddb01.slice/crio-cea559b9bf958c63f0fc56a42e182dca75992f509726fed11dd0ca69ba12b376 WatchSource:0}: Error finding container cea559b9bf958c63f0fc56a42e182dca75992f509726fed11dd0ca69ba12b376: Status 404 returned error can't find the container with id cea559b9bf958c63f0fc56a42e182dca75992f509726fed11dd0ca69ba12b376
Sep 17 21:58:12.020456 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:12.020436    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26382571c0e2f6db815799fbfabf8da2.slice/crio-195fb5c9cec1a288ea13b33f6d55b015e8551749cddb177a33ad253d0284495e WatchSource:0}: Error finding container 195fb5c9cec1a288ea13b33f6d55b015e8551749cddb177a33ad253d0284495e: Status 404 returned error can't find the container with id 195fb5c9cec1a288ea13b33f6d55b015e8551749cddb177a33ad253d0284495e
Sep 17 21:58:12.025993 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.025973    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 21:58:12.338253 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.338172    2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 21:58:12.678912 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.678835    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 21:58:12.728000 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.727973    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 21:58:12.764913 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.764880    2563 apiserver.go:52] "Watching apiserver"
Sep 17 21:58:12.771940 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.771909    2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 21:58:12.772499 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.772440    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-45.ec2.internal","openshift-multus/multus-additional-cni-plugins-wjt9h","openshift-network-operator/iptables-alerter-tkphq","kube-system/global-pull-secret-syncer-wxx9x","kube-system/konnectivity-agent-j6ntd","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flq5r","openshift-cluster-node-tuning-operator/tuned-26gb4","openshift-multus/multus-f6t75","openshift-multus/network-metrics-daemon-xkk4g","openshift-network-diagnostics/network-check-target-bnj94","openshift-ovn-kubernetes/ovnkube-node-nrkd5","kube-system/kube-apiserver-proxy-ip-10-0-138-45.ec2.internal","openshift-dns/node-resolver-n2fp6","openshift-image-registry/node-ca-hzx5m"]
Sep 17 21:58:12.775024 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.775003    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flq5r"
Sep 17 21:58:12.777252 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.777231    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hzx5m"
Sep 17 21:58:12.778769 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.778742    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 21:58:12.778924 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.778903    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-gch4l\""
Sep 17 21:58:12.779208 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.779184    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 21:58:12.779510 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.779494    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 21:58:12.779935 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.779916    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2nzwv\""
Sep 17 21:58:12.781382 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.781363    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-26gb4"
Sep 17 21:58:12.781710 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.781693    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 21:58:12.781867 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.781852    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 21:58:12.782018 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.782000    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 21:58:12.784033 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.783844    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wxx9x"
Sep 17 21:58:12.784033 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:12.783911    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wxx9x" podUID="79a6aec6-e29a-45cf-881f-447b87f04557"
Sep 17 21:58:12.784928 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.784772    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 21:58:12.784928 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.784789    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4prkc\""
Sep 17 21:58:12.785246 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.785229    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 21:58:12.786873 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.786324    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-j6ntd"
Sep 17 21:58:12.788587 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.788570    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 21:58:12.788665 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.788570    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-9pn66\""
Sep 17 21:58:12.788972 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.788831    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 21:58:12.789177 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.789157    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tkphq"
Sep 17 21:58:12.791376 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.791358    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 21:58:12.791576 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.791558    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 21:58:12.791644 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.791630    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-68fdc\""
Sep 17 21:58:12.791762 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.791746    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 21:58:12.791868 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.791853    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-f6t75"
Sep 17 21:58:12.794057 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.793772    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkk4g"
Sep 17 21:58:12.794057 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:12.793843    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkk4g" podUID="45d5f94d-990c-4f9e-b1ab-980021c00716"
Sep 17 21:58:12.794560 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.794340    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 21:58:12.794560 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.794532    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 21:58:12.794699 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.794641    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 21:58:12.794756 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.794720    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 21:58:12.794902 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.794884    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-gkfll\""
Sep 17 21:58:12.796554 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.796536    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bnj94"
Sep 17 21:58:12.796637 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:12.796612    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bnj94" podUID="3b714c1c-9f7e-430e-bc07-94cf3a75921c"
Sep 17 21:58:12.798758 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.798740    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5"
Sep 17 21:58:12.801015 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.800856    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 21:58:12.801665 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.801647    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-n2fp6"
Sep 17 21:58:12.802387 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.802026    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 21:58:12.802387 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.802168    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vcm6l\""
Sep 17 21:58:12.802387 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.802263    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 21:58:12.802387 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.802274    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 21:58:12.802387 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.802346    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 21:58:12.802669 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.802623    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 21:58:12.803971 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.803951    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 21:58:12.804060 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.803984    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-nlfrw\""
Sep 17 21:58:12.804286 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.804269    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 21:58:12.804693 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.804675    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wjt9h"
Sep 17 21:58:12.806603 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.806588    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 21:58:12.806918 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.806801    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 21:58:12.807010 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.806816    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4mjbn\""
Sep 17 21:58:12.817087 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.817068    2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 21:58:12.826951 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.826902    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f004b93f-9438-4008-ad17-92150090d28c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wjt9h\" (UID: \"f004b93f-9438-4008-ad17-92150090d28c\") " pod="openshift-multus/multus-additional-cni-plugins-wjt9h"
Sep 17 21:58:12.826951 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.826938    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/96f98ff0-1bb0-4529-9b06-43019479d9e8-socket-dir\") pod \"aws-ebs-csi-driver-node-flq5r\" (UID: \"96f98ff0-1bb0-4529-9b06-43019479d9e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flq5r"
Sep 17 21:58:12.827222 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.826964    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79a6aec6-e29a-45cf-881f-447b87f04557-original-pull-secret\") pod \"global-pull-secret-syncer-wxx9x\" (UID: \"79a6aec6-e29a-45cf-881f-447b87f04557\") " pod="kube-system/global-pull-secret-syncer-wxx9x"
Sep 17 21:58:12.827222 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.826991    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/992a8c01-f2ce-4523-9a33-56e12b134be2-etc-kubernetes\") pod \"tuned-26gb4\" (UID: \"992a8c01-f2ce-4523-9a33-56e12b134be2\") " pod="openshift-cluster-node-tuning-operator/tuned-26gb4"
Sep 17 21:58:12.827222 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.827014    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/992a8c01-f2ce-4523-9a33-56e12b134be2-etc-systemd\") pod \"tuned-26gb4\" (UID: \"992a8c01-f2ce-4523-9a33-56e12b134be2\") " pod="openshift-cluster-node-tuning-operator/tuned-26gb4"
Sep 17 21:58:12.827222 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.827038    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d5pc\" (UniqueName: \"kubernetes.io/projected/f004b93f-9438-4008-ad17-92150090d28c-kube-api-access-9d5pc\") pod \"multus-additional-cni-plugins-wjt9h\" (UID: \"f004b93f-9438-4008-ad17-92150090d28c\") " pod="openshift-multus/multus-additional-cni-plugins-wjt9h"
Sep 17 21:58:12.827222 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.827062    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/34b8ef2b-97ba-4562-87ee-8da0cb4604c7-host-cni-netd\") pod \"ovnkube-node-nrkd5\" (UID: \"34b8ef2b-97ba-4562-87ee-8da0cb4604c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5"
Sep 17 21:58:12.827222 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.827084    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/34b8ef2b-97ba-4562-87ee-8da0cb4604c7-ovnkube-config\") pod \"ovnkube-node-nrkd5\" (UID: \"34b8ef2b-97ba-4562-87ee-8da0cb4604c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5"
Sep 17 21:58:12.827222 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.827102    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/992a8c01-f2ce-4523-9a33-56e12b134be2-etc-sysctl-d\") pod \"tuned-26gb4\" (UID: \"992a8c01-f2ce-4523-9a33-56e12b134be2\") " pod="openshift-cluster-node-tuning-operator/tuned-26gb4"
Sep 17 21:58:12.827222 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.827124    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/992a8c01-f2ce-4523-9a33-56e12b134be2-var-lib-kubelet\") pod \"tuned-26gb4\" (UID: \"992a8c01-f2ce-4523-9a33-56e12b134be2\") " pod="openshift-cluster-node-tuning-operator/tuned-26gb4"
Sep 17 21:58:12.827222 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.827153    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/992a8c01-f2ce-4523-9a33-56e12b134be2-host\") pod \"tuned-26gb4\" (UID: \"992a8c01-f2ce-4523-9a33-56e12b134be2\") " pod="openshift-cluster-node-tuning-operator/tuned-26gb4"
Sep 17 21:58:12.827222 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.827175    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/992a8c01-f2ce-4523-9a33-56e12b134be2-tmp\") pod \"tuned-26gb4\" (UID: \"992a8c01-f2ce-4523-9a33-56e12b134be2\") " pod="openshift-cluster-node-tuning-operator/tuned-26gb4"
Sep 17 21:58:12.827222 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.827190    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e5348473-b853-4265-8876-9e0b2c8943d8-multus-conf-dir\") pod \"multus-f6t75\" (UID: \"e5348473-b853-4265-8876-9e0b2c8943d8\") " pod="openshift-multus/multus-f6t75"
Sep 17 21:58:12.827222 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.827202    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/992a8c01-f2ce-4523-9a33-56e12b134be2-run\") pod \"tuned-26gb4\" (UID: \"992a8c01-f2ce-4523-9a33-56e12b134be2\") " pod="openshift-cluster-node-tuning-operator/tuned-26gb4"
Sep 17 21:58:12.827222 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.827216    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e5348473-b853-4265-8876-9e0b2c8943d8-cni-binary-copy\") pod \"multus-f6t75\" (UID: \"e5348473-b853-4265-8876-9e0b2c8943d8\") " pod="openshift-multus/multus-f6t75"
Sep 17 21:58:12.827222 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.827230    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e5348473-b853-4265-8876-9e0b2c8943d8-host-run-multus-certs\") pod \"multus-f6t75\" (UID: \"e5348473-b853-4265-8876-9e0b2c8943d8\") " pod="openshift-multus/multus-f6t75"
Sep 17 21:58:12.827982 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.827244    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ddn8\" (UniqueName: \"kubernetes.io/projected/e5348473-b853-4265-8876-9e0b2c8943d8-kube-api-access-4ddn8\") pod \"multus-f6t75\" (UID: \"e5348473-b853-4265-8876-9e0b2c8943d8\") " pod="openshift-multus/multus-f6t75"
Sep 17 21:58:12.827982 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.827272    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f004b93f-9438-4008-ad17-92150090d28c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wjt9h\" (UID: \"f004b93f-9438-4008-ad17-92150090d28c\") " pod="openshift-multus/multus-additional-cni-plugins-wjt9h"
Sep 17 21:58:12.827982 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.827292    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/96f98ff0-1bb0-4529-9b06-43019479d9e8-registration-dir\") pod \"aws-ebs-csi-driver-node-flq5r\" (UID: \"96f98ff0-1bb0-4529-9b06-43019479d9e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flq5r"
Sep 17 21:58:12.827982 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.827332    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/96f98ff0-1bb0-4529-9b06-43019479d9e8-device-dir\") pod \"aws-ebs-csi-driver-node-flq5r\" (UID: \"96f98ff0-1bb0-4529-9b06-43019479d9e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flq5r"
Sep 17 21:58:12.827982 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.827369    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/79a6aec6-e29a-45cf-881f-447b87f04557-kubelet-config\") pod \"global-pull-secret-syncer-wxx9x\" (UID: \"79a6aec6-e29a-45cf-881f-447b87f04557\") " pod="kube-system/global-pull-secret-syncer-wxx9x"
Sep 17 21:58:12.827982 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.827404    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34b8ef2b-97ba-4562-87ee-8da0cb4604c7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nrkd5\" (UID: \"34b8ef2b-97ba-4562-87ee-8da0cb4604c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5"
Sep 17 21:58:12.827982 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.827431    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d2ac593-98b6-4b38-adef-a1bfbf1a168f-tmp-dir\") pod \"node-resolver-n2fp6\" (UID: \"9d2ac593-98b6-4b38-adef-a1bfbf1a168f\") " pod="openshift-dns/node-resolver-n2fp6"
Sep 17 21:58:12.827982 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.827458    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c20c2a19-2b30-48fb-bb14-b5321cf2f4d0-host\") pod \"node-ca-hzx5m\" (UID: \"c20c2a19-2b30-48fb-bb14-b5321cf2f4d0\") " pod="openshift-image-registry/node-ca-hzx5m"
Sep 17 21:58:12.827982 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.827479    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f004b93f-9438-4008-ad17-92150090d28c-cnibin\") pod \"multus-additional-cni-plugins-wjt9h\" (UID: \"f004b93f-9438-4008-ad17-92150090d28c\") " pod="openshift-multus/multus-additional-cni-plugins-wjt9h"
Sep 17 21:58:12.827982 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.827507    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/96f98ff0-1bb0-4529-9b06-43019479d9e8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-flq5r\" (UID: \"96f98ff0-1bb0-4529-9b06-43019479d9e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flq5r"
Sep 17 21:58:12.827982 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.827533    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/34b8ef2b-97ba-4562-87ee-8da0cb4604c7-ovnkube-script-lib\") pod \"ovnkube-node-nrkd5\" (UID: \"34b8ef2b-97ba-4562-87ee-8da0cb4604c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5"
Sep 17 21:58:12.827982 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.827563    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/992a8c01-f2ce-4523-9a33-56e12b134be2-lib-modules\") pod \"tuned-26gb4\" (UID: \"992a8c01-f2ce-4523-9a33-56e12b134be2\") " pod="openshift-cluster-node-tuning-operator/tuned-26gb4"
Sep 17 21:58:12.827982 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.827584    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/992a8c01-f2ce-4523-9a33-56e12b134be2-etc-tuned\") pod \"tuned-26gb4\" (UID: \"992a8c01-f2ce-4523-9a33-56e12b134be2\") " pod="openshift-cluster-node-tuning-operator/tuned-26gb4"
Sep 17 21:58:12.827982 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.827624    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mttzt\" (UniqueName: \"kubernetes.io/projected/d6f6a71b-842d-48f0-93fc-28006b5c122f-kube-api-access-mttzt\") pod \"iptables-alerter-tkphq\" (UID: \"d6f6a71b-842d-48f0-93fc-28006b5c122f\") " pod="openshift-network-operator/iptables-alerter-tkphq"
Sep 17 21:58:12.827982 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.827648    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e5348473-b853-4265-8876-9e0b2c8943d8-multus-socket-dir-parent\") pod \"multus-f6t75\" (UID: \"e5348473-b853-4265-8876-9e0b2c8943d8\") " pod="openshift-multus/multus-f6t75"
Sep 17 21:58:12.827982 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.827675    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5348473-b853-4265-8876-9e0b2c8943d8-host-var-lib-kubelet\") pod \"multus-f6t75\" (UID: \"e5348473-b853-4265-8876-9e0b2c8943d8\") " pod="openshift-multus/multus-f6t75"
Sep 17 21:58:12.828823 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.827705    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f004b93f-9438-4008-ad17-92150090d28c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wjt9h\" (UID: \"f004b93f-9438-4008-ad17-92150090d28c\") " pod="openshift-multus/multus-additional-cni-plugins-wjt9h"
Sep 17 21:58:12.828823 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.827728    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/34b8ef2b-97ba-4562-87ee-8da0cb4604c7-host-slash\") pod \"ovnkube-node-nrkd5\" (UID: \"34b8ef2b-97ba-4562-87ee-8da0cb4604c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5"
Sep 17 21:58:12.828823 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.827743    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/34b8ef2b-97ba-4562-87ee-8da0cb4604c7-run-openvswitch\") pod \"ovnkube-node-nrkd5\" (UID: \"34b8ef2b-97ba-4562-87ee-8da0cb4604c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5"
Sep 17 21:58:12.828823 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.827757    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9d2ac593-98b6-4b38-adef-a1bfbf1a168f-hosts-file\") pod \"node-resolver-n2fp6\" (UID: \"9d2ac593-98b6-4b38-adef-a1bfbf1a168f\") " pod="openshift-dns/node-resolver-n2fp6"
Sep 17 21:58:12.828823 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.827773    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d6f6a71b-842d-48f0-93fc-28006b5c122f-host-slash\") pod \"iptables-alerter-tkphq\" (UID: \"d6f6a71b-842d-48f0-93fc-28006b5c122f\") " pod="openshift-network-operator/iptables-alerter-tkphq"
Sep 17 21:58:12.828823 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.827794    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5348473-b853-4265-8876-9e0b2c8943d8-system-cni-dir\") pod \"multus-f6t75\" (UID: \"e5348473-b853-4265-8876-9e0b2c8943d8\") " pod="openshift-multus/multus-f6t75"
Sep 17 21:58:12.828823 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.827840    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/96f98ff0-1bb0-4529-9b06-43019479d9e8-etc-selinux\") pod \"aws-ebs-csi-driver-node-flq5r\" (UID: \"96f98ff0-1bb0-4529-9b06-43019479d9e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flq5r"
Sep 17 21:58:12.828823 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.827878    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/34b8ef2b-97ba-4562-87ee-8da0cb4604c7-systemd-units\") pod \"ovnkube-node-nrkd5\" (UID: \"34b8ef2b-97ba-4562-87ee-8da0cb4604c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5"
Sep 17 21:58:12.828823 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.827901    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/34b8ef2b-97ba-4562-87ee-8da0cb4604c7-host-run-netns\") pod \"ovnkube-node-nrkd5\" (UID: \"34b8ef2b-97ba-4562-87ee-8da0cb4604c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5"
Sep 17 21:58:12.828823 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.827934    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/34b8ef2b-97ba-4562-87ee-8da0cb4604c7-etc-openvswitch\") pod \"ovnkube-node-nrkd5\" (UID: \"34b8ef2b-97ba-4562-87ee-8da0cb4604c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5"
Sep 17 21:58:12.828823 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.827959    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlgr7\" (UniqueName: \"kubernetes.io/projected/992a8c01-f2ce-4523-9a33-56e12b134be2-kube-api-access-mlgr7\") pod \"tuned-26gb4\" (UID: \"992a8c01-f2ce-4523-9a33-56e12b134be2\") " pod="openshift-cluster-node-tuning-operator/tuned-26gb4"
Sep 17 21:58:12.828823 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.827994    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e5348473-b853-4265-8876-9e0b2c8943d8-cnibin\") pod \"multus-f6t75\" (UID: \"e5348473-b853-4265-8876-9e0b2c8943d8\") " pod="openshift-multus/multus-f6t75"
Sep 17 21:58:12.828823 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.828016    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e5348473-b853-4265-8876-9e0b2c8943d8-host-run-netns\") pod \"multus-f6t75\" (UID: \"e5348473-b853-4265-8876-9e0b2c8943d8\") " pod="openshift-multus/multus-f6t75"
Sep 17 21:58:12.828823 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.828040    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/34b8ef2b-97ba-4562-87ee-8da0cb4604c7-host-kubelet\") pod \"ovnkube-node-nrkd5\" (UID: \"34b8ef2b-97ba-4562-87ee-8da0cb4604c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5"
Sep 17 21:58:12.828823 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.828082    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/34b8ef2b-97ba-4562-87ee-8da0cb4604c7-node-log\") pod \"ovnkube-node-nrkd5\" (UID: \"34b8ef2b-97ba-4562-87ee-8da0cb4604c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5"
Sep 17 21:58:12.828823 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.828106    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/34b8ef2b-97ba-4562-87ee-8da0cb4604c7-host-cni-bin\") pod \"ovnkube-node-nrkd5\" (UID: \"34b8ef2b-97ba-4562-87ee-8da0cb4604c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5"
Sep 17 21:58:12.829874 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.828131    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lvhl\" (UniqueName: \"kubernetes.io/projected/9d2ac593-98b6-4b38-adef-a1bfbf1a168f-kube-api-access-7lvhl\") pod \"node-resolver-n2fp6\" (UID: \"9d2ac593-98b6-4b38-adef-a1bfbf1a168f\") " pod="openshift-dns/node-resolver-n2fp6"
Sep 17 21:58:12.829874 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.828177    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c20c2a19-2b30-48fb-bb14-b5321cf2f4d0-serviceca\") pod \"node-ca-hzx5m\" (UID: \"c20c2a19-2b30-48fb-bb14-b5321cf2f4d0\") " pod="openshift-image-registry/node-ca-hzx5m"
Sep 17 21:58:12.829874 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.828207    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g9xt\" (UniqueName: \"kubernetes.io/projected/c20c2a19-2b30-48fb-bb14-b5321cf2f4d0-kube-api-access-8g9xt\") pod \"node-ca-hzx5m\" (UID: \"c20c2a19-2b30-48fb-bb14-b5321cf2f4d0\") " pod="openshift-image-registry/node-ca-hzx5m"
Sep 17 21:58:12.829874 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.828231    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d6f6a71b-842d-48f0-93fc-28006b5c122f-iptables-alerter-script\") pod \"iptables-alerter-tkphq\" (UID: \"d6f6a71b-842d-48f0-93fc-28006b5c122f\") " pod="openshift-network-operator/iptables-alerter-tkphq"
Sep 17 21:58:12.829874 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.828254    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f004b93f-9438-4008-ad17-92150090d28c-system-cni-dir\") pod \"multus-additional-cni-plugins-wjt9h\" (UID: \"f004b93f-9438-4008-ad17-92150090d28c\") " pod="openshift-multus/multus-additional-cni-plugins-wjt9h"
Sep 17 21:58:12.829874 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.828281    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34b8ef2b-97ba-4562-87ee-8da0cb4604c7-host-run-ovn-kubernetes\") pod \"ovnkube-node-nrkd5\" (UID: \"34b8ef2b-97ba-4562-87ee-8da0cb4604c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5"
Sep 17 21:58:12.829874 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.828302    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/992a8c01-f2ce-4523-9a33-56e12b134be2-etc-sysconfig\") pod \"tuned-26gb4\" (UID: \"992a8c01-f2ce-4523-9a33-56e12b134be2\") " pod="openshift-cluster-node-tuning-operator/tuned-26gb4"
Sep 17 21:58:12.829874 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.828343    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrkvs\" (UniqueName: \"kubernetes.io/projected/3b714c1c-9f7e-430e-bc07-94cf3a75921c-kube-api-access-hrkvs\") pod \"network-check-target-bnj94\" (UID: \"3b714c1c-9f7e-430e-bc07-94cf3a75921c\") " pod="openshift-network-diagnostics/network-check-target-bnj94"
Sep 17 21:58:12.829874 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.828376    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4gmz\" (UniqueName: \"kubernetes.io/projected/96f98ff0-1bb0-4529-9b06-43019479d9e8-kube-api-access-d4gmz\") pod \"aws-ebs-csi-driver-node-flq5r\" (UID: \"96f98ff0-1bb0-4529-9b06-43019479d9e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flq5r"
Sep 17 21:58:12.829874 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.828410    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b93158dd-d718-446e-8e3b-38a2235ed7a7-agent-certs\") pod \"konnectivity-agent-j6ntd\" (UID: \"b93158dd-d718-446e-8e3b-38a2235ed7a7\") " pod="kube-system/konnectivity-agent-j6ntd"
Sep 17 21:58:12.829874 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.828432    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e5348473-b853-4265-8876-9e0b2c8943d8-multus-daemon-config\") pod \"multus-f6t75\" (UID: \"e5348473-b853-4265-8876-9e0b2c8943d8\") " pod="openshift-multus/multus-f6t75"
Sep 17 21:58:12.829874 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.828456    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/34b8ef2b-97ba-4562-87ee-8da0cb4604c7-run-systemd\") pod \"ovnkube-node-nrkd5\" (UID: \"34b8ef2b-97ba-4562-87ee-8da0cb4604c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5"
Sep 17 21:58:12.829874 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.828478    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5348473-b853-4265-8876-9e0b2c8943d8-etc-kubernetes\") pod \"multus-f6t75\" (UID: \"e5348473-b853-4265-8876-9e0b2c8943d8\") " pod="openshift-multus/multus-f6t75"
Sep 17 21:58:12.829874 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.828501    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45d5f94d-990c-4f9e-b1ab-980021c00716-metrics-certs\") pod \"network-metrics-daemon-xkk4g\" (UID: \"45d5f94d-990c-4f9e-b1ab-980021c00716\") " pod="openshift-multus/network-metrics-daemon-xkk4g"
Sep 17 21:58:12.829874 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.828533    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/34b8ef2b-97ba-4562-87ee-8da0cb4604c7-var-lib-openvswitch\") pod \"ovnkube-node-nrkd5\" (UID: \"34b8ef2b-97ba-4562-87ee-8da0cb4604c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5"
Sep 17 21:58:12.829874 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.828557    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfs4v\" (UniqueName: \"kubernetes.io/projected/34b8ef2b-97ba-4562-87ee-8da0cb4604c7-kube-api-access-zfs4v\") pod \"ovnkube-node-nrkd5\" (UID: \"34b8ef2b-97ba-4562-87ee-8da0cb4604c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5"
Sep 17 21:58:12.830756 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.828591    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/79a6aec6-e29a-45cf-881f-447b87f04557-dbus\") pod \"global-pull-secret-syncer-wxx9x\" (UID: \"79a6aec6-e29a-45cf-881f-447b87f04557\") " pod="kube-system/global-pull-secret-syncer-wxx9x"
Sep 17 21:58:12.830756 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.828616    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgbm2\" (UniqueName: \"kubernetes.io/projected/45d5f94d-990c-4f9e-b1ab-980021c00716-kube-api-access-cgbm2\") pod \"network-metrics-daemon-xkk4g\" (UID: \"45d5f94d-990c-4f9e-b1ab-980021c00716\") " pod="openshift-multus/network-metrics-daemon-xkk4g"
Sep 17 21:58:12.830756 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.828638    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/34b8ef2b-97ba-4562-87ee-8da0cb4604c7-run-ovn\") pod \"ovnkube-node-nrkd5\" (UID: \"34b8ef2b-97ba-4562-87ee-8da0cb4604c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5"
Sep 17 21:58:12.830756 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.828693    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/34b8ef2b-97ba-4562-87ee-8da0cb4604c7-log-socket\") pod \"ovnkube-node-nrkd5\" (UID: \"34b8ef2b-97ba-4562-87ee-8da0cb4604c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5"
Sep 17 21:58:12.830756 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.828725    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/34b8ef2b-97ba-4562-87ee-8da0cb4604c7-env-overrides\") pod \"ovnkube-node-nrkd5\" (UID: \"34b8ef2b-97ba-4562-87ee-8da0cb4604c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5"
Sep 17 21:58:12.830756 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.828768    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/34b8ef2b-97ba-4562-87ee-8da0cb4604c7-ovn-node-metrics-cert\") pod \"ovnkube-node-nrkd5\" (UID: \"34b8ef2b-97ba-4562-87ee-8da0cb4604c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5"
Sep 17 21:58:12.830756 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.828805    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/992a8c01-f2ce-4523-9a33-56e12b134be2-sys\") pod \"tuned-26gb4\" (UID: \"992a8c01-f2ce-4523-9a33-56e12b134be2\") " pod="openshift-cluster-node-tuning-operator/tuned-26gb4"
Sep 17 21:58:12.830756 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.828832    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e5348473-b853-4265-8876-9e0b2c8943d8-host-run-k8s-cni-cncf-io\") pod \"multus-f6t75\" (UID: \"e5348473-b853-4265-8876-9e0b2c8943d8\") " pod="openshift-multus/multus-f6t75"
Sep 17 21:58:12.830756 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.828859    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f004b93f-9438-4008-ad17-92150090d28c-cni-binary-copy\") pod \"multus-additional-cni-plugins-wjt9h\" (UID: \"f004b93f-9438-4008-ad17-92150090d28c\") " pod="openshift-multus/multus-additional-cni-plugins-wjt9h"
Sep 17 21:58:12.830756 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.828883    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/96f98ff0-1bb0-4529-9b06-43019479d9e8-sys-fs\") pod \"aws-ebs-csi-driver-node-flq5r\" (UID: \"96f98ff0-1bb0-4529-9b06-43019479d9e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flq5r"
Sep 17 21:58:12.830756 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.828909    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/992a8c01-f2ce-4523-9a33-56e12b134be2-etc-sysctl-conf\") pod \"tuned-26gb4\" (UID: \"992a8c01-f2ce-4523-9a33-56e12b134be2\") " pod="openshift-cluster-node-tuning-operator/tuned-26gb4"
Sep 17 21:58:12.830756 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.828933    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e5348473-b853-4265-8876-9e0b2c8943d8-host-var-lib-cni-bin\") pod \"multus-f6t75\" (UID: \"e5348473-b853-4265-8876-9e0b2c8943d8\") " pod="openshift-multus/multus-f6t75"
Sep 17 21:58:12.830756 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.828955    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e5348473-b853-4265-8876-9e0b2c8943d8-hostroot\") pod \"multus-f6t75\" (UID: \"e5348473-b853-4265-8876-9e0b2c8943d8\") " pod="openshift-multus/multus-f6t75"
Sep 17 21:58:12.830756 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.828980    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b93158dd-d718-446e-8e3b-38a2235ed7a7-konnectivity-ca\") pod \"konnectivity-agent-j6ntd\" (UID: \"b93158dd-d718-446e-8e3b-38a2235ed7a7\") " pod="kube-system/konnectivity-agent-j6ntd"
Sep 17 21:58:12.830756 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.829003    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/992a8c01-f2ce-4523-9a33-56e12b134be2-etc-modprobe-d\") pod \"tuned-26gb4\" (UID: \"992a8c01-f2ce-4523-9a33-56e12b134be2\") " pod="openshift-cluster-node-tuning-operator/tuned-26gb4"
Sep 17 21:58:12.830756 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.829025    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5348473-b853-4265-8876-9e0b2c8943d8-multus-cni-dir\") pod \"multus-f6t75\" (UID: \"e5348473-b853-4265-8876-9e0b2c8943d8\") " pod="openshift-multus/multus-f6t75"
Sep 17 21:58:12.831516 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.829049    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e5348473-b853-4265-8876-9e0b2c8943d8-host-var-lib-cni-multus\") pod \"multus-f6t75\" (UID: \"e5348473-b853-4265-8876-9e0b2c8943d8\") " pod="openshift-multus/multus-f6t75"
Sep 17 21:58:12.857731 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.857629    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 21:53:11 +0000 UTC" deadline="2028-06-05 21:04:19.572856614 +0000 UTC"
Sep 17 21:58:12.857731 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.857683    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15047h6m6.715177862s"
Sep 17 21:58:12.914066 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.914006    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-45.ec2.internal" event={"ID":"2e7e38f229d83af38da836b565cddb01","Type":"ContainerStarted","Data":"cea559b9bf958c63f0fc56a42e182dca75992f509726fed11dd0ca69ba12b376"}
Sep 17 21:58:12.915443 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.915415    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-45.ec2.internal" event={"ID":"26382571c0e2f6db815799fbfabf8da2","Type":"ContainerStarted","Data":"195fb5c9cec1a288ea13b33f6d55b015e8551749cddb177a33ad253d0284495e"}
Sep 17 21:58:12.929440 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.929349    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/34b8ef2b-97ba-4562-87ee-8da0cb4604c7-log-socket\") pod \"ovnkube-node-nrkd5\" (UID: \"34b8ef2b-97ba-4562-87ee-8da0cb4604c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5"
Sep 17 21:58:12.929440 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.929398    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/34b8ef2b-97ba-4562-87ee-8da0cb4604c7-env-overrides\") pod \"ovnkube-node-nrkd5\" (UID: \"34b8ef2b-97ba-4562-87ee-8da0cb4604c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5"
Sep 17 21:58:12.929440 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.929426    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/34b8ef2b-97ba-4562-87ee-8da0cb4604c7-ovn-node-metrics-cert\") pod \"ovnkube-node-nrkd5\" (UID: \"34b8ef2b-97ba-4562-87ee-8da0cb4604c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5"
Sep 17 21:58:12.929699 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.929453    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/992a8c01-f2ce-4523-9a33-56e12b134be2-sys\") pod \"tuned-26gb4\" (UID: \"992a8c01-f2ce-4523-9a33-56e12b134be2\") " pod="openshift-cluster-node-tuning-operator/tuned-26gb4"
Sep 17 21:58:12.929699 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.929478    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e5348473-b853-4265-8876-9e0b2c8943d8-host-run-k8s-cni-cncf-io\") pod \"multus-f6t75\" (UID: \"e5348473-b853-4265-8876-9e0b2c8943d8\") " pod="openshift-multus/multus-f6t75"
Sep 17 21:58:12.929699 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.929480    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/34b8ef2b-97ba-4562-87ee-8da0cb4604c7-log-socket\") pod \"ovnkube-node-nrkd5\" (UID: \"34b8ef2b-97ba-4562-87ee-8da0cb4604c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5"
Sep 17 21:58:12.929699 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.929501    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f004b93f-9438-4008-ad17-92150090d28c-cni-binary-copy\") pod \"multus-additional-cni-plugins-wjt9h\" (UID: \"f004b93f-9438-4008-ad17-92150090d28c\") " pod="openshift-multus/multus-additional-cni-plugins-wjt9h"
Sep 17 21:58:12.929699 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.929525    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/96f98ff0-1bb0-4529-9b06-43019479d9e8-sys-fs\") pod \"aws-ebs-csi-driver-node-flq5r\" (UID: \"96f98ff0-1bb0-4529-9b06-43019479d9e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flq5r"
Sep 17 21:58:12.929699 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.929547    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/992a8c01-f2ce-4523-9a33-56e12b134be2-etc-sysctl-conf\") pod \"tuned-26gb4\" (UID: \"992a8c01-f2ce-4523-9a33-56e12b134be2\") " pod="openshift-cluster-node-tuning-operator/tuned-26gb4"
Sep 17 21:58:12.929699 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.929570    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e5348473-b853-4265-8876-9e0b2c8943d8-host-var-lib-cni-bin\") pod \"multus-f6t75\" (UID: \"e5348473-b853-4265-8876-9e0b2c8943d8\") " pod="openshift-multus/multus-f6t75"
Sep 17 21:58:12.929699 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.929595    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e5348473-b853-4265-8876-9e0b2c8943d8-hostroot\") pod \"multus-f6t75\" (UID: \"e5348473-b853-4265-8876-9e0b2c8943d8\") " pod="openshift-multus/multus-f6t75"
Sep 17 21:58:12.929699 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.929620    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b93158dd-d718-446e-8e3b-38a2235ed7a7-konnectivity-ca\") pod \"konnectivity-agent-j6ntd\" (UID: \"b93158dd-d718-446e-8e3b-38a2235ed7a7\") " pod="kube-system/konnectivity-agent-j6ntd"
Sep 17 21:58:12.929699 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.929644    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/992a8c01-f2ce-4523-9a33-56e12b134be2-etc-modprobe-d\") pod \"tuned-26gb4\" (UID: \"992a8c01-f2ce-4523-9a33-56e12b134be2\") " pod="openshift-cluster-node-tuning-operator/tuned-26gb4"
Sep 17 21:58:12.929699 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.929677    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5348473-b853-4265-8876-9e0b2c8943d8-multus-cni-dir\") pod \"multus-f6t75\" (UID: \"e5348473-b853-4265-8876-9e0b2c8943d8\") " pod="openshift-multus/multus-f6t75"
Sep 17 21:58:12.929699 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.929699    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e5348473-b853-4265-8876-9e0b2c8943d8-host-var-lib-cni-multus\") pod \"multus-f6t75\" (UID: \"e5348473-b853-4265-8876-9e0b2c8943d8\") " pod="openshift-multus/multus-f6t75"
Sep 17 21:58:12.930360 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.929725    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f004b93f-9438-4008-ad17-92150090d28c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wjt9h\" (UID: \"f004b93f-9438-4008-ad17-92150090d28c\") " pod="openshift-multus/multus-additional-cni-plugins-wjt9h"
Sep 17 21:58:12.930360 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.929748    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/96f98ff0-1bb0-4529-9b06-43019479d9e8-socket-dir\") pod \"aws-ebs-csi-driver-node-flq5r\" (UID: \"96f98ff0-1bb0-4529-9b06-43019479d9e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flq5r"
Sep 17 21:58:12.930360 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.929766    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79a6aec6-e29a-45cf-881f-447b87f04557-original-pull-secret\") pod \"global-pull-secret-syncer-wxx9x\" (UID: \"79a6aec6-e29a-45cf-881f-447b87f04557\") " pod="kube-system/global-pull-secret-syncer-wxx9x"
Sep 17 21:58:12.930360 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.929790    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/992a8c01-f2ce-4523-9a33-56e12b134be2-etc-kubernetes\") pod \"tuned-26gb4\" (UID: \"992a8c01-f2ce-4523-9a33-56e12b134be2\") " pod="openshift-cluster-node-tuning-operator/tuned-26gb4"
Sep 17 21:58:12.930360 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.929806    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/992a8c01-f2ce-4523-9a33-56e12b134be2-etc-systemd\") pod \"tuned-26gb4\" (UID: \"992a8c01-f2ce-4523-9a33-56e12b134be2\") " pod="openshift-cluster-node-tuning-operator/tuned-26gb4"
Sep 17 21:58:12.930360 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.929827    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d5pc\" (UniqueName: \"kubernetes.io/projected/f004b93f-9438-4008-ad17-92150090d28c-kube-api-access-9d5pc\") pod \"multus-additional-cni-plugins-wjt9h\" (UID: \"f004b93f-9438-4008-ad17-92150090d28c\") " pod="openshift-multus/multus-additional-cni-plugins-wjt9h"
Sep 17 21:58:12.930360 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.929847    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/34b8ef2b-97ba-4562-87ee-8da0cb4604c7-host-cni-netd\") pod \"ovnkube-node-nrkd5\" (UID: \"34b8ef2b-97ba-4562-87ee-8da0cb4604c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5"
Sep 17 21:58:12.930360 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.929869    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/34b8ef2b-97ba-4562-87ee-8da0cb4604c7-ovnkube-config\") pod \"ovnkube-node-nrkd5\" (UID: \"34b8ef2b-97ba-4562-87ee-8da0cb4604c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5"
Sep 17 21:58:12.930360 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.929889    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/992a8c01-f2ce-4523-9a33-56e12b134be2-etc-sysctl-d\") pod \"tuned-26gb4\" (UID: \"992a8c01-f2ce-4523-9a33-56e12b134be2\") " pod="openshift-cluster-node-tuning-operator/tuned-26gb4"
Sep 17 21:58:12.930360 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.929909    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/992a8c01-f2ce-4523-9a33-56e12b134be2-var-lib-kubelet\") pod \"tuned-26gb4\" (UID: \"992a8c01-f2ce-4523-9a33-56e12b134be2\") " pod="openshift-cluster-node-tuning-operator/tuned-26gb4"
Sep 17 21:58:12.930360 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.929929    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/992a8c01-f2ce-4523-9a33-56e12b134be2-host\") pod \"tuned-26gb4\" (UID: \"992a8c01-f2ce-4523-9a33-56e12b134be2\") " pod="openshift-cluster-node-tuning-operator/tuned-26gb4"
Sep 17 21:58:12.930360 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.929970    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/992a8c01-f2ce-4523-9a33-56e12b134be2-tmp\") pod \"tuned-26gb4\" (UID: \"992a8c01-f2ce-4523-9a33-56e12b134be2\") " pod="openshift-cluster-node-tuning-operator/tuned-26gb4"
Sep 17 21:58:12.930360 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.929977    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e5348473-b853-4265-8876-9e0b2c8943d8-host-var-lib-cni-multus\") pod \"multus-f6t75\" (UID: \"e5348473-b853-4265-8876-9e0b2c8943d8\") " pod="openshift-multus/multus-f6t75"
Sep 17 21:58:12.930360 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.929993    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e5348473-b853-4265-8876-9e0b2c8943d8-multus-conf-dir\") pod \"multus-f6t75\" (UID: \"e5348473-b853-4265-8876-9e0b2c8943d8\") " pod="openshift-multus/multus-f6t75"
Sep 17 21:58:12.930360 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.930014    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/992a8c01-f2ce-4523-9a33-56e12b134be2-run\") pod \"tuned-26gb4\" (UID: \"992a8c01-f2ce-4523-9a33-56e12b134be2\") " pod="openshift-cluster-node-tuning-operator/tuned-26gb4"
Sep 17 21:58:12.930360 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.930038    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e5348473-b853-4265-8876-9e0b2c8943d8-cni-binary-copy\") pod \"multus-f6t75\" (UID: \"e5348473-b853-4265-8876-9e0b2c8943d8\") " pod="openshift-multus/multus-f6t75"
Sep 17 21:58:12.930360 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.930062    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e5348473-b853-4265-8876-9e0b2c8943d8-host-run-multus-certs\") pod \"multus-f6t75\" (UID: \"e5348473-b853-4265-8876-9e0b2c8943d8\") " pod="openshift-multus/multus-f6t75"
Sep 17 21:58:12.931024 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.930085    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ddn8\" (UniqueName: \"kubernetes.io/projected/e5348473-b853-4265-8876-9e0b2c8943d8-kube-api-access-4ddn8\") pod \"multus-f6t75\" (UID: \"e5348473-b853-4265-8876-9e0b2c8943d8\") " pod="openshift-multus/multus-f6t75"
Sep 17 21:58:12.931024 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.930108    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f004b93f-9438-4008-ad17-92150090d28c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wjt9h\" (UID: \"f004b93f-9438-4008-ad17-92150090d28c\") " pod="openshift-multus/multus-additional-cni-plugins-wjt9h"
Sep 17 21:58:12.931024 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.930111    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/34b8ef2b-97ba-4562-87ee-8da0cb4604c7-env-overrides\") pod \"ovnkube-node-nrkd5\" (UID: \"34b8ef2b-97ba-4562-87ee-8da0cb4604c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5"
Sep 17 21:58:12.931024 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.930132    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/96f98ff0-1bb0-4529-9b06-43019479d9e8-registration-dir\") pod \"aws-ebs-csi-driver-node-flq5r\" (UID: \"96f98ff0-1bb0-4529-9b06-43019479d9e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flq5r"
Sep 17 21:58:12.931024 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.930175    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/96f98ff0-1bb0-4529-9b06-43019479d9e8-device-dir\") pod \"aws-ebs-csi-driver-node-flq5r\" (UID: \"96f98ff0-1bb0-4529-9b06-43019479d9e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flq5r"
Sep 17 21:58:12.931024 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.930201    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/79a6aec6-e29a-45cf-881f-447b87f04557-kubelet-config\") pod \"global-pull-secret-syncer-wxx9x\" (UID: \"79a6aec6-e29a-45cf-881f-447b87f04557\") " pod="kube-system/global-pull-secret-syncer-wxx9x"
Sep 17 21:58:12.931024 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.930226    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34b8ef2b-97ba-4562-87ee-8da0cb4604c7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nrkd5\" (UID: \"34b8ef2b-97ba-4562-87ee-8da0cb4604c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5"
Sep 17 21:58:12.931024 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.930254    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d2ac593-98b6-4b38-adef-a1bfbf1a168f-tmp-dir\") pod \"node-resolver-n2fp6\" (UID: \"9d2ac593-98b6-4b38-adef-a1bfbf1a168f\") " pod="openshift-dns/node-resolver-n2fp6"
Sep 17 21:58:12.931024 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.930277    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c20c2a19-2b30-48fb-bb14-b5321cf2f4d0-host\") pod \"node-ca-hzx5m\" (UID: \"c20c2a19-2b30-48fb-bb14-b5321cf2f4d0\") " pod="openshift-image-registry/node-ca-hzx5m"
Sep 17 21:58:12.931024 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.930304    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/992a8c01-f2ce-4523-9a33-56e12b134be2-host\") pod \"tuned-26gb4\" (UID: \"992a8c01-f2ce-4523-9a33-56e12b134be2\") " pod="openshift-cluster-node-tuning-operator/tuned-26gb4"
Sep 17 21:58:12.931024 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.930318    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f004b93f-9438-4008-ad17-92150090d28c-cnibin\") pod \"multus-additional-cni-plugins-wjt9h\" (UID: \"f004b93f-9438-4008-ad17-92150090d28c\") " pod="openshift-multus/multus-additional-cni-plugins-wjt9h"
Sep 17 21:58:12.931024 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.930347    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/96f98ff0-1bb0-4529-9b06-43019479d9e8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-flq5r\" (UID: \"96f98ff0-1bb0-4529-9b06-43019479d9e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flq5r"
Sep 17 21:58:12.931024 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.930372    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/34b8ef2b-97ba-4562-87ee-8da0cb4604c7-ovnkube-script-lib\") pod \"ovnkube-node-nrkd5\" (UID: \"34b8ef2b-97ba-4562-87ee-8da0cb4604c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5"
Sep 17 21:58:12.931024 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.930418    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/992a8c01-f2ce-4523-9a33-56e12b134be2-lib-modules\") pod \"tuned-26gb4\" (UID: \"992a8c01-f2ce-4523-9a33-56e12b134be2\") " pod="openshift-cluster-node-tuning-operator/tuned-26gb4"
Sep 17 21:58:12.931024 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.930455    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f004b93f-9438-4008-ad17-92150090d28c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wjt9h\" (UID: \"f004b93f-9438-4008-ad17-92150090d28c\") " pod="openshift-multus/multus-additional-cni-plugins-wjt9h"
Sep 17 21:58:12.931024 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.930458    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/992a8c01-f2ce-4523-9a33-56e12b134be2-etc-tuned\") pod \"tuned-26gb4\" (UID: \"992a8c01-f2ce-4523-9a33-56e12b134be2\") " pod="openshift-cluster-node-tuning-operator/tuned-26gb4"
Sep 17 21:58:12.931024 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.930508    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mttzt\" (UniqueName: \"kubernetes.io/projected/d6f6a71b-842d-48f0-93fc-28006b5c122f-kube-api-access-mttzt\") pod \"iptables-alerter-tkphq\" (UID: \"d6f6a71b-842d-48f0-93fc-28006b5c122f\") " pod="openshift-network-operator/iptables-alerter-tkphq"
Sep 17 21:58:12.931718 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.930534    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e5348473-b853-4265-8876-9e0b2c8943d8-multus-socket-dir-parent\") pod \"multus-f6t75\" (UID: \"e5348473-b853-4265-8876-9e0b2c8943d8\") " pod="openshift-multus/multus-f6t75"
Sep 17 21:58:12.931718 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.930560    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5348473-b853-4265-8876-9e0b2c8943d8-host-var-lib-kubelet\") pod \"multus-f6t75\" (UID: \"e5348473-b853-4265-8876-9e0b2c8943d8\") " pod="openshift-multus/multus-f6t75"
Sep 17 21:58:12.931718 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.930577    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f004b93f-9438-4008-ad17-92150090d28c-cni-binary-copy\") pod \"multus-additional-cni-plugins-wjt9h\" (UID: \"f004b93f-9438-4008-ad17-92150090d28c\") " pod="openshift-multus/multus-additional-cni-plugins-wjt9h"
Sep 17 21:58:12.931718 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.930587    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f004b93f-9438-4008-ad17-92150090d28c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wjt9h\" (UID: \"f004b93f-9438-4008-ad17-92150090d28c\") " pod="openshift-multus/multus-additional-cni-plugins-wjt9h"
Sep 17 21:58:12.931718 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.930615    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/34b8ef2b-97ba-4562-87ee-8da0cb4604c7-host-slash\") pod \"ovnkube-node-nrkd5\" (UID: \"34b8ef2b-97ba-4562-87ee-8da0cb4604c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5"
Sep 17 21:58:12.931718 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.930637    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/34b8ef2b-97ba-4562-87ee-8da0cb4604c7-run-openvswitch\") pod \"ovnkube-node-nrkd5\" (UID: \"34b8ef2b-97ba-4562-87ee-8da0cb4604c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5"
Sep 17 21:58:12.931718 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.930650    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/96f98ff0-1bb0-4529-9b06-43019479d9e8-sys-fs\") pod \"aws-ebs-csi-driver-node-flq5r\" (UID: \"96f98ff0-1bb0-4529-9b06-43019479d9e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flq5r"
Sep 17 21:58:12.931718 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.930658    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9d2ac593-98b6-4b38-adef-a1bfbf1a168f-hosts-file\") pod \"node-resolver-n2fp6\" (UID: \"9d2ac593-98b6-4b38-adef-a1bfbf1a168f\") " pod="openshift-dns/node-resolver-n2fp6"
Sep 17 21:58:12.931718 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.930703    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9d2ac593-98b6-4b38-adef-a1bfbf1a168f-hosts-file\") pod \"node-resolver-n2fp6\" (UID: \"9d2ac593-98b6-4b38-adef-a1bfbf1a168f\") " pod="openshift-dns/node-resolver-n2fp6"
Sep 17 21:58:12.931718 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.930727    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d6f6a71b-842d-48f0-93fc-28006b5c122f-host-slash\") pod \"iptables-alerter-tkphq\" (UID: \"d6f6a71b-842d-48f0-93fc-28006b5c122f\") " pod="openshift-network-operator/iptables-alerter-tkphq"
Sep 17 21:58:12.931718 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.930736    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e5348473-b853-4265-8876-9e0b2c8943d8-multus-conf-dir\") pod \"multus-f6t75\" (UID: \"e5348473-b853-4265-8876-9e0b2c8943d8\") " pod="openshift-multus/multus-f6t75"
Sep 17 21:58:12.931718 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.930749    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5348473-b853-4265-8876-9e0b2c8943d8-system-cni-dir\") pod \"multus-f6t75\" (UID: \"e5348473-b853-4265-8876-9e0b2c8943d8\") " pod="openshift-multus/multus-f6t75"
Sep 17 21:58:12.931718 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.930770    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/96f98ff0-1bb0-4529-9b06-43019479d9e8-etc-selinux\") pod \"aws-ebs-csi-driver-node-flq5r\" (UID: \"96f98ff0-1bb0-4529-9b06-43019479d9e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flq5r"
Sep 17 21:58:12.931718 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.930779    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e5348473-b853-4265-8876-9e0b2c8943d8-host-run-k8s-cni-cncf-io\") pod \"multus-f6t75\" (UID: \"e5348473-b853-4265-8876-9e0b2c8943d8\") " pod="openshift-multus/multus-f6t75"
Sep 17 21:58:12.931718 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.930785    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/992a8c01-f2ce-4523-9a33-56e12b134be2-etc-sysctl-conf\") pod \"tuned-26gb4\" (UID: \"992a8c01-f2ce-4523-9a33-56e12b134be2\") " pod="openshift-cluster-node-tuning-operator/tuned-26gb4"
Sep 17 21:58:12.931718 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.930792    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/34b8ef2b-97ba-4562-87ee-8da0cb4604c7-systemd-units\") pod \"ovnkube-node-nrkd5\" (UID: \"34b8ef2b-97ba-4562-87ee-8da0cb4604c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5"
Sep 17 21:58:12.931718 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.929571    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/992a8c01-f2ce-4523-9a33-56e12b134be2-sys\") pod \"tuned-26gb4\" (UID: \"992a8c01-f2ce-4523-9a33-56e12b134be2\") " pod="openshift-cluster-node-tuning-operator/tuned-26gb4"
Sep 17 21:58:12.932514 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.930836    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e5348473-b853-4265-8876-9e0b2c8943d8-host-var-lib-cni-bin\") pod \"multus-f6t75\" (UID: \"e5348473-b853-4265-8876-9e0b2c8943d8\") " pod="openshift-multus/multus-f6t75"
Sep 17 21:58:12.932514 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.930893    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/992a8c01-f2ce-4523-9a33-56e12b134be2-run\") pod \"tuned-26gb4\" (UID: \"992a8c01-f2ce-4523-9a33-56e12b134be2\") " pod="openshift-cluster-node-tuning-operator/tuned-26gb4"
Sep 17 21:58:12.932514 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.930918    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/96f98ff0-1bb0-4529-9b06-43019479d9e8-socket-dir\") pod \"aws-ebs-csi-driver-node-flq5r\" (UID: \"96f98ff0-1bb0-4529-9b06-43019479d9e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flq5r"
Sep 17 21:58:12.932514 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:12.930926    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:12.932514 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.930960    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e5348473-b853-4265-8876-9e0b2c8943d8-hostroot\") pod \"multus-f6t75\" (UID: \"e5348473-b853-4265-8876-9e0b2c8943d8\") " pod="openshift-multus/multus-f6t75"
Sep 17 21:58:12.932514 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.931046    2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 21:58:12.932514 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.931090    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/992a8c01-f2ce-4523-9a33-56e12b134be2-etc-systemd\") pod \"tuned-26gb4\" (UID: \"992a8c01-f2ce-4523-9a33-56e12b134be2\") " pod="openshift-cluster-node-tuning-operator/tuned-26gb4"
Sep 17 21:58:12.932514 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.931422    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e5348473-b853-4265-8876-9e0b2c8943d8-cni-binary-copy\") pod \"multus-f6t75\" (UID: \"e5348473-b853-4265-8876-9e0b2c8943d8\") " pod="openshift-multus/multus-f6t75"
Sep 17 21:58:12.932514 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.931429    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/34b8ef2b-97ba-4562-87ee-8da0cb4604c7-host-cni-netd\") pod \"ovnkube-node-nrkd5\" (UID: \"34b8ef2b-97ba-4562-87ee-8da0cb4604c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5"
Sep 17 21:58:12.932514 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.931430    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b93158dd-d718-446e-8e3b-38a2235ed7a7-konnectivity-ca\") pod \"konnectivity-agent-j6ntd\" (UID: \"b93158dd-d718-446e-8e3b-38a2235ed7a7\") " pod="kube-system/konnectivity-agent-j6ntd"
Sep 17 21:58:12.932514 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.931475    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e5348473-b853-4265-8876-9e0b2c8943d8-host-run-multus-certs\") pod \"multus-f6t75\" (UID: \"e5348473-b853-4265-8876-9e0b2c8943d8\") " pod="openshift-multus/multus-f6t75"
Sep 17 21:58:12.932514 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.931506    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/992a8c01-f2ce-4523-9a33-56e12b134be2-etc-modprobe-d\") pod \"tuned-26gb4\" (UID: \"992a8c01-f2ce-4523-9a33-56e12b134be2\") " pod="openshift-cluster-node-tuning-operator/tuned-26gb4"
Sep 17 21:58:12.932514 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.931802    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5348473-b853-4265-8876-9e0b2c8943d8-multus-cni-dir\") pod \"multus-f6t75\" (UID: \"e5348473-b853-4265-8876-9e0b2c8943d8\") " pod="openshift-multus/multus-f6t75"
Sep 17 21:58:12.932514 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.931955    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f004b93f-9438-4008-ad17-92150090d28c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wjt9h\" (UID: \"f004b93f-9438-4008-ad17-92150090d28c\") " pod="openshift-multus/multus-additional-cni-plugins-wjt9h"
Sep 17 21:58:12.932514 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.931999    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/34b8ef2b-97ba-4562-87ee-8da0cb4604c7-ovnkube-config\") pod \"ovnkube-node-nrkd5\" (UID: \"34b8ef2b-97ba-4562-87ee-8da0cb4604c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5"
Sep 17 21:58:12.932514 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.932026    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/96f98ff0-1bb0-4529-9b06-43019479d9e8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-flq5r\" (UID: \"96f98ff0-1bb0-4529-9b06-43019479d9e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flq5r"
Sep 17 21:58:12.932514 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.932080    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c20c2a19-2b30-48fb-bb14-b5321cf2f4d0-host\") pod \"node-ca-hzx5m\" (UID: \"c20c2a19-2b30-48fb-bb14-b5321cf2f4d0\") " pod="openshift-image-registry/node-ca-hzx5m"
Sep 17 21:58:12.932514 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.932094    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d2ac593-98b6-4b38-adef-a1bfbf1a168f-tmp-dir\") pod \"node-resolver-n2fp6\" (UID: \"9d2ac593-98b6-4b38-adef-a1bfbf1a168f\") " pod="openshift-dns/node-resolver-n2fp6"
Sep 17 21:58:12.932514 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.932111    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/992a8c01-f2ce-4523-9a33-56e12b134be2-etc-sysctl-d\") pod \"tuned-26gb4\" (UID: \"992a8c01-f2ce-4523-9a33-56e12b134be2\") " pod="openshift-cluster-node-tuning-operator/tuned-26gb4"
Sep 17 21:58:12.933343 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.932152    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f004b93f-9438-4008-ad17-92150090d28c-cnibin\") pod \"multus-additional-cni-plugins-wjt9h\" (UID: \"f004b93f-9438-4008-ad17-92150090d28c\") " pod="openshift-multus/multus-additional-cni-plugins-wjt9h"
Sep 17 21:58:12.933343 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.932190    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/992a8c01-f2ce-4523-9a33-56e12b134be2-var-lib-kubelet\") pod \"tuned-26gb4\" (UID: \"992a8c01-f2ce-4523-9a33-56e12b134be2\") " pod="openshift-cluster-node-tuning-operator/tuned-26gb4"
Sep 17 21:58:12.933343 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.932213    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/79a6aec6-e29a-45cf-881f-447b87f04557-kubelet-config\") pod \"global-pull-secret-syncer-wxx9x\" (UID: \"79a6aec6-e29a-45cf-881f-447b87f04557\") " pod="kube-system/global-pull-secret-syncer-wxx9x"
Sep 17 21:58:12.933343 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.932209    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/96f98ff0-1bb0-4529-9b06-43019479d9e8-registration-dir\") pod \"aws-ebs-csi-driver-node-flq5r\" (UID: \"96f98ff0-1bb0-4529-9b06-43019479d9e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flq5r"
Sep 17 21:58:12.933343 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.932269    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/96f98ff0-1bb0-4529-9b06-43019479d9e8-device-dir\") pod \"aws-ebs-csi-driver-node-flq5r\" (UID: \"96f98ff0-1bb0-4529-9b06-43019479d9e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flq5r"
Sep 17 21:58:12.933343 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.932290    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34b8ef2b-97ba-4562-87ee-8da0cb4604c7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nrkd5\" (UID: \"34b8ef2b-97ba-4562-87ee-8da0cb4604c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5"
Sep 17 21:58:12.933343 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.932314    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d6f6a71b-842d-48f0-93fc-28006b5c122f-host-slash\") pod \"iptables-alerter-tkphq\" (UID: \"d6f6a71b-842d-48f0-93fc-28006b5c122f\") " pod="openshift-network-operator/iptables-alerter-tkphq"
Sep 17 21:58:12.933343 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.932362    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e5348473-b853-4265-8876-9e0b2c8943d8-multus-socket-dir-parent\") pod \"multus-f6t75\" (UID: \"e5348473-b853-4265-8876-9e0b2c8943d8\") " pod="openshift-multus/multus-f6t75"
Sep 17 21:58:12.933343 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.932416    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5348473-b853-4265-8876-9e0b2c8943d8-host-var-lib-kubelet\") pod \"multus-f6t75\" (UID: \"e5348473-b853-4265-8876-9e0b2c8943d8\") " pod="openshift-multus/multus-f6t75"
Sep 17 21:58:12.933343 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.931053    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/992a8c01-f2ce-4523-9a33-56e12b134be2-etc-kubernetes\") pod \"tuned-26gb4\" (UID: \"992a8c01-f2ce-4523-9a33-56e12b134be2\") " pod="openshift-cluster-node-tuning-operator/tuned-26gb4"
Sep 17 21:58:12.933343 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.932453    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/34b8ef2b-97ba-4562-87ee-8da0cb4604c7-ovnkube-script-lib\") pod \"ovnkube-node-nrkd5\" (UID: \"34b8ef2b-97ba-4562-87ee-8da0cb4604c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5"
Sep 17 21:58:12.933343 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.932472    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/34b8ef2b-97ba-4562-87ee-8da0cb4604c7-systemd-units\") pod \"ovnkube-node-nrkd5\" (UID: \"34b8ef2b-97ba-4562-87ee-8da0cb4604c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5"
Sep 17 21:58:12.933343 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.932489    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/34b8ef2b-97ba-4562-87ee-8da0cb4604c7-host-run-netns\") pod \"ovnkube-node-nrkd5\" (UID: \"34b8ef2b-97ba-4562-87ee-8da0cb4604c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5"
Sep 17 21:58:12.933343 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.932522    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5348473-b853-4265-8876-9e0b2c8943d8-system-cni-dir\") pod \"multus-f6t75\" (UID: \"e5348473-b853-4265-8876-9e0b2c8943d8\") " pod="openshift-multus/multus-f6t75"
Sep 17 21:58:12.933343 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.932538    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/34b8ef2b-97ba-4562-87ee-8da0cb4604c7-etc-openvswitch\") pod \"ovnkube-node-nrkd5\" (UID: \"34b8ef2b-97ba-4562-87ee-8da0cb4604c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5"
Sep 17 21:58:12.933343 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.932543    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/992a8c01-f2ce-4523-9a33-56e12b134be2-lib-modules\") pod \"tuned-26gb4\" (UID: \"992a8c01-f2ce-4523-9a33-56e12b134be2\") " pod="openshift-cluster-node-tuning-operator/tuned-26gb4"
Sep 17 21:58:12.933343 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.932574    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlgr7\" (UniqueName: \"kubernetes.io/projected/992a8c01-f2ce-4523-9a33-56e12b134be2-kube-api-access-mlgr7\") pod \"tuned-26gb4\" (UID: \"992a8c01-f2ce-4523-9a33-56e12b134be2\") " pod="openshift-cluster-node-tuning-operator/tuned-26gb4"
Sep 17 21:58:12.934260 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.932596    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/96f98ff0-1bb0-4529-9b06-43019479d9e8-etc-selinux\") pod \"aws-ebs-csi-driver-node-flq5r\" (UID: \"96f98ff0-1bb0-4529-9b06-43019479d9e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flq5r"
Sep 17 21:58:12.934260 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.932602    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e5348473-b853-4265-8876-9e0b2c8943d8-cnibin\") pod \"multus-f6t75\" (UID: \"e5348473-b853-4265-8876-9e0b2c8943d8\") " pod="openshift-multus/multus-f6t75"
Sep 17 21:58:12.934260 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.932630    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e5348473-b853-4265-8876-9e0b2c8943d8-host-run-netns\") pod \"multus-f6t75\" (UID: \"e5348473-b853-4265-8876-9e0b2c8943d8\") " pod="openshift-multus/multus-f6t75"
Sep 17 21:58:12.934260 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.932643    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f004b93f-9438-4008-ad17-92150090d28c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wjt9h\" (UID: \"f004b93f-9438-4008-ad17-92150090d28c\") " pod="openshift-multus/multus-additional-cni-plugins-wjt9h"
Sep 17 21:58:12.934260 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.932640    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/34b8ef2b-97ba-4562-87ee-8da0cb4604c7-host-slash\") pod \"ovnkube-node-nrkd5\" (UID: \"34b8ef2b-97ba-4562-87ee-8da0cb4604c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5"
Sep 17 21:58:12.934260 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.932654    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/34b8ef2b-97ba-4562-87ee-8da0cb4604c7-host-kubelet\") pod \"ovnkube-node-nrkd5\" (UID: \"34b8ef2b-97ba-4562-87ee-8da0cb4604c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5"
Sep 17 21:58:12.934260 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.932691    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/34b8ef2b-97ba-4562-87ee-8da0cb4604c7-etc-openvswitch\") pod \"ovnkube-node-nrkd5\" (UID: \"34b8ef2b-97ba-4562-87ee-8da0cb4604c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5"
Sep 17 21:58:12.934260 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.932695    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/34b8ef2b-97ba-4562-87ee-8da0cb4604c7-node-log\") pod \"ovnkube-node-nrkd5\" (UID: \"34b8ef2b-97ba-4562-87ee-8da0cb4604c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5"
Sep 17 21:58:12.934260 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:12.932731    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79a6aec6-e29a-45cf-881f-447b87f04557-original-pull-secret podName:79a6aec6-e29a-45cf-881f-447b87f04557 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:13.432710726 +0000 UTC m=+4.600663690 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/79a6aec6-e29a-45cf-881f-447b87f04557-original-pull-secret") pod "global-pull-secret-syncer-wxx9x" (UID: "79a6aec6-e29a-45cf-881f-447b87f04557") : object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:12.934260 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.932737    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/34b8ef2b-97ba-4562-87ee-8da0cb4604c7-node-log\") pod \"ovnkube-node-nrkd5\" (UID: \"34b8ef2b-97ba-4562-87ee-8da0cb4604c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5"
Sep 17 21:58:12.934260 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.932750    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/34b8ef2b-97ba-4562-87ee-8da0cb4604c7-host-kubelet\") pod \"ovnkube-node-nrkd5\" (UID: \"34b8ef2b-97ba-4562-87ee-8da0cb4604c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5"
Sep 17 21:58:12.934260 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.932758    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/34b8ef2b-97ba-4562-87ee-8da0cb4604c7-host-run-netns\") pod \"ovnkube-node-nrkd5\" (UID: \"34b8ef2b-97ba-4562-87ee-8da0cb4604c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5"
Sep 17 21:58:12.934260 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.932785    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e5348473-b853-4265-8876-9e0b2c8943d8-cnibin\") pod \"multus-f6t75\" (UID: \"e5348473-b853-4265-8876-9e0b2c8943d8\") " pod="openshift-multus/multus-f6t75"
Sep 17 21:58:12.934260 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.932805    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/34b8ef2b-97ba-4562-87ee-8da0cb4604c7-host-cni-bin\") pod \"ovnkube-node-nrkd5\" (UID: \"34b8ef2b-97ba-4562-87ee-8da0cb4604c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5"
Sep 17 21:58:12.934260 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.932808    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e5348473-b853-4265-8876-9e0b2c8943d8-host-run-netns\") pod \"multus-f6t75\" (UID: \"e5348473-b853-4265-8876-9e0b2c8943d8\") " pod="openshift-multus/multus-f6t75"
Sep 17 21:58:12.934260 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.932822    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lvhl\" (UniqueName: \"kubernetes.io/projected/9d2ac593-98b6-4b38-adef-a1bfbf1a168f-kube-api-access-7lvhl\") pod \"node-resolver-n2fp6\" (UID: \"9d2ac593-98b6-4b38-adef-a1bfbf1a168f\") " pod="openshift-dns/node-resolver-n2fp6"
Sep 17 21:58:12.934260 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.932839    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c20c2a19-2b30-48fb-bb14-b5321cf2f4d0-serviceca\") pod \"node-ca-hzx5m\" (UID: \"c20c2a19-2b30-48fb-bb14-b5321cf2f4d0\") " pod="openshift-image-registry/node-ca-hzx5m"
Sep 17 21:58:12.935170 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.932847    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/34b8ef2b-97ba-4562-87ee-8da0cb4604c7-host-cni-bin\") pod \"ovnkube-node-nrkd5\" (UID: \"34b8ef2b-97ba-4562-87ee-8da0cb4604c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5"
Sep 17 21:58:12.935170 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.932853    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g9xt\" (UniqueName: \"kubernetes.io/projected/c20c2a19-2b30-48fb-bb14-b5321cf2f4d0-kube-api-access-8g9xt\") pod \"node-ca-hzx5m\" (UID: \"c20c2a19-2b30-48fb-bb14-b5321cf2f4d0\") " pod="openshift-image-registry/node-ca-hzx5m"
Sep 17 21:58:12.935170 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.932851    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/34b8ef2b-97ba-4562-87ee-8da0cb4604c7-run-openvswitch\") pod \"ovnkube-node-nrkd5\" (UID: \"34b8ef2b-97ba-4562-87ee-8da0cb4604c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5"
Sep 17 21:58:12.935170 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.932977    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d6f6a71b-842d-48f0-93fc-28006b5c122f-iptables-alerter-script\") pod \"iptables-alerter-tkphq\" (UID: \"d6f6a71b-842d-48f0-93fc-28006b5c122f\") " pod="openshift-network-operator/iptables-alerter-tkphq"
Sep 17 21:58:12.935170 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.933008    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f004b93f-9438-4008-ad17-92150090d28c-system-cni-dir\") pod \"multus-additional-cni-plugins-wjt9h\" (UID: \"f004b93f-9438-4008-ad17-92150090d28c\") " pod="openshift-multus/multus-additional-cni-plugins-wjt9h"
Sep 17 21:58:12.935170 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.933050    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34b8ef2b-97ba-4562-87ee-8da0cb4604c7-host-run-ovn-kubernetes\") pod \"ovnkube-node-nrkd5\" (UID: \"34b8ef2b-97ba-4562-87ee-8da0cb4604c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5"
Sep 17 21:58:12.935170 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.933091    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34b8ef2b-97ba-4562-87ee-8da0cb4604c7-host-run-ovn-kubernetes\") pod \"ovnkube-node-nrkd5\" (UID: \"34b8ef2b-97ba-4562-87ee-8da0cb4604c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5"
Sep 17 21:58:12.935170 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.933092    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/992a8c01-f2ce-4523-9a33-56e12b134be2-etc-sysconfig\") pod \"tuned-26gb4\" (UID: \"992a8c01-f2ce-4523-9a33-56e12b134be2\") " pod="openshift-cluster-node-tuning-operator/tuned-26gb4"
Sep 17 21:58:12.935170 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.933131    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrkvs\" (UniqueName: \"kubernetes.io/projected/3b714c1c-9f7e-430e-bc07-94cf3a75921c-kube-api-access-hrkvs\") pod \"network-check-target-bnj94\" (UID: \"3b714c1c-9f7e-430e-bc07-94cf3a75921c\") " pod="openshift-network-diagnostics/network-check-target-bnj94"
Sep 17 21:58:12.935170 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.933174    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4gmz\" (UniqueName: \"kubernetes.io/projected/96f98ff0-1bb0-4529-9b06-43019479d9e8-kube-api-access-d4gmz\") pod \"aws-ebs-csi-driver-node-flq5r\" (UID: \"96f98ff0-1bb0-4529-9b06-43019479d9e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flq5r"
Sep 17 21:58:12.935170 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.933179    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f004b93f-9438-4008-ad17-92150090d28c-system-cni-dir\") pod \"multus-additional-cni-plugins-wjt9h\" (UID: \"f004b93f-9438-4008-ad17-92150090d28c\") " pod="openshift-multus/multus-additional-cni-plugins-wjt9h"
Sep 17 21:58:12.935170 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.933217    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b93158dd-d718-446e-8e3b-38a2235ed7a7-agent-certs\") pod \"konnectivity-agent-j6ntd\" (UID: \"b93158dd-d718-446e-8e3b-38a2235ed7a7\") " pod="kube-system/konnectivity-agent-j6ntd"
Sep 17 21:58:12.935170 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.933250    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e5348473-b853-4265-8876-9e0b2c8943d8-multus-daemon-config\") pod \"multus-f6t75\" (UID: \"e5348473-b853-4265-8876-9e0b2c8943d8\") " pod="openshift-multus/multus-f6t75"
Sep 17 21:58:12.935170 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.933278    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/34b8ef2b-97ba-4562-87ee-8da0cb4604c7-run-systemd\") pod \"ovnkube-node-nrkd5\" (UID: \"34b8ef2b-97ba-4562-87ee-8da0cb4604c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5"
Sep 17 21:58:12.935170 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.933134    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/992a8c01-f2ce-4523-9a33-56e12b134be2-etc-sysconfig\") pod \"tuned-26gb4\" (UID: \"992a8c01-f2ce-4523-9a33-56e12b134be2\") " pod="openshift-cluster-node-tuning-operator/tuned-26gb4"
Sep 17 21:58:12.935170 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.933303    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5348473-b853-4265-8876-9e0b2c8943d8-etc-kubernetes\") pod \"multus-f6t75\" (UID: \"e5348473-b853-4265-8876-9e0b2c8943d8\") " pod="openshift-multus/multus-f6t75"
Sep 17 21:58:12.935170 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.933252    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c20c2a19-2b30-48fb-bb14-b5321cf2f4d0-serviceca\") pod \"node-ca-hzx5m\" (UID: \"c20c2a19-2b30-48fb-bb14-b5321cf2f4d0\") " pod="openshift-image-registry/node-ca-hzx5m"
Sep 17 21:58:12.937326 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.933328    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45d5f94d-990c-4f9e-b1ab-980021c00716-metrics-certs\") pod \"network-metrics-daemon-xkk4g\" (UID: \"45d5f94d-990c-4f9e-b1ab-980021c00716\") " pod="openshift-multus/network-metrics-daemon-xkk4g"
Sep 17 21:58:12.937326 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.933352    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/34b8ef2b-97ba-4562-87ee-8da0cb4604c7-var-lib-openvswitch\") pod \"ovnkube-node-nrkd5\" (UID: \"34b8ef2b-97ba-4562-87ee-8da0cb4604c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5"
Sep 17 21:58:12.937326 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.933378    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfs4v\" (UniqueName: \"kubernetes.io/projected/34b8ef2b-97ba-4562-87ee-8da0cb4604c7-kube-api-access-zfs4v\") pod \"ovnkube-node-nrkd5\" (UID: \"34b8ef2b-97ba-4562-87ee-8da0cb4604c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5"
Sep 17 21:58:12.937326 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.933401    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/79a6aec6-e29a-45cf-881f-447b87f04557-dbus\") pod \"global-pull-secret-syncer-wxx9x\" (UID: \"79a6aec6-e29a-45cf-881f-447b87f04557\") " pod="kube-system/global-pull-secret-syncer-wxx9x"
Sep 17 21:58:12.937326 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.933427    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgbm2\" (UniqueName: \"kubernetes.io/projected/45d5f94d-990c-4f9e-b1ab-980021c00716-kube-api-access-cgbm2\") pod \"network-metrics-daemon-xkk4g\" (UID: \"45d5f94d-990c-4f9e-b1ab-980021c00716\") " pod="openshift-multus/network-metrics-daemon-xkk4g"
Sep 17 21:58:12.937326 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.933453    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/34b8ef2b-97ba-4562-87ee-8da0cb4604c7-run-ovn\") pod \"ovnkube-node-nrkd5\" (UID: \"34b8ef2b-97ba-4562-87ee-8da0cb4604c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5"
Sep 17 21:58:12.937326 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.933456    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/34b8ef2b-97ba-4562-87ee-8da0cb4604c7-run-systemd\") pod \"ovnkube-node-nrkd5\" (UID: \"34b8ef2b-97ba-4562-87ee-8da0cb4604c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5"
Sep 17 21:58:12.937326 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:12.933568    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:12.937326 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.933574    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d6f6a71b-842d-48f0-93fc-28006b5c122f-iptables-alerter-script\") pod \"iptables-alerter-tkphq\" (UID: \"d6f6a71b-842d-48f0-93fc-28006b5c122f\") " pod="openshift-network-operator/iptables-alerter-tkphq"
Sep 17 21:58:12.937326 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:12.933627    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45d5f94d-990c-4f9e-b1ab-980021c00716-metrics-certs podName:45d5f94d-990c-4f9e-b1ab-980021c00716 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:13.433612466 +0000 UTC m=+4.601565442 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/45d5f94d-990c-4f9e-b1ab-980021c00716-metrics-certs") pod "network-metrics-daemon-xkk4g" (UID: "45d5f94d-990c-4f9e-b1ab-980021c00716") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:12.937326 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.933634    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/34b8ef2b-97ba-4562-87ee-8da0cb4604c7-var-lib-openvswitch\") pod \"ovnkube-node-nrkd5\" (UID: \"34b8ef2b-97ba-4562-87ee-8da0cb4604c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5"
Sep 17 21:58:12.937326 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.933427    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5348473-b853-4265-8876-9e0b2c8943d8-etc-kubernetes\") pod \"multus-f6t75\" (UID: \"e5348473-b853-4265-8876-9e0b2c8943d8\") " pod="openshift-multus/multus-f6t75"
Sep 17 21:58:12.937326 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.933710    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/34b8ef2b-97ba-4562-87ee-8da0cb4604c7-run-ovn\") pod \"ovnkube-node-nrkd5\" (UID: \"34b8ef2b-97ba-4562-87ee-8da0cb4604c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5"
Sep 17 21:58:12.937326 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.933727    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/79a6aec6-e29a-45cf-881f-447b87f04557-dbus\") pod \"global-pull-secret-syncer-wxx9x\" (UID: \"79a6aec6-e29a-45cf-881f-447b87f04557\") " pod="kube-system/global-pull-secret-syncer-wxx9x"
Sep 17 21:58:12.937326 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.933757    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e5348473-b853-4265-8876-9e0b2c8943d8-multus-daemon-config\") pod \"multus-f6t75\" (UID: \"e5348473-b853-4265-8876-9e0b2c8943d8\") " pod="openshift-multus/multus-f6t75"
Sep 17 21:58:12.937326 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.936226    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b93158dd-d718-446e-8e3b-38a2235ed7a7-agent-certs\") pod \"konnectivity-agent-j6ntd\" (UID: \"b93158dd-d718-446e-8e3b-38a2235ed7a7\") " pod="kube-system/konnectivity-agent-j6ntd"
Sep 17 21:58:12.937326 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.936526    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/34b8ef2b-97ba-4562-87ee-8da0cb4604c7-ovn-node-metrics-cert\") pod \"ovnkube-node-nrkd5\" (UID: \"34b8ef2b-97ba-4562-87ee-8da0cb4604c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5"
Sep 17 21:58:12.938113 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.936785    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/992a8c01-f2ce-4523-9a33-56e12b134be2-etc-tuned\") pod \"tuned-26gb4\" (UID: \"992a8c01-f2ce-4523-9a33-56e12b134be2\") " pod="openshift-cluster-node-tuning-operator/tuned-26gb4"
Sep 17 21:58:12.938113 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.937666    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/992a8c01-f2ce-4523-9a33-56e12b134be2-tmp\") pod \"tuned-26gb4\" (UID: \"992a8c01-f2ce-4523-9a33-56e12b134be2\") " pod="openshift-cluster-node-tuning-operator/tuned-26gb4"
Sep 17 21:58:12.940175 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.940134    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ddn8\" (UniqueName: \"kubernetes.io/projected/e5348473-b853-4265-8876-9e0b2c8943d8-kube-api-access-4ddn8\") pod \"multus-f6t75\" (UID: \"e5348473-b853-4265-8876-9e0b2c8943d8\") " pod="openshift-multus/multus-f6t75"
Sep 17 21:58:12.940542 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.940526    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mttzt\" (UniqueName: \"kubernetes.io/projected/d6f6a71b-842d-48f0-93fc-28006b5c122f-kube-api-access-mttzt\") pod \"iptables-alerter-tkphq\" (UID: \"d6f6a71b-842d-48f0-93fc-28006b5c122f\") " pod="openshift-network-operator/iptables-alerter-tkphq"
Sep 17 21:58:12.941401 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.941358    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d5pc\" (UniqueName: \"kubernetes.io/projected/f004b93f-9438-4008-ad17-92150090d28c-kube-api-access-9d5pc\") pod \"multus-additional-cni-plugins-wjt9h\" (UID: \"f004b93f-9438-4008-ad17-92150090d28c\") " pod="openshift-multus/multus-additional-cni-plugins-wjt9h"
Sep 17 21:58:12.943187 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:12.942559    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 21:58:12.943187 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:12.942583    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 21:58:12.943187 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:12.942598    2563 projected.go:196] Error preparing data for projected volume kube-api-access-hrkvs for pod openshift-network-diagnostics/network-check-target-bnj94: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:12.943187 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:12.942682    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b714c1c-9f7e-430e-bc07-94cf3a75921c-kube-api-access-hrkvs podName:3b714c1c-9f7e-430e-bc07-94cf3a75921c nodeName:}" failed. No retries permitted until 2026-09-17 21:58:13.442664243 +0000 UTC m=+4.610617219 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hrkvs" (UniqueName: "kubernetes.io/projected/3b714c1c-9f7e-430e-bc07-94cf3a75921c-kube-api-access-hrkvs") pod "network-check-target-bnj94" (UID: "3b714c1c-9f7e-430e-bc07-94cf3a75921c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:12.943791 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.943766    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfs4v\" (UniqueName: \"kubernetes.io/projected/34b8ef2b-97ba-4562-87ee-8da0cb4604c7-kube-api-access-zfs4v\") pod \"ovnkube-node-nrkd5\" (UID: \"34b8ef2b-97ba-4562-87ee-8da0cb4604c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5"
Sep 17 21:58:12.944376 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.944352    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lvhl\" (UniqueName: \"kubernetes.io/projected/9d2ac593-98b6-4b38-adef-a1bfbf1a168f-kube-api-access-7lvhl\") pod \"node-resolver-n2fp6\" (UID: \"9d2ac593-98b6-4b38-adef-a1bfbf1a168f\") " pod="openshift-dns/node-resolver-n2fp6"
Sep 17 21:58:12.945385 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.945366    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4gmz\" (UniqueName: \"kubernetes.io/projected/96f98ff0-1bb0-4529-9b06-43019479d9e8-kube-api-access-d4gmz\") pod \"aws-ebs-csi-driver-node-flq5r\" (UID: \"96f98ff0-1bb0-4529-9b06-43019479d9e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flq5r"
Sep 17 21:58:12.946668 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.946645    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlgr7\" (UniqueName: \"kubernetes.io/projected/992a8c01-f2ce-4523-9a33-56e12b134be2-kube-api-access-mlgr7\") pod \"tuned-26gb4\" (UID: \"992a8c01-f2ce-4523-9a33-56e12b134be2\") " pod="openshift-cluster-node-tuning-operator/tuned-26gb4"
Sep 17 21:58:12.946859 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.946841    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g9xt\" (UniqueName: \"kubernetes.io/projected/c20c2a19-2b30-48fb-bb14-b5321cf2f4d0-kube-api-access-8g9xt\") pod \"node-ca-hzx5m\" (UID: \"c20c2a19-2b30-48fb-bb14-b5321cf2f4d0\") " pod="openshift-image-registry/node-ca-hzx5m"
Sep 17 21:58:12.947420 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:12.947399    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgbm2\" (UniqueName: \"kubernetes.io/projected/45d5f94d-990c-4f9e-b1ab-980021c00716-kube-api-access-cgbm2\") pod \"network-metrics-daemon-xkk4g\" (UID: \"45d5f94d-990c-4f9e-b1ab-980021c00716\") " pod="openshift-multus/network-metrics-daemon-xkk4g"
Sep 17 21:58:13.091115 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:13.091072    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flq5r"
Sep 17 21:58:13.099211 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:13.099181    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hzx5m"
Sep 17 21:58:13.109002 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:13.108980    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-26gb4"
Sep 17 21:58:13.114813 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:13.114793    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-j6ntd"
Sep 17 21:58:13.124559 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:13.124534    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tkphq"
Sep 17 21:58:13.139236 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:13.139219    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-f6t75"
Sep 17 21:58:13.147924 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:13.147901    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5"
Sep 17 21:58:13.156476 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:13.156452    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-n2fp6"
Sep 17 21:58:13.162920 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:13.162901    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wjt9h"
Sep 17 21:58:13.435895 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:13.435860    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79a6aec6-e29a-45cf-881f-447b87f04557-original-pull-secret\") pod \"global-pull-secret-syncer-wxx9x\" (UID: \"79a6aec6-e29a-45cf-881f-447b87f04557\") " pod="kube-system/global-pull-secret-syncer-wxx9x"
Sep 17 21:58:13.436069 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:13.435957    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45d5f94d-990c-4f9e-b1ab-980021c00716-metrics-certs\") pod \"network-metrics-daemon-xkk4g\" (UID: \"45d5f94d-990c-4f9e-b1ab-980021c00716\") " pod="openshift-multus/network-metrics-daemon-xkk4g"
Sep 17 21:58:13.436069 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:13.436044    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:13.436202 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:13.436126    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79a6aec6-e29a-45cf-881f-447b87f04557-original-pull-secret podName:79a6aec6-e29a-45cf-881f-447b87f04557 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:14.43610372 +0000 UTC m=+5.604056687 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/79a6aec6-e29a-45cf-881f-447b87f04557-original-pull-secret") pod "global-pull-secret-syncer-wxx9x" (UID: "79a6aec6-e29a-45cf-881f-447b87f04557") : object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:13.436202 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:13.436053    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:13.436294 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:13.436210    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45d5f94d-990c-4f9e-b1ab-980021c00716-metrics-certs podName:45d5f94d-990c-4f9e-b1ab-980021c00716 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:14.4361906 +0000 UTC m=+5.604143565 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/45d5f94d-990c-4f9e-b1ab-980021c00716-metrics-certs") pod "network-metrics-daemon-xkk4g" (UID: "45d5f94d-990c-4f9e-b1ab-980021c00716") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:13.536857 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:13.536819    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrkvs\" (UniqueName: \"kubernetes.io/projected/3b714c1c-9f7e-430e-bc07-94cf3a75921c-kube-api-access-hrkvs\") pod \"network-check-target-bnj94\" (UID: \"3b714c1c-9f7e-430e-bc07-94cf3a75921c\") " pod="openshift-network-diagnostics/network-check-target-bnj94"
Sep 17 21:58:13.537022 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:13.536997    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 21:58:13.537060 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:13.537028    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 21:58:13.537060 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:13.537043    2563 projected.go:196] Error preparing data for projected volume kube-api-access-hrkvs for pod openshift-network-diagnostics/network-check-target-bnj94: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:13.537174 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:13.537151    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b714c1c-9f7e-430e-bc07-94cf3a75921c-kube-api-access-hrkvs podName:3b714c1c-9f7e-430e-bc07-94cf3a75921c nodeName:}" failed. No retries permitted until 2026-09-17 21:58:14.537120651 +0000 UTC m=+5.705073644 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hrkvs" (UniqueName: "kubernetes.io/projected/3b714c1c-9f7e-430e-bc07-94cf3a75921c-kube-api-access-hrkvs") pod "network-check-target-bnj94" (UID: "3b714c1c-9f7e-430e-bc07-94cf3a75921c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:13.858528 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:13.858429    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 21:53:11 +0000 UTC" deadline="2028-05-11 21:37:11.139814615 +0000 UTC"
Sep 17 21:58:13.858528 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:13.858476    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14447h38m57.281342284s"
Sep 17 21:58:13.990031 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:13.989823    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf004b93f_9438_4008_ad17_92150090d28c.slice/crio-1031b20450abb93ac9a2b256823f734bd845ce948282d8297a3d4d0460dceb66 WatchSource:0}: Error finding container 1031b20450abb93ac9a2b256823f734bd845ce948282d8297a3d4d0460dceb66: Status 404 returned error can't find the container with id 1031b20450abb93ac9a2b256823f734bd845ce948282d8297a3d4d0460dceb66
Sep 17 21:58:13.991415 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:13.991368    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod992a8c01_f2ce_4523_9a33_56e12b134be2.slice/crio-c3997305b2e49b020dae79d4b8fcb0b52b224589852c8c304bb0e6b081e7bf23 WatchSource:0}: Error finding container c3997305b2e49b020dae79d4b8fcb0b52b224589852c8c304bb0e6b081e7bf23: Status 404 returned error can't find the container with id c3997305b2e49b020dae79d4b8fcb0b52b224589852c8c304bb0e6b081e7bf23
Sep 17 21:58:13.992362 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:13.992335    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d2ac593_98b6_4b38_adef_a1bfbf1a168f.slice/crio-b8b2e19e0016ef3018da652cd39cd59ca459482116c4ce1abd42735c975db3ea WatchSource:0}: Error finding container b8b2e19e0016ef3018da652cd39cd59ca459482116c4ce1abd42735c975db3ea: Status 404 returned error can't find the container with id b8b2e19e0016ef3018da652cd39cd59ca459482116c4ce1abd42735c975db3ea
Sep 17 21:58:13.995166 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:13.995117    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6f6a71b_842d_48f0_93fc_28006b5c122f.slice/crio-0b1d83f14e8afb5299cb256738a670c065b5c72b02f1a0eefea1f406cc8862cb WatchSource:0}: Error finding container 0b1d83f14e8afb5299cb256738a670c065b5c72b02f1a0eefea1f406cc8862cb: Status 404 returned error can't find the container with id 0b1d83f14e8afb5299cb256738a670c065b5c72b02f1a0eefea1f406cc8862cb
Sep 17 21:58:13.996459 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:13.996438    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34b8ef2b_97ba_4562_87ee_8da0cb4604c7.slice/crio-b9985595b95041aabb005a712d0d6cb91dc269a85e7d1953f9bd831e2e3d9e87 WatchSource:0}: Error finding container b9985595b95041aabb005a712d0d6cb91dc269a85e7d1953f9bd831e2e3d9e87: Status 404 returned error can't find the container with id b9985595b95041aabb005a712d0d6cb91dc269a85e7d1953f9bd831e2e3d9e87
Sep 17 21:58:13.997274 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:13.997250    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96f98ff0_1bb0_4529_9b06_43019479d9e8.slice/crio-e5de853bc33a36edc7d03a71a6b0c3e53db8f5177df27f8a7429f34435e4a2dd WatchSource:0}: Error finding container e5de853bc33a36edc7d03a71a6b0c3e53db8f5177df27f8a7429f34435e4a2dd: Status 404 returned error can't find the container with id e5de853bc33a36edc7d03a71a6b0c3e53db8f5177df27f8a7429f34435e4a2dd
Sep 17 21:58:13.999128 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:13.999105    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb93158dd_d718_446e_8e3b_38a2235ed7a7.slice/crio-81b83729aa0e341fd0105a2d46ddad3a105f53e3b7371ea05cd919ad8425ca2c WatchSource:0}: Error finding container 81b83729aa0e341fd0105a2d46ddad3a105f53e3b7371ea05cd919ad8425ca2c: Status 404 returned error can't find the container with id 81b83729aa0e341fd0105a2d46ddad3a105f53e3b7371ea05cd919ad8425ca2c
Sep 17 21:58:14.441687 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:14.441646    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45d5f94d-990c-4f9e-b1ab-980021c00716-metrics-certs\") pod \"network-metrics-daemon-xkk4g\" (UID: \"45d5f94d-990c-4f9e-b1ab-980021c00716\") " pod="openshift-multus/network-metrics-daemon-xkk4g"
Sep 17 21:58:14.441856 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:14.441722    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79a6aec6-e29a-45cf-881f-447b87f04557-original-pull-secret\") pod \"global-pull-secret-syncer-wxx9x\" (UID: \"79a6aec6-e29a-45cf-881f-447b87f04557\") " pod="kube-system/global-pull-secret-syncer-wxx9x"
Sep 17 21:58:14.441936 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:14.441868    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:14.441988 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:14.441958    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79a6aec6-e29a-45cf-881f-447b87f04557-original-pull-secret podName:79a6aec6-e29a-45cf-881f-447b87f04557 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:16.441938722 +0000 UTC m=+7.609891687 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/79a6aec6-e29a-45cf-881f-447b87f04557-original-pull-secret") pod "global-pull-secret-syncer-wxx9x" (UID: "79a6aec6-e29a-45cf-881f-447b87f04557") : object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:14.442457 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:14.442438    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:14.442526 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:14.442506    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45d5f94d-990c-4f9e-b1ab-980021c00716-metrics-certs podName:45d5f94d-990c-4f9e-b1ab-980021c00716 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:16.442482938 +0000 UTC m=+7.610435920 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/45d5f94d-990c-4f9e-b1ab-980021c00716-metrics-certs") pod "network-metrics-daemon-xkk4g" (UID: "45d5f94d-990c-4f9e-b1ab-980021c00716") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:14.542410 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:14.542329    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrkvs\" (UniqueName: \"kubernetes.io/projected/3b714c1c-9f7e-430e-bc07-94cf3a75921c-kube-api-access-hrkvs\") pod \"network-check-target-bnj94\" (UID: \"3b714c1c-9f7e-430e-bc07-94cf3a75921c\") " pod="openshift-network-diagnostics/network-check-target-bnj94"
Sep 17 21:58:14.542578 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:14.542546    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 21:58:14.542578 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:14.542571    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 21:58:14.542678 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:14.542584    2563 projected.go:196] Error preparing data for projected volume kube-api-access-hrkvs for pod openshift-network-diagnostics/network-check-target-bnj94: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:14.542678 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:14.542646    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b714c1c-9f7e-430e-bc07-94cf3a75921c-kube-api-access-hrkvs podName:3b714c1c-9f7e-430e-bc07-94cf3a75921c nodeName:}" failed. No retries permitted until 2026-09-17 21:58:16.542625684 +0000 UTC m=+7.710578650 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hrkvs" (UniqueName: "kubernetes.io/projected/3b714c1c-9f7e-430e-bc07-94cf3a75921c-kube-api-access-hrkvs") pod "network-check-target-bnj94" (UID: "3b714c1c-9f7e-430e-bc07-94cf3a75921c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:14.909332 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:14.909258    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bnj94"
Sep 17 21:58:14.909332 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:14.909299    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkk4g"
Sep 17 21:58:14.910065 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:14.909396    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bnj94" podUID="3b714c1c-9f7e-430e-bc07-94cf3a75921c"
Sep 17 21:58:14.910065 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:14.909821    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkk4g" podUID="45d5f94d-990c-4f9e-b1ab-980021c00716"
Sep 17 21:58:14.910065 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:14.909881    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wxx9x"
Sep 17 21:58:14.910065 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:14.909955    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wxx9x" podUID="79a6aec6-e29a-45cf-881f-447b87f04557"
Sep 17 21:58:14.938303 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:14.938249    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-45.ec2.internal" event={"ID":"2e7e38f229d83af38da836b565cddb01","Type":"ContainerStarted","Data":"fc7b270dbd93c29d8bca83cf74405eee86507984fb7dcc765b450195d2528432"}
Sep 17 21:58:14.942764 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:14.942625    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5" event={"ID":"34b8ef2b-97ba-4562-87ee-8da0cb4604c7","Type":"ContainerStarted","Data":"b9985595b95041aabb005a712d0d6cb91dc269a85e7d1953f9bd831e2e3d9e87"}
Sep 17 21:58:14.947467 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:14.947437    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-j6ntd" event={"ID":"b93158dd-d718-446e-8e3b-38a2235ed7a7","Type":"ContainerStarted","Data":"81b83729aa0e341fd0105a2d46ddad3a105f53e3b7371ea05cd919ad8425ca2c"}
Sep 17 21:58:14.961251 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:14.961189    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flq5r" event={"ID":"96f98ff0-1bb0-4529-9b06-43019479d9e8","Type":"ContainerStarted","Data":"e5de853bc33a36edc7d03a71a6b0c3e53db8f5177df27f8a7429f34435e4a2dd"}
Sep 17 21:58:14.982060 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:14.981960    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tkphq" event={"ID":"d6f6a71b-842d-48f0-93fc-28006b5c122f","Type":"ContainerStarted","Data":"0b1d83f14e8afb5299cb256738a670c065b5c72b02f1a0eefea1f406cc8862cb"}
Sep 17 21:58:14.988088 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:14.987999    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-n2fp6" event={"ID":"9d2ac593-98b6-4b38-adef-a1bfbf1a168f","Type":"ContainerStarted","Data":"b8b2e19e0016ef3018da652cd39cd59ca459482116c4ce1abd42735c975db3ea"}
Sep 17 21:58:14.998207 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:14.997493    2563 generic.go:356] "Generic (PLEG): container finished" podID="26382571c0e2f6db815799fbfabf8da2" containerID="5aab1b2a37e66efdc6dfe4b1e90d87a251dea5483bfa567c840c939c7bf98008" exitCode=0
Sep 17 21:58:14.998207 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:14.997817    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-45.ec2.internal" event={"ID":"26382571c0e2f6db815799fbfabf8da2","Type":"ContainerDied","Data":"5aab1b2a37e66efdc6dfe4b1e90d87a251dea5483bfa567c840c939c7bf98008"}
Sep 17 21:58:15.013047 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:15.011902    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-45.ec2.internal" podStartSLOduration=4.011880974 podStartE2EDuration="4.011880974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:14.955915601 +0000 UTC m=+6.123868582" watchObservedRunningTime="2026-09-17 21:58:15.011880974 +0000 UTC m=+6.179833972"
Sep 17 21:58:15.019085 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:15.019004    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-f6t75" event={"ID":"e5348473-b853-4265-8876-9e0b2c8943d8","Type":"ContainerStarted","Data":"e119ea293e64f90056f1759bd39307dbba77495359026434eaad097428056051"}
Sep 17 21:58:15.046641 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:15.046604    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hzx5m" event={"ID":"c20c2a19-2b30-48fb-bb14-b5321cf2f4d0","Type":"ContainerStarted","Data":"1af32b4398a1fc8a6cc0f1289bbc1189298aa137293d78e2a1fbab2d10d77120"}
Sep 17 21:58:15.050429 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:15.049190    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-26gb4" event={"ID":"992a8c01-f2ce-4523-9a33-56e12b134be2","Type":"ContainerStarted","Data":"c3997305b2e49b020dae79d4b8fcb0b52b224589852c8c304bb0e6b081e7bf23"}
Sep 17 21:58:15.061918 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:15.061885    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wjt9h" event={"ID":"f004b93f-9438-4008-ad17-92150090d28c","Type":"ContainerStarted","Data":"1031b20450abb93ac9a2b256823f734bd845ce948282d8297a3d4d0460dceb66"}
Sep 17 21:58:16.083495 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:16.083202    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-45.ec2.internal" event={"ID":"26382571c0e2f6db815799fbfabf8da2","Type":"ContainerStarted","Data":"d7897935d4e83786cbad257fc3c917e196472c0f6a7ff865968f154ed4ef2bb3"}
Sep 17 21:58:16.456883 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:16.456764    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45d5f94d-990c-4f9e-b1ab-980021c00716-metrics-certs\") pod \"network-metrics-daemon-xkk4g\" (UID: \"45d5f94d-990c-4f9e-b1ab-980021c00716\") " pod="openshift-multus/network-metrics-daemon-xkk4g"
Sep 17 21:58:16.456883 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:16.456828    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79a6aec6-e29a-45cf-881f-447b87f04557-original-pull-secret\") pod \"global-pull-secret-syncer-wxx9x\" (UID: \"79a6aec6-e29a-45cf-881f-447b87f04557\") " pod="kube-system/global-pull-secret-syncer-wxx9x"
Sep 17 21:58:16.457111 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:16.456966    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:16.457111 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:16.457029    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79a6aec6-e29a-45cf-881f-447b87f04557-original-pull-secret podName:79a6aec6-e29a-45cf-881f-447b87f04557 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:20.45701083 +0000 UTC m=+11.624963796 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/79a6aec6-e29a-45cf-881f-447b87f04557-original-pull-secret") pod "global-pull-secret-syncer-wxx9x" (UID: "79a6aec6-e29a-45cf-881f-447b87f04557") : object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:16.457449 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:16.457431    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:16.457522 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:16.457486    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45d5f94d-990c-4f9e-b1ab-980021c00716-metrics-certs podName:45d5f94d-990c-4f9e-b1ab-980021c00716 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:20.457470793 +0000 UTC m=+11.625423759 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/45d5f94d-990c-4f9e-b1ab-980021c00716-metrics-certs") pod "network-metrics-daemon-xkk4g" (UID: "45d5f94d-990c-4f9e-b1ab-980021c00716") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:16.557963 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:16.557915    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrkvs\" (UniqueName: \"kubernetes.io/projected/3b714c1c-9f7e-430e-bc07-94cf3a75921c-kube-api-access-hrkvs\") pod \"network-check-target-bnj94\" (UID: \"3b714c1c-9f7e-430e-bc07-94cf3a75921c\") " pod="openshift-network-diagnostics/network-check-target-bnj94"
Sep 17 21:58:16.558191 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:16.558127    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 21:58:16.558191 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:16.558170    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 21:58:16.558191 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:16.558185    2563 projected.go:196] Error preparing data for projected volume kube-api-access-hrkvs for pod openshift-network-diagnostics/network-check-target-bnj94: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:16.558371 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:16.558245    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b714c1c-9f7e-430e-bc07-94cf3a75921c-kube-api-access-hrkvs podName:3b714c1c-9f7e-430e-bc07-94cf3a75921c nodeName:}" failed. No retries permitted until 2026-09-17 21:58:20.558227365 +0000 UTC m=+11.726180348 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hrkvs" (UniqueName: "kubernetes.io/projected/3b714c1c-9f7e-430e-bc07-94cf3a75921c-kube-api-access-hrkvs") pod "network-check-target-bnj94" (UID: "3b714c1c-9f7e-430e-bc07-94cf3a75921c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:16.910185 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:16.909305    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bnj94"
Sep 17 21:58:16.910185 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:16.909517    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bnj94" podUID="3b714c1c-9f7e-430e-bc07-94cf3a75921c"
Sep 17 21:58:16.910185 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:16.909961    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wxx9x"
Sep 17 21:58:16.910185 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:16.910071    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wxx9x" podUID="79a6aec6-e29a-45cf-881f-447b87f04557"
Sep 17 21:58:16.910799 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:16.910608    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkk4g"
Sep 17 21:58:16.910799 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:16.910736    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkk4g" podUID="45d5f94d-990c-4f9e-b1ab-980021c00716"
Sep 17 21:58:18.909032 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:18.908982    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bnj94"
Sep 17 21:58:18.909553 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:18.909116    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bnj94" podUID="3b714c1c-9f7e-430e-bc07-94cf3a75921c"
Sep 17 21:58:18.909553 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:18.909274    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkk4g"
Sep 17 21:58:18.909553 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:18.909275    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wxx9x"
Sep 17 21:58:18.909553 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:18.909408    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkk4g" podUID="45d5f94d-990c-4f9e-b1ab-980021c00716"
Sep 17 21:58:18.910436 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:18.910391    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wxx9x" podUID="79a6aec6-e29a-45cf-881f-447b87f04557"
Sep 17 21:58:20.492194 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:20.492156    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45d5f94d-990c-4f9e-b1ab-980021c00716-metrics-certs\") pod \"network-metrics-daemon-xkk4g\" (UID: \"45d5f94d-990c-4f9e-b1ab-980021c00716\") " pod="openshift-multus/network-metrics-daemon-xkk4g"
Sep 17 21:58:20.492789 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:20.492215    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79a6aec6-e29a-45cf-881f-447b87f04557-original-pull-secret\") pod \"global-pull-secret-syncer-wxx9x\" (UID: \"79a6aec6-e29a-45cf-881f-447b87f04557\") " pod="kube-system/global-pull-secret-syncer-wxx9x"
Sep 17 21:58:20.492789 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:20.492338    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:20.492789 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:20.492410    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79a6aec6-e29a-45cf-881f-447b87f04557-original-pull-secret podName:79a6aec6-e29a-45cf-881f-447b87f04557 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:28.49238048 +0000 UTC m=+19.660333456 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/79a6aec6-e29a-45cf-881f-447b87f04557-original-pull-secret") pod "global-pull-secret-syncer-wxx9x" (UID: "79a6aec6-e29a-45cf-881f-447b87f04557") : object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:20.492789 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:20.492777    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:20.493019 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:20.492822    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45d5f94d-990c-4f9e-b1ab-980021c00716-metrics-certs podName:45d5f94d-990c-4f9e-b1ab-980021c00716 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:28.492807664 +0000 UTC m=+19.660760631 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/45d5f94d-990c-4f9e-b1ab-980021c00716-metrics-certs") pod "network-metrics-daemon-xkk4g" (UID: "45d5f94d-990c-4f9e-b1ab-980021c00716") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:20.593293 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:20.593249    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrkvs\" (UniqueName: \"kubernetes.io/projected/3b714c1c-9f7e-430e-bc07-94cf3a75921c-kube-api-access-hrkvs\") pod \"network-check-target-bnj94\" (UID: \"3b714c1c-9f7e-430e-bc07-94cf3a75921c\") " pod="openshift-network-diagnostics/network-check-target-bnj94"
Sep 17 21:58:20.593486 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:20.593446    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 21:58:20.593486 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:20.593475    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 21:58:20.593604 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:20.593489    2563 projected.go:196] Error preparing data for projected volume kube-api-access-hrkvs for pod openshift-network-diagnostics/network-check-target-bnj94: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:20.593604 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:20.593561    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b714c1c-9f7e-430e-bc07-94cf3a75921c-kube-api-access-hrkvs podName:3b714c1c-9f7e-430e-bc07-94cf3a75921c nodeName:}" failed. No retries permitted until 2026-09-17 21:58:28.593542774 +0000 UTC m=+19.761495748 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hrkvs" (UniqueName: "kubernetes.io/projected/3b714c1c-9f7e-430e-bc07-94cf3a75921c-kube-api-access-hrkvs") pod "network-check-target-bnj94" (UID: "3b714c1c-9f7e-430e-bc07-94cf3a75921c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:20.910498 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:20.909697    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wxx9x"
Sep 17 21:58:20.910498 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:20.909809    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wxx9x" podUID="79a6aec6-e29a-45cf-881f-447b87f04557"
Sep 17 21:58:20.910498 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:20.910190    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkk4g"
Sep 17 21:58:20.910498 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:20.910295    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkk4g" podUID="45d5f94d-990c-4f9e-b1ab-980021c00716"
Sep 17 21:58:20.910498 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:20.910350    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bnj94"
Sep 17 21:58:20.910498 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:20.910414    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bnj94" podUID="3b714c1c-9f7e-430e-bc07-94cf3a75921c"
Sep 17 21:58:22.909542 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:22.909413    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bnj94"
Sep 17 21:58:22.909542 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:22.909458    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkk4g"
Sep 17 21:58:22.909542 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:22.909421    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wxx9x"
Sep 17 21:58:22.910105 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:22.909551    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bnj94" podUID="3b714c1c-9f7e-430e-bc07-94cf3a75921c"
Sep 17 21:58:22.910105 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:22.909640    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wxx9x" podUID="79a6aec6-e29a-45cf-881f-447b87f04557"
Sep 17 21:58:22.910105 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:22.909723    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkk4g" podUID="45d5f94d-990c-4f9e-b1ab-980021c00716"
Sep 17 21:58:24.909235 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:24.909193    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wxx9x"
Sep 17 21:58:24.909235 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:24.909230    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkk4g"
Sep 17 21:58:24.909823 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:24.909230    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bnj94"
Sep 17 21:58:24.909823 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:24.909335    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wxx9x" podUID="79a6aec6-e29a-45cf-881f-447b87f04557"
Sep 17 21:58:24.909823 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:24.909436    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkk4g" podUID="45d5f94d-990c-4f9e-b1ab-980021c00716"
Sep 17 21:58:24.909823 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:24.909535    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bnj94" podUID="3b714c1c-9f7e-430e-bc07-94cf3a75921c"
Sep 17 21:58:26.909049 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:26.909015    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wxx9x"
Sep 17 21:58:26.909535 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:26.909028    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bnj94"
Sep 17 21:58:26.909535 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:26.909160    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wxx9x" podUID="79a6aec6-e29a-45cf-881f-447b87f04557"
Sep 17 21:58:26.909535 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:26.909028    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkk4g"
Sep 17 21:58:26.909535 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:26.909238    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bnj94" podUID="3b714c1c-9f7e-430e-bc07-94cf3a75921c"
Sep 17 21:58:26.909535 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:26.909296    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkk4g" podUID="45d5f94d-990c-4f9e-b1ab-980021c00716"
Sep 17 21:58:28.551640 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:28.551602    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45d5f94d-990c-4f9e-b1ab-980021c00716-metrics-certs\") pod \"network-metrics-daemon-xkk4g\" (UID: \"45d5f94d-990c-4f9e-b1ab-980021c00716\") " pod="openshift-multus/network-metrics-daemon-xkk4g"
Sep 17 21:58:28.552129 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:28.551651    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79a6aec6-e29a-45cf-881f-447b87f04557-original-pull-secret\") pod \"global-pull-secret-syncer-wxx9x\" (UID: \"79a6aec6-e29a-45cf-881f-447b87f04557\") " pod="kube-system/global-pull-secret-syncer-wxx9x"
Sep 17 21:58:28.552129 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:28.551758    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:28.552129 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:28.551819    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79a6aec6-e29a-45cf-881f-447b87f04557-original-pull-secret podName:79a6aec6-e29a-45cf-881f-447b87f04557 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:44.55180426 +0000 UTC m=+35.719757221 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/79a6aec6-e29a-45cf-881f-447b87f04557-original-pull-secret") pod "global-pull-secret-syncer-wxx9x" (UID: "79a6aec6-e29a-45cf-881f-447b87f04557") : object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:28.552129 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:28.551758    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:28.552129 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:28.551883    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45d5f94d-990c-4f9e-b1ab-980021c00716-metrics-certs podName:45d5f94d-990c-4f9e-b1ab-980021c00716 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:44.551870434 +0000 UTC m=+35.719823412 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/45d5f94d-990c-4f9e-b1ab-980021c00716-metrics-certs") pod "network-metrics-daemon-xkk4g" (UID: "45d5f94d-990c-4f9e-b1ab-980021c00716") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:28.652360 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:28.652321    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrkvs\" (UniqueName: \"kubernetes.io/projected/3b714c1c-9f7e-430e-bc07-94cf3a75921c-kube-api-access-hrkvs\") pod \"network-check-target-bnj94\" (UID: \"3b714c1c-9f7e-430e-bc07-94cf3a75921c\") " pod="openshift-network-diagnostics/network-check-target-bnj94"
Sep 17 21:58:28.652532 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:28.652469    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 21:58:28.652532 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:28.652491    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 21:58:28.652532 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:28.652500    2563 projected.go:196] Error preparing data for projected volume kube-api-access-hrkvs for pod openshift-network-diagnostics/network-check-target-bnj94: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:28.652665 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:28.652564    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b714c1c-9f7e-430e-bc07-94cf3a75921c-kube-api-access-hrkvs podName:3b714c1c-9f7e-430e-bc07-94cf3a75921c nodeName:}" failed. No retries permitted until 2026-09-17 21:58:44.652543007 +0000 UTC m=+35.820495969 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hrkvs" (UniqueName: "kubernetes.io/projected/3b714c1c-9f7e-430e-bc07-94cf3a75921c-kube-api-access-hrkvs") pod "network-check-target-bnj94" (UID: "3b714c1c-9f7e-430e-bc07-94cf3a75921c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:28.909441 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:28.909357    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bnj94"
Sep 17 21:58:28.909606 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:28.909358    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wxx9x"
Sep 17 21:58:28.909606 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:28.909491    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkk4g"
Sep 17 21:58:28.909606 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:28.909487    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bnj94" podUID="3b714c1c-9f7e-430e-bc07-94cf3a75921c"
Sep 17 21:58:28.909779 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:28.909603    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wxx9x" podUID="79a6aec6-e29a-45cf-881f-447b87f04557"
Sep 17 21:58:28.909779 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:28.909681    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkk4g" podUID="45d5f94d-990c-4f9e-b1ab-980021c00716"
Sep 17 21:58:30.909562 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:30.909477    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bnj94"
Sep 17 21:58:30.909562 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:30.909500    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wxx9x"
Sep 17 21:58:30.910074 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:30.909581    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkk4g"
Sep 17 21:58:30.910074 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:30.909625    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bnj94" podUID="3b714c1c-9f7e-430e-bc07-94cf3a75921c"
Sep 17 21:58:30.910074 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:30.909673    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkk4g" podUID="45d5f94d-990c-4f9e-b1ab-980021c00716"
Sep 17 21:58:30.910074 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:30.909743    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wxx9x" podUID="79a6aec6-e29a-45cf-881f-447b87f04557"
Sep 17 21:58:32.118010 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:32.117789    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-f6t75" event={"ID":"e5348473-b853-4265-8876-9e0b2c8943d8","Type":"ContainerStarted","Data":"4b24eb590e31d72cb81e019f81fd97aaff68f5b624c537772cb124481db87e49"}
Sep 17 21:58:32.119463 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:32.119440    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hzx5m" event={"ID":"c20c2a19-2b30-48fb-bb14-b5321cf2f4d0","Type":"ContainerStarted","Data":"28cb7cfcd7268f48ab8b50455809997b2a5cd7ae4c39820f825024e3f4864047"}
Sep 17 21:58:32.120918 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:32.120887    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-26gb4" event={"ID":"992a8c01-f2ce-4523-9a33-56e12b134be2","Type":"ContainerStarted","Data":"8fcc580a1f94c007e4235bf28db20480f51b42d8e87b8aacdc29eeefad1ed24e"}
Sep 17 21:58:32.122341 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:32.122317    2563 generic.go:356] "Generic (PLEG): container finished" podID="f004b93f-9438-4008-ad17-92150090d28c" containerID="6e318da9fe93a81b84e8b895850335686db44de57a9c849505642512a9e7e0cd" exitCode=0
Sep 17 21:58:32.122429 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:32.122389    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wjt9h" event={"ID":"f004b93f-9438-4008-ad17-92150090d28c","Type":"ContainerDied","Data":"6e318da9fe93a81b84e8b895850335686db44de57a9c849505642512a9e7e0cd"}
Sep 17 21:58:32.123934 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:32.123920    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nrkd5_34b8ef2b-97ba-4562-87ee-8da0cb4604c7/ovn-acl-logging/0.log"
Sep 17 21:58:32.124328 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:32.124307    2563 generic.go:356] "Generic (PLEG): container finished" podID="34b8ef2b-97ba-4562-87ee-8da0cb4604c7" containerID="71f5100ab90a76fee68ddbe374f7a28e60b1cfd51d5e9d1b78ab9b9440bb829a" exitCode=1
Sep 17 21:58:32.124387 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:32.124370    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5" event={"ID":"34b8ef2b-97ba-4562-87ee-8da0cb4604c7","Type":"ContainerDied","Data":"71f5100ab90a76fee68ddbe374f7a28e60b1cfd51d5e9d1b78ab9b9440bb829a"}
Sep 17 21:58:32.124461 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:32.124445    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5" event={"ID":"34b8ef2b-97ba-4562-87ee-8da0cb4604c7","Type":"ContainerStarted","Data":"03f6b3365ef82fdd8b3336be5e4a0eeb53458d04128fce2fc56752493b5bcf9e"}
Sep 17 21:58:32.125776 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:32.125758    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-j6ntd" event={"ID":"b93158dd-d718-446e-8e3b-38a2235ed7a7","Type":"ContainerStarted","Data":"3f62ea772322404fe8bd3c9a4e55cd0f7602d99598ce72623dd6928a6b6a4689"}
Sep 17 21:58:32.127193 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:32.127171    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flq5r" event={"ID":"96f98ff0-1bb0-4529-9b06-43019479d9e8","Type":"ContainerStarted","Data":"d117fcee0381665283c70465a7d4f8547da0584437a6bc3f337b9a8ea31f2f84"}
Sep 17 21:58:32.128476 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:32.128456    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-n2fp6" event={"ID":"9d2ac593-98b6-4b38-adef-a1bfbf1a168f","Type":"ContainerStarted","Data":"bd4c08da880ca7e694ce15ce9f2a978ac569fb32d43519709c6e6f323926071f"}
Sep 17 21:58:32.133507 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:32.133470    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-45.ec2.internal" podStartSLOduration=21.133458731 podStartE2EDuration="21.133458731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:16.099640988 +0000 UTC m=+7.267593973" watchObservedRunningTime="2026-09-17 21:58:32.133458731 +0000 UTC m=+23.301411713"
Sep 17 21:58:32.133837 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:32.133811    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-f6t75" podStartSLOduration=3.499548028 podStartE2EDuration="21.133803174s" totalImagesPullingTime="17.634255146s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:32.133082038 +0000 UTC m=+23.301035020" watchObservedRunningTime="2026-09-17 21:58:32.133803174 +0000 UTC m=+23.301756178"
Sep 17 21:58:32.149777 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:32.147907    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-hzx5m" podStartSLOduration=4.552179302 podStartE2EDuration="22.147891432s" totalImagesPullingTime="17.59571213s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:32.147579303 +0000 UTC m=+23.315532287" watchObservedRunningTime="2026-09-17 21:58:32.147891432 +0000 UTC m=+23.315844415"
Sep 17 21:58:32.178637 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:32.178545    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-j6ntd" podStartSLOduration=7.615456582 podStartE2EDuration="21.178527221s" totalImagesPullingTime="13.563070639s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:58:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:32.178527273 +0000 UTC m=+23.346480257" watchObservedRunningTime="2026-09-17 21:58:32.178527221 +0000 UTC m=+23.346480205"
Sep 17 21:58:32.178808 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:32.178717    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-26gb4" podStartSLOduration=3.5478915559999997 podStartE2EDuration="21.17870935s" totalImagesPullingTime="17.630817794s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:32.164458387 +0000 UTC m=+23.332411370" watchObservedRunningTime="2026-09-17 21:58:32.17870935 +0000 UTC m=+23.346662334"
Sep 17 21:58:32.195808 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:32.195766    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-n2fp6" podStartSLOduration=3.590825534 podStartE2EDuration="21.195750922s" totalImagesPullingTime="17.604925388s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:32.195484094 +0000 UTC m=+23.363437077" watchObservedRunningTime="2026-09-17 21:58:32.195750922 +0000 UTC m=+23.363703905"
Sep 17 21:58:32.909440 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:32.909252    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkk4g"
Sep 17 21:58:32.909673 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:32.909291    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bnj94"
Sep 17 21:58:32.909673 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:32.909548    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkk4g" podUID="45d5f94d-990c-4f9e-b1ab-980021c00716"
Sep 17 21:58:32.909673 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:32.909335    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wxx9x"
Sep 17 21:58:32.909673 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:32.909630    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bnj94" podUID="3b714c1c-9f7e-430e-bc07-94cf3a75921c"
Sep 17 21:58:32.909869 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:32.909697    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wxx9x" podUID="79a6aec6-e29a-45cf-881f-447b87f04557"
Sep 17 21:58:33.133373 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:33.133348    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nrkd5_34b8ef2b-97ba-4562-87ee-8da0cb4604c7/ovn-acl-logging/0.log"
Sep 17 21:58:33.133777 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:33.133752    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5" event={"ID":"34b8ef2b-97ba-4562-87ee-8da0cb4604c7","Type":"ContainerStarted","Data":"7dcd822785341865e6da7508b5ca4e64f7bdac6012919676848ff8c00c21b8be"}
Sep 17 21:58:33.133868 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:33.133845    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5" event={"ID":"34b8ef2b-97ba-4562-87ee-8da0cb4604c7","Type":"ContainerStarted","Data":"567cc32dba06a561c361926ff88f0ac0997800ea25d466106503303f5ba43112"}
Sep 17 21:58:33.133960 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:33.133899    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5" event={"ID":"34b8ef2b-97ba-4562-87ee-8da0cb4604c7","Type":"ContainerStarted","Data":"e69863fa6d8049a537a84f466f7289837d5c2e9805bfe0d20a572bd0ddf0a71c"}
Sep 17 21:58:33.133960 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:33.133948    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5" event={"ID":"34b8ef2b-97ba-4562-87ee-8da0cb4604c7","Type":"ContainerStarted","Data":"8574e1aba072b824df844ff1df83dd665caaaa71502df39db736f9272f60af68"}
Sep 17 21:58:33.252701 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:33.252673    2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 21:58:33.895501 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:33.895373    2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T21:58:33.252697911Z","UUID":"2a0d8b79-ca62-4cf7-ab39-c32b72d41550","Handler":null,"Name":"","Endpoint":""}
Sep 17 21:58:33.897213 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:33.897182    2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 21:58:33.897213 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:33.897219    2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 21:58:34.139956 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:34.139910    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flq5r" event={"ID":"96f98ff0-1bb0-4529-9b06-43019479d9e8","Type":"ContainerStarted","Data":"8b69c3479f5a1b3077ce99e55130372741aae0855673e5f4854ec2436ac2dc73"}
Sep 17 21:58:34.141846 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:34.141807    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tkphq" event={"ID":"d6f6a71b-842d-48f0-93fc-28006b5c122f","Type":"ContainerStarted","Data":"b6c28484af491f0e5d86bd39505fad1ae8aec78b1a00da1e16b5831e814c0825"}
Sep 17 21:58:34.156427 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:34.156336    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-tkphq" podStartSLOduration=5.530429029 podStartE2EDuration="23.156321516s" totalImagesPullingTime="17.625892487s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:34.155998212 +0000 UTC m=+25.323951189" watchObservedRunningTime="2026-09-17 21:58:34.156321516 +0000 UTC m=+25.324274579"
Sep 17 21:58:34.909003 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:34.908913    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bnj94"
Sep 17 21:58:34.909176 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:34.908913    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wxx9x"
Sep 17 21:58:34.909176 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:34.909043    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bnj94" podUID="3b714c1c-9f7e-430e-bc07-94cf3a75921c"
Sep 17 21:58:34.909176 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:34.909116    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wxx9x" podUID="79a6aec6-e29a-45cf-881f-447b87f04557"
Sep 17 21:58:34.909176 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:34.908921    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkk4g"
Sep 17 21:58:34.909360 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:34.909251    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkk4g" podUID="45d5f94d-990c-4f9e-b1ab-980021c00716"
Sep 17 21:58:35.147313 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:35.147282    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nrkd5_34b8ef2b-97ba-4562-87ee-8da0cb4604c7/ovn-acl-logging/0.log"
Sep 17 21:58:35.147815 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:35.147695    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5" event={"ID":"34b8ef2b-97ba-4562-87ee-8da0cb4604c7","Type":"ContainerStarted","Data":"96ee564f86df921fb04672f2025922d6586cd0fd3006c71180cf584223d39948"}
Sep 17 21:58:35.150035 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:35.150007    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flq5r" event={"ID":"96f98ff0-1bb0-4529-9b06-43019479d9e8","Type":"ContainerStarted","Data":"133b5d7a3c9b22d5a1cf5c32be3968be119fa1a7a8daaa04cf4eb0b7f22abed7"}
Sep 17 21:58:35.168559 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:35.168462    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flq5r" podStartSLOduration=5.110732367 podStartE2EDuration="25.168439867s" totalImagesPullingTime="20.0577075s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:35.168199416 +0000 UTC m=+26.336152399" watchObservedRunningTime="2026-09-17 21:58:35.168439867 +0000 UTC m=+26.336392854"
Sep 17 21:58:36.705162 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:36.705031    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-j6ntd"
Sep 17 21:58:36.705773 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:36.705722    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-j6ntd"
Sep 17 21:58:36.908817 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:36.908649    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bnj94"
Sep 17 21:58:36.908951 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:36.908648    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wxx9x"
Sep 17 21:58:36.908951 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:36.908908    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bnj94" podUID="3b714c1c-9f7e-430e-bc07-94cf3a75921c"
Sep 17 21:58:36.909134 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:36.908695    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkk4g"
Sep 17 21:58:36.909134 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:36.908996    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wxx9x" podUID="79a6aec6-e29a-45cf-881f-447b87f04557"
Sep 17 21:58:36.909134 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:36.909075    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkk4g" podUID="45d5f94d-990c-4f9e-b1ab-980021c00716"
Sep 17 21:58:37.159025 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:37.158945    2563 generic.go:356] "Generic (PLEG): container finished" podID="f004b93f-9438-4008-ad17-92150090d28c" containerID="68ef9b6ae2a3da4b387c768486954a6b02776bb3fc603e864bf918fed4ab02fd" exitCode=0
Sep 17 21:58:37.159192 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:37.159022    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wjt9h" event={"ID":"f004b93f-9438-4008-ad17-92150090d28c","Type":"ContainerDied","Data":"68ef9b6ae2a3da4b387c768486954a6b02776bb3fc603e864bf918fed4ab02fd"}
Sep 17 21:58:37.162624 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:37.162608    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nrkd5_34b8ef2b-97ba-4562-87ee-8da0cb4604c7/ovn-acl-logging/0.log"
Sep 17 21:58:37.163043 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:37.163020    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5" event={"ID":"34b8ef2b-97ba-4562-87ee-8da0cb4604c7","Type":"ContainerStarted","Data":"ccc764e776efbc3877d53767743789d01f494bfd699399359964aa42c63509f3"}
Sep 17 21:58:37.163151 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:37.163101    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-j6ntd"
Sep 17 21:58:37.163435 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:37.163418    2563 scope.go:117] "RemoveContainer" containerID="71f5100ab90a76fee68ddbe374f7a28e60b1cfd51d5e9d1b78ab9b9440bb829a"
Sep 17 21:58:37.163626 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:37.163609    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-j6ntd"
Sep 17 21:58:38.168676 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:38.168648    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nrkd5_34b8ef2b-97ba-4562-87ee-8da0cb4604c7/ovn-acl-logging/0.log"
Sep 17 21:58:38.169558 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:38.169041    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5" event={"ID":"34b8ef2b-97ba-4562-87ee-8da0cb4604c7","Type":"ContainerStarted","Data":"7c772858da11e4c76b9886b6a80df7b5a782cdf57ac49df74b8a8d791b92680b"}
Sep 17 21:58:38.169558 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:38.169317    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5"
Sep 17 21:58:38.169558 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:38.169415    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5"
Sep 17 21:58:38.169558 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:38.169455    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5"
Sep 17 21:58:38.184338 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:38.184314    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5"
Sep 17 21:58:38.184449 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:38.184410    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5"
Sep 17 21:58:38.197538 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:38.197499    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5" podStartSLOduration=9.523202258 podStartE2EDuration="27.197487157s" totalImagesPullingTime="17.674284899s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:38.196992397 +0000 UTC m=+29.364945385" watchObservedRunningTime="2026-09-17 21:58:38.197487157 +0000 UTC m=+29.365440143"
Sep 17 21:58:38.751049 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:38.750735    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wxx9x"]
Sep 17 21:58:38.751237 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:38.751080    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wxx9x"
Sep 17 21:58:38.751237 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:38.751194    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wxx9x" podUID="79a6aec6-e29a-45cf-881f-447b87f04557"
Sep 17 21:58:38.753866 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:38.753840    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xkk4g"]
Sep 17 21:58:38.753997 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:38.753956    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkk4g"
Sep 17 21:58:38.754100 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:38.754068    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkk4g" podUID="45d5f94d-990c-4f9e-b1ab-980021c00716"
Sep 17 21:58:38.754658 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:38.754638    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bnj94"]
Sep 17 21:58:38.754739 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:38.754724    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bnj94"
Sep 17 21:58:38.754842 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:38.754824    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bnj94" podUID="3b714c1c-9f7e-430e-bc07-94cf3a75921c"
Sep 17 21:58:39.173523 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:39.173491    2563 generic.go:356] "Generic (PLEG): container finished" podID="f004b93f-9438-4008-ad17-92150090d28c" containerID="a4da0daab1b6eff4938672825ae0d094b1ebe9cf03d9f5dd07e962ce655f04ef" exitCode=0
Sep 17 21:58:39.173927 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:39.173567    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wjt9h" event={"ID":"f004b93f-9438-4008-ad17-92150090d28c","Type":"ContainerDied","Data":"a4da0daab1b6eff4938672825ae0d094b1ebe9cf03d9f5dd07e962ce655f04ef"}
Sep 17 21:58:40.909825 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:40.909798    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bnj94"
Sep 17 21:58:40.910261 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:40.909902    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bnj94" podUID="3b714c1c-9f7e-430e-bc07-94cf3a75921c"
Sep 17 21:58:40.910261 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:40.909931    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wxx9x"
Sep 17 21:58:40.910261 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:40.910003    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wxx9x" podUID="79a6aec6-e29a-45cf-881f-447b87f04557"
Sep 17 21:58:40.910261 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:40.910059    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkk4g"
Sep 17 21:58:40.910261 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:40.910185    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkk4g" podUID="45d5f94d-990c-4f9e-b1ab-980021c00716"
Sep 17 21:58:41.180731 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:41.180695    2563 generic.go:356] "Generic (PLEG): container finished" podID="f004b93f-9438-4008-ad17-92150090d28c" containerID="2a813e1b86eed7bcb487511c7e27888ffa3081de42c6f88e7e8c517d6ff7dad1" exitCode=0
Sep 17 21:58:41.180889 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:41.180739    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wjt9h" event={"ID":"f004b93f-9438-4008-ad17-92150090d28c","Type":"ContainerDied","Data":"2a813e1b86eed7bcb487511c7e27888ffa3081de42c6f88e7e8c517d6ff7dad1"}
Sep 17 21:58:42.908874 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:42.908840    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bnj94"
Sep 17 21:58:42.909351 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:42.908953    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bnj94" podUID="3b714c1c-9f7e-430e-bc07-94cf3a75921c"
Sep 17 21:58:42.909351 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:42.908840    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wxx9x"
Sep 17 21:58:42.909351 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:42.908840    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkk4g"
Sep 17 21:58:42.909351 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:42.909054    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wxx9x" podUID="79a6aec6-e29a-45cf-881f-447b87f04557"
Sep 17 21:58:42.909351 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:42.909121    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkk4g" podUID="45d5f94d-990c-4f9e-b1ab-980021c00716"
Sep 17 21:58:44.564919 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:44.564837    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79a6aec6-e29a-45cf-881f-447b87f04557-original-pull-secret\") pod \"global-pull-secret-syncer-wxx9x\" (UID: \"79a6aec6-e29a-45cf-881f-447b87f04557\") " pod="kube-system/global-pull-secret-syncer-wxx9x"
Sep 17 21:58:44.564919 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:44.564918    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45d5f94d-990c-4f9e-b1ab-980021c00716-metrics-certs\") pod \"network-metrics-daemon-xkk4g\" (UID: \"45d5f94d-990c-4f9e-b1ab-980021c00716\") " pod="openshift-multus/network-metrics-daemon-xkk4g"
Sep 17 21:58:44.565399 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:44.565043    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:44.565399 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:44.565049    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:44.565399 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:44.565121    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45d5f94d-990c-4f9e-b1ab-980021c00716-metrics-certs podName:45d5f94d-990c-4f9e-b1ab-980021c00716 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:16.565098819 +0000 UTC m=+67.733051806 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/45d5f94d-990c-4f9e-b1ab-980021c00716-metrics-certs") pod "network-metrics-daemon-xkk4g" (UID: "45d5f94d-990c-4f9e-b1ab-980021c00716") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:44.565399 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:44.565151    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79a6aec6-e29a-45cf-881f-447b87f04557-original-pull-secret podName:79a6aec6-e29a-45cf-881f-447b87f04557 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:16.565131197 +0000 UTC m=+67.733084159 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/79a6aec6-e29a-45cf-881f-447b87f04557-original-pull-secret") pod "global-pull-secret-syncer-wxx9x" (UID: "79a6aec6-e29a-45cf-881f-447b87f04557") : object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:44.665579 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:44.665540    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrkvs\" (UniqueName: \"kubernetes.io/projected/3b714c1c-9f7e-430e-bc07-94cf3a75921c-kube-api-access-hrkvs\") pod \"network-check-target-bnj94\" (UID: \"3b714c1c-9f7e-430e-bc07-94cf3a75921c\") " pod="openshift-network-diagnostics/network-check-target-bnj94"
Sep 17 21:58:44.665731 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:44.665710    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 21:58:44.665785 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:44.665734    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 21:58:44.665785 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:44.665745    2563 projected.go:196] Error preparing data for projected volume kube-api-access-hrkvs for pod openshift-network-diagnostics/network-check-target-bnj94: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:44.665862 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:44.665812    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b714c1c-9f7e-430e-bc07-94cf3a75921c-kube-api-access-hrkvs podName:3b714c1c-9f7e-430e-bc07-94cf3a75921c nodeName:}" failed. No retries permitted until 2026-09-17 21:59:16.665795885 +0000 UTC m=+67.833748859 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-hrkvs" (UniqueName: "kubernetes.io/projected/3b714c1c-9f7e-430e-bc07-94cf3a75921c-kube-api-access-hrkvs") pod "network-check-target-bnj94" (UID: "3b714c1c-9f7e-430e-bc07-94cf3a75921c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:44.773490 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:44.773459    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-45.ec2.internal" event="NodeReady"
Sep 17 21:58:44.773674 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:44.773600    2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 21:58:44.848644 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:44.848560    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-kcnsx"]
Sep 17 21:58:44.875858 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:44.875820    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-p6kpt"]
Sep 17 21:58:44.876026 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:44.875970    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kcnsx"
Sep 17 21:58:44.880768 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:44.879688    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-kb28p\""
Sep 17 21:58:44.887055 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:44.885071    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 21:58:44.887055 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:44.885874    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 21:58:44.890985 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:44.890960    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5d5df9c674-7d924"]
Sep 17 21:58:44.891495 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:44.891479    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-p6kpt"
Sep 17 21:58:44.894045 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:44.894027    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 21:58:44.894134 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:44.894029    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-5ht6s\""
Sep 17 21:58:44.894134 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:44.894095    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 21:58:44.894324 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:44.894297    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 21:58:44.911258 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:44.911231    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d5df9c674-7d924"
Sep 17 21:58:44.914303 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:44.914285    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 21:58:44.914411 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:44.914310    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 21:58:44.914569 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:44.914552    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 21:58:44.914743 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:44.914719    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vcbvz\""
Sep 17 21:58:44.914982 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:44.914965    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kcnsx"]
Sep 17 21:58:44.915050 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:44.915008    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-p6kpt"]
Sep 17 21:58:44.915050 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:44.915025    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d5df9c674-7d924"]
Sep 17 21:58:44.915170 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:44.915123    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkk4g"
Sep 17 21:58:44.915170 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:44.915131    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wxx9x"
Sep 17 21:58:44.915418 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:44.915398    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bnj94"
Sep 17 21:58:44.919202 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:44.918893    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cxqmr\""
Sep 17 21:58:44.919202 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:44.918931    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 21:58:44.919506 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:44.919486    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 21:58:44.919797 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:44.919772    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 21:58:44.919891 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:44.919772    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 21:58:44.920107 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:44.920081    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4zg8g\""
Sep 17 21:58:44.921902 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:44.921880    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 21:58:44.967986 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:44.967953    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0e64d86f-9913-4a28-a1dc-1810710456bc-config-volume\") pod \"dns-default-kcnsx\" (UID: \"0e64d86f-9913-4a28-a1dc-1810710456bc\") " pod="openshift-dns/dns-default-kcnsx"
Sep 17 21:58:44.968123 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:44.967997    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/22e6cd51-4d91-400b-b942-32e59b89a0be-registry-certificates\") pod \"image-registry-5d5df9c674-7d924\" (UID: \"22e6cd51-4d91-400b-b942-32e59b89a0be\") " pod="openshift-image-registry/image-registry-5d5df9c674-7d924"
Sep 17 21:58:44.968123 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:44.968029    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/22e6cd51-4d91-400b-b942-32e59b89a0be-installation-pull-secrets\") pod \"image-registry-5d5df9c674-7d924\" (UID: \"22e6cd51-4d91-400b-b942-32e59b89a0be\") " pod="openshift-image-registry/image-registry-5d5df9c674-7d924"
Sep 17 21:58:44.968123 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:44.968088    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/01db16ed-7748-4c99-a5ce-7f98df2f5e7a-cert\") pod \"ingress-canary-p6kpt\" (UID: \"01db16ed-7748-4c99-a5ce-7f98df2f5e7a\") " pod="openshift-ingress-canary/ingress-canary-p6kpt"
Sep 17 21:58:44.968123 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:44.968116    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/22e6cd51-4d91-400b-b942-32e59b89a0be-trusted-ca\") pod \"image-registry-5d5df9c674-7d924\" (UID: \"22e6cd51-4d91-400b-b942-32e59b89a0be\") " pod="openshift-image-registry/image-registry-5d5df9c674-7d924"
Sep 17 21:58:44.968379 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:44.968160    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e64d86f-9913-4a28-a1dc-1810710456bc-tmp-dir\") pod \"dns-default-kcnsx\" (UID: \"0e64d86f-9913-4a28-a1dc-1810710456bc\") " pod="openshift-dns/dns-default-kcnsx"
Sep 17 21:58:44.968379 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:44.968186    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/22e6cd51-4d91-400b-b942-32e59b89a0be-image-registry-private-configuration\") pod \"image-registry-5d5df9c674-7d924\" (UID: \"22e6cd51-4d91-400b-b942-32e59b89a0be\") " pod="openshift-image-registry/image-registry-5d5df9c674-7d924"
Sep 17 21:58:44.968379 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:44.968207    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k654d\" (UniqueName: \"kubernetes.io/projected/22e6cd51-4d91-400b-b942-32e59b89a0be-kube-api-access-k654d\") pod \"image-registry-5d5df9c674-7d924\" (UID: \"22e6cd51-4d91-400b-b942-32e59b89a0be\") " pod="openshift-image-registry/image-registry-5d5df9c674-7d924"
Sep 17 21:58:44.968379 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:44.968235    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/22e6cd51-4d91-400b-b942-32e59b89a0be-registry-tls\") pod \"image-registry-5d5df9c674-7d924\" (UID: \"22e6cd51-4d91-400b-b942-32e59b89a0be\") " pod="openshift-image-registry/image-registry-5d5df9c674-7d924"
Sep 17 21:58:44.968379 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:44.968258    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cc22\" (UniqueName: \"kubernetes.io/projected/01db16ed-7748-4c99-a5ce-7f98df2f5e7a-kube-api-access-8cc22\") pod \"ingress-canary-p6kpt\" (UID: \"01db16ed-7748-4c99-a5ce-7f98df2f5e7a\") " pod="openshift-ingress-canary/ingress-canary-p6kpt"
Sep 17 21:58:44.968379 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:44.968284    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/22e6cd51-4d91-400b-b942-32e59b89a0be-ca-trust-extracted\") pod \"image-registry-5d5df9c674-7d924\" (UID: \"22e6cd51-4d91-400b-b942-32e59b89a0be\") " pod="openshift-image-registry/image-registry-5d5df9c674-7d924"
Sep 17 21:58:44.968379 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:44.968323    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0e64d86f-9913-4a28-a1dc-1810710456bc-metrics-tls\") pod \"dns-default-kcnsx\" (UID: \"0e64d86f-9913-4a28-a1dc-1810710456bc\") " pod="openshift-dns/dns-default-kcnsx"
Sep 17 21:58:44.968379 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:44.968371    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knf7q\" (UniqueName: \"kubernetes.io/projected/0e64d86f-9913-4a28-a1dc-1810710456bc-kube-api-access-knf7q\") pod \"dns-default-kcnsx\" (UID: \"0e64d86f-9913-4a28-a1dc-1810710456bc\") " pod="openshift-dns/dns-default-kcnsx"
Sep 17 21:58:44.968723 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:44.968400    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/22e6cd51-4d91-400b-b942-32e59b89a0be-bound-sa-token\") pod \"image-registry-5d5df9c674-7d924\" (UID: \"22e6cd51-4d91-400b-b942-32e59b89a0be\") " pod="openshift-image-registry/image-registry-5d5df9c674-7d924"
Sep 17 21:58:45.069594 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:45.069564    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0e64d86f-9913-4a28-a1dc-1810710456bc-config-volume\") pod \"dns-default-kcnsx\" (UID: \"0e64d86f-9913-4a28-a1dc-1810710456bc\") " pod="openshift-dns/dns-default-kcnsx"
Sep 17 21:58:45.069594 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:45.069602    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/22e6cd51-4d91-400b-b942-32e59b89a0be-registry-certificates\") pod \"image-registry-5d5df9c674-7d924\" (UID: \"22e6cd51-4d91-400b-b942-32e59b89a0be\") " pod="openshift-image-registry/image-registry-5d5df9c674-7d924"
Sep 17 21:58:45.069824 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:45.069621    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/22e6cd51-4d91-400b-b942-32e59b89a0be-installation-pull-secrets\") pod \"image-registry-5d5df9c674-7d924\" (UID: \"22e6cd51-4d91-400b-b942-32e59b89a0be\") " pod="openshift-image-registry/image-registry-5d5df9c674-7d924"
Sep 17 21:58:45.069824 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:45.069655    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/01db16ed-7748-4c99-a5ce-7f98df2f5e7a-cert\") pod \"ingress-canary-p6kpt\" (UID: \"01db16ed-7748-4c99-a5ce-7f98df2f5e7a\") " pod="openshift-ingress-canary/ingress-canary-p6kpt"
Sep 17 21:58:45.069824 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:45.069679    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/22e6cd51-4d91-400b-b942-32e59b89a0be-trusted-ca\") pod \"image-registry-5d5df9c674-7d924\" (UID: \"22e6cd51-4d91-400b-b942-32e59b89a0be\") " pod="openshift-image-registry/image-registry-5d5df9c674-7d924"
Sep 17 21:58:45.069824 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:45.069707    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e64d86f-9913-4a28-a1dc-1810710456bc-tmp-dir\") pod \"dns-default-kcnsx\" (UID: \"0e64d86f-9913-4a28-a1dc-1810710456bc\") " pod="openshift-dns/dns-default-kcnsx"
Sep 17 21:58:45.069824 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:45.069734    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/22e6cd51-4d91-400b-b942-32e59b89a0be-image-registry-private-configuration\") pod \"image-registry-5d5df9c674-7d924\" (UID: \"22e6cd51-4d91-400b-b942-32e59b89a0be\") " pod="openshift-image-registry/image-registry-5d5df9c674-7d924"
Sep 17 21:58:45.069824 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:45.069758    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k654d\" (UniqueName: \"kubernetes.io/projected/22e6cd51-4d91-400b-b942-32e59b89a0be-kube-api-access-k654d\") pod \"image-registry-5d5df9c674-7d924\" (UID: \"22e6cd51-4d91-400b-b942-32e59b89a0be\") " pod="openshift-image-registry/image-registry-5d5df9c674-7d924"
Sep 17 21:58:45.069824 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:45.069786    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/22e6cd51-4d91-400b-b942-32e59b89a0be-registry-tls\") pod \"image-registry-5d5df9c674-7d924\" (UID: \"22e6cd51-4d91-400b-b942-32e59b89a0be\") " pod="openshift-image-registry/image-registry-5d5df9c674-7d924"
Sep 17 21:58:45.069824 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:45.069810    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cc22\" (UniqueName: \"kubernetes.io/projected/01db16ed-7748-4c99-a5ce-7f98df2f5e7a-kube-api-access-8cc22\") pod \"ingress-canary-p6kpt\" (UID: \"01db16ed-7748-4c99-a5ce-7f98df2f5e7a\") " pod="openshift-ingress-canary/ingress-canary-p6kpt"
Sep 17 21:58:45.070247 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:45.069836    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/22e6cd51-4d91-400b-b942-32e59b89a0be-ca-trust-extracted\") pod \"image-registry-5d5df9c674-7d924\" (UID: \"22e6cd51-4d91-400b-b942-32e59b89a0be\") " pod="openshift-image-registry/image-registry-5d5df9c674-7d924"
Sep 17 21:58:45.070247 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:45.069875    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0e64d86f-9913-4a28-a1dc-1810710456bc-metrics-tls\") pod \"dns-default-kcnsx\" (UID: \"0e64d86f-9913-4a28-a1dc-1810710456bc\") " pod="openshift-dns/dns-default-kcnsx"
Sep 17 21:58:45.070247 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:45.069931    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knf7q\" (UniqueName: \"kubernetes.io/projected/0e64d86f-9913-4a28-a1dc-1810710456bc-kube-api-access-knf7q\") pod \"dns-default-kcnsx\" (UID: \"0e64d86f-9913-4a28-a1dc-1810710456bc\") " pod="openshift-dns/dns-default-kcnsx"
Sep 17 21:58:45.070247 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:45.069950    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/22e6cd51-4d91-400b-b942-32e59b89a0be-bound-sa-token\") pod \"image-registry-5d5df9c674-7d924\" (UID: \"22e6cd51-4d91-400b-b942-32e59b89a0be\") " pod="openshift-image-registry/image-registry-5d5df9c674-7d924"
Sep 17 21:58:45.070444 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:45.070426    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0e64d86f-9913-4a28-a1dc-1810710456bc-config-volume\") pod \"dns-default-kcnsx\" (UID: \"0e64d86f-9913-4a28-a1dc-1810710456bc\") " pod="openshift-dns/dns-default-kcnsx"
Sep 17 21:58:45.071319 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:45.070496    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/22e6cd51-4d91-400b-b942-32e59b89a0be-registry-certificates\") pod \"image-registry-5d5df9c674-7d924\" (UID: \"22e6cd51-4d91-400b-b942-32e59b89a0be\") " pod="openshift-image-registry/image-registry-5d5df9c674-7d924"
Sep 17 21:58:45.071319 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:45.070514    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 21:58:45.071319 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:45.070528    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d5df9c674-7d924: secret "image-registry-tls" not found
Sep 17 21:58:45.071319 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:45.070578    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/22e6cd51-4d91-400b-b942-32e59b89a0be-registry-tls podName:22e6cd51-4d91-400b-b942-32e59b89a0be nodeName:}" failed. No retries permitted until 2026-09-17 21:58:45.570560574 +0000 UTC m=+36.738513535 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/22e6cd51-4d91-400b-b942-32e59b89a0be-registry-tls") pod "image-registry-5d5df9c674-7d924" (UID: "22e6cd51-4d91-400b-b942-32e59b89a0be") : secret "image-registry-tls" not found
Sep 17 21:58:45.071319 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:45.070641    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 21:58:45.071319 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:45.070708    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e64d86f-9913-4a28-a1dc-1810710456bc-metrics-tls podName:0e64d86f-9913-4a28-a1dc-1810710456bc nodeName:}" failed. No retries permitted until 2026-09-17 21:58:45.570689269 +0000 UTC m=+36.738642235 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0e64d86f-9913-4a28-a1dc-1810710456bc-metrics-tls") pod "dns-default-kcnsx" (UID: "0e64d86f-9913-4a28-a1dc-1810710456bc") : secret "dns-default-metrics-tls" not found
Sep 17 21:58:45.071319 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:45.070867    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/22e6cd51-4d91-400b-b942-32e59b89a0be-ca-trust-extracted\") pod \"image-registry-5d5df9c674-7d924\" (UID: \"22e6cd51-4d91-400b-b942-32e59b89a0be\") " pod="openshift-image-registry/image-registry-5d5df9c674-7d924"
Sep 17 21:58:45.071319 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:45.071044    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e64d86f-9913-4a28-a1dc-1810710456bc-tmp-dir\") pod \"dns-default-kcnsx\" (UID: \"0e64d86f-9913-4a28-a1dc-1810710456bc\") " pod="openshift-dns/dns-default-kcnsx"
Sep 17 21:58:45.071831 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:45.071685    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/22e6cd51-4d91-400b-b942-32e59b89a0be-trusted-ca\") pod \"image-registry-5d5df9c674-7d924\" (UID: \"22e6cd51-4d91-400b-b942-32e59b89a0be\") " pod="openshift-image-registry/image-registry-5d5df9c674-7d924"
Sep 17 21:58:45.074561 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:45.074539    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/22e6cd51-4d91-400b-b942-32e59b89a0be-installation-pull-secrets\") pod \"image-registry-5d5df9c674-7d924\" (UID: \"22e6cd51-4d91-400b-b942-32e59b89a0be\") " pod="openshift-image-registry/image-registry-5d5df9c674-7d924"
Sep 17 21:58:45.074665 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:45.074599    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/22e6cd51-4d91-400b-b942-32e59b89a0be-image-registry-private-configuration\") pod \"image-registry-5d5df9c674-7d924\" (UID: \"22e6cd51-4d91-400b-b942-32e59b89a0be\") " pod="openshift-image-registry/image-registry-5d5df9c674-7d924"
Sep 17 21:58:45.074723 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:45.074682    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/01db16ed-7748-4c99-a5ce-7f98df2f5e7a-cert\") pod \"ingress-canary-p6kpt\" (UID: \"01db16ed-7748-4c99-a5ce-7f98df2f5e7a\") " pod="openshift-ingress-canary/ingress-canary-p6kpt"
Sep 17 21:58:45.080882 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:45.080851    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cc22\" (UniqueName: \"kubernetes.io/projected/01db16ed-7748-4c99-a5ce-7f98df2f5e7a-kube-api-access-8cc22\") pod \"ingress-canary-p6kpt\" (UID: \"01db16ed-7748-4c99-a5ce-7f98df2f5e7a\") " pod="openshift-ingress-canary/ingress-canary-p6kpt"
Sep 17 21:58:45.080972 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:45.080853    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knf7q\" (UniqueName: \"kubernetes.io/projected/0e64d86f-9913-4a28-a1dc-1810710456bc-kube-api-access-knf7q\") pod \"dns-default-kcnsx\" (UID: \"0e64d86f-9913-4a28-a1dc-1810710456bc\") " pod="openshift-dns/dns-default-kcnsx"
Sep 17 21:58:45.081449 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:45.081401    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/22e6cd51-4d91-400b-b942-32e59b89a0be-bound-sa-token\") pod \"image-registry-5d5df9c674-7d924\" (UID: \"22e6cd51-4d91-400b-b942-32e59b89a0be\") " pod="openshift-image-registry/image-registry-5d5df9c674-7d924"
Sep 17 21:58:45.081560 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:45.081539    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k654d\" (UniqueName: \"kubernetes.io/projected/22e6cd51-4d91-400b-b942-32e59b89a0be-kube-api-access-k654d\") pod \"image-registry-5d5df9c674-7d924\" (UID: \"22e6cd51-4d91-400b-b942-32e59b89a0be\") " pod="openshift-image-registry/image-registry-5d5df9c674-7d924"
Sep 17 21:58:45.200803 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:45.200771    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-p6kpt"
Sep 17 21:58:45.573549 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:45.573476    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0e64d86f-9913-4a28-a1dc-1810710456bc-metrics-tls\") pod \"dns-default-kcnsx\" (UID: \"0e64d86f-9913-4a28-a1dc-1810710456bc\") " pod="openshift-dns/dns-default-kcnsx"
Sep 17 21:58:45.574241 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:45.573626    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/22e6cd51-4d91-400b-b942-32e59b89a0be-registry-tls\") pod \"image-registry-5d5df9c674-7d924\" (UID: \"22e6cd51-4d91-400b-b942-32e59b89a0be\") " pod="openshift-image-registry/image-registry-5d5df9c674-7d924"
Sep 17 21:58:45.574241 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:45.573647    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 21:58:45.574241 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:45.573706    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 21:58:45.574241 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:45.573716    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d5df9c674-7d924: secret "image-registry-tls" not found
Sep 17 21:58:45.574241 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:45.573724    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e64d86f-9913-4a28-a1dc-1810710456bc-metrics-tls podName:0e64d86f-9913-4a28-a1dc-1810710456bc nodeName:}" failed. No retries permitted until 2026-09-17 21:58:46.573703676 +0000 UTC m=+37.741656637 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0e64d86f-9913-4a28-a1dc-1810710456bc-metrics-tls") pod "dns-default-kcnsx" (UID: "0e64d86f-9913-4a28-a1dc-1810710456bc") : secret "dns-default-metrics-tls" not found
Sep 17 21:58:45.574241 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:45.573755    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/22e6cd51-4d91-400b-b942-32e59b89a0be-registry-tls podName:22e6cd51-4d91-400b-b942-32e59b89a0be nodeName:}" failed. No retries permitted until 2026-09-17 21:58:46.57374269 +0000 UTC m=+37.741695651 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/22e6cd51-4d91-400b-b942-32e59b89a0be-registry-tls") pod "image-registry-5d5df9c674-7d924" (UID: "22e6cd51-4d91-400b-b942-32e59b89a0be") : secret "image-registry-tls" not found
Sep 17 21:58:45.654825 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:45.653283    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-p6kpt"]
Sep 17 21:58:45.657175 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:45.657127    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01db16ed_7748_4c99_a5ce_7f98df2f5e7a.slice/crio-5a5869db0e0a1a4f0ae2f9b7c8b17c49764a30f47265c7e984de4dd07742d9b1 WatchSource:0}: Error finding container 5a5869db0e0a1a4f0ae2f9b7c8b17c49764a30f47265c7e984de4dd07742d9b1: Status 404 returned error can't find the container with id 5a5869db0e0a1a4f0ae2f9b7c8b17c49764a30f47265c7e984de4dd07742d9b1
Sep 17 21:58:46.194286 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:46.194256    2563 generic.go:356] "Generic (PLEG): container finished" podID="f004b93f-9438-4008-ad17-92150090d28c" containerID="45aba00f7a841022c9303d49c1927968aaffb66517bda5a4bfaf9ba7375dcd7c" exitCode=0
Sep 17 21:58:46.194476 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:46.194335    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wjt9h" event={"ID":"f004b93f-9438-4008-ad17-92150090d28c","Type":"ContainerDied","Data":"45aba00f7a841022c9303d49c1927968aaffb66517bda5a4bfaf9ba7375dcd7c"}
Sep 17 21:58:46.195429 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:46.195410    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-p6kpt" event={"ID":"01db16ed-7748-4c99-a5ce-7f98df2f5e7a","Type":"ContainerStarted","Data":"5a5869db0e0a1a4f0ae2f9b7c8b17c49764a30f47265c7e984de4dd07742d9b1"}
Sep 17 21:58:46.582385 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:46.582308    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/22e6cd51-4d91-400b-b942-32e59b89a0be-registry-tls\") pod \"image-registry-5d5df9c674-7d924\" (UID: \"22e6cd51-4d91-400b-b942-32e59b89a0be\") " pod="openshift-image-registry/image-registry-5d5df9c674-7d924"
Sep 17 21:58:46.582385 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:46.582356    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0e64d86f-9913-4a28-a1dc-1810710456bc-metrics-tls\") pod \"dns-default-kcnsx\" (UID: \"0e64d86f-9913-4a28-a1dc-1810710456bc\") " pod="openshift-dns/dns-default-kcnsx"
Sep 17 21:58:46.583250 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:46.582486    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 21:58:46.583250 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:46.582495    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 21:58:46.583250 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:46.582517    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d5df9c674-7d924: secret "image-registry-tls" not found
Sep 17 21:58:46.583250 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:46.582543    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e64d86f-9913-4a28-a1dc-1810710456bc-metrics-tls podName:0e64d86f-9913-4a28-a1dc-1810710456bc nodeName:}" failed. No retries permitted until 2026-09-17 21:58:48.582527104 +0000 UTC m=+39.750480086 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0e64d86f-9913-4a28-a1dc-1810710456bc-metrics-tls") pod "dns-default-kcnsx" (UID: "0e64d86f-9913-4a28-a1dc-1810710456bc") : secret "dns-default-metrics-tls" not found
Sep 17 21:58:46.583250 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:46.582577    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/22e6cd51-4d91-400b-b942-32e59b89a0be-registry-tls podName:22e6cd51-4d91-400b-b942-32e59b89a0be nodeName:}" failed. No retries permitted until 2026-09-17 21:58:48.582558619 +0000 UTC m=+39.750511589 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/22e6cd51-4d91-400b-b942-32e59b89a0be-registry-tls") pod "image-registry-5d5df9c674-7d924" (UID: "22e6cd51-4d91-400b-b942-32e59b89a0be") : secret "image-registry-tls" not found
Sep 17 21:58:47.201177 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:47.201150    2563 generic.go:356] "Generic (PLEG): container finished" podID="f004b93f-9438-4008-ad17-92150090d28c" containerID="4d3a27ffa39e96c9665fcb9b11861dedc93208314088317d997c27c8a64b1b78" exitCode=0
Sep 17 21:58:47.201409 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:47.201184    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wjt9h" event={"ID":"f004b93f-9438-4008-ad17-92150090d28c","Type":"ContainerDied","Data":"4d3a27ffa39e96c9665fcb9b11861dedc93208314088317d997c27c8a64b1b78"}
Sep 17 21:58:48.207076 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:48.207047    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wjt9h" event={"ID":"f004b93f-9438-4008-ad17-92150090d28c","Type":"ContainerStarted","Data":"415573d5b9783607ecc6cdb90783c63324043e3ca91171bf1815c946ad803ad5"}
Sep 17 21:58:48.208597 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:48.208567    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-p6kpt" event={"ID":"01db16ed-7748-4c99-a5ce-7f98df2f5e7a","Type":"ContainerStarted","Data":"295d1dfb599c431aacd81cdc9c8f24919687020d645079dfc9f666dc86e0394d"}
Sep 17 21:58:48.231237 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:48.229587    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-wjt9h" podStartSLOduration=8.254616837 podStartE2EDuration="37.229570407s" totalImagesPullingTime="28.97495357s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:11 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:48.227544614 +0000 UTC m=+39.395497598" watchObservedRunningTime="2026-09-17 21:58:48.229570407 +0000 UTC m=+39.397523403"
Sep 17 21:58:48.244439 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:48.244389    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-p6kpt" podStartSLOduration=1.960849988 podStartE2EDuration="4.244378925s" totalImagesPullingTime="2.283528937s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:58:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:48.243306775 +0000 UTC m=+39.411259755" watchObservedRunningTime="2026-09-17 21:58:48.244378925 +0000 UTC m=+39.412331905"
Sep 17 21:58:48.599900 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:48.599816    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/22e6cd51-4d91-400b-b942-32e59b89a0be-registry-tls\") pod \"image-registry-5d5df9c674-7d924\" (UID: \"22e6cd51-4d91-400b-b942-32e59b89a0be\") " pod="openshift-image-registry/image-registry-5d5df9c674-7d924"
Sep 17 21:58:48.599900 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:48.599868    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0e64d86f-9913-4a28-a1dc-1810710456bc-metrics-tls\") pod \"dns-default-kcnsx\" (UID: \"0e64d86f-9913-4a28-a1dc-1810710456bc\") " pod="openshift-dns/dns-default-kcnsx"
Sep 17 21:58:48.600097 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:48.599964    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 21:58:48.600097 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:48.599970    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 21:58:48.600097 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:48.599989    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d5df9c674-7d924: secret "image-registry-tls" not found
Sep 17 21:58:48.600097 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:48.600021    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e64d86f-9913-4a28-a1dc-1810710456bc-metrics-tls podName:0e64d86f-9913-4a28-a1dc-1810710456bc nodeName:}" failed. No retries permitted until 2026-09-17 21:58:52.600002664 +0000 UTC m=+43.767955639 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0e64d86f-9913-4a28-a1dc-1810710456bc-metrics-tls") pod "dns-default-kcnsx" (UID: "0e64d86f-9913-4a28-a1dc-1810710456bc") : secret "dns-default-metrics-tls" not found
Sep 17 21:58:48.600097 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:48.600034    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/22e6cd51-4d91-400b-b942-32e59b89a0be-registry-tls podName:22e6cd51-4d91-400b-b942-32e59b89a0be nodeName:}" failed. No retries permitted until 2026-09-17 21:58:52.600028346 +0000 UTC m=+43.767981307 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/22e6cd51-4d91-400b-b942-32e59b89a0be-registry-tls") pod "image-registry-5d5df9c674-7d924" (UID: "22e6cd51-4d91-400b-b942-32e59b89a0be") : secret "image-registry-tls" not found
Sep 17 21:58:49.799242 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:49.799208    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-l9x89"]
Sep 17 21:58:49.803640 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:49.803621    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-l9x89"
Sep 17 21:58:49.805813 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:49.805796    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 21:58:49.805975 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:49.805955    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 21:58:49.806029 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:49.805998    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 21:58:49.806826 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:49.806800    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qxfkf\""
Sep 17 21:58:49.806826 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:49.806802    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 21:58:49.811984 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:49.811965    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-l9x89"]
Sep 17 21:58:49.910111 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:49.910085    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cd8b7977-5b29-4e94-b08c-b4de9c50deaf-data-volume\") pod \"insights-runtime-extractor-l9x89\" (UID: \"cd8b7977-5b29-4e94-b08c-b4de9c50deaf\") " pod="openshift-insights/insights-runtime-extractor-l9x89"
Sep 17 21:58:49.910111 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:49.910114    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cd8b7977-5b29-4e94-b08c-b4de9c50deaf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l9x89\" (UID: \"cd8b7977-5b29-4e94-b08c-b4de9c50deaf\") " pod="openshift-insights/insights-runtime-extractor-l9x89"
Sep 17 21:58:49.910323 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:49.910133    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cd8b7977-5b29-4e94-b08c-b4de9c50deaf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l9x89\" (UID: \"cd8b7977-5b29-4e94-b08c-b4de9c50deaf\") " pod="openshift-insights/insights-runtime-extractor-l9x89"
Sep 17 21:58:49.910323 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:49.910179    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwh4h\" (UniqueName: \"kubernetes.io/projected/cd8b7977-5b29-4e94-b08c-b4de9c50deaf-kube-api-access-hwh4h\") pod \"insights-runtime-extractor-l9x89\" (UID: \"cd8b7977-5b29-4e94-b08c-b4de9c50deaf\") " pod="openshift-insights/insights-runtime-extractor-l9x89"
Sep 17 21:58:49.910323 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:49.910298    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cd8b7977-5b29-4e94-b08c-b4de9c50deaf-crio-socket\") pod \"insights-runtime-extractor-l9x89\" (UID: \"cd8b7977-5b29-4e94-b08c-b4de9c50deaf\") " pod="openshift-insights/insights-runtime-extractor-l9x89"
Sep 17 21:58:50.010980 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:50.010944    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cd8b7977-5b29-4e94-b08c-b4de9c50deaf-data-volume\") pod \"insights-runtime-extractor-l9x89\" (UID: \"cd8b7977-5b29-4e94-b08c-b4de9c50deaf\") " pod="openshift-insights/insights-runtime-extractor-l9x89"
Sep 17 21:58:50.010980 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:50.010979    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cd8b7977-5b29-4e94-b08c-b4de9c50deaf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l9x89\" (UID: \"cd8b7977-5b29-4e94-b08c-b4de9c50deaf\") " pod="openshift-insights/insights-runtime-extractor-l9x89"
Sep 17 21:58:50.011177 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:50.010999    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cd8b7977-5b29-4e94-b08c-b4de9c50deaf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l9x89\" (UID: \"cd8b7977-5b29-4e94-b08c-b4de9c50deaf\") " pod="openshift-insights/insights-runtime-extractor-l9x89"
Sep 17 21:58:50.011177 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:50.011026    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwh4h\" (UniqueName: \"kubernetes.io/projected/cd8b7977-5b29-4e94-b08c-b4de9c50deaf-kube-api-access-hwh4h\") pod \"insights-runtime-extractor-l9x89\" (UID: \"cd8b7977-5b29-4e94-b08c-b4de9c50deaf\") " pod="openshift-insights/insights-runtime-extractor-l9x89"
Sep 17 21:58:50.011177 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:50.011090    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 21:58:50.011177 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:50.011135    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 21:58:50.011310 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:50.011189    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cd8b7977-5b29-4e94-b08c-b4de9c50deaf-kube-rbac-proxy-cm podName:cd8b7977-5b29-4e94-b08c-b4de9c50deaf nodeName:}" failed. No retries permitted until 2026-09-17 21:58:50.511166006 +0000 UTC m=+41.679118989 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cd8b7977-5b29-4e94-b08c-b4de9c50deaf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l9x89" (UID: "cd8b7977-5b29-4e94-b08c-b4de9c50deaf") : configmap "kube-rbac-proxy" not found
Sep 17 21:58:50.011310 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:50.011208    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd8b7977-5b29-4e94-b08c-b4de9c50deaf-insights-runtime-extractor-tls podName:cd8b7977-5b29-4e94-b08c-b4de9c50deaf nodeName:}" failed. No retries permitted until 2026-09-17 21:58:50.51120014 +0000 UTC m=+41.679153101 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cd8b7977-5b29-4e94-b08c-b4de9c50deaf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l9x89" (UID: "cd8b7977-5b29-4e94-b08c-b4de9c50deaf") : secret "insights-runtime-extractor-tls" not found
Sep 17 21:58:50.011382 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:50.011318    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cd8b7977-5b29-4e94-b08c-b4de9c50deaf-crio-socket\") pod \"insights-runtime-extractor-l9x89\" (UID: \"cd8b7977-5b29-4e94-b08c-b4de9c50deaf\") " pod="openshift-insights/insights-runtime-extractor-l9x89"
Sep 17 21:58:50.011382 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:50.011325    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cd8b7977-5b29-4e94-b08c-b4de9c50deaf-data-volume\") pod \"insights-runtime-extractor-l9x89\" (UID: \"cd8b7977-5b29-4e94-b08c-b4de9c50deaf\") " pod="openshift-insights/insights-runtime-extractor-l9x89"
Sep 17 21:58:50.011513 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:50.011506    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cd8b7977-5b29-4e94-b08c-b4de9c50deaf-crio-socket\") pod \"insights-runtime-extractor-l9x89\" (UID: \"cd8b7977-5b29-4e94-b08c-b4de9c50deaf\") " pod="openshift-insights/insights-runtime-extractor-l9x89"
Sep 17 21:58:50.021105 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:50.021076    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwh4h\" (UniqueName: \"kubernetes.io/projected/cd8b7977-5b29-4e94-b08c-b4de9c50deaf-kube-api-access-hwh4h\") pod \"insights-runtime-extractor-l9x89\" (UID: \"cd8b7977-5b29-4e94-b08c-b4de9c50deaf\") " pod="openshift-insights/insights-runtime-extractor-l9x89"
Sep 17 21:58:50.514758 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:50.514725    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cd8b7977-5b29-4e94-b08c-b4de9c50deaf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l9x89\" (UID: \"cd8b7977-5b29-4e94-b08c-b4de9c50deaf\") " pod="openshift-insights/insights-runtime-extractor-l9x89"
Sep 17 21:58:50.514758 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:50.514759    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cd8b7977-5b29-4e94-b08c-b4de9c50deaf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l9x89\" (UID: \"cd8b7977-5b29-4e94-b08c-b4de9c50deaf\") " pod="openshift-insights/insights-runtime-extractor-l9x89"
Sep 17 21:58:50.514963 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:50.514850    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 21:58:50.514963 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:50.514867    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 21:58:50.514963 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:50.514917    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd8b7977-5b29-4e94-b08c-b4de9c50deaf-insights-runtime-extractor-tls podName:cd8b7977-5b29-4e94-b08c-b4de9c50deaf nodeName:}" failed. No retries permitted until 2026-09-17 21:58:51.514902015 +0000 UTC m=+42.682854976 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cd8b7977-5b29-4e94-b08c-b4de9c50deaf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l9x89" (UID: "cd8b7977-5b29-4e94-b08c-b4de9c50deaf") : secret "insights-runtime-extractor-tls" not found
Sep 17 21:58:50.514963 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:50.514932    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cd8b7977-5b29-4e94-b08c-b4de9c50deaf-kube-rbac-proxy-cm podName:cd8b7977-5b29-4e94-b08c-b4de9c50deaf nodeName:}" failed. No retries permitted until 2026-09-17 21:58:51.514926035 +0000 UTC m=+42.682878996 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cd8b7977-5b29-4e94-b08c-b4de9c50deaf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l9x89" (UID: "cd8b7977-5b29-4e94-b08c-b4de9c50deaf") : configmap "kube-rbac-proxy" not found
Sep 17 21:58:50.993636 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:50.993601    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-r4fmm"]
Sep 17 21:58:50.997223 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:50.997199    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-r4fmm"
Sep 17 21:58:51.000352 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:51.000329    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 17 21:58:51.000662 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:51.000646    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-k5m4c\""
Sep 17 21:58:51.000867 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:51.000852    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 17 21:58:51.007833 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:51.007814    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-r4fmm"]
Sep 17 21:58:51.120412 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:51.120375    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29w28\" (UniqueName: \"kubernetes.io/projected/ac585f9d-01e3-403f-a392-6081f15a9d9b-kube-api-access-29w28\") pod \"migrator-76794cbd6c-r4fmm\" (UID: \"ac585f9d-01e3-403f-a392-6081f15a9d9b\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-r4fmm"
Sep 17 21:58:51.221375 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:51.221344    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29w28\" (UniqueName: \"kubernetes.io/projected/ac585f9d-01e3-403f-a392-6081f15a9d9b-kube-api-access-29w28\") pod \"migrator-76794cbd6c-r4fmm\" (UID: \"ac585f9d-01e3-403f-a392-6081f15a9d9b\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-r4fmm"
Sep 17 21:58:51.229734 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:51.229708    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29w28\" (UniqueName: \"kubernetes.io/projected/ac585f9d-01e3-403f-a392-6081f15a9d9b-kube-api-access-29w28\") pod \"migrator-76794cbd6c-r4fmm\" (UID: \"ac585f9d-01e3-403f-a392-6081f15a9d9b\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-r4fmm"
Sep 17 21:58:51.305715 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:51.305640    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-r4fmm"
Sep 17 21:58:51.525052 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:51.525018    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cd8b7977-5b29-4e94-b08c-b4de9c50deaf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l9x89\" (UID: \"cd8b7977-5b29-4e94-b08c-b4de9c50deaf\") " pod="openshift-insights/insights-runtime-extractor-l9x89"
Sep 17 21:58:51.525052 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:51.525053    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cd8b7977-5b29-4e94-b08c-b4de9c50deaf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l9x89\" (UID: \"cd8b7977-5b29-4e94-b08c-b4de9c50deaf\") " pod="openshift-insights/insights-runtime-extractor-l9x89"
Sep 17 21:58:51.525330 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:51.525187    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 21:58:51.525330 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:51.525271    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cd8b7977-5b29-4e94-b08c-b4de9c50deaf-kube-rbac-proxy-cm podName:cd8b7977-5b29-4e94-b08c-b4de9c50deaf nodeName:}" failed. No retries permitted until 2026-09-17 21:58:53.525250167 +0000 UTC m=+44.693203142 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cd8b7977-5b29-4e94-b08c-b4de9c50deaf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l9x89" (UID: "cd8b7977-5b29-4e94-b08c-b4de9c50deaf") : configmap "kube-rbac-proxy" not found
Sep 17 21:58:51.525330 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:51.525284    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 21:58:51.525482 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:51.525335    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd8b7977-5b29-4e94-b08c-b4de9c50deaf-insights-runtime-extractor-tls podName:cd8b7977-5b29-4e94-b08c-b4de9c50deaf nodeName:}" failed. No retries permitted until 2026-09-17 21:58:53.525320428 +0000 UTC m=+44.693273393 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cd8b7977-5b29-4e94-b08c-b4de9c50deaf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l9x89" (UID: "cd8b7977-5b29-4e94-b08c-b4de9c50deaf") : secret "insights-runtime-extractor-tls" not found
Sep 17 21:58:51.553154 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:51.553116    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-n2fp6_9d2ac593-98b6-4b38-adef-a1bfbf1a168f/dns-node-resolver/0.log"
Sep 17 21:58:51.574547 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:51.574493    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-r4fmm"]
Sep 17 21:58:51.578024 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:58:51.578002    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac585f9d_01e3_403f_a392_6081f15a9d9b.slice/crio-dea61b18588f310988fbd1ce83ab789461bb00525c11349547a878ce3d448f9b WatchSource:0}: Error finding container dea61b18588f310988fbd1ce83ab789461bb00525c11349547a878ce3d448f9b: Status 404 returned error can't find the container with id dea61b18588f310988fbd1ce83ab789461bb00525c11349547a878ce3d448f9b
Sep 17 21:58:52.219307 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:52.219270    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-r4fmm" event={"ID":"ac585f9d-01e3-403f-a392-6081f15a9d9b","Type":"ContainerStarted","Data":"dea61b18588f310988fbd1ce83ab789461bb00525c11349547a878ce3d448f9b"}
Sep 17 21:58:52.353308 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:52.353278    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hzx5m_c20c2a19-2b30-48fb-bb14-b5321cf2f4d0/node-ca/0.log"
Sep 17 21:58:52.634698 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:52.634673    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/22e6cd51-4d91-400b-b942-32e59b89a0be-registry-tls\") pod \"image-registry-5d5df9c674-7d924\" (UID: \"22e6cd51-4d91-400b-b942-32e59b89a0be\") " pod="openshift-image-registry/image-registry-5d5df9c674-7d924"
Sep 17 21:58:52.634806 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:52.634713    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0e64d86f-9913-4a28-a1dc-1810710456bc-metrics-tls\") pod \"dns-default-kcnsx\" (UID: \"0e64d86f-9913-4a28-a1dc-1810710456bc\") " pod="openshift-dns/dns-default-kcnsx"
Sep 17 21:58:52.634870 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:52.634836    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 21:58:52.634870 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:52.634840    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 21:58:52.634870 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:52.634860    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d5df9c674-7d924: secret "image-registry-tls" not found
Sep 17 21:58:52.634990 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:52.634877    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e64d86f-9913-4a28-a1dc-1810710456bc-metrics-tls podName:0e64d86f-9913-4a28-a1dc-1810710456bc nodeName:}" failed. No retries permitted until 2026-09-17 21:59:00.634864662 +0000 UTC m=+51.802817624 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0e64d86f-9913-4a28-a1dc-1810710456bc-metrics-tls") pod "dns-default-kcnsx" (UID: "0e64d86f-9913-4a28-a1dc-1810710456bc") : secret "dns-default-metrics-tls" not found
Sep 17 21:58:52.634990 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:52.634912    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/22e6cd51-4d91-400b-b942-32e59b89a0be-registry-tls podName:22e6cd51-4d91-400b-b942-32e59b89a0be nodeName:}" failed. No retries permitted until 2026-09-17 21:59:00.634893376 +0000 UTC m=+51.802846353 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/22e6cd51-4d91-400b-b942-32e59b89a0be-registry-tls") pod "image-registry-5d5df9c674-7d924" (UID: "22e6cd51-4d91-400b-b942-32e59b89a0be") : secret "image-registry-tls" not found
Sep 17 21:58:53.154507 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:53.154478    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-p6kpt_01db16ed-7748-4c99-a5ce-7f98df2f5e7a/serve-healthcheck-canary/0.log"
Sep 17 21:58:53.223684 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:53.223656    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-r4fmm" event={"ID":"ac585f9d-01e3-403f-a392-6081f15a9d9b","Type":"ContainerStarted","Data":"a81d3a359c34a8add269e6b90f3d85e5fb50e483c3390970c7f7a6409658da37"}
Sep 17 21:58:53.224010 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:53.223711    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-r4fmm" event={"ID":"ac585f9d-01e3-403f-a392-6081f15a9d9b","Type":"ContainerStarted","Data":"a446fe0b288472e5396e1c71ae3f0afb93c670d603f6e3d1e56c14dffa250431"}
Sep 17 21:58:53.239204 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:53.239136    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-r4fmm" podStartSLOduration=2.304564058 podStartE2EDuration="3.239117934s" totalImagesPullingTime="934.553876ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:58:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:53.238973483 +0000 UTC m=+44.406926467" watchObservedRunningTime="2026-09-17 21:58:53.239117934 +0000 UTC m=+44.407070918"
Sep 17 21:58:53.540487 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:53.540384    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cd8b7977-5b29-4e94-b08c-b4de9c50deaf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l9x89\" (UID: \"cd8b7977-5b29-4e94-b08c-b4de9c50deaf\") " pod="openshift-insights/insights-runtime-extractor-l9x89"
Sep 17 21:58:53.540487 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:53.540433    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cd8b7977-5b29-4e94-b08c-b4de9c50deaf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l9x89\" (UID: \"cd8b7977-5b29-4e94-b08c-b4de9c50deaf\") " pod="openshift-insights/insights-runtime-extractor-l9x89"
Sep 17 21:58:53.540694 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:53.540509    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 21:58:53.540694 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:53.540536    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 21:58:53.540694 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:53.540585    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cd8b7977-5b29-4e94-b08c-b4de9c50deaf-kube-rbac-proxy-cm podName:cd8b7977-5b29-4e94-b08c-b4de9c50deaf nodeName:}" failed. No retries permitted until 2026-09-17 21:58:57.540566373 +0000 UTC m=+48.708519355 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cd8b7977-5b29-4e94-b08c-b4de9c50deaf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l9x89" (UID: "cd8b7977-5b29-4e94-b08c-b4de9c50deaf") : configmap "kube-rbac-proxy" not found
Sep 17 21:58:53.540694 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:53.540650    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd8b7977-5b29-4e94-b08c-b4de9c50deaf-insights-runtime-extractor-tls podName:cd8b7977-5b29-4e94-b08c-b4de9c50deaf nodeName:}" failed. No retries permitted until 2026-09-17 21:58:57.540630933 +0000 UTC m=+48.708583898 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cd8b7977-5b29-4e94-b08c-b4de9c50deaf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l9x89" (UID: "cd8b7977-5b29-4e94-b08c-b4de9c50deaf") : secret "insights-runtime-extractor-tls" not found
Sep 17 21:58:57.571920 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:57.571860    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cd8b7977-5b29-4e94-b08c-b4de9c50deaf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l9x89\" (UID: \"cd8b7977-5b29-4e94-b08c-b4de9c50deaf\") " pod="openshift-insights/insights-runtime-extractor-l9x89"
Sep 17 21:58:57.571920 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:58:57.571919    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cd8b7977-5b29-4e94-b08c-b4de9c50deaf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l9x89\" (UID: \"cd8b7977-5b29-4e94-b08c-b4de9c50deaf\") " pod="openshift-insights/insights-runtime-extractor-l9x89"
Sep 17 21:58:57.572401 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:57.572016    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 21:58:57.572401 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:57.572069    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd8b7977-5b29-4e94-b08c-b4de9c50deaf-insights-runtime-extractor-tls podName:cd8b7977-5b29-4e94-b08c-b4de9c50deaf nodeName:}" failed. No retries permitted until 2026-09-17 21:59:05.572055363 +0000 UTC m=+56.740008324 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cd8b7977-5b29-4e94-b08c-b4de9c50deaf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l9x89" (UID: "cd8b7977-5b29-4e94-b08c-b4de9c50deaf") : secret "insights-runtime-extractor-tls" not found
Sep 17 21:58:57.572401 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:57.572073    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 21:58:57.572401 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:58:57.572162    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cd8b7977-5b29-4e94-b08c-b4de9c50deaf-kube-rbac-proxy-cm podName:cd8b7977-5b29-4e94-b08c-b4de9c50deaf nodeName:}" failed. No retries permitted until 2026-09-17 21:59:05.572123613 +0000 UTC m=+56.740076577 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cd8b7977-5b29-4e94-b08c-b4de9c50deaf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l9x89" (UID: "cd8b7977-5b29-4e94-b08c-b4de9c50deaf") : configmap "kube-rbac-proxy" not found
Sep 17 21:59:00.697120 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:00.697077    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/22e6cd51-4d91-400b-b942-32e59b89a0be-registry-tls\") pod \"image-registry-5d5df9c674-7d924\" (UID: \"22e6cd51-4d91-400b-b942-32e59b89a0be\") " pod="openshift-image-registry/image-registry-5d5df9c674-7d924"
Sep 17 21:59:00.697525 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:00.697136    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0e64d86f-9913-4a28-a1dc-1810710456bc-metrics-tls\") pod \"dns-default-kcnsx\" (UID: \"0e64d86f-9913-4a28-a1dc-1810710456bc\") " pod="openshift-dns/dns-default-kcnsx"
Sep 17 21:59:00.697525 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:59:00.697258    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 21:59:00.697525 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:59:00.697324    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e64d86f-9913-4a28-a1dc-1810710456bc-metrics-tls podName:0e64d86f-9913-4a28-a1dc-1810710456bc nodeName:}" failed. No retries permitted until 2026-09-17 21:59:16.697310041 +0000 UTC m=+67.865263001 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0e64d86f-9913-4a28-a1dc-1810710456bc-metrics-tls") pod "dns-default-kcnsx" (UID: "0e64d86f-9913-4a28-a1dc-1810710456bc") : secret "dns-default-metrics-tls" not found
Sep 17 21:59:00.697525 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:59:00.697260    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 21:59:00.697525 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:59:00.697363    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d5df9c674-7d924: secret "image-registry-tls" not found
Sep 17 21:59:00.697525 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:59:00.697404    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/22e6cd51-4d91-400b-b942-32e59b89a0be-registry-tls podName:22e6cd51-4d91-400b-b942-32e59b89a0be nodeName:}" failed. No retries permitted until 2026-09-17 21:59:16.697392255 +0000 UTC m=+67.865345231 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/22e6cd51-4d91-400b-b942-32e59b89a0be-registry-tls") pod "image-registry-5d5df9c674-7d924" (UID: "22e6cd51-4d91-400b-b942-32e59b89a0be") : secret "image-registry-tls" not found
Sep 17 21:59:05.635803 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:05.635766    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cd8b7977-5b29-4e94-b08c-b4de9c50deaf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l9x89\" (UID: \"cd8b7977-5b29-4e94-b08c-b4de9c50deaf\") " pod="openshift-insights/insights-runtime-extractor-l9x89"
Sep 17 21:59:05.635803 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:05.635803    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cd8b7977-5b29-4e94-b08c-b4de9c50deaf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l9x89\" (UID: \"cd8b7977-5b29-4e94-b08c-b4de9c50deaf\") " pod="openshift-insights/insights-runtime-extractor-l9x89"
Sep 17 21:59:05.636322 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:59:05.635946    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 21:59:05.636322 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:59:05.636004    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd8b7977-5b29-4e94-b08c-b4de9c50deaf-insights-runtime-extractor-tls podName:cd8b7977-5b29-4e94-b08c-b4de9c50deaf nodeName:}" failed. No retries permitted until 2026-09-17 21:59:21.635990675 +0000 UTC m=+72.803943649 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cd8b7977-5b29-4e94-b08c-b4de9c50deaf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l9x89" (UID: "cd8b7977-5b29-4e94-b08c-b4de9c50deaf") : secret "insights-runtime-extractor-tls" not found
Sep 17 21:59:05.636322 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:05.636299    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cd8b7977-5b29-4e94-b08c-b4de9c50deaf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l9x89\" (UID: \"cd8b7977-5b29-4e94-b08c-b4de9c50deaf\") " pod="openshift-insights/insights-runtime-extractor-l9x89"
Sep 17 21:59:10.186662 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:10.186625    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nrkd5"
Sep 17 21:59:11.995401 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:11.995366    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-58cf7bb5c8-l8j5t"]
Sep 17 21:59:11.999307 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:11.999289    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58cf7bb5c8-l8j5t"
Sep 17 21:59:12.004729 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:12.004708    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 17 21:59:12.004729 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:12.004720    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 17 21:59:12.005408 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:12.005378    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 17 21:59:12.005477 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:12.005387    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 17 21:59:12.010885 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:12.010866    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-58cf7bb5c8-l8j5t"]
Sep 17 21:59:12.034666 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:12.034639    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5d5df9c674-7d924"]
Sep 17 21:59:12.034830 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:59:12.034811    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-5d5df9c674-7d924" podUID="22e6cd51-4d91-400b-b942-32e59b89a0be"
Sep 17 21:59:12.081065 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:12.081036    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-988484c4f-ggmxg"]
Sep 17 21:59:12.083038 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:12.083024    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-988484c4f-ggmxg"
Sep 17 21:59:12.102224 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:12.102196    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-988484c4f-ggmxg"]
Sep 17 21:59:12.186557 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:12.186522    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/84f54040-ce25-49f7-a00d-d0231b434a33-installation-pull-secrets\") pod \"image-registry-988484c4f-ggmxg\" (UID: \"84f54040-ce25-49f7-a00d-d0231b434a33\") " pod="openshift-image-registry/image-registry-988484c4f-ggmxg"
Sep 17 21:59:12.186715 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:12.186567    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/84f54040-ce25-49f7-a00d-d0231b434a33-registry-certificates\") pod \"image-registry-988484c4f-ggmxg\" (UID: \"84f54040-ce25-49f7-a00d-d0231b434a33\") " pod="openshift-image-registry/image-registry-988484c4f-ggmxg"
Sep 17 21:59:12.186715 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:12.186620    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/84f54040-ce25-49f7-a00d-d0231b434a33-registry-tls\") pod \"image-registry-988484c4f-ggmxg\" (UID: \"84f54040-ce25-49f7-a00d-d0231b434a33\") " pod="openshift-image-registry/image-registry-988484c4f-ggmxg"
Sep 17 21:59:12.186715 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:12.186672    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/84f54040-ce25-49f7-a00d-d0231b434a33-bound-sa-token\") pod \"image-registry-988484c4f-ggmxg\" (UID: \"84f54040-ce25-49f7-a00d-d0231b434a33\") " pod="openshift-image-registry/image-registry-988484c4f-ggmxg"
Sep 17 21:59:12.186715 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:12.186699    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82p9k\" (UniqueName: \"kubernetes.io/projected/84f54040-ce25-49f7-a00d-d0231b434a33-kube-api-access-82p9k\") pod \"image-registry-988484c4f-ggmxg\" (UID: \"84f54040-ce25-49f7-a00d-d0231b434a33\") " pod="openshift-image-registry/image-registry-988484c4f-ggmxg"
Sep 17 21:59:12.186865 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:12.186740    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/84f54040-ce25-49f7-a00d-d0231b434a33-trusted-ca\") pod \"image-registry-988484c4f-ggmxg\" (UID: \"84f54040-ce25-49f7-a00d-d0231b434a33\") " pod="openshift-image-registry/image-registry-988484c4f-ggmxg"
Sep 17 21:59:12.186865 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:12.186761    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d2c263ad-7b64-4c3d-a2f5-97ff8a1b1734-tmp\") pod \"klusterlet-addon-workmgr-58cf7bb5c8-l8j5t\" (UID: \"d2c263ad-7b64-4c3d-a2f5-97ff8a1b1734\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58cf7bb5c8-l8j5t"
Sep 17 21:59:12.186865 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:12.186779    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcz29\" (UniqueName: \"kubernetes.io/projected/d2c263ad-7b64-4c3d-a2f5-97ff8a1b1734-kube-api-access-hcz29\") pod \"klusterlet-addon-workmgr-58cf7bb5c8-l8j5t\" (UID: \"d2c263ad-7b64-4c3d-a2f5-97ff8a1b1734\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58cf7bb5c8-l8j5t"
Sep 17 21:59:12.186865 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:12.186806    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/84f54040-ce25-49f7-a00d-d0231b434a33-image-registry-private-configuration\") pod \"image-registry-988484c4f-ggmxg\" (UID: \"84f54040-ce25-49f7-a00d-d0231b434a33\") " pod="openshift-image-registry/image-registry-988484c4f-ggmxg"
Sep 17 21:59:12.186865 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:12.186842    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d2c263ad-7b64-4c3d-a2f5-97ff8a1b1734-klusterlet-config\") pod \"klusterlet-addon-workmgr-58cf7bb5c8-l8j5t\" (UID: \"d2c263ad-7b64-4c3d-a2f5-97ff8a1b1734\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58cf7bb5c8-l8j5t"
Sep 17 21:59:12.187032 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:12.186867    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/84f54040-ce25-49f7-a00d-d0231b434a33-ca-trust-extracted\") pod \"image-registry-988484c4f-ggmxg\" (UID: \"84f54040-ce25-49f7-a00d-d0231b434a33\") " pod="openshift-image-registry/image-registry-988484c4f-ggmxg"
Sep 17 21:59:12.269356 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:12.269276    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d5df9c674-7d924"
Sep 17 21:59:12.275264 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:12.275240    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d5df9c674-7d924"
Sep 17 21:59:12.287699 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:12.287677    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/84f54040-ce25-49f7-a00d-d0231b434a33-trusted-ca\") pod \"image-registry-988484c4f-ggmxg\" (UID: \"84f54040-ce25-49f7-a00d-d0231b434a33\") " pod="openshift-image-registry/image-registry-988484c4f-ggmxg"
Sep 17 21:59:12.287810 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:12.287710    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d2c263ad-7b64-4c3d-a2f5-97ff8a1b1734-tmp\") pod \"klusterlet-addon-workmgr-58cf7bb5c8-l8j5t\" (UID: \"d2c263ad-7b64-4c3d-a2f5-97ff8a1b1734\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58cf7bb5c8-l8j5t"
Sep 17 21:59:12.287870 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:12.287804    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcz29\" (UniqueName: \"kubernetes.io/projected/d2c263ad-7b64-4c3d-a2f5-97ff8a1b1734-kube-api-access-hcz29\") pod \"klusterlet-addon-workmgr-58cf7bb5c8-l8j5t\" (UID: \"d2c263ad-7b64-4c3d-a2f5-97ff8a1b1734\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58cf7bb5c8-l8j5t"
Sep 17 21:59:12.287870 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:12.287838    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/84f54040-ce25-49f7-a00d-d0231b434a33-image-registry-private-configuration\") pod \"image-registry-988484c4f-ggmxg\" (UID: \"84f54040-ce25-49f7-a00d-d0231b434a33\") " pod="openshift-image-registry/image-registry-988484c4f-ggmxg"
Sep 17 21:59:12.287973 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:12.287876    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d2c263ad-7b64-4c3d-a2f5-97ff8a1b1734-klusterlet-config\") pod \"klusterlet-addon-workmgr-58cf7bb5c8-l8j5t\" (UID: \"d2c263ad-7b64-4c3d-a2f5-97ff8a1b1734\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58cf7bb5c8-l8j5t"
Sep 17 21:59:12.287973 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:12.287903    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/84f54040-ce25-49f7-a00d-d0231b434a33-ca-trust-extracted\") pod \"image-registry-988484c4f-ggmxg\" (UID: \"84f54040-ce25-49f7-a00d-d0231b434a33\") " pod="openshift-image-registry/image-registry-988484c4f-ggmxg"
Sep 17 21:59:12.287973 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:12.287925    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/84f54040-ce25-49f7-a00d-d0231b434a33-installation-pull-secrets\") pod \"image-registry-988484c4f-ggmxg\" (UID: \"84f54040-ce25-49f7-a00d-d0231b434a33\") " pod="openshift-image-registry/image-registry-988484c4f-ggmxg"
Sep 17 21:59:12.287973 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:12.287953    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/84f54040-ce25-49f7-a00d-d0231b434a33-registry-certificates\") pod \"image-registry-988484c4f-ggmxg\" (UID: \"84f54040-ce25-49f7-a00d-d0231b434a33\") " pod="openshift-image-registry/image-registry-988484c4f-ggmxg"
Sep 17 21:59:12.288198 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:12.288060    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d2c263ad-7b64-4c3d-a2f5-97ff8a1b1734-tmp\") pod \"klusterlet-addon-workmgr-58cf7bb5c8-l8j5t\" (UID: \"d2c263ad-7b64-4c3d-a2f5-97ff8a1b1734\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58cf7bb5c8-l8j5t"
Sep 17 21:59:12.288198 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:12.288112    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/84f54040-ce25-49f7-a00d-d0231b434a33-registry-tls\") pod \"image-registry-988484c4f-ggmxg\" (UID: \"84f54040-ce25-49f7-a00d-d0231b434a33\") " pod="openshift-image-registry/image-registry-988484c4f-ggmxg"
Sep 17 21:59:12.288198 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:12.288182    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/84f54040-ce25-49f7-a00d-d0231b434a33-bound-sa-token\") pod \"image-registry-988484c4f-ggmxg\" (UID: \"84f54040-ce25-49f7-a00d-d0231b434a33\") " pod="openshift-image-registry/image-registry-988484c4f-ggmxg"
Sep 17 21:59:12.288347 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:12.288228    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82p9k\" (UniqueName: \"kubernetes.io/projected/84f54040-ce25-49f7-a00d-d0231b434a33-kube-api-access-82p9k\") pod \"image-registry-988484c4f-ggmxg\" (UID: \"84f54040-ce25-49f7-a00d-d0231b434a33\") " pod="openshift-image-registry/image-registry-988484c4f-ggmxg"
Sep 17 21:59:12.288405 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:12.288347    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/84f54040-ce25-49f7-a00d-d0231b434a33-ca-trust-extracted\") pod \"image-registry-988484c4f-ggmxg\" (UID: \"84f54040-ce25-49f7-a00d-d0231b434a33\") " pod="openshift-image-registry/image-registry-988484c4f-ggmxg"
Sep 17 21:59:12.288746 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:12.288721    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/84f54040-ce25-49f7-a00d-d0231b434a33-trusted-ca\") pod \"image-registry-988484c4f-ggmxg\" (UID: \"84f54040-ce25-49f7-a00d-d0231b434a33\") " pod="openshift-image-registry/image-registry-988484c4f-ggmxg"
Sep 17 21:59:12.288830 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:12.288753    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/84f54040-ce25-49f7-a00d-d0231b434a33-registry-certificates\") pod \"image-registry-988484c4f-ggmxg\" (UID: \"84f54040-ce25-49f7-a00d-d0231b434a33\") " pod="openshift-image-registry/image-registry-988484c4f-ggmxg"
Sep 17 21:59:12.290349 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:12.290305    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/84f54040-ce25-49f7-a00d-d0231b434a33-image-registry-private-configuration\") pod \"image-registry-988484c4f-ggmxg\" (UID: \"84f54040-ce25-49f7-a00d-d0231b434a33\") " pod="openshift-image-registry/image-registry-988484c4f-ggmxg"
Sep 17 21:59:12.290448 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:12.290361    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/84f54040-ce25-49f7-a00d-d0231b434a33-installation-pull-secrets\") pod \"image-registry-988484c4f-ggmxg\" (UID: \"84f54040-ce25-49f7-a00d-d0231b434a33\") " pod="openshift-image-registry/image-registry-988484c4f-ggmxg"
Sep 17 21:59:12.290780 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:12.290756    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/84f54040-ce25-49f7-a00d-d0231b434a33-registry-tls\") pod \"image-registry-988484c4f-ggmxg\" (UID: \"84f54040-ce25-49f7-a00d-d0231b434a33\") " pod="openshift-image-registry/image-registry-988484c4f-ggmxg"
Sep 17 21:59:12.290871 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:12.290855    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d2c263ad-7b64-4c3d-a2f5-97ff8a1b1734-klusterlet-config\") pod \"klusterlet-addon-workmgr-58cf7bb5c8-l8j5t\" (UID: \"d2c263ad-7b64-4c3d-a2f5-97ff8a1b1734\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58cf7bb5c8-l8j5t"
Sep 17 21:59:12.311171 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:12.311135    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcz29\" (UniqueName: \"kubernetes.io/projected/d2c263ad-7b64-4c3d-a2f5-97ff8a1b1734-kube-api-access-hcz29\") pod \"klusterlet-addon-workmgr-58cf7bb5c8-l8j5t\" (UID: \"d2c263ad-7b64-4c3d-a2f5-97ff8a1b1734\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58cf7bb5c8-l8j5t"
Sep 17 21:59:12.312575 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:12.312555    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82p9k\" (UniqueName: \"kubernetes.io/projected/84f54040-ce25-49f7-a00d-d0231b434a33-kube-api-access-82p9k\") pod \"image-registry-988484c4f-ggmxg\" (UID: \"84f54040-ce25-49f7-a00d-d0231b434a33\") " pod="openshift-image-registry/image-registry-988484c4f-ggmxg"
Sep 17 21:59:12.312686 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:12.312628    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/84f54040-ce25-49f7-a00d-d0231b434a33-bound-sa-token\") pod \"image-registry-988484c4f-ggmxg\" (UID: \"84f54040-ce25-49f7-a00d-d0231b434a33\") " pod="openshift-image-registry/image-registry-988484c4f-ggmxg"
Sep 17 21:59:12.388493 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:12.388466    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/22e6cd51-4d91-400b-b942-32e59b89a0be-trusted-ca\") pod \"22e6cd51-4d91-400b-b942-32e59b89a0be\" (UID: \"22e6cd51-4d91-400b-b942-32e59b89a0be\") "
Sep 17 21:59:12.388659 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:12.388531    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/22e6cd51-4d91-400b-b942-32e59b89a0be-image-registry-private-configuration\") pod \"22e6cd51-4d91-400b-b942-32e59b89a0be\" (UID: \"22e6cd51-4d91-400b-b942-32e59b89a0be\") "
Sep 17 21:59:12.388659 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:12.388551    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/22e6cd51-4d91-400b-b942-32e59b89a0be-ca-trust-extracted\") pod \"22e6cd51-4d91-400b-b942-32e59b89a0be\" (UID: \"22e6cd51-4d91-400b-b942-32e59b89a0be\") "
Sep 17 21:59:12.388659 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:12.388578    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/22e6cd51-4d91-400b-b942-32e59b89a0be-installation-pull-secrets\") pod \"22e6cd51-4d91-400b-b942-32e59b89a0be\" (UID: \"22e6cd51-4d91-400b-b942-32e59b89a0be\") "
Sep 17 21:59:12.388659 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:12.388594    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k654d\" (UniqueName: \"kubernetes.io/projected/22e6cd51-4d91-400b-b942-32e59b89a0be-kube-api-access-k654d\") pod \"22e6cd51-4d91-400b-b942-32e59b89a0be\" (UID: \"22e6cd51-4d91-400b-b942-32e59b89a0be\") "
Sep 17 21:59:12.388659 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:12.388615    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/22e6cd51-4d91-400b-b942-32e59b89a0be-bound-sa-token\") pod \"22e6cd51-4d91-400b-b942-32e59b89a0be\" (UID: \"22e6cd51-4d91-400b-b942-32e59b89a0be\") "
Sep 17 21:59:12.388659 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:12.388652    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/22e6cd51-4d91-400b-b942-32e59b89a0be-registry-certificates\") pod \"22e6cd51-4d91-400b-b942-32e59b89a0be\" (UID: \"22e6cd51-4d91-400b-b942-32e59b89a0be\") "
Sep 17 21:59:12.388982 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:12.388866    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22e6cd51-4d91-400b-b942-32e59b89a0be-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "22e6cd51-4d91-400b-b942-32e59b89a0be" (UID: "22e6cd51-4d91-400b-b942-32e59b89a0be"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:59:12.388982 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:12.388910    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22e6cd51-4d91-400b-b942-32e59b89a0be-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "22e6cd51-4d91-400b-b942-32e59b89a0be" (UID: "22e6cd51-4d91-400b-b942-32e59b89a0be"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:59:12.389103 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:12.389057    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22e6cd51-4d91-400b-b942-32e59b89a0be-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "22e6cd51-4d91-400b-b942-32e59b89a0be" (UID: "22e6cd51-4d91-400b-b942-32e59b89a0be"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:59:12.390851 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:12.390820    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22e6cd51-4d91-400b-b942-32e59b89a0be-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "22e6cd51-4d91-400b-b942-32e59b89a0be" (UID: "22e6cd51-4d91-400b-b942-32e59b89a0be"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:59:12.390968 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:12.390865    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22e6cd51-4d91-400b-b942-32e59b89a0be-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "22e6cd51-4d91-400b-b942-32e59b89a0be" (UID: "22e6cd51-4d91-400b-b942-32e59b89a0be"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:59:12.391108 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:12.391078    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22e6cd51-4d91-400b-b942-32e59b89a0be-kube-api-access-k654d" (OuterVolumeSpecName: "kube-api-access-k654d") pod "22e6cd51-4d91-400b-b942-32e59b89a0be" (UID: "22e6cd51-4d91-400b-b942-32e59b89a0be"). InnerVolumeSpecName "kube-api-access-k654d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:59:12.391244 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:12.391103    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22e6cd51-4d91-400b-b942-32e59b89a0be-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "22e6cd51-4d91-400b-b942-32e59b89a0be" (UID: "22e6cd51-4d91-400b-b942-32e59b89a0be"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:59:12.394294 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:12.394272    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vcbvz\""
Sep 17 21:59:12.403200 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:12.403184    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-988484c4f-ggmxg"
Sep 17 21:59:12.489667 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:12.489637    2563 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/22e6cd51-4d91-400b-b942-32e59b89a0be-installation-pull-secrets\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 21:59:12.489667 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:12.489661    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k654d\" (UniqueName: \"kubernetes.io/projected/22e6cd51-4d91-400b-b942-32e59b89a0be-kube-api-access-k654d\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 21:59:12.489667 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:12.489670    2563 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/22e6cd51-4d91-400b-b942-32e59b89a0be-bound-sa-token\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 21:59:12.489870 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:12.489679    2563 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/22e6cd51-4d91-400b-b942-32e59b89a0be-registry-certificates\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 21:59:12.489870 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:12.489687    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/22e6cd51-4d91-400b-b942-32e59b89a0be-trusted-ca\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 21:59:12.489870 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:12.489696    2563 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/22e6cd51-4d91-400b-b942-32e59b89a0be-image-registry-private-configuration\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 21:59:12.489870 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:12.489704    2563 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/22e6cd51-4d91-400b-b942-32e59b89a0be-ca-trust-extracted\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 21:59:12.608036 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:12.607955    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58cf7bb5c8-l8j5t"
Sep 17 21:59:12.672226 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:12.672156    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-988484c4f-ggmxg"]
Sep 17 21:59:12.675083 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:59:12.675056    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84f54040_ce25_49f7_a00d_d0231b434a33.slice/crio-2f9b531a04cac86390a1c32ecf276f0cf043895ad572c6cbb92200ec005b2c81 WatchSource:0}: Error finding container 2f9b531a04cac86390a1c32ecf276f0cf043895ad572c6cbb92200ec005b2c81: Status 404 returned error can't find the container with id 2f9b531a04cac86390a1c32ecf276f0cf043895ad572c6cbb92200ec005b2c81
Sep 17 21:59:12.876124 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:12.876034    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-58cf7bb5c8-l8j5t"]
Sep 17 21:59:12.878707 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:59:12.878673    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2c263ad_7b64_4c3d_a2f5_97ff8a1b1734.slice/crio-dfdd150d94e035d817ba3be65192fd1f3830df907de873c449825c0ac4554695 WatchSource:0}: Error finding container dfdd150d94e035d817ba3be65192fd1f3830df907de873c449825c0ac4554695: Status 404 returned error can't find the container with id dfdd150d94e035d817ba3be65192fd1f3830df907de873c449825c0ac4554695
Sep 17 21:59:13.272285 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:13.272246    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58cf7bb5c8-l8j5t" event={"ID":"d2c263ad-7b64-4c3d-a2f5-97ff8a1b1734","Type":"ContainerStarted","Data":"dfdd150d94e035d817ba3be65192fd1f3830df907de873c449825c0ac4554695"}
Sep 17 21:59:13.273650 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:13.273620    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-988484c4f-ggmxg" event={"ID":"84f54040-ce25-49f7-a00d-d0231b434a33","Type":"ContainerStarted","Data":"a836196226121ad85f64a7a2a846ed43a04b140543064d02374efa3f9ccd26e6"}
Sep 17 21:59:13.273650 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:13.273645    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d5df9c674-7d924"
Sep 17 21:59:13.273841 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:13.273668    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-988484c4f-ggmxg" event={"ID":"84f54040-ce25-49f7-a00d-d0231b434a33","Type":"ContainerStarted","Data":"2f9b531a04cac86390a1c32ecf276f0cf043895ad572c6cbb92200ec005b2c81"}
Sep 17 21:59:13.273841 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:13.273746    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-988484c4f-ggmxg"
Sep 17 21:59:13.293770 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:13.293722    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-988484c4f-ggmxg" podStartSLOduration=1.2936956720000001 podStartE2EDuration="1.293695672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:59:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:13.292570395 +0000 UTC m=+64.460523395" watchObservedRunningTime="2026-09-17 21:59:13.293695672 +0000 UTC m=+64.461648654"
Sep 17 21:59:13.317612 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:13.317588    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5d5df9c674-7d924"]
Sep 17 21:59:13.323682 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:13.323637    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5d5df9c674-7d924"]
Sep 17 21:59:13.396268 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:13.396242    2563 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/22e6cd51-4d91-400b-b942-32e59b89a0be-registry-tls\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 21:59:14.912043 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:14.912011    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22e6cd51-4d91-400b-b942-32e59b89a0be" path="/var/lib/kubelet/pods/22e6cd51-4d91-400b-b942-32e59b89a0be/volumes"
Sep 17 21:59:16.621545 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:16.621500    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45d5f94d-990c-4f9e-b1ab-980021c00716-metrics-certs\") pod \"network-metrics-daemon-xkk4g\" (UID: \"45d5f94d-990c-4f9e-b1ab-980021c00716\") " pod="openshift-multus/network-metrics-daemon-xkk4g"
Sep 17 21:59:16.621545 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:16.621540    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79a6aec6-e29a-45cf-881f-447b87f04557-original-pull-secret\") pod \"global-pull-secret-syncer-wxx9x\" (UID: \"79a6aec6-e29a-45cf-881f-447b87f04557\") " pod="kube-system/global-pull-secret-syncer-wxx9x"
Sep 17 21:59:16.624222 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:16.624201    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 21:59:16.624751 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:16.624732    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 21:59:16.634916 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:16.634894    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79a6aec6-e29a-45cf-881f-447b87f04557-original-pull-secret\") pod \"global-pull-secret-syncer-wxx9x\" (UID: \"79a6aec6-e29a-45cf-881f-447b87f04557\") " pod="kube-system/global-pull-secret-syncer-wxx9x"
Sep 17 21:59:16.635023 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:16.634923    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45d5f94d-990c-4f9e-b1ab-980021c00716-metrics-certs\") pod \"network-metrics-daemon-xkk4g\" (UID: \"45d5f94d-990c-4f9e-b1ab-980021c00716\") " pod="openshift-multus/network-metrics-daemon-xkk4g"
Sep 17 21:59:16.722753 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:16.722717    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0e64d86f-9913-4a28-a1dc-1810710456bc-metrics-tls\") pod \"dns-default-kcnsx\" (UID: \"0e64d86f-9913-4a28-a1dc-1810710456bc\") " pod="openshift-dns/dns-default-kcnsx"
Sep 17 21:59:16.722917 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:16.722783    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrkvs\" (UniqueName: \"kubernetes.io/projected/3b714c1c-9f7e-430e-bc07-94cf3a75921c-kube-api-access-hrkvs\") pod \"network-check-target-bnj94\" (UID: \"3b714c1c-9f7e-430e-bc07-94cf3a75921c\") " pod="openshift-network-diagnostics/network-check-target-bnj94"
Sep 17 21:59:16.725125 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:16.725098    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 21:59:16.725657 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:16.725629    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0e64d86f-9913-4a28-a1dc-1810710456bc-metrics-tls\") pod \"dns-default-kcnsx\" (UID: \"0e64d86f-9913-4a28-a1dc-1810710456bc\") " pod="openshift-dns/dns-default-kcnsx"
Sep 17 21:59:16.730976 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:16.730955    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cxqmr\""
Sep 17 21:59:16.735788 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:16.735766    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wxx9x"
Sep 17 21:59:16.735948 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:16.735931    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 21:59:16.739968 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:16.739950    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkk4g"
Sep 17 21:59:16.747001 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:16.746971    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrkvs\" (UniqueName: \"kubernetes.io/projected/3b714c1c-9f7e-430e-bc07-94cf3a75921c-kube-api-access-hrkvs\") pod \"network-check-target-bnj94\" (UID: \"3b714c1c-9f7e-430e-bc07-94cf3a75921c\") " pod="openshift-network-diagnostics/network-check-target-bnj94"
Sep 17 21:59:16.994855 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:16.994828    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-kb28p\""
Sep 17 21:59:17.003605 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:17.003569    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kcnsx"
Sep 17 21:59:17.043508 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:17.043475    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4zg8g\""
Sep 17 21:59:17.051501 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:17.051471    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bnj94"
Sep 17 21:59:17.151432 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:17.151406    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wxx9x"]
Sep 17 21:59:17.154039 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:59:17.154011    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79a6aec6_e29a_45cf_881f_447b87f04557.slice/crio-5d370e03d1564c1062c617650fa379c26f76eb98517b1e07817510f22c987588 WatchSource:0}: Error finding container 5d370e03d1564c1062c617650fa379c26f76eb98517b1e07817510f22c987588: Status 404 returned error can't find the container with id 5d370e03d1564c1062c617650fa379c26f76eb98517b1e07817510f22c987588
Sep 17 21:59:17.154337 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:17.154318    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xkk4g"]
Sep 17 21:59:17.157050 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:59:17.157031    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45d5f94d_990c_4f9e_b1ab_980021c00716.slice/crio-54c731a59c97420f57fb31210f484a92a734e1236e96de4b9ad7403e78ab14dd WatchSource:0}: Error finding container 54c731a59c97420f57fb31210f484a92a734e1236e96de4b9ad7403e78ab14dd: Status 404 returned error can't find the container with id 54c731a59c97420f57fb31210f484a92a734e1236e96de4b9ad7403e78ab14dd
Sep 17 21:59:17.273972 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:17.273935    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kcnsx"]
Sep 17 21:59:17.276479 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:59:17.276452    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e64d86f_9913_4a28_a1dc_1810710456bc.slice/crio-1c73b4c9b64df6a994a808361e20127ad5411aa7547f72ac4f46253c6029a1e2 WatchSource:0}: Error finding container 1c73b4c9b64df6a994a808361e20127ad5411aa7547f72ac4f46253c6029a1e2: Status 404 returned error can't find the container with id 1c73b4c9b64df6a994a808361e20127ad5411aa7547f72ac4f46253c6029a1e2
Sep 17 21:59:17.285090 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:17.285062    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kcnsx" event={"ID":"0e64d86f-9913-4a28-a1dc-1810710456bc","Type":"ContainerStarted","Data":"1c73b4c9b64df6a994a808361e20127ad5411aa7547f72ac4f46253c6029a1e2"}
Sep 17 21:59:17.286212 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:17.286183    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xkk4g" event={"ID":"45d5f94d-990c-4f9e-b1ab-980021c00716","Type":"ContainerStarted","Data":"54c731a59c97420f57fb31210f484a92a734e1236e96de4b9ad7403e78ab14dd"}
Sep 17 21:59:17.287182 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:17.287159    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wxx9x" event={"ID":"79a6aec6-e29a-45cf-881f-447b87f04557","Type":"ContainerStarted","Data":"5d370e03d1564c1062c617650fa379c26f76eb98517b1e07817510f22c987588"}
Sep 17 21:59:17.325524 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:17.325487    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bnj94"]
Sep 17 21:59:17.329657 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:59:17.329633    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b714c1c_9f7e_430e_bc07_94cf3a75921c.slice/crio-15ed7cdf2bc8977d4c0a162254eef462cefe4beed61ea97e4dd09a823e0d8cea WatchSource:0}: Error finding container 15ed7cdf2bc8977d4c0a162254eef462cefe4beed61ea97e4dd09a823e0d8cea: Status 404 returned error can't find the container with id 15ed7cdf2bc8977d4c0a162254eef462cefe4beed61ea97e4dd09a823e0d8cea
Sep 17 21:59:18.120762 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:18.120729    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-zwlxc"]
Sep 17 21:59:18.123974 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:18.123950    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-zwlxc"
Sep 17 21:59:18.127616 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:18.127595    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 17 21:59:18.127892 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:18.127861    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-d9rkr\""
Sep 17 21:59:18.131953 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:18.131926    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-zwlxc"]
Sep 17 21:59:18.135590 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:18.135544    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7cc7320d-2524-475c-abc8-1b97ca677476-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-zwlxc\" (UID: \"7cc7320d-2524-475c-abc8-1b97ca677476\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-zwlxc"
Sep 17 21:59:18.236651 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:18.236610    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7cc7320d-2524-475c-abc8-1b97ca677476-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-zwlxc\" (UID: \"7cc7320d-2524-475c-abc8-1b97ca677476\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-zwlxc"
Sep 17 21:59:18.236826 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:59:18.236758    2563 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 17 21:59:18.236826 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:59:18.236825    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cc7320d-2524-475c-abc8-1b97ca677476-tls-certificates podName:7cc7320d-2524-475c-abc8-1b97ca677476 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:18.736803434 +0000 UTC m=+69.904756410 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/7cc7320d-2524-475c-abc8-1b97ca677476-tls-certificates") pod "prometheus-operator-admission-webhook-5c6747dc7b-zwlxc" (UID: "7cc7320d-2524-475c-abc8-1b97ca677476") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 17 21:59:18.292702 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:18.292629    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bnj94" event={"ID":"3b714c1c-9f7e-430e-bc07-94cf3a75921c","Type":"ContainerStarted","Data":"15ed7cdf2bc8977d4c0a162254eef462cefe4beed61ea97e4dd09a823e0d8cea"}
Sep 17 21:59:18.741154 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:18.741013    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7cc7320d-2524-475c-abc8-1b97ca677476-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-zwlxc\" (UID: \"7cc7320d-2524-475c-abc8-1b97ca677476\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-zwlxc"
Sep 17 21:59:18.756826 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:18.751060    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7cc7320d-2524-475c-abc8-1b97ca677476-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-zwlxc\" (UID: \"7cc7320d-2524-475c-abc8-1b97ca677476\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-zwlxc"
Sep 17 21:59:19.040223 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:19.039905    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-zwlxc"
Sep 17 21:59:19.300920 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:19.300824    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xkk4g" event={"ID":"45d5f94d-990c-4f9e-b1ab-980021c00716","Type":"ContainerStarted","Data":"783ae61c14b05c8159df1f9f5e188ec6fc70d813241f6007770d78b82bfb8c6f"}
Sep 17 21:59:19.300920 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:19.300883    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xkk4g" event={"ID":"45d5f94d-990c-4f9e-b1ab-980021c00716","Type":"ContainerStarted","Data":"69c7be5f45b055fe8abd390f88ce7d0acbefc5cd32dc301696aefdcd4d4e8656"}
Sep 17 21:59:19.319324 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:19.319272    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xkk4g" podStartSLOduration=66.867614794 podStartE2EDuration="1m8.319254067s" totalImagesPullingTime="1.451639273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:58:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:19.318227342 +0000 UTC m=+70.486180328" watchObservedRunningTime="2026-09-17 21:59:19.319254067 +0000 UTC m=+70.487207063"
Sep 17 21:59:21.662811 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:21.662762    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cd8b7977-5b29-4e94-b08c-b4de9c50deaf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l9x89\" (UID: \"cd8b7977-5b29-4e94-b08c-b4de9c50deaf\") " pod="openshift-insights/insights-runtime-extractor-l9x89"
Sep 17 21:59:21.665152 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:21.665110    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cd8b7977-5b29-4e94-b08c-b4de9c50deaf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l9x89\" (UID: \"cd8b7977-5b29-4e94-b08c-b4de9c50deaf\") " pod="openshift-insights/insights-runtime-extractor-l9x89"
Sep 17 21:59:21.915850 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:21.915777    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qxfkf\""
Sep 17 21:59:21.923732 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:21.923708    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-l9x89"
Sep 17 21:59:23.489826 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:23.489796    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-zwlxc"]
Sep 17 21:59:23.491483 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:59:23.491459    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cc7320d_2524_475c_abc8_1b97ca677476.slice/crio-fec941106ea5bd197791c9a275f75f7464642cd95557fcc967f6c0fe34819054 WatchSource:0}: Error finding container fec941106ea5bd197791c9a275f75f7464642cd95557fcc967f6c0fe34819054: Status 404 returned error can't find the container with id fec941106ea5bd197791c9a275f75f7464642cd95557fcc967f6c0fe34819054
Sep 17 21:59:23.516182 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:23.516159    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-l9x89"]
Sep 17 21:59:23.521018 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:59:23.520996    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd8b7977_5b29_4e94_b08c_b4de9c50deaf.slice/crio-558d49c614a1ebfd436b0a045b600394ad1005d3dad679068ad4b76660b9b8db WatchSource:0}: Error finding container 558d49c614a1ebfd436b0a045b600394ad1005d3dad679068ad4b76660b9b8db: Status 404 returned error can't find the container with id 558d49c614a1ebfd436b0a045b600394ad1005d3dad679068ad4b76660b9b8db
Sep 17 21:59:24.337616 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:24.337583    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kcnsx" event={"ID":"0e64d86f-9913-4a28-a1dc-1810710456bc","Type":"ContainerStarted","Data":"4a1d4c8a7e8a0d59f502c6dfcd14ef626853d3335746bdcb2b551dd4f6ad4257"}
Sep 17 21:59:24.337876 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:24.337856    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-kcnsx"
Sep 17 21:59:24.337968 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:24.337894    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kcnsx" event={"ID":"0e64d86f-9913-4a28-a1dc-1810710456bc","Type":"ContainerStarted","Data":"e016bb1ad50e9de26f5529976c3a957206c5156ee38a98402e8e8f5bbcbb30fe"}
Sep 17 21:59:24.339541 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:24.339513    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bnj94" event={"ID":"3b714c1c-9f7e-430e-bc07-94cf3a75921c","Type":"ContainerStarted","Data":"50515fa5d92f78467ec8885a03cedc496de652740d28a595f0a61aa13dd131bd"}
Sep 17 21:59:24.339719 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:24.339697    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bnj94"
Sep 17 21:59:24.341152 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:24.341116    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58cf7bb5c8-l8j5t" event={"ID":"d2c263ad-7b64-4c3d-a2f5-97ff8a1b1734","Type":"ContainerStarted","Data":"6177f4f4030aa0317060f49ba7896ec33253a4da9ab087c2d6edbceef5efc110"}
Sep 17 21:59:24.341289 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:24.341271    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58cf7bb5c8-l8j5t"
Sep 17 21:59:24.342891 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:24.342870    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wxx9x" event={"ID":"79a6aec6-e29a-45cf-881f-447b87f04557","Type":"ContainerStarted","Data":"5c571490d3da3d21646053a5e24bb5a6f47d0bb799c685b7e043d4e2a1ed7d7e"}
Sep 17 21:59:24.343082 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:24.343046    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58cf7bb5c8-l8j5t"
Sep 17 21:59:24.344617 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:24.344590    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l9x89" event={"ID":"cd8b7977-5b29-4e94-b08c-b4de9c50deaf","Type":"ContainerStarted","Data":"861dcb4e368594a1c5b3660b20280c7317e8890b2d4da3c9d4fc36cbede94e8f"}
Sep 17 21:59:24.344705 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:24.344634    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l9x89" event={"ID":"cd8b7977-5b29-4e94-b08c-b4de9c50deaf","Type":"ContainerStarted","Data":"558d49c614a1ebfd436b0a045b600394ad1005d3dad679068ad4b76660b9b8db"}
Sep 17 21:59:24.345875 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:24.345855    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-zwlxc" event={"ID":"7cc7320d-2524-475c-abc8-1b97ca677476","Type":"ContainerStarted","Data":"fec941106ea5bd197791c9a275f75f7464642cd95557fcc967f6c0fe34819054"}
Sep 17 21:59:24.358608 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:24.358555    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-kcnsx" podStartSLOduration=34.431161156 podStartE2EDuration="40.358536962s" totalImagesPullingTime="5.927375806s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:58:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:24.357837534 +0000 UTC m=+75.525790518" watchObservedRunningTime="2026-09-17 21:59:24.358536962 +0000 UTC m=+75.526489948"
Sep 17 21:59:24.374670 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:24.374629    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-wxx9x" podStartSLOduration=67.311333432 podStartE2EDuration="1m13.374613148s" totalImagesPullingTime="6.063279716s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:24.373682685 +0000 UTC m=+75.541635668" watchObservedRunningTime="2026-09-17 21:59:24.374613148 +0000 UTC m=+75.542566111"
Sep 17 21:59:24.414748 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:24.414701    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-bnj94" podStartSLOduration=67.526745369 podStartE2EDuration="1m13.414683516s" totalImagesPullingTime="5.887938147s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:58:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:24.39325435 +0000 UTC m=+75.561207334" watchObservedRunningTime="2026-09-17 21:59:24.414683516 +0000 UTC m=+75.582636500"
Sep 17 21:59:24.415284 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:24.415253    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58cf7bb5c8-l8j5t" podStartSLOduration=3.081913042 podStartE2EDuration="13.41524408s" totalImagesPullingTime="10.333331038s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:59:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:24.414491386 +0000 UTC m=+75.582444368" watchObservedRunningTime="2026-09-17 21:59:24.41524408 +0000 UTC m=+75.583197067"
Sep 17 21:59:25.350945 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:25.350866    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l9x89" event={"ID":"cd8b7977-5b29-4e94-b08c-b4de9c50deaf","Type":"ContainerStarted","Data":"41f1eff80c8ab26338aadf5836cf34cd796bc2f84270320374f4765c20b4a8cf"}
Sep 17 21:59:25.352753 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:25.352726    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-zwlxc" event={"ID":"7cc7320d-2524-475c-abc8-1b97ca677476","Type":"ContainerStarted","Data":"df3ac445b44cb7515eac38f4afe98ea87ce4d10c3a6e0c98ae9daf02af1e8511"}
Sep 17 21:59:25.352896 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:25.352878    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-zwlxc"
Sep 17 21:59:25.357953 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:25.357932    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-zwlxc"
Sep 17 21:59:25.388970 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:25.388927    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-zwlxc" podStartSLOduration=6.388283783 podStartE2EDuration="7.388914387s" totalImagesPullingTime="1.000630604s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:59:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:25.370957969 +0000 UTC m=+76.538910978" watchObservedRunningTime="2026-09-17 21:59:25.388914387 +0000 UTC m=+76.556867407"
Sep 17 21:59:26.320334 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:26.320305    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-sjdcx"]
Sep 17 21:59:26.324291 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:26.324276    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-sjdcx"
Sep 17 21:59:26.330780 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:26.330752    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 17 21:59:26.330914 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:26.330795    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 17 21:59:26.331361 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:26.331338    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 21:59:26.331470 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:26.331337    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 21:59:26.331621 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:26.331603    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 21:59:26.331669 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:26.331635    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-pszlp\""
Sep 17 21:59:26.345313 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:26.345292    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-sjdcx"]
Sep 17 21:59:26.358065 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:26.358045    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l9x89" event={"ID":"cd8b7977-5b29-4e94-b08c-b4de9c50deaf","Type":"ContainerStarted","Data":"36d2802ea90d64b55f8db2bb91b6e4a3d16d45a85353b412ff3a386594e0429f"}
Sep 17 21:59:26.399885 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:26.399859    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/12b8424b-4eee-4a25-a2a2-407d9fbcf929-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-sjdcx\" (UID: \"12b8424b-4eee-4a25-a2a2-407d9fbcf929\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-sjdcx"
Sep 17 21:59:26.399979 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:26.399897    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjcv9\" (UniqueName: \"kubernetes.io/projected/12b8424b-4eee-4a25-a2a2-407d9fbcf929-kube-api-access-cjcv9\") pod \"prometheus-operator-7bd5d56d6-sjdcx\" (UID: \"12b8424b-4eee-4a25-a2a2-407d9fbcf929\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-sjdcx"
Sep 17 21:59:26.400090 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:26.400074    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/12b8424b-4eee-4a25-a2a2-407d9fbcf929-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-sjdcx\" (UID: \"12b8424b-4eee-4a25-a2a2-407d9fbcf929\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-sjdcx"
Sep 17 21:59:26.400302 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:26.400200    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/12b8424b-4eee-4a25-a2a2-407d9fbcf929-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-sjdcx\" (UID: \"12b8424b-4eee-4a25-a2a2-407d9fbcf929\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-sjdcx"
Sep 17 21:59:26.403037 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:26.402996    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-l9x89" podStartSLOduration=35.318661926 podStartE2EDuration="37.402983393s" totalImagesPullingTime="2.084321467s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:26.401959045 +0000 UTC m=+77.569912028" watchObservedRunningTime="2026-09-17 21:59:26.402983393 +0000 UTC m=+77.570936375"
Sep 17 21:59:26.501213 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:26.501189    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/12b8424b-4eee-4a25-a2a2-407d9fbcf929-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-sjdcx\" (UID: \"12b8424b-4eee-4a25-a2a2-407d9fbcf929\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-sjdcx"
Sep 17 21:59:26.501286 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:26.501217    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/12b8424b-4eee-4a25-a2a2-407d9fbcf929-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-sjdcx\" (UID: \"12b8424b-4eee-4a25-a2a2-407d9fbcf929\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-sjdcx"
Sep 17 21:59:26.501286 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:26.501234    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/12b8424b-4eee-4a25-a2a2-407d9fbcf929-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-sjdcx\" (UID: \"12b8424b-4eee-4a25-a2a2-407d9fbcf929\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-sjdcx"
Sep 17 21:59:26.501286 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:26.501250    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjcv9\" (UniqueName: \"kubernetes.io/projected/12b8424b-4eee-4a25-a2a2-407d9fbcf929-kube-api-access-cjcv9\") pod \"prometheus-operator-7bd5d56d6-sjdcx\" (UID: \"12b8424b-4eee-4a25-a2a2-407d9fbcf929\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-sjdcx"
Sep 17 21:59:26.501475 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:59:26.501459    2563 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 17 21:59:26.501539 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:59:26.501528    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12b8424b-4eee-4a25-a2a2-407d9fbcf929-prometheus-operator-tls podName:12b8424b-4eee-4a25-a2a2-407d9fbcf929 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:27.001507363 +0000 UTC m=+78.169460343 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/12b8424b-4eee-4a25-a2a2-407d9fbcf929-prometheus-operator-tls") pod "prometheus-operator-7bd5d56d6-sjdcx" (UID: "12b8424b-4eee-4a25-a2a2-407d9fbcf929") : secret "prometheus-operator-tls" not found
Sep 17 21:59:26.501885 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:26.501866    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/12b8424b-4eee-4a25-a2a2-407d9fbcf929-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-sjdcx\" (UID: \"12b8424b-4eee-4a25-a2a2-407d9fbcf929\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-sjdcx"
Sep 17 21:59:26.503478 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:26.503453    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/12b8424b-4eee-4a25-a2a2-407d9fbcf929-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-sjdcx\" (UID: \"12b8424b-4eee-4a25-a2a2-407d9fbcf929\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-sjdcx"
Sep 17 21:59:26.511696 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:26.511678    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjcv9\" (UniqueName: \"kubernetes.io/projected/12b8424b-4eee-4a25-a2a2-407d9fbcf929-kube-api-access-cjcv9\") pod \"prometheus-operator-7bd5d56d6-sjdcx\" (UID: \"12b8424b-4eee-4a25-a2a2-407d9fbcf929\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-sjdcx"
Sep 17 21:59:27.004795 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:27.004761    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/12b8424b-4eee-4a25-a2a2-407d9fbcf929-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-sjdcx\" (UID: \"12b8424b-4eee-4a25-a2a2-407d9fbcf929\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-sjdcx"
Sep 17 21:59:27.006944 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:27.006919    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/12b8424b-4eee-4a25-a2a2-407d9fbcf929-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-sjdcx\" (UID: \"12b8424b-4eee-4a25-a2a2-407d9fbcf929\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-sjdcx"
Sep 17 21:59:27.233415 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:27.233383    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-sjdcx"
Sep 17 21:59:27.501639 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:27.501615    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-sjdcx"]
Sep 17 21:59:27.503938 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:59:27.503912    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12b8424b_4eee_4a25_a2a2_407d9fbcf929.slice/crio-d1a71e93c4e6d7c4247a34675ec212033a8f648ef7b2e04e70753632d05761bd WatchSource:0}: Error finding container d1a71e93c4e6d7c4247a34675ec212033a8f648ef7b2e04e70753632d05761bd: Status 404 returned error can't find the container with id d1a71e93c4e6d7c4247a34675ec212033a8f648ef7b2e04e70753632d05761bd
Sep 17 21:59:28.366243 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:28.366196    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-sjdcx" event={"ID":"12b8424b-4eee-4a25-a2a2-407d9fbcf929","Type":"ContainerStarted","Data":"d1a71e93c4e6d7c4247a34675ec212033a8f648ef7b2e04e70753632d05761bd"}
Sep 17 21:59:29.371868 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:29.371838    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-sjdcx" event={"ID":"12b8424b-4eee-4a25-a2a2-407d9fbcf929","Type":"ContainerStarted","Data":"9ecd140cb779ad638e8ecaa34d940c9388ff36be62ef87b4fb90aa0d076d6e37"}
Sep 17 21:59:29.372263 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:29.371893    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-sjdcx" event={"ID":"12b8424b-4eee-4a25-a2a2-407d9fbcf929","Type":"ContainerStarted","Data":"d0b98eccc574f4810e1f34eb755bf5510c0064c15076c37ef7bcb6c4799cf146"}
Sep 17 21:59:29.399785 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:29.399739    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-sjdcx" podStartSLOduration=2.240257644 podStartE2EDuration="3.399726917s" totalImagesPullingTime="1.159469273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:59:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:29.399273695 +0000 UTC m=+80.567226679" watchObservedRunningTime="2026-09-17 21:59:29.399726917 +0000 UTC m=+80.567679900"
Sep 17 21:59:29.441845 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:29.441823    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6594f4c75-drqjx"]
Sep 17 21:59:29.444832 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:29.444818    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-drqjx"
Sep 17 21:59:29.447107 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:29.447083    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 21:59:29.447213 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:29.447121    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 21:59:29.447348 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:29.447332    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-jp25d\""
Sep 17 21:59:29.465825 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:29.465807    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-drqjx"]
Sep 17 21:59:29.524608 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:29.524587    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6qfb\" (UniqueName: \"kubernetes.io/projected/3befbbf9-41ff-4f57-9cfd-aee679772893-kube-api-access-w6qfb\") pod \"downloads-6594f4c75-drqjx\" (UID: \"3befbbf9-41ff-4f57-9cfd-aee679772893\") " pod="openshift-console/downloads-6594f4c75-drqjx"
Sep 17 21:59:29.625522 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:29.625463    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6qfb\" (UniqueName: \"kubernetes.io/projected/3befbbf9-41ff-4f57-9cfd-aee679772893-kube-api-access-w6qfb\") pod \"downloads-6594f4c75-drqjx\" (UID: \"3befbbf9-41ff-4f57-9cfd-aee679772893\") " pod="openshift-console/downloads-6594f4c75-drqjx"
Sep 17 21:59:29.634562 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:29.634534    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6qfb\" (UniqueName: \"kubernetes.io/projected/3befbbf9-41ff-4f57-9cfd-aee679772893-kube-api-access-w6qfb\") pod \"downloads-6594f4c75-drqjx\" (UID: \"3befbbf9-41ff-4f57-9cfd-aee679772893\") " pod="openshift-console/downloads-6594f4c75-drqjx"
Sep 17 21:59:29.753301 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:29.753275    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-drqjx"
Sep 17 21:59:30.038287 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:30.038262    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-drqjx"]
Sep 17 21:59:30.041058 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:59:30.041033    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3befbbf9_41ff_4f57_9cfd_aee679772893.slice/crio-7fdf2883f56a502d60ff7c99705f1cce0e6ffe3b43e8b5fccdaa4fac65a8ab80 WatchSource:0}: Error finding container 7fdf2883f56a502d60ff7c99705f1cce0e6ffe3b43e8b5fccdaa4fac65a8ab80: Status 404 returned error can't find the container with id 7fdf2883f56a502d60ff7c99705f1cce0e6ffe3b43e8b5fccdaa4fac65a8ab80
Sep 17 21:59:30.376780 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:30.376709    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-drqjx" event={"ID":"3befbbf9-41ff-4f57-9cfd-aee679772893","Type":"ContainerStarted","Data":"7fdf2883f56a502d60ff7c99705f1cce0e6ffe3b43e8b5fccdaa4fac65a8ab80"}
Sep 17 21:59:31.719051 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:31.719015    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-jqq5f"]
Sep 17 21:59:31.722982 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:31.722960    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-jqq5f"
Sep 17 21:59:31.725757 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:31.725731    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 17 21:59:31.726027 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:31.726011    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-xvkmg\""
Sep 17 21:59:31.726094 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:31.726063    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 17 21:59:31.738548 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:31.738527    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-klbh9"]
Sep 17 21:59:31.742227 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:31.742208    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-jqq5f"]
Sep 17 21:59:31.742337 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:31.742312    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-klbh9"
Sep 17 21:59:31.744477 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:31.744456    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5x5d5\""
Sep 17 21:59:31.744618 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:31.744590    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 21:59:31.744716 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:31.744700    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 21:59:31.744779 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:31.744714    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 21:59:31.841264 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:31.841234    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3eebb3bc-62b7-464c-8199-d97145f2c8cb-node-exporter-accelerators-collector-config\") pod \"node-exporter-klbh9\" (UID: \"3eebb3bc-62b7-464c-8199-d97145f2c8cb\") " pod="openshift-monitoring/node-exporter-klbh9"
Sep 17 21:59:31.841396 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:31.841279    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3eebb3bc-62b7-464c-8199-d97145f2c8cb-sys\") pod \"node-exporter-klbh9\" (UID: \"3eebb3bc-62b7-464c-8199-d97145f2c8cb\") " pod="openshift-monitoring/node-exporter-klbh9"
Sep 17 21:59:31.841396 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:31.841321    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3eebb3bc-62b7-464c-8199-d97145f2c8cb-node-exporter-wtmp\") pod \"node-exporter-klbh9\" (UID: \"3eebb3bc-62b7-464c-8199-d97145f2c8cb\") " pod="openshift-monitoring/node-exporter-klbh9"
Sep 17 21:59:31.841396 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:31.841344    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3eebb3bc-62b7-464c-8199-d97145f2c8cb-root\") pod \"node-exporter-klbh9\" (UID: \"3eebb3bc-62b7-464c-8199-d97145f2c8cb\") " pod="openshift-monitoring/node-exporter-klbh9"
Sep 17 21:59:31.841396 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:31.841358    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3eebb3bc-62b7-464c-8199-d97145f2c8cb-node-exporter-tls\") pod \"node-exporter-klbh9\" (UID: \"3eebb3bc-62b7-464c-8199-d97145f2c8cb\") " pod="openshift-monitoring/node-exporter-klbh9"
Sep 17 21:59:31.841396 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:31.841381    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fn6t\" (UniqueName: \"kubernetes.io/projected/3eebb3bc-62b7-464c-8199-d97145f2c8cb-kube-api-access-7fn6t\") pod \"node-exporter-klbh9\" (UID: \"3eebb3bc-62b7-464c-8199-d97145f2c8cb\") " pod="openshift-monitoring/node-exporter-klbh9"
Sep 17 21:59:31.841692 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:31.841431    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2123e624-b230-4c2b-a8c4-eaad63f9fbcb-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-jqq5f\" (UID: \"2123e624-b230-4c2b-a8c4-eaad63f9fbcb\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-jqq5f"
Sep 17 21:59:31.841692 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:31.841464    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9xrh\" (UniqueName: \"kubernetes.io/projected/2123e624-b230-4c2b-a8c4-eaad63f9fbcb-kube-api-access-p9xrh\") pod \"openshift-state-metrics-86c5bc79d6-jqq5f\" (UID: \"2123e624-b230-4c2b-a8c4-eaad63f9fbcb\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-jqq5f"
Sep 17 21:59:31.841692 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:31.841508    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2123e624-b230-4c2b-a8c4-eaad63f9fbcb-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-jqq5f\" (UID: \"2123e624-b230-4c2b-a8c4-eaad63f9fbcb\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-jqq5f"
Sep 17 21:59:31.841692 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:31.841528    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2123e624-b230-4c2b-a8c4-eaad63f9fbcb-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-jqq5f\" (UID: \"2123e624-b230-4c2b-a8c4-eaad63f9fbcb\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-jqq5f"
Sep 17 21:59:31.841692 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:31.841571    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3eebb3bc-62b7-464c-8199-d97145f2c8cb-metrics-client-ca\") pod \"node-exporter-klbh9\" (UID: \"3eebb3bc-62b7-464c-8199-d97145f2c8cb\") " pod="openshift-monitoring/node-exporter-klbh9"
Sep 17 21:59:31.841692 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:31.841608    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3eebb3bc-62b7-464c-8199-d97145f2c8cb-node-exporter-textfile\") pod \"node-exporter-klbh9\" (UID: \"3eebb3bc-62b7-464c-8199-d97145f2c8cb\") " pod="openshift-monitoring/node-exporter-klbh9"
Sep 17 21:59:31.841692 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:31.841635    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3eebb3bc-62b7-464c-8199-d97145f2c8cb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-klbh9\" (UID: \"3eebb3bc-62b7-464c-8199-d97145f2c8cb\") " pod="openshift-monitoring/node-exporter-klbh9"
Sep 17 21:59:31.942329 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:31.942299    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3eebb3bc-62b7-464c-8199-d97145f2c8cb-node-exporter-accelerators-collector-config\") pod \"node-exporter-klbh9\" (UID: \"3eebb3bc-62b7-464c-8199-d97145f2c8cb\") " pod="openshift-monitoring/node-exporter-klbh9"
Sep 17 21:59:31.942329 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:31.942335    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3eebb3bc-62b7-464c-8199-d97145f2c8cb-sys\") pod \"node-exporter-klbh9\" (UID: \"3eebb3bc-62b7-464c-8199-d97145f2c8cb\") " pod="openshift-monitoring/node-exporter-klbh9"
Sep 17 21:59:31.942538 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:31.942352    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3eebb3bc-62b7-464c-8199-d97145f2c8cb-node-exporter-wtmp\") pod \"node-exporter-klbh9\" (UID: \"3eebb3bc-62b7-464c-8199-d97145f2c8cb\") " pod="openshift-monitoring/node-exporter-klbh9"
Sep 17 21:59:31.942538 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:31.942373    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3eebb3bc-62b7-464c-8199-d97145f2c8cb-root\") pod \"node-exporter-klbh9\" (UID: \"3eebb3bc-62b7-464c-8199-d97145f2c8cb\") " pod="openshift-monitoring/node-exporter-klbh9"
Sep 17 21:59:31.942538 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:31.942406    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3eebb3bc-62b7-464c-8199-d97145f2c8cb-node-exporter-tls\") pod \"node-exporter-klbh9\" (UID: \"3eebb3bc-62b7-464c-8199-d97145f2c8cb\") " pod="openshift-monitoring/node-exporter-klbh9"
Sep 17 21:59:31.942538 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:31.942430    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3eebb3bc-62b7-464c-8199-d97145f2c8cb-root\") pod \"node-exporter-klbh9\" (UID: \"3eebb3bc-62b7-464c-8199-d97145f2c8cb\") " pod="openshift-monitoring/node-exporter-klbh9"
Sep 17 21:59:31.942538 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:31.942443    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fn6t\" (UniqueName: \"kubernetes.io/projected/3eebb3bc-62b7-464c-8199-d97145f2c8cb-kube-api-access-7fn6t\") pod \"node-exporter-klbh9\" (UID: \"3eebb3bc-62b7-464c-8199-d97145f2c8cb\") " pod="openshift-monitoring/node-exporter-klbh9"
Sep 17 21:59:31.942538 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:31.942482    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2123e624-b230-4c2b-a8c4-eaad63f9fbcb-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-jqq5f\" (UID: \"2123e624-b230-4c2b-a8c4-eaad63f9fbcb\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-jqq5f"
Sep 17 21:59:31.942538 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:31.942512    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9xrh\" (UniqueName: \"kubernetes.io/projected/2123e624-b230-4c2b-a8c4-eaad63f9fbcb-kube-api-access-p9xrh\") pod \"openshift-state-metrics-86c5bc79d6-jqq5f\" (UID: \"2123e624-b230-4c2b-a8c4-eaad63f9fbcb\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-jqq5f"
Sep 17 21:59:31.942538 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:31.942519    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3eebb3bc-62b7-464c-8199-d97145f2c8cb-node-exporter-wtmp\") pod \"node-exporter-klbh9\" (UID: \"3eebb3bc-62b7-464c-8199-d97145f2c8cb\") " pod="openshift-monitoring/node-exporter-klbh9"
Sep 17 21:59:31.942538 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:59:31.942532    2563 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 17 21:59:31.942966 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:31.942546    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2123e624-b230-4c2b-a8c4-eaad63f9fbcb-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-jqq5f\" (UID: \"2123e624-b230-4c2b-a8c4-eaad63f9fbcb\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-jqq5f"
Sep 17 21:59:31.942966 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:31.942573    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2123e624-b230-4c2b-a8c4-eaad63f9fbcb-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-jqq5f\" (UID: \"2123e624-b230-4c2b-a8c4-eaad63f9fbcb\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-jqq5f"
Sep 17 21:59:31.942966 ip-10-0-138-45 kubenswrapper[2563]: E0917 21:59:31.942591    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3eebb3bc-62b7-464c-8199-d97145f2c8cb-node-exporter-tls podName:3eebb3bc-62b7-464c-8199-d97145f2c8cb nodeName:}" failed. No retries permitted until 2026-09-17 21:59:32.442569877 +0000 UTC m=+83.610522852 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/3eebb3bc-62b7-464c-8199-d97145f2c8cb-node-exporter-tls") pod "node-exporter-klbh9" (UID: "3eebb3bc-62b7-464c-8199-d97145f2c8cb") : secret "node-exporter-tls" not found
Sep 17 21:59:31.942966 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:31.942624    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3eebb3bc-62b7-464c-8199-d97145f2c8cb-metrics-client-ca\") pod \"node-exporter-klbh9\" (UID: \"3eebb3bc-62b7-464c-8199-d97145f2c8cb\") " pod="openshift-monitoring/node-exporter-klbh9"
Sep 17 21:59:31.942966 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:31.942666    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3eebb3bc-62b7-464c-8199-d97145f2c8cb-node-exporter-textfile\") pod \"node-exporter-klbh9\" (UID: \"3eebb3bc-62b7-464c-8199-d97145f2c8cb\") " pod="openshift-monitoring/node-exporter-klbh9"
Sep 17 21:59:31.942966 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:31.942693    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3eebb3bc-62b7-464c-8199-d97145f2c8cb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-klbh9\" (UID: \"3eebb3bc-62b7-464c-8199-d97145f2c8cb\") " pod="openshift-monitoring/node-exporter-klbh9"
Sep 17 21:59:31.942966 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:31.942948    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3eebb3bc-62b7-464c-8199-d97145f2c8cb-node-exporter-accelerators-collector-config\") pod \"node-exporter-klbh9\" (UID: \"3eebb3bc-62b7-464c-8199-d97145f2c8cb\") " pod="openshift-monitoring/node-exporter-klbh9"
Sep 17 21:59:31.943321 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:31.943010    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3eebb3bc-62b7-464c-8199-d97145f2c8cb-sys\") pod \"node-exporter-klbh9\" (UID: \"3eebb3bc-62b7-464c-8199-d97145f2c8cb\") " pod="openshift-monitoring/node-exporter-klbh9"
Sep 17 21:59:31.943321 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:31.943057    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3eebb3bc-62b7-464c-8199-d97145f2c8cb-node-exporter-textfile\") pod \"node-exporter-klbh9\" (UID: \"3eebb3bc-62b7-464c-8199-d97145f2c8cb\") " pod="openshift-monitoring/node-exporter-klbh9"
Sep 17 21:59:31.943321 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:31.943287    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3eebb3bc-62b7-464c-8199-d97145f2c8cb-metrics-client-ca\") pod \"node-exporter-klbh9\" (UID: \"3eebb3bc-62b7-464c-8199-d97145f2c8cb\") " pod="openshift-monitoring/node-exporter-klbh9"
Sep 17 21:59:31.943734 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:31.943708    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2123e624-b230-4c2b-a8c4-eaad63f9fbcb-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-jqq5f\" (UID: \"2123e624-b230-4c2b-a8c4-eaad63f9fbcb\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-jqq5f"
Sep 17 21:59:31.945075 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:31.945052    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2123e624-b230-4c2b-a8c4-eaad63f9fbcb-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-jqq5f\" (UID: \"2123e624-b230-4c2b-a8c4-eaad63f9fbcb\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-jqq5f"
Sep 17 21:59:31.946421 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:31.946401    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3eebb3bc-62b7-464c-8199-d97145f2c8cb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-klbh9\" (UID: \"3eebb3bc-62b7-464c-8199-d97145f2c8cb\") " pod="openshift-monitoring/node-exporter-klbh9"
Sep 17 21:59:31.946618 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:31.946596    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2123e624-b230-4c2b-a8c4-eaad63f9fbcb-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-jqq5f\" (UID: \"2123e624-b230-4c2b-a8c4-eaad63f9fbcb\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-jqq5f"
Sep 17 21:59:31.952593 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:31.952570    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fn6t\" (UniqueName: \"kubernetes.io/projected/3eebb3bc-62b7-464c-8199-d97145f2c8cb-kube-api-access-7fn6t\") pod \"node-exporter-klbh9\" (UID: \"3eebb3bc-62b7-464c-8199-d97145f2c8cb\") " pod="openshift-monitoring/node-exporter-klbh9"
Sep 17 21:59:31.952786 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:31.952760    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9xrh\" (UniqueName: \"kubernetes.io/projected/2123e624-b230-4c2b-a8c4-eaad63f9fbcb-kube-api-access-p9xrh\") pod \"openshift-state-metrics-86c5bc79d6-jqq5f\" (UID: \"2123e624-b230-4c2b-a8c4-eaad63f9fbcb\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-jqq5f"
Sep 17 21:59:32.033967 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:32.033904    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-jqq5f"
Sep 17 21:59:32.315959 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:32.315881    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-jqq5f"]
Sep 17 21:59:32.320059 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:59:32.320031    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2123e624_b230_4c2b_a8c4_eaad63f9fbcb.slice/crio-bc7c76297a249fbadb2eaf2fddd1df65c82e756d5eed3e69fcd0aaa7017941fa WatchSource:0}: Error finding container bc7c76297a249fbadb2eaf2fddd1df65c82e756d5eed3e69fcd0aaa7017941fa: Status 404 returned error can't find the container with id bc7c76297a249fbadb2eaf2fddd1df65c82e756d5eed3e69fcd0aaa7017941fa
Sep 17 21:59:32.387293 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:32.387205    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-jqq5f" event={"ID":"2123e624-b230-4c2b-a8c4-eaad63f9fbcb","Type":"ContainerStarted","Data":"bc7c76297a249fbadb2eaf2fddd1df65c82e756d5eed3e69fcd0aaa7017941fa"}
Sep 17 21:59:32.446804 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:32.446775    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3eebb3bc-62b7-464c-8199-d97145f2c8cb-node-exporter-tls\") pod \"node-exporter-klbh9\" (UID: \"3eebb3bc-62b7-464c-8199-d97145f2c8cb\") " pod="openshift-monitoring/node-exporter-klbh9"
Sep 17 21:59:32.449465 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:32.449445    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3eebb3bc-62b7-464c-8199-d97145f2c8cb-node-exporter-tls\") pod \"node-exporter-klbh9\" (UID: \"3eebb3bc-62b7-464c-8199-d97145f2c8cb\") " pod="openshift-monitoring/node-exporter-klbh9"
Sep 17 21:59:32.652686 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:32.652613    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-klbh9"
Sep 17 21:59:32.664003 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:59:32.663975    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eebb3bc_62b7_464c_8199_d97145f2c8cb.slice/crio-891aac8179480c86a16d04a1b2e2c167d970cdc180e42928b965291b271aeed4 WatchSource:0}: Error finding container 891aac8179480c86a16d04a1b2e2c167d970cdc180e42928b965291b271aeed4: Status 404 returned error can't find the container with id 891aac8179480c86a16d04a1b2e2c167d970cdc180e42928b965291b271aeed4
Sep 17 21:59:33.393430 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:33.393398    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-jqq5f" event={"ID":"2123e624-b230-4c2b-a8c4-eaad63f9fbcb","Type":"ContainerStarted","Data":"644ddfc482724cbae48cff91de5a8af180d8f6541140e31243ff8cd1463d148b"}
Sep 17 21:59:33.393969 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:33.393467    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-jqq5f" event={"ID":"2123e624-b230-4c2b-a8c4-eaad63f9fbcb","Type":"ContainerStarted","Data":"92d11a12507e4094d5cc1ee4fbb4b45c65fb065f8119e53efa62ec0e2c6fe588"}
Sep 17 21:59:33.394819 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:33.394760    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-klbh9" event={"ID":"3eebb3bc-62b7-464c-8199-d97145f2c8cb","Type":"ContainerStarted","Data":"891aac8179480c86a16d04a1b2e2c167d970cdc180e42928b965291b271aeed4"}
Sep 17 21:59:34.280842 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:34.280799    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-988484c4f-ggmxg"
Sep 17 21:59:34.354704 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:34.354679    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-kcnsx"
Sep 17 21:59:34.402274 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:34.402182    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-jqq5f" event={"ID":"2123e624-b230-4c2b-a8c4-eaad63f9fbcb","Type":"ContainerStarted","Data":"294a9ce11f0eed8473fffe36ded0ffc2fd2020a47aad6ae21e95d6af32694d28"}
Sep 17 21:59:34.404501 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:34.404472    2563 generic.go:356] "Generic (PLEG): container finished" podID="3eebb3bc-62b7-464c-8199-d97145f2c8cb" containerID="c9457f999bf661af465c05191d963f87aff67b8bdb4b8bd6496cdc20420bab26" exitCode=0
Sep 17 21:59:34.404625 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:34.404513    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-klbh9" event={"ID":"3eebb3bc-62b7-464c-8199-d97145f2c8cb","Type":"ContainerDied","Data":"c9457f999bf661af465c05191d963f87aff67b8bdb4b8bd6496cdc20420bab26"}
Sep 17 21:59:34.458215 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:34.458161    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-jqq5f" podStartSLOduration=2.273776318 podStartE2EDuration="3.458127155s" totalImagesPullingTime="1.184350837s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:59:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:34.425547493 +0000 UTC m=+85.593500479" watchObservedRunningTime="2026-09-17 21:59:34.458127155 +0000 UTC m=+85.626080139"
Sep 17 21:59:35.411433 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:35.411395    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-klbh9" event={"ID":"3eebb3bc-62b7-464c-8199-d97145f2c8cb","Type":"ContainerStarted","Data":"6c268d69d9b069019c1183be8ced86ccb3a4addec0bb50874c6ec4e41e5a31f1"}
Sep 17 21:59:35.411834 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:35.411462    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-klbh9" event={"ID":"3eebb3bc-62b7-464c-8199-d97145f2c8cb","Type":"ContainerStarted","Data":"efed54c3285d22b6ffe110ce6026c405af52fbefc78db71f8da67c554d655f2d"}
Sep 17 21:59:35.432356 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:35.432306    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-klbh9" podStartSLOduration=3.330006911 podStartE2EDuration="4.432285555s" totalImagesPullingTime="1.028969557s" totalInitContainerRuntime="73.309087ms" isStatefulPod=true podCreationTimestamp="2026-09-17 21:59:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:35.431033506 +0000 UTC m=+86.598986515" watchObservedRunningTime="2026-09-17 21:59:35.432285555 +0000 UTC m=+86.600238550"
Sep 17 21:59:35.700361 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:35.700278    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-69d87d4967-6w259"]
Sep 17 21:59:35.705315 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:35.704679    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69d87d4967-6w259"
Sep 17 21:59:35.708235 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:35.708213    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 21:59:35.708407 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:35.708385    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 21:59:35.709431 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:35.709067    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-rhsvk\""
Sep 17 21:59:35.709431 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:35.709216    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 21:59:35.709431 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:35.709385    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 21:59:35.709431 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:35.709397    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 21:59:35.721991 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:35.721971    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69d87d4967-6w259"]
Sep 17 21:59:35.773402 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:35.773368    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01ac47d5-3c66-40fa-adc5-294dbf851172-console-serving-cert\") pod \"console-69d87d4967-6w259\" (UID: \"01ac47d5-3c66-40fa-adc5-294dbf851172\") " pod="openshift-console/console-69d87d4967-6w259"
Sep 17 21:59:35.773569 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:35.773411    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01ac47d5-3c66-40fa-adc5-294dbf851172-console-oauth-config\") pod \"console-69d87d4967-6w259\" (UID: \"01ac47d5-3c66-40fa-adc5-294dbf851172\") " pod="openshift-console/console-69d87d4967-6w259"
Sep 17 21:59:35.773569 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:35.773455    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp2sm\" (UniqueName: \"kubernetes.io/projected/01ac47d5-3c66-40fa-adc5-294dbf851172-kube-api-access-jp2sm\") pod \"console-69d87d4967-6w259\" (UID: \"01ac47d5-3c66-40fa-adc5-294dbf851172\") " pod="openshift-console/console-69d87d4967-6w259"
Sep 17 21:59:35.773569 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:35.773488    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/01ac47d5-3c66-40fa-adc5-294dbf851172-console-config\") pod \"console-69d87d4967-6w259\" (UID: \"01ac47d5-3c66-40fa-adc5-294dbf851172\") " pod="openshift-console/console-69d87d4967-6w259"
Sep 17 21:59:35.773569 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:35.773518    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01ac47d5-3c66-40fa-adc5-294dbf851172-service-ca\") pod \"console-69d87d4967-6w259\" (UID: \"01ac47d5-3c66-40fa-adc5-294dbf851172\") " pod="openshift-console/console-69d87d4967-6w259"
Sep 17 21:59:35.773569 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:35.773539    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01ac47d5-3c66-40fa-adc5-294dbf851172-oauth-serving-cert\") pod \"console-69d87d4967-6w259\" (UID: \"01ac47d5-3c66-40fa-adc5-294dbf851172\") " pod="openshift-console/console-69d87d4967-6w259"
Sep 17 21:59:35.874349 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:35.874317    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/01ac47d5-3c66-40fa-adc5-294dbf851172-console-config\") pod \"console-69d87d4967-6w259\" (UID: \"01ac47d5-3c66-40fa-adc5-294dbf851172\") " pod="openshift-console/console-69d87d4967-6w259"
Sep 17 21:59:35.874503 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:35.874366    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01ac47d5-3c66-40fa-adc5-294dbf851172-service-ca\") pod \"console-69d87d4967-6w259\" (UID: \"01ac47d5-3c66-40fa-adc5-294dbf851172\") " pod="openshift-console/console-69d87d4967-6w259"
Sep 17 21:59:35.874503 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:35.874393    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01ac47d5-3c66-40fa-adc5-294dbf851172-oauth-serving-cert\") pod \"console-69d87d4967-6w259\" (UID: \"01ac47d5-3c66-40fa-adc5-294dbf851172\") " pod="openshift-console/console-69d87d4967-6w259"
Sep 17 21:59:35.874503 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:35.874435    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01ac47d5-3c66-40fa-adc5-294dbf851172-console-serving-cert\") pod \"console-69d87d4967-6w259\" (UID: \"01ac47d5-3c66-40fa-adc5-294dbf851172\") " pod="openshift-console/console-69d87d4967-6w259"
Sep 17 21:59:35.874503 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:35.874457    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01ac47d5-3c66-40fa-adc5-294dbf851172-console-oauth-config\") pod \"console-69d87d4967-6w259\" (UID: \"01ac47d5-3c66-40fa-adc5-294dbf851172\") " pod="openshift-console/console-69d87d4967-6w259"
Sep 17 21:59:35.874503 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:35.874488    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp2sm\" (UniqueName: \"kubernetes.io/projected/01ac47d5-3c66-40fa-adc5-294dbf851172-kube-api-access-jp2sm\") pod \"console-69d87d4967-6w259\" (UID: \"01ac47d5-3c66-40fa-adc5-294dbf851172\") " pod="openshift-console/console-69d87d4967-6w259"
Sep 17 21:59:35.875263 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:35.875216    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01ac47d5-3c66-40fa-adc5-294dbf851172-service-ca\") pod \"console-69d87d4967-6w259\" (UID: \"01ac47d5-3c66-40fa-adc5-294dbf851172\") " pod="openshift-console/console-69d87d4967-6w259"
Sep 17 21:59:35.875263 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:35.875238    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/01ac47d5-3c66-40fa-adc5-294dbf851172-console-config\") pod \"console-69d87d4967-6w259\" (UID: \"01ac47d5-3c66-40fa-adc5-294dbf851172\") " pod="openshift-console/console-69d87d4967-6w259"
Sep 17 21:59:35.875467 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:35.875339    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01ac47d5-3c66-40fa-adc5-294dbf851172-oauth-serving-cert\") pod \"console-69d87d4967-6w259\" (UID: \"01ac47d5-3c66-40fa-adc5-294dbf851172\") " pod="openshift-console/console-69d87d4967-6w259"
Sep 17 21:59:35.877536 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:35.877512    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01ac47d5-3c66-40fa-adc5-294dbf851172-console-oauth-config\") pod \"console-69d87d4967-6w259\" (UID: \"01ac47d5-3c66-40fa-adc5-294dbf851172\") " pod="openshift-console/console-69d87d4967-6w259"
Sep 17 21:59:35.877636 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:35.877562    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01ac47d5-3c66-40fa-adc5-294dbf851172-console-serving-cert\") pod \"console-69d87d4967-6w259\" (UID: \"01ac47d5-3c66-40fa-adc5-294dbf851172\") " pod="openshift-console/console-69d87d4967-6w259"
Sep 17 21:59:35.883674 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:35.883651    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp2sm\" (UniqueName: \"kubernetes.io/projected/01ac47d5-3c66-40fa-adc5-294dbf851172-kube-api-access-jp2sm\") pod \"console-69d87d4967-6w259\" (UID: \"01ac47d5-3c66-40fa-adc5-294dbf851172\") " pod="openshift-console/console-69d87d4967-6w259"
Sep 17 21:59:36.015794 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:36.015712    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69d87d4967-6w259"
Sep 17 21:59:36.133875 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:36.133837    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-58ccf7d5fc-sg9zb"]
Sep 17 21:59:36.138294 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:36.138270    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-58ccf7d5fc-sg9zb"
Sep 17 21:59:36.141800 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:36.141715    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 17 21:59:36.141800 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:36.141715    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 17 21:59:36.141800 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:36.141767    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-czq5s\""
Sep 17 21:59:36.142026 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:36.141847    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 17 21:59:36.142026 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:36.141868    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 17 21:59:36.142026 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:36.141898    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-c34vn48arpi0t\""
Sep 17 21:59:36.148579 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:36.148562    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-58ccf7d5fc-sg9zb"]
Sep 17 21:59:36.177451 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:36.177424    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49ac9268-c95a-4df8-a119-9843598e243d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-58ccf7d5fc-sg9zb\" (UID: \"49ac9268-c95a-4df8-a119-9843598e243d\") " pod="openshift-monitoring/metrics-server-58ccf7d5fc-sg9zb"
Sep 17 21:59:36.177553 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:36.177473    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/49ac9268-c95a-4df8-a119-9843598e243d-secret-metrics-server-client-certs\") pod \"metrics-server-58ccf7d5fc-sg9zb\" (UID: \"49ac9268-c95a-4df8-a119-9843598e243d\") " pod="openshift-monitoring/metrics-server-58ccf7d5fc-sg9zb"
Sep 17 21:59:36.177632 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:36.177559    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/49ac9268-c95a-4df8-a119-9843598e243d-secret-metrics-server-tls\") pod \"metrics-server-58ccf7d5fc-sg9zb\" (UID: \"49ac9268-c95a-4df8-a119-9843598e243d\") " pod="openshift-monitoring/metrics-server-58ccf7d5fc-sg9zb"
Sep 17 21:59:36.177632 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:36.177623    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/49ac9268-c95a-4df8-a119-9843598e243d-audit-log\") pod \"metrics-server-58ccf7d5fc-sg9zb\" (UID: \"49ac9268-c95a-4df8-a119-9843598e243d\") " pod="openshift-monitoring/metrics-server-58ccf7d5fc-sg9zb"
Sep 17 21:59:36.177734 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:36.177652    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/49ac9268-c95a-4df8-a119-9843598e243d-metrics-server-audit-profiles\") pod \"metrics-server-58ccf7d5fc-sg9zb\" (UID: \"49ac9268-c95a-4df8-a119-9843598e243d\") " pod="openshift-monitoring/metrics-server-58ccf7d5fc-sg9zb"
Sep 17 21:59:36.177734 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:36.177685    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4txwv\" (UniqueName: \"kubernetes.io/projected/49ac9268-c95a-4df8-a119-9843598e243d-kube-api-access-4txwv\") pod \"metrics-server-58ccf7d5fc-sg9zb\" (UID: \"49ac9268-c95a-4df8-a119-9843598e243d\") " pod="openshift-monitoring/metrics-server-58ccf7d5fc-sg9zb"
Sep 17 21:59:36.177734 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:36.177716    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/49ac9268-c95a-4df8-a119-9843598e243d-client-ca-bundle\") pod \"metrics-server-58ccf7d5fc-sg9zb\" (UID: \"49ac9268-c95a-4df8-a119-9843598e243d\") " pod="openshift-monitoring/metrics-server-58ccf7d5fc-sg9zb"
Sep 17 21:59:36.278087 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:36.278027    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/49ac9268-c95a-4df8-a119-9843598e243d-audit-log\") pod \"metrics-server-58ccf7d5fc-sg9zb\" (UID: \"49ac9268-c95a-4df8-a119-9843598e243d\") " pod="openshift-monitoring/metrics-server-58ccf7d5fc-sg9zb"
Sep 17 21:59:36.278087 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:36.278063    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/49ac9268-c95a-4df8-a119-9843598e243d-metrics-server-audit-profiles\") pod \"metrics-server-58ccf7d5fc-sg9zb\" (UID: \"49ac9268-c95a-4df8-a119-9843598e243d\") " pod="openshift-monitoring/metrics-server-58ccf7d5fc-sg9zb"
Sep 17 21:59:36.278266 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:36.278127    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4txwv\" (UniqueName: \"kubernetes.io/projected/49ac9268-c95a-4df8-a119-9843598e243d-kube-api-access-4txwv\") pod \"metrics-server-58ccf7d5fc-sg9zb\" (UID: \"49ac9268-c95a-4df8-a119-9843598e243d\") " pod="openshift-monitoring/metrics-server-58ccf7d5fc-sg9zb"
Sep 17 21:59:36.278266 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:36.278178    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/49ac9268-c95a-4df8-a119-9843598e243d-client-ca-bundle\") pod \"metrics-server-58ccf7d5fc-sg9zb\" (UID: \"49ac9268-c95a-4df8-a119-9843598e243d\") " pod="openshift-monitoring/metrics-server-58ccf7d5fc-sg9zb"
Sep 17 21:59:36.278266 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:36.278218    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49ac9268-c95a-4df8-a119-9843598e243d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-58ccf7d5fc-sg9zb\" (UID: \"49ac9268-c95a-4df8-a119-9843598e243d\") " pod="openshift-monitoring/metrics-server-58ccf7d5fc-sg9zb"
Sep 17 21:59:36.278266 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:36.278249    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/49ac9268-c95a-4df8-a119-9843598e243d-secret-metrics-server-client-certs\") pod \"metrics-server-58ccf7d5fc-sg9zb\" (UID: \"49ac9268-c95a-4df8-a119-9843598e243d\") " pod="openshift-monitoring/metrics-server-58ccf7d5fc-sg9zb"
Sep 17 21:59:36.278462 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:36.278290    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/49ac9268-c95a-4df8-a119-9843598e243d-secret-metrics-server-tls\") pod \"metrics-server-58ccf7d5fc-sg9zb\" (UID: \"49ac9268-c95a-4df8-a119-9843598e243d\") " pod="openshift-monitoring/metrics-server-58ccf7d5fc-sg9zb"
Sep 17 21:59:36.278944 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:36.278848    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/49ac9268-c95a-4df8-a119-9843598e243d-audit-log\") pod \"metrics-server-58ccf7d5fc-sg9zb\" (UID: \"49ac9268-c95a-4df8-a119-9843598e243d\") " pod="openshift-monitoring/metrics-server-58ccf7d5fc-sg9zb"
Sep 17 21:59:36.278944 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:36.278860    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49ac9268-c95a-4df8-a119-9843598e243d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-58ccf7d5fc-sg9zb\" (UID: \"49ac9268-c95a-4df8-a119-9843598e243d\") " pod="openshift-monitoring/metrics-server-58ccf7d5fc-sg9zb"
Sep 17 21:59:36.279127 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:36.279104    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/49ac9268-c95a-4df8-a119-9843598e243d-metrics-server-audit-profiles\") pod \"metrics-server-58ccf7d5fc-sg9zb\" (UID: \"49ac9268-c95a-4df8-a119-9843598e243d\") " pod="openshift-monitoring/metrics-server-58ccf7d5fc-sg9zb"
Sep 17 21:59:36.281054 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:36.281031    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/49ac9268-c95a-4df8-a119-9843598e243d-client-ca-bundle\") pod \"metrics-server-58ccf7d5fc-sg9zb\" (UID: \"49ac9268-c95a-4df8-a119-9843598e243d\") " pod="openshift-monitoring/metrics-server-58ccf7d5fc-sg9zb"
Sep 17 21:59:36.281207 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:36.281182    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/49ac9268-c95a-4df8-a119-9843598e243d-secret-metrics-server-tls\") pod \"metrics-server-58ccf7d5fc-sg9zb\" (UID: \"49ac9268-c95a-4df8-a119-9843598e243d\") " pod="openshift-monitoring/metrics-server-58ccf7d5fc-sg9zb"
Sep 17 21:59:36.281756 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:36.281737    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/49ac9268-c95a-4df8-a119-9843598e243d-secret-metrics-server-client-certs\") pod \"metrics-server-58ccf7d5fc-sg9zb\" (UID: \"49ac9268-c95a-4df8-a119-9843598e243d\") " pod="openshift-monitoring/metrics-server-58ccf7d5fc-sg9zb"
Sep 17 21:59:36.289345 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:36.289323    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4txwv\" (UniqueName: \"kubernetes.io/projected/49ac9268-c95a-4df8-a119-9843598e243d-kube-api-access-4txwv\") pod \"metrics-server-58ccf7d5fc-sg9zb\" (UID: \"49ac9268-c95a-4df8-a119-9843598e243d\") " pod="openshift-monitoring/metrics-server-58ccf7d5fc-sg9zb"
Sep 17 21:59:36.297865 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:36.297839    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69d87d4967-6w259"]
Sep 17 21:59:36.300207 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:59:36.300183    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01ac47d5_3c66_40fa_adc5_294dbf851172.slice/crio-3b237ec881c2612fd3eceedfdf013fe91944bd2e40816c8877ccb327a871239f WatchSource:0}: Error finding container 3b237ec881c2612fd3eceedfdf013fe91944bd2e40816c8877ccb327a871239f: Status 404 returned error can't find the container with id 3b237ec881c2612fd3eceedfdf013fe91944bd2e40816c8877ccb327a871239f
Sep 17 21:59:36.417426 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:36.417058    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69d87d4967-6w259" event={"ID":"01ac47d5-3c66-40fa-adc5-294dbf851172","Type":"ContainerStarted","Data":"3b237ec881c2612fd3eceedfdf013fe91944bd2e40816c8877ccb327a871239f"}
Sep 17 21:59:36.450417 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:36.450364    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-58ccf7d5fc-sg9zb"
Sep 17 21:59:36.734741 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:36.733775    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-58ccf7d5fc-sg9zb"]
Sep 17 21:59:36.739857 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:59:36.739652    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49ac9268_c95a_4df8_a119_9843598e243d.slice/crio-8ac0a1fbc585712e667bf13aaee75a3304c5b3390e522cc8d6e8c631201265fb WatchSource:0}: Error finding container 8ac0a1fbc585712e667bf13aaee75a3304c5b3390e522cc8d6e8c631201265fb: Status 404 returned error can't find the container with id 8ac0a1fbc585712e667bf13aaee75a3304c5b3390e522cc8d6e8c631201265fb
Sep 17 21:59:37.424886 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:37.424810    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-58ccf7d5fc-sg9zb" event={"ID":"49ac9268-c95a-4df8-a119-9843598e243d","Type":"ContainerStarted","Data":"8ac0a1fbc585712e667bf13aaee75a3304c5b3390e522cc8d6e8c631201265fb"}
Sep 17 21:59:38.069117 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.067653    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 21:59:38.074176 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.073813    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 21:59:38.079459 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.076737    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-cgkpc\""
Sep 17 21:59:38.079459 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.076952    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 17 21:59:38.079459 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.077133    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 17 21:59:38.079459 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.077339    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 17 21:59:38.079459 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.077797    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 17 21:59:38.079459 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.078060    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 17 21:59:38.079459 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.078244    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 17 21:59:38.079459 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.078466    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9rr5phjncvf5d\""
Sep 17 21:59:38.079459 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.078663    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 17 21:59:38.079459 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.078826    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 17 21:59:38.079459 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.078994    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 17 21:59:38.079459 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.079175    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 17 21:59:38.079459 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.079346    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 17 21:59:38.080313 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.079530    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 17 21:59:38.080313 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.079839    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 17 21:59:38.092845 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.091373    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 17 21:59:38.100914 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.100889    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 21:59:38.196198 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.196159    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 21:59:38.196345 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.196236    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 21:59:38.196345 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.196265    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 21:59:38.196345 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.196289    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 21:59:38.196345 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.196318    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 21:59:38.196345 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.196344    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 21:59:38.196591 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.196395    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 21:59:38.196591 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.196454    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 21:59:38.196591 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.196498    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 21:59:38.196591 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.196525    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 21:59:38.196591 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.196562    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 21:59:38.196591 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.196588    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 21:59:38.196880 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.196622    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 21:59:38.196880 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.196669    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w47l7\" (UniqueName: \"kubernetes.io/projected/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-kube-api-access-w47l7\") pod \"prometheus-k8s-0\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 21:59:38.196880 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.196698    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-config-out\") pod \"prometheus-k8s-0\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 21:59:38.196880 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.196742    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-config\") pod \"prometheus-k8s-0\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 21:59:38.196880 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.196778    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 21:59:38.196880 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.196804    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 21:59:38.196880 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.196866    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 21:59:38.197244 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.196902    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-web-config\") pod \"prometheus-k8s-0\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 21:59:38.297524 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.297486    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 21:59:38.297695 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.297534    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 21:59:38.297695 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.297560    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 21:59:38.297695 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.297588    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 21:59:38.297695 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.297614    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 21:59:38.297695 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.297637    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 21:59:38.297695 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.297661    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 21:59:38.297695 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.297688    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 21:59:38.298063 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.297716    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 21:59:38.298063 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.297750    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 21:59:38.298063 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.297773    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 21:59:38.298063 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.297809    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 21:59:38.298063 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.297842    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w47l7\" (UniqueName: \"kubernetes.io/projected/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-kube-api-access-w47l7\") pod \"prometheus-k8s-0\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 21:59:38.298063 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.297870    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-config-out\") pod \"prometheus-k8s-0\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 21:59:38.298063 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.297905    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-config\") pod \"prometheus-k8s-0\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 21:59:38.298063 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.297933    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 21:59:38.298063 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.297961    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 21:59:38.298063 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.297994    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 21:59:38.298063 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.298020    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-web-config\") pod \"prometheus-k8s-0\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 21:59:38.298063 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.298058    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 21:59:38.298665 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.298484    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 21:59:38.298977 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.298949    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 21:59:38.299292 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.299259    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 21:59:38.299931 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.299902    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 21:59:38.300463 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.300432    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 21:59:38.305167 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.304383    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 21:59:38.305167 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.304725    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 21:59:38.307118 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.307090    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 21:59:38.307863 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.307836    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 21:59:38.308378 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.308352    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 21:59:38.308994 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.308947    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w47l7\" (UniqueName: \"kubernetes.io/projected/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-kube-api-access-w47l7\") pod \"prometheus-k8s-0\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 21:59:38.309575 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.309550    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-config-out\") pod \"prometheus-k8s-0\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 21:59:38.310365 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.310322    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 21:59:38.310889 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.310844    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 21:59:38.314038 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.313088    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 21:59:38.314038 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.313595    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 21:59:38.314038 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.313976    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 21:59:38.314289 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.314087    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 21:59:38.314558 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.314536    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-web-config\") pod \"prometheus-k8s-0\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 21:59:38.316337 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.316297    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-config\") pod \"prometheus-k8s-0\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 21:59:38.409864 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:38.409645    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 21:59:49.398806 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:49.398780    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 21:59:49.407268 ip-10-0-138-45 kubenswrapper[2563]: W0917 21:59:49.407239    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5b72cd3_2884_4e2d_b3e8_2b19349d5fa9.slice/crio-646fb24c23b83b8c3a35b7f95aeaf6c6c842a70368daa451677461302983e398 WatchSource:0}: Error finding container 646fb24c23b83b8c3a35b7f95aeaf6c6c842a70368daa451677461302983e398: Status 404 returned error can't find the container with id 646fb24c23b83b8c3a35b7f95aeaf6c6c842a70368daa451677461302983e398
Sep 17 21:59:49.477761 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:49.477729    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-58ccf7d5fc-sg9zb" event={"ID":"49ac9268-c95a-4df8-a119-9843598e243d","Type":"ContainerStarted","Data":"450a3ac8284f87abc97e23b4acdeb4e28ad50b2c79d3295acc616245669fd751"}
Sep 17 21:59:49.481006 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:49.480974    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9","Type":"ContainerStarted","Data":"646fb24c23b83b8c3a35b7f95aeaf6c6c842a70368daa451677461302983e398"}
Sep 17 21:59:50.489030 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:50.488219    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-drqjx" event={"ID":"3befbbf9-41ff-4f57-9cfd-aee679772893","Type":"ContainerStarted","Data":"1eff12afe98978e64c38ab70ec5e9762a7bed32d29600c403f0670628528c392"}
Sep 17 21:59:50.489030 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:50.488977    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6594f4c75-drqjx"
Sep 17 21:59:50.491967 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:50.491938    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69d87d4967-6w259" event={"ID":"01ac47d5-3c66-40fa-adc5-294dbf851172","Type":"ContainerStarted","Data":"c17af31b23e66df0280d35a70922c79410d13c214ad21b10648d0f8e9a90467b"}
Sep 17 21:59:50.502055 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:50.502032    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6594f4c75-drqjx"
Sep 17 21:59:50.507211 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:50.507164    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-6594f4c75-drqjx" podStartSLOduration=2.131587367 podStartE2EDuration="21.507123111s" totalImagesPullingTime="19.375535744s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:59:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:50.506503347 +0000 UTC m=+101.674456333" watchObservedRunningTime="2026-09-17 21:59:50.507123111 +0000 UTC m=+101.675076092"
Sep 17 21:59:50.541086 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:50.541033    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-58ccf7d5fc-sg9zb" podStartSLOduration=1.981030412 podStartE2EDuration="14.541013497s" totalImagesPullingTime="12.559983085s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:59:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:50.539260258 +0000 UTC m=+101.707213242" watchObservedRunningTime="2026-09-17 21:59:50.541013497 +0000 UTC m=+101.708966481"
Sep 17 21:59:50.557966 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:50.557556    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-69d87d4967-6w259" podStartSLOduration=2.5016943449999998 podStartE2EDuration="15.557536293s" totalImagesPullingTime="13.055841948s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:59:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:50.555836922 +0000 UTC m=+101.723789920" watchObservedRunningTime="2026-09-17 21:59:50.557536293 +0000 UTC m=+101.725489277"
Sep 17 21:59:51.498560 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:51.498527    2563 generic.go:356] "Generic (PLEG): container finished" podID="a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9" containerID="a50b0dd6f055b2a4ed19bafb063544001afb41844a3e8c6153a3e1b6bdca558d" exitCode=0
Sep 17 21:59:51.498969 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:51.498620    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9","Type":"ContainerDied","Data":"a50b0dd6f055b2a4ed19bafb063544001afb41844a3e8c6153a3e1b6bdca558d"}
Sep 17 21:59:55.357325 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:55.357288    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-bnj94"
Sep 17 21:59:55.521559 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:55.521524    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9","Type":"ContainerStarted","Data":"b2abdfe0f022fc1f214e2b5fa16678460f2ca81f860790e82122f14492962662"}
Sep 17 21:59:55.521771 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:55.521593    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9","Type":"ContainerStarted","Data":"57aa6161cfafa84feca3294b48af0652080ac0b9bfaa9cc337a4acc42f6fea64"}
Sep 17 21:59:55.945050 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:55.944377    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-69d87d4967-6w259"]
Sep 17 21:59:56.016043 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:56.016012    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-69d87d4967-6w259"
Sep 17 21:59:56.451076 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:56.450994    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-58ccf7d5fc-sg9zb"
Sep 17 21:59:56.452112 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:56.451319    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-58ccf7d5fc-sg9zb"
Sep 17 21:59:58.539872 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:58.539830    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9","Type":"ContainerStarted","Data":"832cc61345e4b80de54616a8b371913d97222a3dd75492595dae2a5062ccb4f9"}
Sep 17 21:59:58.540308 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:58.539905    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9","Type":"ContainerStarted","Data":"10e22e4921ab467ee928faa8ee6052af01ea45fc0286c2d0f852d86e24f03d78"}
Sep 17 21:59:58.540308 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:58.539936    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9","Type":"ContainerStarted","Data":"1d27f621f939a1411bc81104721b1c93f25090f17b89ccde466863cf0ea27d46"}
Sep 17 21:59:58.540308 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:58.539964    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9","Type":"ContainerStarted","Data":"58300c3bc1661cadd488ffef6361e57dbbb66d46863480510119187c02ebfd02"}
Sep 17 21:59:58.571908 ip-10-0-138-45 kubenswrapper[2563]: I0917 21:59:58.571805    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=12.680490183 podStartE2EDuration="20.571785265s" totalImagesPullingTime="7.870316026s" totalInitContainerRuntime="20.979056ms" isStatefulPod=false podCreationTimestamp="2026-09-17 21:59:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:58.56959179 +0000 UTC m=+109.737544776" watchObservedRunningTime="2026-09-17 21:59:58.571785265 +0000 UTC m=+109.739738249"
Sep 17 22:00:03.409910 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:03.409869    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:16.456336 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:16.456307    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-58ccf7d5fc-sg9zb"
Sep 17 22:00:16.460055 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:16.460030    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-58ccf7d5fc-sg9zb"
Sep 17 22:00:20.971324 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:20.971258    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-69d87d4967-6w259" podUID="01ac47d5-3c66-40fa-adc5-294dbf851172" containerName="console" containerID="cri-o://c17af31b23e66df0280d35a70922c79410d13c214ad21b10648d0f8e9a90467b" gracePeriod=15
Sep 17 22:00:21.238843 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:21.238781    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69d87d4967-6w259"
Sep 17 22:00:21.364820 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:21.364789    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01ac47d5-3c66-40fa-adc5-294dbf851172-console-serving-cert\") pod \"01ac47d5-3c66-40fa-adc5-294dbf851172\" (UID: \"01ac47d5-3c66-40fa-adc5-294dbf851172\") "
Sep 17 22:00:21.364962 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:21.364846    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01ac47d5-3c66-40fa-adc5-294dbf851172-console-oauth-config\") pod \"01ac47d5-3c66-40fa-adc5-294dbf851172\" (UID: \"01ac47d5-3c66-40fa-adc5-294dbf851172\") "
Sep 17 22:00:21.364962 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:21.364877    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01ac47d5-3c66-40fa-adc5-294dbf851172-oauth-serving-cert\") pod \"01ac47d5-3c66-40fa-adc5-294dbf851172\" (UID: \"01ac47d5-3c66-40fa-adc5-294dbf851172\") "
Sep 17 22:00:21.364962 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:21.364905    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01ac47d5-3c66-40fa-adc5-294dbf851172-service-ca\") pod \"01ac47d5-3c66-40fa-adc5-294dbf851172\" (UID: \"01ac47d5-3c66-40fa-adc5-294dbf851172\") "
Sep 17 22:00:21.364962 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:21.364949    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/01ac47d5-3c66-40fa-adc5-294dbf851172-console-config\") pod \"01ac47d5-3c66-40fa-adc5-294dbf851172\" (UID: \"01ac47d5-3c66-40fa-adc5-294dbf851172\") "
Sep 17 22:00:21.365163 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:21.365030    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jp2sm\" (UniqueName: \"kubernetes.io/projected/01ac47d5-3c66-40fa-adc5-294dbf851172-kube-api-access-jp2sm\") pod \"01ac47d5-3c66-40fa-adc5-294dbf851172\" (UID: \"01ac47d5-3c66-40fa-adc5-294dbf851172\") "
Sep 17 22:00:21.365298 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:21.365268    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01ac47d5-3c66-40fa-adc5-294dbf851172-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "01ac47d5-3c66-40fa-adc5-294dbf851172" (UID: "01ac47d5-3c66-40fa-adc5-294dbf851172"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:00:21.365430 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:21.365328    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01ac47d5-3c66-40fa-adc5-294dbf851172-service-ca" (OuterVolumeSpecName: "service-ca") pod "01ac47d5-3c66-40fa-adc5-294dbf851172" (UID: "01ac47d5-3c66-40fa-adc5-294dbf851172"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:00:21.365430 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:21.365354    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01ac47d5-3c66-40fa-adc5-294dbf851172-oauth-serving-cert\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:00:21.365430 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:21.365394    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01ac47d5-3c66-40fa-adc5-294dbf851172-console-config" (OuterVolumeSpecName: "console-config") pod "01ac47d5-3c66-40fa-adc5-294dbf851172" (UID: "01ac47d5-3c66-40fa-adc5-294dbf851172"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:00:21.367254 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:21.367214    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01ac47d5-3c66-40fa-adc5-294dbf851172-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "01ac47d5-3c66-40fa-adc5-294dbf851172" (UID: "01ac47d5-3c66-40fa-adc5-294dbf851172"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:00:21.367360 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:21.367298    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01ac47d5-3c66-40fa-adc5-294dbf851172-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "01ac47d5-3c66-40fa-adc5-294dbf851172" (UID: "01ac47d5-3c66-40fa-adc5-294dbf851172"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:00:21.367360 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:21.367318    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01ac47d5-3c66-40fa-adc5-294dbf851172-kube-api-access-jp2sm" (OuterVolumeSpecName: "kube-api-access-jp2sm") pod "01ac47d5-3c66-40fa-adc5-294dbf851172" (UID: "01ac47d5-3c66-40fa-adc5-294dbf851172"). InnerVolumeSpecName "kube-api-access-jp2sm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:00:21.465916 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:21.465885    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jp2sm\" (UniqueName: \"kubernetes.io/projected/01ac47d5-3c66-40fa-adc5-294dbf851172-kube-api-access-jp2sm\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:00:21.465916 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:21.465915    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01ac47d5-3c66-40fa-adc5-294dbf851172-console-serving-cert\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:00:21.465916 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:21.465924    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01ac47d5-3c66-40fa-adc5-294dbf851172-console-oauth-config\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:00:21.466124 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:21.465932    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01ac47d5-3c66-40fa-adc5-294dbf851172-service-ca\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:00:21.466124 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:21.465941    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/01ac47d5-3c66-40fa-adc5-294dbf851172-console-config\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:00:21.628553 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:21.628468    2563 generic.go:356] "Generic (PLEG): container finished" podID="01ac47d5-3c66-40fa-adc5-294dbf851172" containerID="c17af31b23e66df0280d35a70922c79410d13c214ad21b10648d0f8e9a90467b" exitCode=0
Sep 17 22:00:21.628553 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:21.628523    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69d87d4967-6w259"
Sep 17 22:00:21.628723 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:21.628565    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69d87d4967-6w259" event={"ID":"01ac47d5-3c66-40fa-adc5-294dbf851172","Type":"ContainerDied","Data":"c17af31b23e66df0280d35a70922c79410d13c214ad21b10648d0f8e9a90467b"}
Sep 17 22:00:21.628723 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:21.628621    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69d87d4967-6w259" event={"ID":"01ac47d5-3c66-40fa-adc5-294dbf851172","Type":"ContainerDied","Data":"3b237ec881c2612fd3eceedfdf013fe91944bd2e40816c8877ccb327a871239f"}
Sep 17 22:00:21.628723 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:21.628644    2563 scope.go:117] "RemoveContainer" containerID="c17af31b23e66df0280d35a70922c79410d13c214ad21b10648d0f8e9a90467b"
Sep 17 22:00:21.637863 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:21.637842    2563 scope.go:117] "RemoveContainer" containerID="c17af31b23e66df0280d35a70922c79410d13c214ad21b10648d0f8e9a90467b"
Sep 17 22:00:21.638191 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:00:21.638165    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c17af31b23e66df0280d35a70922c79410d13c214ad21b10648d0f8e9a90467b\": container with ID starting with c17af31b23e66df0280d35a70922c79410d13c214ad21b10648d0f8e9a90467b not found: ID does not exist" containerID="c17af31b23e66df0280d35a70922c79410d13c214ad21b10648d0f8e9a90467b"
Sep 17 22:00:21.638258 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:21.638200    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c17af31b23e66df0280d35a70922c79410d13c214ad21b10648d0f8e9a90467b"} err="failed to get container status \"c17af31b23e66df0280d35a70922c79410d13c214ad21b10648d0f8e9a90467b\": rpc error: code = NotFound desc = could not find container \"c17af31b23e66df0280d35a70922c79410d13c214ad21b10648d0f8e9a90467b\": container with ID starting with c17af31b23e66df0280d35a70922c79410d13c214ad21b10648d0f8e9a90467b not found: ID does not exist"
Sep 17 22:00:21.651007 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:21.650971    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-69d87d4967-6w259"]
Sep 17 22:00:21.652925 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:21.652905    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-69d87d4967-6w259"]
Sep 17 22:00:22.914774 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:22.914738    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01ac47d5-3c66-40fa-adc5-294dbf851172" path="/var/lib/kubelet/pods/01ac47d5-3c66-40fa-adc5-294dbf851172/volumes"
Sep 17 22:00:38.410783 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:38.410748    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:38.487318 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:38.487291    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:38.707559 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:38.707481    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:56.370920 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.370833    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 22:00:56.371430 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.371236    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9" containerName="prometheus" containerID="cri-o://57aa6161cfafa84feca3294b48af0652080ac0b9bfaa9cc337a4acc42f6fea64" gracePeriod=600
Sep 17 22:00:56.371430 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.371300    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9" containerName="kube-rbac-proxy-thanos" containerID="cri-o://832cc61345e4b80de54616a8b371913d97222a3dd75492595dae2a5062ccb4f9" gracePeriod=600
Sep 17 22:00:56.371430 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.371419    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9" containerName="kube-rbac-proxy" containerID="cri-o://10e22e4921ab467ee928faa8ee6052af01ea45fc0286c2d0f852d86e24f03d78" gracePeriod=600
Sep 17 22:00:56.371584 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.371506    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9" containerName="kube-rbac-proxy-web" containerID="cri-o://1d27f621f939a1411bc81104721b1c93f25090f17b89ccde466863cf0ea27d46" gracePeriod=600
Sep 17 22:00:56.371637 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.371611    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9" containerName="thanos-sidecar" containerID="cri-o://58300c3bc1661cadd488ffef6361e57dbbb66d46863480510119187c02ebfd02" gracePeriod=600
Sep 17 22:00:56.371686 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.371628    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9" containerName="config-reloader" containerID="cri-o://b2abdfe0f022fc1f214e2b5fa16678460f2ca81f860790e82122f14492962662" gracePeriod=600
Sep 17 22:00:56.628061 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.626937    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:56.721529 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.721498    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-secret-metrics-client-certs\") pod \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") "
Sep 17 22:00:56.721680 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.721552    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-prometheus-k8s-db\") pod \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") "
Sep 17 22:00:56.721680 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.721583    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-configmap-kubelet-serving-ca-bundle\") pod \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") "
Sep 17 22:00:56.721680 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.721609    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-thanos-prometheus-http-client-file\") pod \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") "
Sep 17 22:00:56.721680 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.721644    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-configmap-metrics-client-ca\") pod \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") "
Sep 17 22:00:56.721680 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.721669    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-prometheus-k8s-rulefiles-0\") pod \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") "
Sep 17 22:00:56.721979 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.721702    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-prometheus-k8s-rulefiles-1\") pod \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") "
Sep 17 22:00:56.721979 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.721739    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") "
Sep 17 22:00:56.721979 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.721768    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-config\") pod \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") "
Sep 17 22:00:56.721979 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.721796    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-prometheus-k8s-rulefiles-2\") pod \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") "
Sep 17 22:00:56.721979 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.721825    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-tls-assets\") pod \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") "
Sep 17 22:00:56.721979 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.721860    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-configmap-serving-certs-ca-bundle\") pod \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") "
Sep 17 22:00:56.721979 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.721895    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-secret-grpc-tls\") pod \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") "
Sep 17 22:00:56.721979 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.721920    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w47l7\" (UniqueName: \"kubernetes.io/projected/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-kube-api-access-w47l7\") pod \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") "
Sep 17 22:00:56.721979 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.721913    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9" (UID: "a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:00:56.721979 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.721945    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-secret-prometheus-k8s-tls\") pod \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") "
Sep 17 22:00:56.721979 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.721976    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-prometheus-trusted-ca-bundle\") pod \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") "
Sep 17 22:00:56.722546 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.722009    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") "
Sep 17 22:00:56.722546 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.722033    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-config-out\") pod \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") "
Sep 17 22:00:56.722546 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.722090    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-web-config\") pod \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") "
Sep 17 22:00:56.722546 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.722095    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9" (UID: "a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:00:56.722546 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.722128    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-secret-kube-rbac-proxy\") pod \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\" (UID: \"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9\") "
Sep 17 22:00:56.722546 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.722193    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9" (UID: "a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:00:56.722546 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.722393    2563 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:00:56.722546 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.722412    2563 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-configmap-metrics-client-ca\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:00:56.722546 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.722427    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:00:56.722965 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.722617    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9" (UID: "a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:00:56.722965 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.722791    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9" (UID: "a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:00:56.723104 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.723081    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9" (UID: "a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:00:56.724107 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.723427    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9" (UID: "a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:00:56.724107 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.723854    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9" (UID: "a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:00:56.724330 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.724266    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9" (UID: "a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:00:56.724392 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.724357    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9" (UID: "a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:00:56.724681 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.724650    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9" (UID: "a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:00:56.724819 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.724781    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9" (UID: "a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:00:56.725508 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.725481    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-config" (OuterVolumeSpecName: "config") pod "a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9" (UID: "a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:00:56.726038 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.726010    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9" (UID: "a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:00:56.726583 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.726557    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9" (UID: "a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:00:56.726677 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.726602    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9" (UID: "a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:00:56.726677 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.726623    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-kube-api-access-w47l7" (OuterVolumeSpecName: "kube-api-access-w47l7") pod "a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9" (UID: "a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9"). InnerVolumeSpecName "kube-api-access-w47l7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:00:56.726875 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.726851    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9" (UID: "a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:00:56.727055 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.727037    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-config-out" (OuterVolumeSpecName: "config-out") pod "a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9" (UID: "a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:00:56.739018 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.738997    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-web-config" (OuterVolumeSpecName: "web-config") pod "a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9" (UID: "a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:00:56.761424 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.761399    2563 generic.go:356] "Generic (PLEG): container finished" podID="a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9" containerID="832cc61345e4b80de54616a8b371913d97222a3dd75492595dae2a5062ccb4f9" exitCode=0
Sep 17 22:00:56.761424 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.761417    2563 generic.go:356] "Generic (PLEG): container finished" podID="a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9" containerID="10e22e4921ab467ee928faa8ee6052af01ea45fc0286c2d0f852d86e24f03d78" exitCode=0
Sep 17 22:00:56.761424 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.761425    2563 generic.go:356] "Generic (PLEG): container finished" podID="a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9" containerID="1d27f621f939a1411bc81104721b1c93f25090f17b89ccde466863cf0ea27d46" exitCode=0
Sep 17 22:00:56.761624 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.761431    2563 generic.go:356] "Generic (PLEG): container finished" podID="a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9" containerID="58300c3bc1661cadd488ffef6361e57dbbb66d46863480510119187c02ebfd02" exitCode=0
Sep 17 22:00:56.761624 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.761439    2563 generic.go:356] "Generic (PLEG): container finished" podID="a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9" containerID="b2abdfe0f022fc1f214e2b5fa16678460f2ca81f860790e82122f14492962662" exitCode=0
Sep 17 22:00:56.761624 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.761444    2563 generic.go:356] "Generic (PLEG): container finished" podID="a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9" containerID="57aa6161cfafa84feca3294b48af0652080ac0b9bfaa9cc337a4acc42f6fea64" exitCode=0
Sep 17 22:00:56.761624 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.761475    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9","Type":"ContainerDied","Data":"832cc61345e4b80de54616a8b371913d97222a3dd75492595dae2a5062ccb4f9"}
Sep 17 22:00:56.761624 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.761494    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:56.761624 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.761538    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9","Type":"ContainerDied","Data":"10e22e4921ab467ee928faa8ee6052af01ea45fc0286c2d0f852d86e24f03d78"}
Sep 17 22:00:56.761624 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.761566    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9","Type":"ContainerDied","Data":"1d27f621f939a1411bc81104721b1c93f25090f17b89ccde466863cf0ea27d46"}
Sep 17 22:00:56.761624 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.761592    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9","Type":"ContainerDied","Data":"58300c3bc1661cadd488ffef6361e57dbbb66d46863480510119187c02ebfd02"}
Sep 17 22:00:56.761624 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.761604    2563 scope.go:117] "RemoveContainer" containerID="832cc61345e4b80de54616a8b371913d97222a3dd75492595dae2a5062ccb4f9"
Sep 17 22:00:56.761624 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.761623    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9","Type":"ContainerDied","Data":"b2abdfe0f022fc1f214e2b5fa16678460f2ca81f860790e82122f14492962662"}
Sep 17 22:00:56.762095 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.761648    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9","Type":"ContainerDied","Data":"57aa6161cfafa84feca3294b48af0652080ac0b9bfaa9cc337a4acc42f6fea64"}
Sep 17 22:00:56.762095 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.761678    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9","Type":"ContainerDied","Data":"646fb24c23b83b8c3a35b7f95aeaf6c6c842a70368daa451677461302983e398"}
Sep 17 22:00:56.769004 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.768918    2563 scope.go:117] "RemoveContainer" containerID="10e22e4921ab467ee928faa8ee6052af01ea45fc0286c2d0f852d86e24f03d78"
Sep 17 22:00:56.775500 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.775481    2563 scope.go:117] "RemoveContainer" containerID="1d27f621f939a1411bc81104721b1c93f25090f17b89ccde466863cf0ea27d46"
Sep 17 22:00:56.782647 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.782575    2563 scope.go:117] "RemoveContainer" containerID="58300c3bc1661cadd488ffef6361e57dbbb66d46863480510119187c02ebfd02"
Sep 17 22:00:56.784685 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.784663    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 22:00:56.790083 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.790069    2563 scope.go:117] "RemoveContainer" containerID="b2abdfe0f022fc1f214e2b5fa16678460f2ca81f860790e82122f14492962662"
Sep 17 22:00:56.790133 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.790114    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 22:00:56.796161 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.796130    2563 scope.go:117] "RemoveContainer" containerID="57aa6161cfafa84feca3294b48af0652080ac0b9bfaa9cc337a4acc42f6fea64"
Sep 17 22:00:56.803091 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.803076    2563 scope.go:117] "RemoveContainer" containerID="a50b0dd6f055b2a4ed19bafb063544001afb41844a3e8c6153a3e1b6bdca558d"
Sep 17 22:00:56.808976 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.808960    2563 scope.go:117] "RemoveContainer" containerID="832cc61345e4b80de54616a8b371913d97222a3dd75492595dae2a5062ccb4f9"
Sep 17 22:00:56.809254 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:00:56.809236    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"832cc61345e4b80de54616a8b371913d97222a3dd75492595dae2a5062ccb4f9\": container with ID starting with 832cc61345e4b80de54616a8b371913d97222a3dd75492595dae2a5062ccb4f9 not found: ID does not exist" containerID="832cc61345e4b80de54616a8b371913d97222a3dd75492595dae2a5062ccb4f9"
Sep 17 22:00:56.809319 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.809264    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"832cc61345e4b80de54616a8b371913d97222a3dd75492595dae2a5062ccb4f9"} err="failed to get container status \"832cc61345e4b80de54616a8b371913d97222a3dd75492595dae2a5062ccb4f9\": rpc error: code = NotFound desc = could not find container \"832cc61345e4b80de54616a8b371913d97222a3dd75492595dae2a5062ccb4f9\": container with ID starting with 832cc61345e4b80de54616a8b371913d97222a3dd75492595dae2a5062ccb4f9 not found: ID does not exist"
Sep 17 22:00:56.809319 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.809280    2563 scope.go:117] "RemoveContainer" containerID="10e22e4921ab467ee928faa8ee6052af01ea45fc0286c2d0f852d86e24f03d78"
Sep 17 22:00:56.809539 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:00:56.809522    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10e22e4921ab467ee928faa8ee6052af01ea45fc0286c2d0f852d86e24f03d78\": container with ID starting with 10e22e4921ab467ee928faa8ee6052af01ea45fc0286c2d0f852d86e24f03d78 not found: ID does not exist" containerID="10e22e4921ab467ee928faa8ee6052af01ea45fc0286c2d0f852d86e24f03d78"
Sep 17 22:00:56.809578 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.809545    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10e22e4921ab467ee928faa8ee6052af01ea45fc0286c2d0f852d86e24f03d78"} err="failed to get container status \"10e22e4921ab467ee928faa8ee6052af01ea45fc0286c2d0f852d86e24f03d78\": rpc error: code = NotFound desc = could not find container \"10e22e4921ab467ee928faa8ee6052af01ea45fc0286c2d0f852d86e24f03d78\": container with ID starting with 10e22e4921ab467ee928faa8ee6052af01ea45fc0286c2d0f852d86e24f03d78 not found: ID does not exist"
Sep 17 22:00:56.809578 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.809561    2563 scope.go:117] "RemoveContainer" containerID="1d27f621f939a1411bc81104721b1c93f25090f17b89ccde466863cf0ea27d46"
Sep 17 22:00:56.809800 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:00:56.809782    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d27f621f939a1411bc81104721b1c93f25090f17b89ccde466863cf0ea27d46\": container with ID starting with 1d27f621f939a1411bc81104721b1c93f25090f17b89ccde466863cf0ea27d46 not found: ID does not exist" containerID="1d27f621f939a1411bc81104721b1c93f25090f17b89ccde466863cf0ea27d46"
Sep 17 22:00:56.809856 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.809802    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d27f621f939a1411bc81104721b1c93f25090f17b89ccde466863cf0ea27d46"} err="failed to get container status \"1d27f621f939a1411bc81104721b1c93f25090f17b89ccde466863cf0ea27d46\": rpc error: code = NotFound desc = could not find container \"1d27f621f939a1411bc81104721b1c93f25090f17b89ccde466863cf0ea27d46\": container with ID starting with 1d27f621f939a1411bc81104721b1c93f25090f17b89ccde466863cf0ea27d46 not found: ID does not exist"
Sep 17 22:00:56.809856 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.809815    2563 scope.go:117] "RemoveContainer" containerID="58300c3bc1661cadd488ffef6361e57dbbb66d46863480510119187c02ebfd02"
Sep 17 22:00:56.810047 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:00:56.810030    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58300c3bc1661cadd488ffef6361e57dbbb66d46863480510119187c02ebfd02\": container with ID starting with 58300c3bc1661cadd488ffef6361e57dbbb66d46863480510119187c02ebfd02 not found: ID does not exist" containerID="58300c3bc1661cadd488ffef6361e57dbbb66d46863480510119187c02ebfd02"
Sep 17 22:00:56.810089 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.810052    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58300c3bc1661cadd488ffef6361e57dbbb66d46863480510119187c02ebfd02"} err="failed to get container status \"58300c3bc1661cadd488ffef6361e57dbbb66d46863480510119187c02ebfd02\": rpc error: code = NotFound desc = could not find container \"58300c3bc1661cadd488ffef6361e57dbbb66d46863480510119187c02ebfd02\": container with ID starting with 58300c3bc1661cadd488ffef6361e57dbbb66d46863480510119187c02ebfd02 not found: ID does not exist"
Sep 17 22:00:56.810089 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.810066    2563 scope.go:117] "RemoveContainer" containerID="b2abdfe0f022fc1f214e2b5fa16678460f2ca81f860790e82122f14492962662"
Sep 17 22:00:56.810324 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:00:56.810309    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2abdfe0f022fc1f214e2b5fa16678460f2ca81f860790e82122f14492962662\": container with ID starting with b2abdfe0f022fc1f214e2b5fa16678460f2ca81f860790e82122f14492962662 not found: ID does not exist" containerID="b2abdfe0f022fc1f214e2b5fa16678460f2ca81f860790e82122f14492962662"
Sep 17 22:00:56.810368 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.810327    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2abdfe0f022fc1f214e2b5fa16678460f2ca81f860790e82122f14492962662"} err="failed to get container status \"b2abdfe0f022fc1f214e2b5fa16678460f2ca81f860790e82122f14492962662\": rpc error: code = NotFound desc = could not find container \"b2abdfe0f022fc1f214e2b5fa16678460f2ca81f860790e82122f14492962662\": container with ID starting with b2abdfe0f022fc1f214e2b5fa16678460f2ca81f860790e82122f14492962662 not found: ID does not exist"
Sep 17 22:00:56.810368 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.810340    2563 scope.go:117] "RemoveContainer" containerID="57aa6161cfafa84feca3294b48af0652080ac0b9bfaa9cc337a4acc42f6fea64"
Sep 17 22:00:56.810593 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:00:56.810565    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57aa6161cfafa84feca3294b48af0652080ac0b9bfaa9cc337a4acc42f6fea64\": container with ID starting with 57aa6161cfafa84feca3294b48af0652080ac0b9bfaa9cc337a4acc42f6fea64 not found: ID does not exist" containerID="57aa6161cfafa84feca3294b48af0652080ac0b9bfaa9cc337a4acc42f6fea64"
Sep 17 22:00:56.810684 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.810590    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57aa6161cfafa84feca3294b48af0652080ac0b9bfaa9cc337a4acc42f6fea64"} err="failed to get container status \"57aa6161cfafa84feca3294b48af0652080ac0b9bfaa9cc337a4acc42f6fea64\": rpc error: code = NotFound desc = could not find container \"57aa6161cfafa84feca3294b48af0652080ac0b9bfaa9cc337a4acc42f6fea64\": container with ID starting with 57aa6161cfafa84feca3294b48af0652080ac0b9bfaa9cc337a4acc42f6fea64 not found: ID does not exist"
Sep 17 22:00:56.810684 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.810605    2563 scope.go:117] "RemoveContainer" containerID="a50b0dd6f055b2a4ed19bafb063544001afb41844a3e8c6153a3e1b6bdca558d"
Sep 17 22:00:56.810844 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:00:56.810827    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a50b0dd6f055b2a4ed19bafb063544001afb41844a3e8c6153a3e1b6bdca558d\": container with ID starting with a50b0dd6f055b2a4ed19bafb063544001afb41844a3e8c6153a3e1b6bdca558d not found: ID does not exist" containerID="a50b0dd6f055b2a4ed19bafb063544001afb41844a3e8c6153a3e1b6bdca558d"
Sep 17 22:00:56.810889 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.810848    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a50b0dd6f055b2a4ed19bafb063544001afb41844a3e8c6153a3e1b6bdca558d"} err="failed to get container status \"a50b0dd6f055b2a4ed19bafb063544001afb41844a3e8c6153a3e1b6bdca558d\": rpc error: code = NotFound desc = could not find container \"a50b0dd6f055b2a4ed19bafb063544001afb41844a3e8c6153a3e1b6bdca558d\": container with ID starting with a50b0dd6f055b2a4ed19bafb063544001afb41844a3e8c6153a3e1b6bdca558d not found: ID does not exist"
Sep 17 22:00:56.810889 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.810859    2563 scope.go:117] "RemoveContainer" containerID="832cc61345e4b80de54616a8b371913d97222a3dd75492595dae2a5062ccb4f9"
Sep 17 22:00:56.811064 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.811047    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"832cc61345e4b80de54616a8b371913d97222a3dd75492595dae2a5062ccb4f9"} err="failed to get container status \"832cc61345e4b80de54616a8b371913d97222a3dd75492595dae2a5062ccb4f9\": rpc error: code = NotFound desc = could not find container \"832cc61345e4b80de54616a8b371913d97222a3dd75492595dae2a5062ccb4f9\": container with ID starting with 832cc61345e4b80de54616a8b371913d97222a3dd75492595dae2a5062ccb4f9 not found: ID does not exist"
Sep 17 22:00:56.811105 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.811064    2563 scope.go:117] "RemoveContainer" containerID="10e22e4921ab467ee928faa8ee6052af01ea45fc0286c2d0f852d86e24f03d78"
Sep 17 22:00:56.811321 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.811297    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10e22e4921ab467ee928faa8ee6052af01ea45fc0286c2d0f852d86e24f03d78"} err="failed to get container status \"10e22e4921ab467ee928faa8ee6052af01ea45fc0286c2d0f852d86e24f03d78\": rpc error: code = NotFound desc = could not find container \"10e22e4921ab467ee928faa8ee6052af01ea45fc0286c2d0f852d86e24f03d78\": container with ID starting with 10e22e4921ab467ee928faa8ee6052af01ea45fc0286c2d0f852d86e24f03d78 not found: ID does not exist"
Sep 17 22:00:56.811321 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.811311    2563 scope.go:117] "RemoveContainer" containerID="1d27f621f939a1411bc81104721b1c93f25090f17b89ccde466863cf0ea27d46"
Sep 17 22:00:56.811553 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.811535    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d27f621f939a1411bc81104721b1c93f25090f17b89ccde466863cf0ea27d46"} err="failed to get container status \"1d27f621f939a1411bc81104721b1c93f25090f17b89ccde466863cf0ea27d46\": rpc error: code = NotFound desc = could not find container \"1d27f621f939a1411bc81104721b1c93f25090f17b89ccde466863cf0ea27d46\": container with ID starting with 1d27f621f939a1411bc81104721b1c93f25090f17b89ccde466863cf0ea27d46 not found: ID does not exist"
Sep 17 22:00:56.811598 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.811555    2563 scope.go:117] "RemoveContainer" containerID="58300c3bc1661cadd488ffef6361e57dbbb66d46863480510119187c02ebfd02"
Sep 17 22:00:56.811800 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.811778    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58300c3bc1661cadd488ffef6361e57dbbb66d46863480510119187c02ebfd02"} err="failed to get container status \"58300c3bc1661cadd488ffef6361e57dbbb66d46863480510119187c02ebfd02\": rpc error: code = NotFound desc = could not find container \"58300c3bc1661cadd488ffef6361e57dbbb66d46863480510119187c02ebfd02\": container with ID starting with 58300c3bc1661cadd488ffef6361e57dbbb66d46863480510119187c02ebfd02 not found: ID does not exist"
Sep 17 22:00:56.811867 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.811803    2563 scope.go:117] "RemoveContainer" containerID="b2abdfe0f022fc1f214e2b5fa16678460f2ca81f860790e82122f14492962662"
Sep 17 22:00:56.812174 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.812152    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2abdfe0f022fc1f214e2b5fa16678460f2ca81f860790e82122f14492962662"} err="failed to get container status \"b2abdfe0f022fc1f214e2b5fa16678460f2ca81f860790e82122f14492962662\": rpc error: code = NotFound desc = could not find container \"b2abdfe0f022fc1f214e2b5fa16678460f2ca81f860790e82122f14492962662\": container with ID starting with b2abdfe0f022fc1f214e2b5fa16678460f2ca81f860790e82122f14492962662 not found: ID does not exist"
Sep 17 22:00:56.812174 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.812173    2563 scope.go:117] "RemoveContainer" containerID="57aa6161cfafa84feca3294b48af0652080ac0b9bfaa9cc337a4acc42f6fea64"
Sep 17 22:00:56.812446 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.812426    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57aa6161cfafa84feca3294b48af0652080ac0b9bfaa9cc337a4acc42f6fea64"} err="failed to get container status \"57aa6161cfafa84feca3294b48af0652080ac0b9bfaa9cc337a4acc42f6fea64\": rpc error: code = NotFound desc = could not find container \"57aa6161cfafa84feca3294b48af0652080ac0b9bfaa9cc337a4acc42f6fea64\": container with ID starting with 57aa6161cfafa84feca3294b48af0652080ac0b9bfaa9cc337a4acc42f6fea64 not found: ID does not exist"
Sep 17 22:00:56.812446 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.812447    2563 scope.go:117] "RemoveContainer" containerID="a50b0dd6f055b2a4ed19bafb063544001afb41844a3e8c6153a3e1b6bdca558d"
Sep 17 22:00:56.812756 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.812739    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a50b0dd6f055b2a4ed19bafb063544001afb41844a3e8c6153a3e1b6bdca558d"} err="failed to get container status \"a50b0dd6f055b2a4ed19bafb063544001afb41844a3e8c6153a3e1b6bdca558d\": rpc error: code = NotFound desc = could not find container \"a50b0dd6f055b2a4ed19bafb063544001afb41844a3e8c6153a3e1b6bdca558d\": container with ID starting with a50b0dd6f055b2a4ed19bafb063544001afb41844a3e8c6153a3e1b6bdca558d not found: ID does not exist"
Sep 17 22:00:56.812818 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.812756    2563 scope.go:117] "RemoveContainer" containerID="832cc61345e4b80de54616a8b371913d97222a3dd75492595dae2a5062ccb4f9"
Sep 17 22:00:56.813043 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.813023    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"832cc61345e4b80de54616a8b371913d97222a3dd75492595dae2a5062ccb4f9"} err="failed to get container status \"832cc61345e4b80de54616a8b371913d97222a3dd75492595dae2a5062ccb4f9\": rpc error: code = NotFound desc = could not find container \"832cc61345e4b80de54616a8b371913d97222a3dd75492595dae2a5062ccb4f9\": container with ID starting with 832cc61345e4b80de54616a8b371913d97222a3dd75492595dae2a5062ccb4f9 not found: ID does not exist"
Sep 17 22:00:56.813043 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.813042    2563 scope.go:117] "RemoveContainer" containerID="10e22e4921ab467ee928faa8ee6052af01ea45fc0286c2d0f852d86e24f03d78"
Sep 17 22:00:56.813436 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.813417    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10e22e4921ab467ee928faa8ee6052af01ea45fc0286c2d0f852d86e24f03d78"} err="failed to get container status \"10e22e4921ab467ee928faa8ee6052af01ea45fc0286c2d0f852d86e24f03d78\": rpc error: code = NotFound desc = could not find container \"10e22e4921ab467ee928faa8ee6052af01ea45fc0286c2d0f852d86e24f03d78\": container with ID starting with 10e22e4921ab467ee928faa8ee6052af01ea45fc0286c2d0f852d86e24f03d78 not found: ID does not exist"
Sep 17 22:00:56.813502 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.813437    2563 scope.go:117] "RemoveContainer" containerID="1d27f621f939a1411bc81104721b1c93f25090f17b89ccde466863cf0ea27d46"
Sep 17 22:00:56.813680 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.813665    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d27f621f939a1411bc81104721b1c93f25090f17b89ccde466863cf0ea27d46"} err="failed to get container status \"1d27f621f939a1411bc81104721b1c93f25090f17b89ccde466863cf0ea27d46\": rpc error: code = NotFound desc = could not find container \"1d27f621f939a1411bc81104721b1c93f25090f17b89ccde466863cf0ea27d46\": container with ID starting with 1d27f621f939a1411bc81104721b1c93f25090f17b89ccde466863cf0ea27d46 not found: ID does not exist"
Sep 17 22:00:56.813730 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.813680    2563 scope.go:117] "RemoveContainer" containerID="58300c3bc1661cadd488ffef6361e57dbbb66d46863480510119187c02ebfd02"
Sep 17 22:00:56.813730 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.813707    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 22:00:56.813919 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.813905    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58300c3bc1661cadd488ffef6361e57dbbb66d46863480510119187c02ebfd02"} err="failed to get container status \"58300c3bc1661cadd488ffef6361e57dbbb66d46863480510119187c02ebfd02\": rpc error: code = NotFound desc = could not find container \"58300c3bc1661cadd488ffef6361e57dbbb66d46863480510119187c02ebfd02\": container with ID starting with 58300c3bc1661cadd488ffef6361e57dbbb66d46863480510119187c02ebfd02 not found: ID does not exist"
Sep 17 22:00:56.813964 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.813919    2563 scope.go:117] "RemoveContainer" containerID="b2abdfe0f022fc1f214e2b5fa16678460f2ca81f860790e82122f14492962662"
Sep 17 22:00:56.814004 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.813989    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9" containerName="config-reloader"
Sep 17 22:00:56.814043 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.814008    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9" containerName="config-reloader"
Sep 17 22:00:56.814043 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.814026    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9" containerName="init-config-reloader"
Sep 17 22:00:56.814043 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.814035    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9" containerName="init-config-reloader"
Sep 17 22:00:56.814153 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.814048    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9" containerName="kube-rbac-proxy-thanos"
Sep 17 22:00:56.814153 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.814056    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9" containerName="kube-rbac-proxy-thanos"
Sep 17 22:00:56.814153 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.814064    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9" containerName="kube-rbac-proxy"
Sep 17 22:00:56.814153 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.814069    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9" containerName="kube-rbac-proxy"
Sep 17 22:00:56.814153 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.814079    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01ac47d5-3c66-40fa-adc5-294dbf851172" containerName="console"
Sep 17 22:00:56.814153 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.814084    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="01ac47d5-3c66-40fa-adc5-294dbf851172" containerName="console"
Sep 17 22:00:56.814153 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.814092    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9" containerName="prometheus"
Sep 17 22:00:56.814153 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.814099    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9" containerName="prometheus"
Sep 17 22:00:56.814153 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.814107    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9" containerName="kube-rbac-proxy-web"
Sep 17 22:00:56.814153 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.814115    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9" containerName="kube-rbac-proxy-web"
Sep 17 22:00:56.814153 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.814124    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9" containerName="thanos-sidecar"
Sep 17 22:00:56.814153 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.814131    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9" containerName="thanos-sidecar"
Sep 17 22:00:56.814153 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.814095    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2abdfe0f022fc1f214e2b5fa16678460f2ca81f860790e82122f14492962662"} err="failed to get container status \"b2abdfe0f022fc1f214e2b5fa16678460f2ca81f860790e82122f14492962662\": rpc error: code = NotFound desc = could not find container \"b2abdfe0f022fc1f214e2b5fa16678460f2ca81f860790e82122f14492962662\": container with ID starting with b2abdfe0f022fc1f214e2b5fa16678460f2ca81f860790e82122f14492962662 not found: ID does not exist"
Sep 17 22:00:56.814556 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.814164    2563 scope.go:117] "RemoveContainer" containerID="57aa6161cfafa84feca3294b48af0652080ac0b9bfaa9cc337a4acc42f6fea64"
Sep 17 22:00:56.814556 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.814200    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9" containerName="thanos-sidecar"
Sep 17 22:00:56.814556 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.814209    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="01ac47d5-3c66-40fa-adc5-294dbf851172" containerName="console"
Sep 17 22:00:56.814556 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.814216    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9" containerName="kube-rbac-proxy-web"
Sep 17 22:00:56.814556 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.814223    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9" containerName="prometheus"
Sep 17 22:00:56.814556 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.814228    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9" containerName="config-reloader"
Sep 17 22:00:56.814556 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.814236    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9" containerName="kube-rbac-proxy"
Sep 17 22:00:56.814556 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.814243    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9" containerName="kube-rbac-proxy-thanos"
Sep 17 22:00:56.814556 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.814351    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57aa6161cfafa84feca3294b48af0652080ac0b9bfaa9cc337a4acc42f6fea64"} err="failed to get container status \"57aa6161cfafa84feca3294b48af0652080ac0b9bfaa9cc337a4acc42f6fea64\": rpc error: code = NotFound desc = could not find container \"57aa6161cfafa84feca3294b48af0652080ac0b9bfaa9cc337a4acc42f6fea64\": container with ID starting with 57aa6161cfafa84feca3294b48af0652080ac0b9bfaa9cc337a4acc42f6fea64 not found: ID does not exist"
Sep 17 22:00:56.814556 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.814365    2563 scope.go:117] "RemoveContainer" containerID="a50b0dd6f055b2a4ed19bafb063544001afb41844a3e8c6153a3e1b6bdca558d"
Sep 17 22:00:56.814863 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.814568    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a50b0dd6f055b2a4ed19bafb063544001afb41844a3e8c6153a3e1b6bdca558d"} err="failed to get container status \"a50b0dd6f055b2a4ed19bafb063544001afb41844a3e8c6153a3e1b6bdca558d\": rpc error: code = NotFound desc = could not find container \"a50b0dd6f055b2a4ed19bafb063544001afb41844a3e8c6153a3e1b6bdca558d\": container with ID starting with a50b0dd6f055b2a4ed19bafb063544001afb41844a3e8c6153a3e1b6bdca558d not found: ID does not exist"
Sep 17 22:00:56.814863 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.814582    2563 scope.go:117] "RemoveContainer" containerID="832cc61345e4b80de54616a8b371913d97222a3dd75492595dae2a5062ccb4f9"
Sep 17 22:00:56.814863 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.814756    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"832cc61345e4b80de54616a8b371913d97222a3dd75492595dae2a5062ccb4f9"} err="failed to get container status \"832cc61345e4b80de54616a8b371913d97222a3dd75492595dae2a5062ccb4f9\": rpc error: code = NotFound desc = could not find container \"832cc61345e4b80de54616a8b371913d97222a3dd75492595dae2a5062ccb4f9\": container with ID starting with 832cc61345e4b80de54616a8b371913d97222a3dd75492595dae2a5062ccb4f9 not found: ID does not exist"
Sep 17 22:00:56.814863 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.814769    2563 scope.go:117] "RemoveContainer" containerID="10e22e4921ab467ee928faa8ee6052af01ea45fc0286c2d0f852d86e24f03d78"
Sep 17 22:00:56.815006 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.814986    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10e22e4921ab467ee928faa8ee6052af01ea45fc0286c2d0f852d86e24f03d78"} err="failed to get container status \"10e22e4921ab467ee928faa8ee6052af01ea45fc0286c2d0f852d86e24f03d78\": rpc error: code = NotFound desc = could not find container \"10e22e4921ab467ee928faa8ee6052af01ea45fc0286c2d0f852d86e24f03d78\": container with ID starting with 10e22e4921ab467ee928faa8ee6052af01ea45fc0286c2d0f852d86e24f03d78 not found: ID does not exist"
Sep 17 22:00:56.815044 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.815007    2563 scope.go:117] "RemoveContainer" containerID="1d27f621f939a1411bc81104721b1c93f25090f17b89ccde466863cf0ea27d46"
Sep 17 22:00:56.815244 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.815227    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d27f621f939a1411bc81104721b1c93f25090f17b89ccde466863cf0ea27d46"} err="failed to get container status \"1d27f621f939a1411bc81104721b1c93f25090f17b89ccde466863cf0ea27d46\": rpc error: code = NotFound desc = could not find container \"1d27f621f939a1411bc81104721b1c93f25090f17b89ccde466863cf0ea27d46\": container with ID starting with 1d27f621f939a1411bc81104721b1c93f25090f17b89ccde466863cf0ea27d46 not found: ID does not exist"
Sep 17 22:00:56.815286 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.815245    2563 scope.go:117] "RemoveContainer" containerID="58300c3bc1661cadd488ffef6361e57dbbb66d46863480510119187c02ebfd02"
Sep 17 22:00:56.815470 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.815452    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58300c3bc1661cadd488ffef6361e57dbbb66d46863480510119187c02ebfd02"} err="failed to get container status \"58300c3bc1661cadd488ffef6361e57dbbb66d46863480510119187c02ebfd02\": rpc error: code = NotFound desc = could not find container \"58300c3bc1661cadd488ffef6361e57dbbb66d46863480510119187c02ebfd02\": container with ID starting with 58300c3bc1661cadd488ffef6361e57dbbb66d46863480510119187c02ebfd02 not found: ID does not exist"
Sep 17 22:00:56.815507 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.815471    2563 scope.go:117] "RemoveContainer" containerID="b2abdfe0f022fc1f214e2b5fa16678460f2ca81f860790e82122f14492962662"
Sep 17 22:00:56.815670 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.815657    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2abdfe0f022fc1f214e2b5fa16678460f2ca81f860790e82122f14492962662"} err="failed to get container status \"b2abdfe0f022fc1f214e2b5fa16678460f2ca81f860790e82122f14492962662\": rpc error: code = NotFound desc = could not find container \"b2abdfe0f022fc1f214e2b5fa16678460f2ca81f860790e82122f14492962662\": container with ID starting with b2abdfe0f022fc1f214e2b5fa16678460f2ca81f860790e82122f14492962662 not found: ID does not exist"
Sep 17 22:00:56.815709 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.815670    2563 scope.go:117] "RemoveContainer" containerID="57aa6161cfafa84feca3294b48af0652080ac0b9bfaa9cc337a4acc42f6fea64"
Sep 17 22:00:56.815843 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.815830    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57aa6161cfafa84feca3294b48af0652080ac0b9bfaa9cc337a4acc42f6fea64"} err="failed to get container status \"57aa6161cfafa84feca3294b48af0652080ac0b9bfaa9cc337a4acc42f6fea64\": rpc error: code = NotFound desc = could not find container \"57aa6161cfafa84feca3294b48af0652080ac0b9bfaa9cc337a4acc42f6fea64\": container with ID starting with 57aa6161cfafa84feca3294b48af0652080ac0b9bfaa9cc337a4acc42f6fea64 not found: ID does not exist"
Sep 17 22:00:56.815843 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.815843    2563 scope.go:117] "RemoveContainer" containerID="a50b0dd6f055b2a4ed19bafb063544001afb41844a3e8c6153a3e1b6bdca558d"
Sep 17 22:00:56.816074 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.816056    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a50b0dd6f055b2a4ed19bafb063544001afb41844a3e8c6153a3e1b6bdca558d"} err="failed to get container status \"a50b0dd6f055b2a4ed19bafb063544001afb41844a3e8c6153a3e1b6bdca558d\": rpc error: code = NotFound desc = could not find container \"a50b0dd6f055b2a4ed19bafb063544001afb41844a3e8c6153a3e1b6bdca558d\": container with ID starting with a50b0dd6f055b2a4ed19bafb063544001afb41844a3e8c6153a3e1b6bdca558d not found: ID does not exist"
Sep 17 22:00:56.816074 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.816075    2563 scope.go:117] "RemoveContainer" containerID="832cc61345e4b80de54616a8b371913d97222a3dd75492595dae2a5062ccb4f9"
Sep 17 22:00:56.816358 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.816333    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"832cc61345e4b80de54616a8b371913d97222a3dd75492595dae2a5062ccb4f9"} err="failed to get container status \"832cc61345e4b80de54616a8b371913d97222a3dd75492595dae2a5062ccb4f9\": rpc error: code = NotFound desc = could not find container \"832cc61345e4b80de54616a8b371913d97222a3dd75492595dae2a5062ccb4f9\": container with ID starting with 832cc61345e4b80de54616a8b371913d97222a3dd75492595dae2a5062ccb4f9 not found: ID does not exist"
Sep 17 22:00:56.816447 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.816360    2563 scope.go:117] "RemoveContainer" containerID="10e22e4921ab467ee928faa8ee6052af01ea45fc0286c2d0f852d86e24f03d78"
Sep 17 22:00:56.816625 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.816606    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10e22e4921ab467ee928faa8ee6052af01ea45fc0286c2d0f852d86e24f03d78"} err="failed to get container status \"10e22e4921ab467ee928faa8ee6052af01ea45fc0286c2d0f852d86e24f03d78\": rpc error: code = NotFound desc = could not find container \"10e22e4921ab467ee928faa8ee6052af01ea45fc0286c2d0f852d86e24f03d78\": container with ID starting with 10e22e4921ab467ee928faa8ee6052af01ea45fc0286c2d0f852d86e24f03d78 not found: ID does not exist"
Sep 17 22:00:56.816676 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.816625    2563 scope.go:117] "RemoveContainer" containerID="1d27f621f939a1411bc81104721b1c93f25090f17b89ccde466863cf0ea27d46"
Sep 17 22:00:56.816897 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.816866    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d27f621f939a1411bc81104721b1c93f25090f17b89ccde466863cf0ea27d46"} err="failed to get container status \"1d27f621f939a1411bc81104721b1c93f25090f17b89ccde466863cf0ea27d46\": rpc error: code = NotFound desc = could not find container \"1d27f621f939a1411bc81104721b1c93f25090f17b89ccde466863cf0ea27d46\": container with ID starting with 1d27f621f939a1411bc81104721b1c93f25090f17b89ccde466863cf0ea27d46 not found: ID does not exist"
Sep 17 22:00:56.816897 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.816891    2563 scope.go:117] "RemoveContainer" containerID="58300c3bc1661cadd488ffef6361e57dbbb66d46863480510119187c02ebfd02"
Sep 17 22:00:56.817238 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.817214    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58300c3bc1661cadd488ffef6361e57dbbb66d46863480510119187c02ebfd02"} err="failed to get container status \"58300c3bc1661cadd488ffef6361e57dbbb66d46863480510119187c02ebfd02\": rpc error: code = NotFound desc = could not find container \"58300c3bc1661cadd488ffef6361e57dbbb66d46863480510119187c02ebfd02\": container with ID starting with 58300c3bc1661cadd488ffef6361e57dbbb66d46863480510119187c02ebfd02 not found: ID does not exist"
Sep 17 22:00:56.817303 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.817240    2563 scope.go:117] "RemoveContainer" containerID="b2abdfe0f022fc1f214e2b5fa16678460f2ca81f860790e82122f14492962662"
Sep 17 22:00:56.817471 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.817453    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2abdfe0f022fc1f214e2b5fa16678460f2ca81f860790e82122f14492962662"} err="failed to get container status \"b2abdfe0f022fc1f214e2b5fa16678460f2ca81f860790e82122f14492962662\": rpc error: code = NotFound desc = could not find container \"b2abdfe0f022fc1f214e2b5fa16678460f2ca81f860790e82122f14492962662\": container with ID starting with b2abdfe0f022fc1f214e2b5fa16678460f2ca81f860790e82122f14492962662 not found: ID does not exist"
Sep 17 22:00:56.817525 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.817472    2563 scope.go:117] "RemoveContainer" containerID="57aa6161cfafa84feca3294b48af0652080ac0b9bfaa9cc337a4acc42f6fea64"
Sep 17 22:00:56.817685 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.817670    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57aa6161cfafa84feca3294b48af0652080ac0b9bfaa9cc337a4acc42f6fea64"} err="failed to get container status \"57aa6161cfafa84feca3294b48af0652080ac0b9bfaa9cc337a4acc42f6fea64\": rpc error: code = NotFound desc = could not find container \"57aa6161cfafa84feca3294b48af0652080ac0b9bfaa9cc337a4acc42f6fea64\": container with ID starting with 57aa6161cfafa84feca3294b48af0652080ac0b9bfaa9cc337a4acc42f6fea64 not found: ID does not exist"
Sep 17 22:00:56.817731 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.817684    2563 scope.go:117] "RemoveContainer" containerID="a50b0dd6f055b2a4ed19bafb063544001afb41844a3e8c6153a3e1b6bdca558d"
Sep 17 22:00:56.817932 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.817910    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a50b0dd6f055b2a4ed19bafb063544001afb41844a3e8c6153a3e1b6bdca558d"} err="failed to get container status \"a50b0dd6f055b2a4ed19bafb063544001afb41844a3e8c6153a3e1b6bdca558d\": rpc error: code = NotFound desc = could not find container \"a50b0dd6f055b2a4ed19bafb063544001afb41844a3e8c6153a3e1b6bdca558d\": container with ID starting with a50b0dd6f055b2a4ed19bafb063544001afb41844a3e8c6153a3e1b6bdca558d not found: ID does not exist"
Sep 17 22:00:56.818001 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.817933    2563 scope.go:117] "RemoveContainer" containerID="832cc61345e4b80de54616a8b371913d97222a3dd75492595dae2a5062ccb4f9"
Sep 17 22:00:56.818190 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.818173    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"832cc61345e4b80de54616a8b371913d97222a3dd75492595dae2a5062ccb4f9"} err="failed to get container status \"832cc61345e4b80de54616a8b371913d97222a3dd75492595dae2a5062ccb4f9\": rpc error: code = NotFound desc = could not find container \"832cc61345e4b80de54616a8b371913d97222a3dd75492595dae2a5062ccb4f9\": container with ID starting with 832cc61345e4b80de54616a8b371913d97222a3dd75492595dae2a5062ccb4f9 not found: ID does not exist"
Sep 17 22:00:56.818248 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.818192    2563 scope.go:117] "RemoveContainer" containerID="10e22e4921ab467ee928faa8ee6052af01ea45fc0286c2d0f852d86e24f03d78"
Sep 17 22:00:56.818422 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.818404    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10e22e4921ab467ee928faa8ee6052af01ea45fc0286c2d0f852d86e24f03d78"} err="failed to get container status \"10e22e4921ab467ee928faa8ee6052af01ea45fc0286c2d0f852d86e24f03d78\": rpc error: code = NotFound desc = could not find container \"10e22e4921ab467ee928faa8ee6052af01ea45fc0286c2d0f852d86e24f03d78\": container with ID starting with 10e22e4921ab467ee928faa8ee6052af01ea45fc0286c2d0f852d86e24f03d78 not found: ID does not exist"
Sep 17 22:00:56.818483 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.818424    2563 scope.go:117] "RemoveContainer" containerID="1d27f621f939a1411bc81104721b1c93f25090f17b89ccde466863cf0ea27d46"
Sep 17 22:00:56.818627 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.818611    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d27f621f939a1411bc81104721b1c93f25090f17b89ccde466863cf0ea27d46"} err="failed to get container status \"1d27f621f939a1411bc81104721b1c93f25090f17b89ccde466863cf0ea27d46\": rpc error: code = NotFound desc = could not find container \"1d27f621f939a1411bc81104721b1c93f25090f17b89ccde466863cf0ea27d46\": container with ID starting with 1d27f621f939a1411bc81104721b1c93f25090f17b89ccde466863cf0ea27d46 not found: ID does not exist"
Sep 17 22:00:56.818685 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.818628    2563 scope.go:117] "RemoveContainer" containerID="58300c3bc1661cadd488ffef6361e57dbbb66d46863480510119187c02ebfd02"
Sep 17 22:00:56.818837 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.818821    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58300c3bc1661cadd488ffef6361e57dbbb66d46863480510119187c02ebfd02"} err="failed to get container status \"58300c3bc1661cadd488ffef6361e57dbbb66d46863480510119187c02ebfd02\": rpc error: code = NotFound desc = could not find container \"58300c3bc1661cadd488ffef6361e57dbbb66d46863480510119187c02ebfd02\": container with ID starting with 58300c3bc1661cadd488ffef6361e57dbbb66d46863480510119187c02ebfd02 not found: ID does not exist"
Sep 17 22:00:56.818899 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.818838    2563 scope.go:117] "RemoveContainer" containerID="b2abdfe0f022fc1f214e2b5fa16678460f2ca81f860790e82122f14492962662"
Sep 17 22:00:56.819067 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.819051    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2abdfe0f022fc1f214e2b5fa16678460f2ca81f860790e82122f14492962662"} err="failed to get container status \"b2abdfe0f022fc1f214e2b5fa16678460f2ca81f860790e82122f14492962662\": rpc error: code = NotFound desc = could not find container \"b2abdfe0f022fc1f214e2b5fa16678460f2ca81f860790e82122f14492962662\": container with ID starting with b2abdfe0f022fc1f214e2b5fa16678460f2ca81f860790e82122f14492962662 not found: ID does not exist"
Sep 17 22:00:56.819130 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.819068    2563 scope.go:117] "RemoveContainer" containerID="57aa6161cfafa84feca3294b48af0652080ac0b9bfaa9cc337a4acc42f6fea64"
Sep 17 22:00:56.819192 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.819158    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:56.819292 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.819274    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57aa6161cfafa84feca3294b48af0652080ac0b9bfaa9cc337a4acc42f6fea64"} err="failed to get container status \"57aa6161cfafa84feca3294b48af0652080ac0b9bfaa9cc337a4acc42f6fea64\": rpc error: code = NotFound desc = could not find container \"57aa6161cfafa84feca3294b48af0652080ac0b9bfaa9cc337a4acc42f6fea64\": container with ID starting with 57aa6161cfafa84feca3294b48af0652080ac0b9bfaa9cc337a4acc42f6fea64 not found: ID does not exist"
Sep 17 22:00:56.819354 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.819294    2563 scope.go:117] "RemoveContainer" containerID="a50b0dd6f055b2a4ed19bafb063544001afb41844a3e8c6153a3e1b6bdca558d"
Sep 17 22:00:56.819805 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.819787    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a50b0dd6f055b2a4ed19bafb063544001afb41844a3e8c6153a3e1b6bdca558d"} err="failed to get container status \"a50b0dd6f055b2a4ed19bafb063544001afb41844a3e8c6153a3e1b6bdca558d\": rpc error: code = NotFound desc = could not find container \"a50b0dd6f055b2a4ed19bafb063544001afb41844a3e8c6153a3e1b6bdca558d\": container with ID starting with a50b0dd6f055b2a4ed19bafb063544001afb41844a3e8c6153a3e1b6bdca558d not found: ID does not exist"
Sep 17 22:00:56.821518 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.821494    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 17 22:00:56.821618 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.821541    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 17 22:00:56.821685 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.821646    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 17 22:00:56.822081 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.821855    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 17 22:00:56.822081 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.821909    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-cgkpc\""
Sep 17 22:00:56.822081 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.821925    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 17 22:00:56.822081 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.821947    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 17 22:00:56.822081 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.821972    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 17 22:00:56.822501 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.822112    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9rr5phjncvf5d\""
Sep 17 22:00:56.822501 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.822191    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 17 22:00:56.822501 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.822194    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 17 22:00:56.822501 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.822222    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 17 22:00:56.822501 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.822402    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 17 22:00:56.822501 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.822417    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 17 22:00:56.822893 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.822833    2563 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-secret-grpc-tls\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:00:56.822893 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.822850    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w47l7\" (UniqueName: \"kubernetes.io/projected/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-kube-api-access-w47l7\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:00:56.822893 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.822863    2563 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-secret-prometheus-k8s-tls\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:00:56.822893 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.822879    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-prometheus-trusted-ca-bundle\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:00:56.823079 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.822895    2563 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:00:56.823079 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.822908    2563 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-config-out\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:00:56.823079 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.822920    2563 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-web-config\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:00:56.823079 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.822936    2563 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-secret-kube-rbac-proxy\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:00:56.823079 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.822948    2563 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-secret-metrics-client-certs\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:00:56.823079 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.822961    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-prometheus-k8s-db\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:00:56.823079 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.822976    2563 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-thanos-prometheus-http-client-file\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:00:56.823079 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.822990    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:00:56.823079 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.823003    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:00:56.823079 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.823016    2563 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:00:56.823079 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.823029    2563 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-config\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:00:56.823079 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.823040    2563 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-tls-assets\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:00:56.823079 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.823051    2563 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:00:56.830011 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.829992    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 17 22:00:56.831760 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.831739    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 17 22:00:56.833942 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.833920    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 22:00:56.913673 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.913647    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9" path="/var/lib/kubelet/pods/a5b72cd3-2884-4e2d-b3e8-2b19349d5fa9/volumes"
Sep 17 22:00:56.923730 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.923711    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cd0340a2-0b4d-42e1-8661-f5f15afc9b56-config-out\") pod \"prometheus-k8s-0\" (UID: \"cd0340a2-0b4d-42e1-8661-f5f15afc9b56\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:56.923797 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.923737    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cd0340a2-0b4d-42e1-8661-f5f15afc9b56-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cd0340a2-0b4d-42e1-8661-f5f15afc9b56\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:56.923797 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.923757    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd0340a2-0b4d-42e1-8661-f5f15afc9b56-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cd0340a2-0b4d-42e1-8661-f5f15afc9b56\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:56.923797 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.923782    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cd0340a2-0b4d-42e1-8661-f5f15afc9b56-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cd0340a2-0b4d-42e1-8661-f5f15afc9b56\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:56.923903 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.923816    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd0340a2-0b4d-42e1-8661-f5f15afc9b56-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cd0340a2-0b4d-42e1-8661-f5f15afc9b56\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:56.923903 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.923865    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cd0340a2-0b4d-42e1-8661-f5f15afc9b56-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cd0340a2-0b4d-42e1-8661-f5f15afc9b56\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:56.923903 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.923885    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cd0340a2-0b4d-42e1-8661-f5f15afc9b56-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cd0340a2-0b4d-42e1-8661-f5f15afc9b56\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:56.923903 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.923899    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd0340a2-0b4d-42e1-8661-f5f15afc9b56-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cd0340a2-0b4d-42e1-8661-f5f15afc9b56\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:56.924035 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.923915    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cd0340a2-0b4d-42e1-8661-f5f15afc9b56-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cd0340a2-0b4d-42e1-8661-f5f15afc9b56\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:56.924035 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.923933    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cd0340a2-0b4d-42e1-8661-f5f15afc9b56-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cd0340a2-0b4d-42e1-8661-f5f15afc9b56\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:56.924035 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.923983    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cd0340a2-0b4d-42e1-8661-f5f15afc9b56-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cd0340a2-0b4d-42e1-8661-f5f15afc9b56\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:56.924035 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.924000    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cd0340a2-0b4d-42e1-8661-f5f15afc9b56-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cd0340a2-0b4d-42e1-8661-f5f15afc9b56\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:56.924035 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.924022    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqpwn\" (UniqueName: \"kubernetes.io/projected/cd0340a2-0b4d-42e1-8661-f5f15afc9b56-kube-api-access-rqpwn\") pod \"prometheus-k8s-0\" (UID: \"cd0340a2-0b4d-42e1-8661-f5f15afc9b56\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:56.924218 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.924038    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cd0340a2-0b4d-42e1-8661-f5f15afc9b56-web-config\") pod \"prometheus-k8s-0\" (UID: \"cd0340a2-0b4d-42e1-8661-f5f15afc9b56\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:56.924218 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.924053    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/cd0340a2-0b4d-42e1-8661-f5f15afc9b56-config\") pod \"prometheus-k8s-0\" (UID: \"cd0340a2-0b4d-42e1-8661-f5f15afc9b56\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:56.924218 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.924082    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cd0340a2-0b4d-42e1-8661-f5f15afc9b56-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cd0340a2-0b4d-42e1-8661-f5f15afc9b56\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:56.924218 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.924125    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cd0340a2-0b4d-42e1-8661-f5f15afc9b56-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cd0340a2-0b4d-42e1-8661-f5f15afc9b56\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:56.924218 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.924168    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cd0340a2-0b4d-42e1-8661-f5f15afc9b56-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cd0340a2-0b4d-42e1-8661-f5f15afc9b56\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:56.924218 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.924188    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd0340a2-0b4d-42e1-8661-f5f15afc9b56-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cd0340a2-0b4d-42e1-8661-f5f15afc9b56\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:56.924218 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:56.924205    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cd0340a2-0b4d-42e1-8661-f5f15afc9b56-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cd0340a2-0b4d-42e1-8661-f5f15afc9b56\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:57.024918 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:57.024890    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqpwn\" (UniqueName: \"kubernetes.io/projected/cd0340a2-0b4d-42e1-8661-f5f15afc9b56-kube-api-access-rqpwn\") pod \"prometheus-k8s-0\" (UID: \"cd0340a2-0b4d-42e1-8661-f5f15afc9b56\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:57.025021 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:57.024948    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cd0340a2-0b4d-42e1-8661-f5f15afc9b56-web-config\") pod \"prometheus-k8s-0\" (UID: \"cd0340a2-0b4d-42e1-8661-f5f15afc9b56\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:57.025091 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:57.025069    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/cd0340a2-0b4d-42e1-8661-f5f15afc9b56-config\") pod \"prometheus-k8s-0\" (UID: \"cd0340a2-0b4d-42e1-8661-f5f15afc9b56\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:57.025166 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:57.025135    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cd0340a2-0b4d-42e1-8661-f5f15afc9b56-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cd0340a2-0b4d-42e1-8661-f5f15afc9b56\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:57.025225 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:57.025187    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cd0340a2-0b4d-42e1-8661-f5f15afc9b56-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cd0340a2-0b4d-42e1-8661-f5f15afc9b56\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:57.025225 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:57.025214    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cd0340a2-0b4d-42e1-8661-f5f15afc9b56-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cd0340a2-0b4d-42e1-8661-f5f15afc9b56\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:57.025321 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:57.025251    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd0340a2-0b4d-42e1-8661-f5f15afc9b56-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cd0340a2-0b4d-42e1-8661-f5f15afc9b56\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:57.025321 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:57.025277    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cd0340a2-0b4d-42e1-8661-f5f15afc9b56-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cd0340a2-0b4d-42e1-8661-f5f15afc9b56\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:57.025321 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:57.025311    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cd0340a2-0b4d-42e1-8661-f5f15afc9b56-config-out\") pod \"prometheus-k8s-0\" (UID: \"cd0340a2-0b4d-42e1-8661-f5f15afc9b56\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:57.025465 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:57.025333    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cd0340a2-0b4d-42e1-8661-f5f15afc9b56-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cd0340a2-0b4d-42e1-8661-f5f15afc9b56\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:57.025465 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:57.025362    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd0340a2-0b4d-42e1-8661-f5f15afc9b56-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cd0340a2-0b4d-42e1-8661-f5f15afc9b56\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:57.025465 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:57.025398    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cd0340a2-0b4d-42e1-8661-f5f15afc9b56-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cd0340a2-0b4d-42e1-8661-f5f15afc9b56\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:57.025465 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:57.025420    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd0340a2-0b4d-42e1-8661-f5f15afc9b56-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cd0340a2-0b4d-42e1-8661-f5f15afc9b56\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:57.025465 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:57.025448    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cd0340a2-0b4d-42e1-8661-f5f15afc9b56-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cd0340a2-0b4d-42e1-8661-f5f15afc9b56\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:57.025708 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:57.025477    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cd0340a2-0b4d-42e1-8661-f5f15afc9b56-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cd0340a2-0b4d-42e1-8661-f5f15afc9b56\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:57.025708 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:57.025498    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd0340a2-0b4d-42e1-8661-f5f15afc9b56-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cd0340a2-0b4d-42e1-8661-f5f15afc9b56\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:57.025708 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:57.025524    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cd0340a2-0b4d-42e1-8661-f5f15afc9b56-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cd0340a2-0b4d-42e1-8661-f5f15afc9b56\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:57.025708 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:57.025554    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cd0340a2-0b4d-42e1-8661-f5f15afc9b56-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cd0340a2-0b4d-42e1-8661-f5f15afc9b56\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:57.025708 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:57.025600    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cd0340a2-0b4d-42e1-8661-f5f15afc9b56-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cd0340a2-0b4d-42e1-8661-f5f15afc9b56\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:57.025708 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:57.025626    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cd0340a2-0b4d-42e1-8661-f5f15afc9b56-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cd0340a2-0b4d-42e1-8661-f5f15afc9b56\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:57.026013 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:57.025761    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cd0340a2-0b4d-42e1-8661-f5f15afc9b56-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cd0340a2-0b4d-42e1-8661-f5f15afc9b56\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:57.026013 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:57.025943    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd0340a2-0b4d-42e1-8661-f5f15afc9b56-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cd0340a2-0b4d-42e1-8661-f5f15afc9b56\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:57.027968 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:57.027942    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cd0340a2-0b4d-42e1-8661-f5f15afc9b56-web-config\") pod \"prometheus-k8s-0\" (UID: \"cd0340a2-0b4d-42e1-8661-f5f15afc9b56\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:57.028099 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:57.028077    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/cd0340a2-0b4d-42e1-8661-f5f15afc9b56-config\") pod \"prometheus-k8s-0\" (UID: \"cd0340a2-0b4d-42e1-8661-f5f15afc9b56\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:57.028183 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:57.028161    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cd0340a2-0b4d-42e1-8661-f5f15afc9b56-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cd0340a2-0b4d-42e1-8661-f5f15afc9b56\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:57.028406 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:57.028284    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cd0340a2-0b4d-42e1-8661-f5f15afc9b56-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cd0340a2-0b4d-42e1-8661-f5f15afc9b56\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:57.028406 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:57.028395    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cd0340a2-0b4d-42e1-8661-f5f15afc9b56-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cd0340a2-0b4d-42e1-8661-f5f15afc9b56\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:57.028562 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:57.028407    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cd0340a2-0b4d-42e1-8661-f5f15afc9b56-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cd0340a2-0b4d-42e1-8661-f5f15afc9b56\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:57.028622 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:57.028561    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cd0340a2-0b4d-42e1-8661-f5f15afc9b56-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cd0340a2-0b4d-42e1-8661-f5f15afc9b56\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:57.029159 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:57.028858    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd0340a2-0b4d-42e1-8661-f5f15afc9b56-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cd0340a2-0b4d-42e1-8661-f5f15afc9b56\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:57.029159 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:57.029010    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd0340a2-0b4d-42e1-8661-f5f15afc9b56-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cd0340a2-0b4d-42e1-8661-f5f15afc9b56\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:57.029159 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:57.029081    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cd0340a2-0b4d-42e1-8661-f5f15afc9b56-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cd0340a2-0b4d-42e1-8661-f5f15afc9b56\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:57.029360 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:57.029276    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd0340a2-0b4d-42e1-8661-f5f15afc9b56-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cd0340a2-0b4d-42e1-8661-f5f15afc9b56\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:57.029511 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:57.029490    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cd0340a2-0b4d-42e1-8661-f5f15afc9b56-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cd0340a2-0b4d-42e1-8661-f5f15afc9b56\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:57.029642 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:57.029624    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cd0340a2-0b4d-42e1-8661-f5f15afc9b56-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cd0340a2-0b4d-42e1-8661-f5f15afc9b56\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:57.029954 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:57.029938    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cd0340a2-0b4d-42e1-8661-f5f15afc9b56-config-out\") pod \"prometheus-k8s-0\" (UID: \"cd0340a2-0b4d-42e1-8661-f5f15afc9b56\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:57.030763 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:57.030737    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cd0340a2-0b4d-42e1-8661-f5f15afc9b56-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cd0340a2-0b4d-42e1-8661-f5f15afc9b56\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:57.030840 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:57.030762    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cd0340a2-0b4d-42e1-8661-f5f15afc9b56-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cd0340a2-0b4d-42e1-8661-f5f15afc9b56\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:57.031257 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:57.031240    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cd0340a2-0b4d-42e1-8661-f5f15afc9b56-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cd0340a2-0b4d-42e1-8661-f5f15afc9b56\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:57.032841 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:57.032826    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqpwn\" (UniqueName: \"kubernetes.io/projected/cd0340a2-0b4d-42e1-8661-f5f15afc9b56-kube-api-access-rqpwn\") pod \"prometheus-k8s-0\" (UID: \"cd0340a2-0b4d-42e1-8661-f5f15afc9b56\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:57.131020 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:57.130988    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:00:57.407908 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:57.407878    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 22:00:57.409792 ip-10-0-138-45 kubenswrapper[2563]: W0917 22:00:57.409762    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd0340a2_0b4d_42e1_8661_f5f15afc9b56.slice/crio-c327627edeeed31de51804440b97cff21c46b4f4ad1e167a8729bdb3cbd17c45 WatchSource:0}: Error finding container c327627edeeed31de51804440b97cff21c46b4f4ad1e167a8729bdb3cbd17c45: Status 404 returned error can't find the container with id c327627edeeed31de51804440b97cff21c46b4f4ad1e167a8729bdb3cbd17c45
Sep 17 22:00:57.766543 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:57.766459    2563 generic.go:356] "Generic (PLEG): container finished" podID="cd0340a2-0b4d-42e1-8661-f5f15afc9b56" containerID="717af934901be74a1e9274dee3e08c66533e5eddab6f8dd08e0c99040bb01c63" exitCode=0
Sep 17 22:00:57.766710 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:57.766549    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cd0340a2-0b4d-42e1-8661-f5f15afc9b56","Type":"ContainerDied","Data":"717af934901be74a1e9274dee3e08c66533e5eddab6f8dd08e0c99040bb01c63"}
Sep 17 22:00:57.766710 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:57.766631    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cd0340a2-0b4d-42e1-8661-f5f15afc9b56","Type":"ContainerStarted","Data":"c327627edeeed31de51804440b97cff21c46b4f4ad1e167a8729bdb3cbd17c45"}
Sep 17 22:00:58.779234 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:58.779201    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cd0340a2-0b4d-42e1-8661-f5f15afc9b56","Type":"ContainerStarted","Data":"9b8f9f37d8a03c6caea652c85823652f97e3cc1e1aa0259635cb4c4af4008d94"}
Sep 17 22:00:58.779613 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:58.779282    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cd0340a2-0b4d-42e1-8661-f5f15afc9b56","Type":"ContainerStarted","Data":"ce910b5440a9c7f2ce3e5cc144c83e34fb05bcffb6a1213661a03bd1c8ac086e"}
Sep 17 22:00:58.779613 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:58.779308    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cd0340a2-0b4d-42e1-8661-f5f15afc9b56","Type":"ContainerStarted","Data":"b6c4351ec8a3ea958d18512e648c30717d1537c97efe699096b97480dd716fa9"}
Sep 17 22:00:58.779613 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:58.779332    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cd0340a2-0b4d-42e1-8661-f5f15afc9b56","Type":"ContainerStarted","Data":"047fad99ca1c476821186a1750d93c7f70fa51cac5d14e1e15ba2ec5f172f4ce"}
Sep 17 22:00:58.779613 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:58.779354    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cd0340a2-0b4d-42e1-8661-f5f15afc9b56","Type":"ContainerStarted","Data":"4b094af6630c5fd3c7096964a06f6df8daac4d2d790238772e3e3be3b1e36f5f"}
Sep 17 22:00:58.779613 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:58.779375    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cd0340a2-0b4d-42e1-8661-f5f15afc9b56","Type":"ContainerStarted","Data":"11bd59c33fd7c18d17fee68a94648c5df330f69676799ade7261d6aaeff011b7"}
Sep 17 22:00:58.807818 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:00:58.807774    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.786979261 podStartE2EDuration="2.807759073s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.779812ms" isStatefulPod=false podCreationTimestamp="2026-09-17 22:00:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:00:58.80535011 +0000 UTC m=+169.973303094" watchObservedRunningTime="2026-09-17 22:00:58.807759073 +0000 UTC m=+169.975712055"
Sep 17 22:01:02.131618 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:01:02.131581    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:01:57.132089 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:01:57.132052    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:01:57.147572 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:01:57.147547    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:01:58.008250 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:01:58.008224    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 22:03:10.825484 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:03:10.825451    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nrkd5_34b8ef2b-97ba-4562-87ee-8da0cb4604c7/ovn-acl-logging/0.log"
Sep 17 22:03:10.832841 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:03:10.832818    2563 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 22:03:10.832841 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:03:10.832837    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nrkd5_34b8ef2b-97ba-4562-87ee-8da0cb4604c7/ovn-acl-logging/0.log"
Sep 17 22:06:24.744942 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:06:24.744908    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-qddlq"]
Sep 17 22:06:24.748023 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:06:24.748008    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-qddlq"
Sep 17 22:06:24.750437 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:06:24.750401    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 17 22:06:24.750570 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:06:24.750449    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-ndvdn\""
Sep 17 22:06:24.750570 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:06:24.750412    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 22:06:24.750570 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:06:24.750415    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 17 22:06:24.751251 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:06:24.751235    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 22:06:24.755594 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:06:24.755572    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-qddlq"]
Sep 17 22:06:24.853368 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:06:24.853338    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz2r8\" (UniqueName: \"kubernetes.io/projected/30d8c766-4389-4e58-ba47-812160a0d19d-kube-api-access-sz2r8\") pod \"s3-init-qddlq\" (UID: \"30d8c766-4389-4e58-ba47-812160a0d19d\") " pod="kserve/s3-init-qddlq"
Sep 17 22:06:24.853533 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:06:24.853432    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30d8c766-4389-4e58-ba47-812160a0d19d-model-cache\") pod \"s3-init-qddlq\" (UID: \"30d8c766-4389-4e58-ba47-812160a0d19d\") " pod="kserve/s3-init-qddlq"
Sep 17 22:06:24.954398 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:06:24.954367    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz2r8\" (UniqueName: \"kubernetes.io/projected/30d8c766-4389-4e58-ba47-812160a0d19d-kube-api-access-sz2r8\") pod \"s3-init-qddlq\" (UID: \"30d8c766-4389-4e58-ba47-812160a0d19d\") " pod="kserve/s3-init-qddlq"
Sep 17 22:06:24.954559 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:06:24.954430    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30d8c766-4389-4e58-ba47-812160a0d19d-model-cache\") pod \"s3-init-qddlq\" (UID: \"30d8c766-4389-4e58-ba47-812160a0d19d\") " pod="kserve/s3-init-qddlq"
Sep 17 22:06:24.954829 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:06:24.954812    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30d8c766-4389-4e58-ba47-812160a0d19d-model-cache\") pod \"s3-init-qddlq\" (UID: \"30d8c766-4389-4e58-ba47-812160a0d19d\") " pod="kserve/s3-init-qddlq"
Sep 17 22:06:24.963115 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:06:24.963083    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz2r8\" (UniqueName: \"kubernetes.io/projected/30d8c766-4389-4e58-ba47-812160a0d19d-kube-api-access-sz2r8\") pod \"s3-init-qddlq\" (UID: \"30d8c766-4389-4e58-ba47-812160a0d19d\") " pod="kserve/s3-init-qddlq"
Sep 17 22:06:25.068916 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:06:25.068831    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-qddlq"
Sep 17 22:06:25.334470 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:06:25.334446    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-qddlq"]
Sep 17 22:06:25.336716 ip-10-0-138-45 kubenswrapper[2563]: W0917 22:06:25.336693    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod30d8c766_4389_4e58_ba47_812160a0d19d.slice/crio-0494a0d904b342d8f8555c7a34732808a8b14c03d4370181dbd73b6bd05cc7d5 WatchSource:0}: Error finding container 0494a0d904b342d8f8555c7a34732808a8b14c03d4370181dbd73b6bd05cc7d5: Status 404 returned error can't find the container with id 0494a0d904b342d8f8555c7a34732808a8b14c03d4370181dbd73b6bd05cc7d5
Sep 17 22:06:25.338881 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:06:25.338864    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 22:06:25.952918 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:06:25.952882    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-qddlq" event={"ID":"30d8c766-4389-4e58-ba47-812160a0d19d","Type":"ContainerStarted","Data":"0494a0d904b342d8f8555c7a34732808a8b14c03d4370181dbd73b6bd05cc7d5"}
Sep 17 22:06:28.967871 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:06:28.967843    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-qddlq" event={"ID":"30d8c766-4389-4e58-ba47-812160a0d19d","Type":"ContainerStarted","Data":"fbdcb248f7c05a394182849c2b3d5cf4b23b0a1a81c72a8fe78b0047f390afdb"}
Sep 17 22:06:31.980619 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:06:31.980584    2563 generic.go:356] "Generic (PLEG): container finished" podID="30d8c766-4389-4e58-ba47-812160a0d19d" containerID="fbdcb248f7c05a394182849c2b3d5cf4b23b0a1a81c72a8fe78b0047f390afdb" exitCode=0
Sep 17 22:06:31.981002 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:06:31.980617    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-qddlq" event={"ID":"30d8c766-4389-4e58-ba47-812160a0d19d","Type":"ContainerDied","Data":"fbdcb248f7c05a394182849c2b3d5cf4b23b0a1a81c72a8fe78b0047f390afdb"}
Sep 17 22:06:37.006711 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:06:37.006680    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-qddlq" event={"ID":"30d8c766-4389-4e58-ba47-812160a0d19d","Type":"ContainerStarted","Data":"5b9ca261b760757be83b8461674cb911963341d2bb1c05f21e7a41fec5ba5c2e"}
Sep 17 22:06:37.024568 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:06:37.024478    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-qddlq" podStartSLOduration=2.98754171 podStartE2EDuration="13.024459987s" totalImagesPullingTime="7.938037826s" totalInitContainerRuntime="2.098880451s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:06:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:06:37.022872909 +0000 UTC m=+508.190825891" watchObservedRunningTime="2026-09-17 22:06:37.024459987 +0000 UTC m=+508.192412969"
Sep 17 22:07:41.250360 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:07:41.250329    2563 generic.go:356] "Generic (PLEG): container finished" podID="30d8c766-4389-4e58-ba47-812160a0d19d" containerID="5b9ca261b760757be83b8461674cb911963341d2bb1c05f21e7a41fec5ba5c2e" exitCode=0
Sep 17 22:07:41.250759 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:07:41.250407    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-qddlq" event={"ID":"30d8c766-4389-4e58-ba47-812160a0d19d","Type":"ContainerDied","Data":"5b9ca261b760757be83b8461674cb911963341d2bb1c05f21e7a41fec5ba5c2e"}
Sep 17 22:07:42.378382 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:07:42.378360    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-qddlq"
Sep 17 22:07:42.508744 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:07:42.508655    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sz2r8\" (UniqueName: \"kubernetes.io/projected/30d8c766-4389-4e58-ba47-812160a0d19d-kube-api-access-sz2r8\") pod \"30d8c766-4389-4e58-ba47-812160a0d19d\" (UID: \"30d8c766-4389-4e58-ba47-812160a0d19d\") "
Sep 17 22:07:42.508744 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:07:42.508729    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30d8c766-4389-4e58-ba47-812160a0d19d-model-cache\") pod \"30d8c766-4389-4e58-ba47-812160a0d19d\" (UID: \"30d8c766-4389-4e58-ba47-812160a0d19d\") "
Sep 17 22:07:42.510788 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:07:42.510749    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30d8c766-4389-4e58-ba47-812160a0d19d-kube-api-access-sz2r8" (OuterVolumeSpecName: "kube-api-access-sz2r8") pod "30d8c766-4389-4e58-ba47-812160a0d19d" (UID: "30d8c766-4389-4e58-ba47-812160a0d19d"). InnerVolumeSpecName "kube-api-access-sz2r8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:07:42.530978 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:07:42.530949    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30d8c766-4389-4e58-ba47-812160a0d19d-model-cache" (OuterVolumeSpecName: "model-cache") pod "30d8c766-4389-4e58-ba47-812160a0d19d" (UID: "30d8c766-4389-4e58-ba47-812160a0d19d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:07:42.609460 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:07:42.609430    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sz2r8\" (UniqueName: \"kubernetes.io/projected/30d8c766-4389-4e58-ba47-812160a0d19d-kube-api-access-sz2r8\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:07:42.609460 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:07:42.609455    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30d8c766-4389-4e58-ba47-812160a0d19d-model-cache\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:07:43.258572 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:07:43.258498    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-qddlq"
Sep 17 22:07:43.258731 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:07:43.258500    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-qddlq" event={"ID":"30d8c766-4389-4e58-ba47-812160a0d19d","Type":"ContainerDied","Data":"0494a0d904b342d8f8555c7a34732808a8b14c03d4370181dbd73b6bd05cc7d5"}
Sep 17 22:07:43.258731 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:07:43.258619    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0494a0d904b342d8f8555c7a34732808a8b14c03d4370181dbd73b6bd05cc7d5"
Sep 17 22:07:53.706277 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:07:53.706235    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-6z54j"]
Sep 17 22:07:53.706650 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:07:53.706540    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30d8c766-4389-4e58-ba47-812160a0d19d" containerName="download-hf-model"
Sep 17 22:07:53.706650 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:07:53.706552    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="30d8c766-4389-4e58-ba47-812160a0d19d" containerName="download-hf-model"
Sep 17 22:07:53.706650 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:07:53.706575    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30d8c766-4389-4e58-ba47-812160a0d19d" containerName="s3-init"
Sep 17 22:07:53.706650 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:07:53.706581    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="30d8c766-4389-4e58-ba47-812160a0d19d" containerName="s3-init"
Sep 17 22:07:53.706650 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:07:53.706629    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="30d8c766-4389-4e58-ba47-812160a0d19d" containerName="s3-init"
Sep 17 22:07:53.709779 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:07:53.709763    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-6z54j"
Sep 17 22:07:53.712309 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:07:53.712285    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 22:07:53.712454 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:07:53.712322    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 17 22:07:53.712454 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:07:53.712330    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Sep 17 22:07:53.712454 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:07:53.712429    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-ndvdn\""
Sep 17 22:07:53.712975 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:07:53.712957    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 22:07:53.717087 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:07:53.717065    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-6z54j"]
Sep 17 22:07:53.801637 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:07:53.801595    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9dba7ec-6fad-445d-ae48-429e1d1cd9d0-model-cache\") pod \"s3-tls-init-custom-6z54j\" (UID: \"a9dba7ec-6fad-445d-ae48-429e1d1cd9d0\") " pod="kserve/s3-tls-init-custom-6z54j"
Sep 17 22:07:53.801820 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:07:53.801655    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5gqx\" (UniqueName: \"kubernetes.io/projected/a9dba7ec-6fad-445d-ae48-429e1d1cd9d0-kube-api-access-m5gqx\") pod \"s3-tls-init-custom-6z54j\" (UID: \"a9dba7ec-6fad-445d-ae48-429e1d1cd9d0\") " pod="kserve/s3-tls-init-custom-6z54j"
Sep 17 22:07:53.902470 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:07:53.902432    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5gqx\" (UniqueName: \"kubernetes.io/projected/a9dba7ec-6fad-445d-ae48-429e1d1cd9d0-kube-api-access-m5gqx\") pod \"s3-tls-init-custom-6z54j\" (UID: \"a9dba7ec-6fad-445d-ae48-429e1d1cd9d0\") " pod="kserve/s3-tls-init-custom-6z54j"
Sep 17 22:07:53.902609 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:07:53.902496    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9dba7ec-6fad-445d-ae48-429e1d1cd9d0-model-cache\") pod \"s3-tls-init-custom-6z54j\" (UID: \"a9dba7ec-6fad-445d-ae48-429e1d1cd9d0\") " pod="kserve/s3-tls-init-custom-6z54j"
Sep 17 22:07:53.902814 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:07:53.902800    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9dba7ec-6fad-445d-ae48-429e1d1cd9d0-model-cache\") pod \"s3-tls-init-custom-6z54j\" (UID: \"a9dba7ec-6fad-445d-ae48-429e1d1cd9d0\") " pod="kserve/s3-tls-init-custom-6z54j"
Sep 17 22:07:53.911401 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:07:53.911379    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5gqx\" (UniqueName: \"kubernetes.io/projected/a9dba7ec-6fad-445d-ae48-429e1d1cd9d0-kube-api-access-m5gqx\") pod \"s3-tls-init-custom-6z54j\" (UID: \"a9dba7ec-6fad-445d-ae48-429e1d1cd9d0\") " pod="kserve/s3-tls-init-custom-6z54j"
Sep 17 22:07:54.033062 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:07:54.032978    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-6z54j"
Sep 17 22:07:54.303968 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:07:54.303944    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-6z54j"]
Sep 17 22:07:54.306023 ip-10-0-138-45 kubenswrapper[2563]: W0917 22:07:54.305983    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda9dba7ec_6fad_445d_ae48_429e1d1cd9d0.slice/crio-8563db8078e27b93778a627586132207851517ae92550d9ff3a3a93d5dc0f356 WatchSource:0}: Error finding container 8563db8078e27b93778a627586132207851517ae92550d9ff3a3a93d5dc0f356: Status 404 returned error can't find the container with id 8563db8078e27b93778a627586132207851517ae92550d9ff3a3a93d5dc0f356
Sep 17 22:07:55.303798 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:07:55.303755    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-6z54j" event={"ID":"a9dba7ec-6fad-445d-ae48-429e1d1cd9d0","Type":"ContainerStarted","Data":"837db4e6824ae01b14de7bf234c0cc9bd4ec4bdbd358d6827b375e7b80d5cf41"}
Sep 17 22:07:55.304427 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:07:55.303817    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-6z54j" event={"ID":"a9dba7ec-6fad-445d-ae48-429e1d1cd9d0","Type":"ContainerStarted","Data":"8563db8078e27b93778a627586132207851517ae92550d9ff3a3a93d5dc0f356"}
Sep 17 22:07:56.308723 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:07:56.308692    2563 generic.go:356] "Generic (PLEG): container finished" podID="a9dba7ec-6fad-445d-ae48-429e1d1cd9d0" containerID="837db4e6824ae01b14de7bf234c0cc9bd4ec4bdbd358d6827b375e7b80d5cf41" exitCode=0
Sep 17 22:07:56.309047 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:07:56.308777    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-6z54j" event={"ID":"a9dba7ec-6fad-445d-ae48-429e1d1cd9d0","Type":"ContainerDied","Data":"837db4e6824ae01b14de7bf234c0cc9bd4ec4bdbd358d6827b375e7b80d5cf41"}
Sep 17 22:07:57.314660 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:07:57.314621    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-6z54j" event={"ID":"a9dba7ec-6fad-445d-ae48-429e1d1cd9d0","Type":"ContainerStarted","Data":"b707fccdd77f47cb74d7400688073b2f0f1892d2f67c319feb5d254f8b6de90a"}
Sep 17 22:07:57.333833 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:07:57.333783    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-custom-6z54j" podStartSLOduration=2.403470052 podStartE2EDuration="4.333770034s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.930299982s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:07:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:07:57.332279845 +0000 UTC m=+588.500232827" watchObservedRunningTime="2026-09-17 22:07:57.333770034 +0000 UTC m=+588.501723063"
Sep 17 22:08:04.342056 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:04.342024    2563 generic.go:356] "Generic (PLEG): container finished" podID="a9dba7ec-6fad-445d-ae48-429e1d1cd9d0" containerID="b707fccdd77f47cb74d7400688073b2f0f1892d2f67c319feb5d254f8b6de90a" exitCode=0
Sep 17 22:08:04.342450 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:04.342063    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-6z54j" event={"ID":"a9dba7ec-6fad-445d-ae48-429e1d1cd9d0","Type":"ContainerDied","Data":"b707fccdd77f47cb74d7400688073b2f0f1892d2f67c319feb5d254f8b6de90a"}
Sep 17 22:08:05.479978 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:05.479953    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-6z54j"
Sep 17 22:08:05.598030 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:05.597938    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m5gqx\" (UniqueName: \"kubernetes.io/projected/a9dba7ec-6fad-445d-ae48-429e1d1cd9d0-kube-api-access-m5gqx\") pod \"a9dba7ec-6fad-445d-ae48-429e1d1cd9d0\" (UID: \"a9dba7ec-6fad-445d-ae48-429e1d1cd9d0\") "
Sep 17 22:08:05.598030 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:05.597986    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9dba7ec-6fad-445d-ae48-429e1d1cd9d0-model-cache\") pod \"a9dba7ec-6fad-445d-ae48-429e1d1cd9d0\" (UID: \"a9dba7ec-6fad-445d-ae48-429e1d1cd9d0\") "
Sep 17 22:08:05.600104 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:05.600066    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9dba7ec-6fad-445d-ae48-429e1d1cd9d0-kube-api-access-m5gqx" (OuterVolumeSpecName: "kube-api-access-m5gqx") pod "a9dba7ec-6fad-445d-ae48-429e1d1cd9d0" (UID: "a9dba7ec-6fad-445d-ae48-429e1d1cd9d0"). InnerVolumeSpecName "kube-api-access-m5gqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:08:05.617353 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:05.617318    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9dba7ec-6fad-445d-ae48-429e1d1cd9d0-model-cache" (OuterVolumeSpecName: "model-cache") pod "a9dba7ec-6fad-445d-ae48-429e1d1cd9d0" (UID: "a9dba7ec-6fad-445d-ae48-429e1d1cd9d0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:08:05.698653 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:05.698619    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5gqx\" (UniqueName: \"kubernetes.io/projected/a9dba7ec-6fad-445d-ae48-429e1d1cd9d0-kube-api-access-m5gqx\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:08:05.698653 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:05.698649    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9dba7ec-6fad-445d-ae48-429e1d1cd9d0-model-cache\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:08:06.350539 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:06.350511    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-6z54j"
Sep 17 22:08:06.350539 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:06.350524    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-6z54j" event={"ID":"a9dba7ec-6fad-445d-ae48-429e1d1cd9d0","Type":"ContainerDied","Data":"8563db8078e27b93778a627586132207851517ae92550d9ff3a3a93d5dc0f356"}
Sep 17 22:08:06.350744 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:06.350556    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8563db8078e27b93778a627586132207851517ae92550d9ff3a3a93d5dc0f356"
Sep 17 22:08:10.055803 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:10.055769    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-r46kx"]
Sep 17 22:08:10.056194 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:10.056043    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9dba7ec-6fad-445d-ae48-429e1d1cd9d0" containerName="s3-tls-init-custom"
Sep 17 22:08:10.056194 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:10.056054    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9dba7ec-6fad-445d-ae48-429e1d1cd9d0" containerName="s3-tls-init-custom"
Sep 17 22:08:10.056194 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:10.056071    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9dba7ec-6fad-445d-ae48-429e1d1cd9d0" containerName="download-hf-model"
Sep 17 22:08:10.056194 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:10.056077    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9dba7ec-6fad-445d-ae48-429e1d1cd9d0" containerName="download-hf-model"
Sep 17 22:08:10.057187 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:10.056631    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9dba7ec-6fad-445d-ae48-429e1d1cd9d0" containerName="s3-tls-init-custom"
Sep 17 22:08:10.061415 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:10.060425    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-r46kx"
Sep 17 22:08:10.063179 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:10.063157    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 22:08:10.063319 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:10.063301    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-ndvdn\""
Sep 17 22:08:10.063849 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:10.063832    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 17 22:08:10.063849 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:10.063842    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Sep 17 22:08:10.063945 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:10.063845    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 22:08:10.066542 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:10.066520    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-r46kx"]
Sep 17 22:08:10.137272 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:10.137242    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjvqv\" (UniqueName: \"kubernetes.io/projected/340c5f4c-7f8b-4ac3-8821-ac96bb524c0a-kube-api-access-cjvqv\") pod \"s3-tls-init-serving-r46kx\" (UID: \"340c5f4c-7f8b-4ac3-8821-ac96bb524c0a\") " pod="kserve/s3-tls-init-serving-r46kx"
Sep 17 22:08:10.137456 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:10.137283    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/340c5f4c-7f8b-4ac3-8821-ac96bb524c0a-model-cache\") pod \"s3-tls-init-serving-r46kx\" (UID: \"340c5f4c-7f8b-4ac3-8821-ac96bb524c0a\") " pod="kserve/s3-tls-init-serving-r46kx"
Sep 17 22:08:10.238423 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:10.238385    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjvqv\" (UniqueName: \"kubernetes.io/projected/340c5f4c-7f8b-4ac3-8821-ac96bb524c0a-kube-api-access-cjvqv\") pod \"s3-tls-init-serving-r46kx\" (UID: \"340c5f4c-7f8b-4ac3-8821-ac96bb524c0a\") " pod="kserve/s3-tls-init-serving-r46kx"
Sep 17 22:08:10.238423 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:10.238428    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/340c5f4c-7f8b-4ac3-8821-ac96bb524c0a-model-cache\") pod \"s3-tls-init-serving-r46kx\" (UID: \"340c5f4c-7f8b-4ac3-8821-ac96bb524c0a\") " pod="kserve/s3-tls-init-serving-r46kx"
Sep 17 22:08:10.238768 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:10.238752    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/340c5f4c-7f8b-4ac3-8821-ac96bb524c0a-model-cache\") pod \"s3-tls-init-serving-r46kx\" (UID: \"340c5f4c-7f8b-4ac3-8821-ac96bb524c0a\") " pod="kserve/s3-tls-init-serving-r46kx"
Sep 17 22:08:10.247745 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:10.247723    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjvqv\" (UniqueName: \"kubernetes.io/projected/340c5f4c-7f8b-4ac3-8821-ac96bb524c0a-kube-api-access-cjvqv\") pod \"s3-tls-init-serving-r46kx\" (UID: \"340c5f4c-7f8b-4ac3-8821-ac96bb524c0a\") " pod="kserve/s3-tls-init-serving-r46kx"
Sep 17 22:08:10.384308 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:10.384230    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-r46kx"
Sep 17 22:08:10.650591 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:10.650511    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-r46kx"]
Sep 17 22:08:10.653270 ip-10-0-138-45 kubenswrapper[2563]: W0917 22:08:10.653244    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod340c5f4c_7f8b_4ac3_8821_ac96bb524c0a.slice/crio-2bc39d7db9a1ebe8f63069bc42b249b378ada6bc519fbfe543fcde0537e7ce0e WatchSource:0}: Error finding container 2bc39d7db9a1ebe8f63069bc42b249b378ada6bc519fbfe543fcde0537e7ce0e: Status 404 returned error can't find the container with id 2bc39d7db9a1ebe8f63069bc42b249b378ada6bc519fbfe543fcde0537e7ce0e
Sep 17 22:08:10.853160 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:10.853115    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nrkd5_34b8ef2b-97ba-4562-87ee-8da0cb4604c7/ovn-acl-logging/0.log"
Sep 17 22:08:10.861235 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:10.861211    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nrkd5_34b8ef2b-97ba-4562-87ee-8da0cb4604c7/ovn-acl-logging/0.log"
Sep 17 22:08:11.373584 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:11.373547    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-r46kx" event={"ID":"340c5f4c-7f8b-4ac3-8821-ac96bb524c0a","Type":"ContainerStarted","Data":"f97e2d77e452efe4103eba595a5b95de0ca60243432f9e0b10dad38945636ba6"}
Sep 17 22:08:11.374090 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:11.373602    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-r46kx" event={"ID":"340c5f4c-7f8b-4ac3-8821-ac96bb524c0a","Type":"ContainerStarted","Data":"2bc39d7db9a1ebe8f63069bc42b249b378ada6bc519fbfe543fcde0537e7ce0e"}
Sep 17 22:08:13.383338 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:13.383254    2563 generic.go:356] "Generic (PLEG): container finished" podID="340c5f4c-7f8b-4ac3-8821-ac96bb524c0a" containerID="f97e2d77e452efe4103eba595a5b95de0ca60243432f9e0b10dad38945636ba6" exitCode=0
Sep 17 22:08:13.383338 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:13.383306    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-r46kx" event={"ID":"340c5f4c-7f8b-4ac3-8821-ac96bb524c0a","Type":"ContainerDied","Data":"f97e2d77e452efe4103eba595a5b95de0ca60243432f9e0b10dad38945636ba6"}
Sep 17 22:08:13.392623 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:13.392603    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Sep 17 22:08:14.389014 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:14.388978    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-r46kx" event={"ID":"340c5f4c-7f8b-4ac3-8821-ac96bb524c0a","Type":"ContainerStarted","Data":"4dbc6ce2fa1474f638e29290c704b556783118e834f7f4f22e2e669f864db1e7"}
Sep 17 22:08:14.407841 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:14.407791    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-r46kx" podStartSLOduration=2.070499931 podStartE2EDuration="4.407774109s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.337274178s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:08:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:08:14.407493226 +0000 UTC m=+605.575446209" watchObservedRunningTime="2026-09-17 22:08:14.407774109 +0000 UTC m=+605.575727104"
Sep 17 22:08:20.414234 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:20.414197    2563 generic.go:356] "Generic (PLEG): container finished" podID="340c5f4c-7f8b-4ac3-8821-ac96bb524c0a" containerID="4dbc6ce2fa1474f638e29290c704b556783118e834f7f4f22e2e669f864db1e7" exitCode=0
Sep 17 22:08:20.414607 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:20.414267    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-r46kx" event={"ID":"340c5f4c-7f8b-4ac3-8821-ac96bb524c0a","Type":"ContainerDied","Data":"4dbc6ce2fa1474f638e29290c704b556783118e834f7f4f22e2e669f864db1e7"}
Sep 17 22:08:21.550323 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:21.550295    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-r46kx"
Sep 17 22:08:21.644643 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:21.644608    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cjvqv\" (UniqueName: \"kubernetes.io/projected/340c5f4c-7f8b-4ac3-8821-ac96bb524c0a-kube-api-access-cjvqv\") pod \"340c5f4c-7f8b-4ac3-8821-ac96bb524c0a\" (UID: \"340c5f4c-7f8b-4ac3-8821-ac96bb524c0a\") "
Sep 17 22:08:21.644643 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:21.644661    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/340c5f4c-7f8b-4ac3-8821-ac96bb524c0a-model-cache\") pod \"340c5f4c-7f8b-4ac3-8821-ac96bb524c0a\" (UID: \"340c5f4c-7f8b-4ac3-8821-ac96bb524c0a\") "
Sep 17 22:08:21.646717 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:21.646687    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/340c5f4c-7f8b-4ac3-8821-ac96bb524c0a-kube-api-access-cjvqv" (OuterVolumeSpecName: "kube-api-access-cjvqv") pod "340c5f4c-7f8b-4ac3-8821-ac96bb524c0a" (UID: "340c5f4c-7f8b-4ac3-8821-ac96bb524c0a"). InnerVolumeSpecName "kube-api-access-cjvqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:08:21.668113 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:21.668080    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/340c5f4c-7f8b-4ac3-8821-ac96bb524c0a-model-cache" (OuterVolumeSpecName: "model-cache") pod "340c5f4c-7f8b-4ac3-8821-ac96bb524c0a" (UID: "340c5f4c-7f8b-4ac3-8821-ac96bb524c0a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:08:21.746407 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:21.746329    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjvqv\" (UniqueName: \"kubernetes.io/projected/340c5f4c-7f8b-4ac3-8821-ac96bb524c0a-kube-api-access-cjvqv\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:08:21.746407 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:21.746360    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/340c5f4c-7f8b-4ac3-8821-ac96bb524c0a-model-cache\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:08:22.423917 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:22.423891    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-r46kx"
Sep 17 22:08:22.424087 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:22.423888    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-r46kx" event={"ID":"340c5f4c-7f8b-4ac3-8821-ac96bb524c0a","Type":"ContainerDied","Data":"2bc39d7db9a1ebe8f63069bc42b249b378ada6bc519fbfe543fcde0537e7ce0e"}
Sep 17 22:08:22.424087 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:22.424007    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2bc39d7db9a1ebe8f63069bc42b249b378ada6bc519fbfe543fcde0537e7ce0e"
Sep 17 22:08:34.207384 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:34.207346    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf"]
Sep 17 22:08:34.207763 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:34.207661    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="340c5f4c-7f8b-4ac3-8821-ac96bb524c0a" containerName="download-hf-model"
Sep 17 22:08:34.207763 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:34.207674    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="340c5f4c-7f8b-4ac3-8821-ac96bb524c0a" containerName="download-hf-model"
Sep 17 22:08:34.207763 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:34.207686    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="340c5f4c-7f8b-4ac3-8821-ac96bb524c0a" containerName="s3-tls-init-serving"
Sep 17 22:08:34.207763 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:34.207691    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="340c5f4c-7f8b-4ac3-8821-ac96bb524c0a" containerName="s3-tls-init-serving"
Sep 17 22:08:34.207763 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:34.207734    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="340c5f4c-7f8b-4ac3-8821-ac96bb524c0a" containerName="s3-tls-init-serving"
Sep 17 22:08:34.211061 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:34.211043    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf"
Sep 17 22:08:34.213517 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:34.213493    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 22:08:34.213682 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:34.213523    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Sep 17 22:08:34.213682 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:34.213556    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 22:08:34.213682 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:34.213586    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Sep 17 22:08:34.213682 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:34.213620    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v4p4m\""
Sep 17 22:08:34.213682 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:34.213643    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:08:34.213935 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:34.213705    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 22:08:34.222699 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:34.222676    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf"]
Sep 17 22:08:34.306728 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:34.306692    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-e7c5cc-predictor-77d5c4fb69-c4xkh"]
Sep 17 22:08:34.310258 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:34.310241    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-e7c5cc-predictor-77d5c4fb69-c4xkh"
Sep 17 22:08:34.312890 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:34.312864    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-e7c5cc-dockercfg-66629\""
Sep 17 22:08:34.313015 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:34.312920    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-e7c5cc-predictor-serving-cert\""
Sep 17 22:08:34.313089 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:34.313073    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-e7c5cc\""
Sep 17 22:08:34.313164 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:34.313127    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-e7c5cc-kube-rbac-proxy-sar-config\""
Sep 17 22:08:34.321195 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:34.321175    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-e7c5cc-predictor-77d5c4fb69-c4xkh"]
Sep 17 22:08:34.351085 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:34.351041    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/135e3bde-367b-460d-b41c-b32d5a6b6f5a-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-56c7449565-5kvcf\" (UID: \"135e3bde-367b-460d-b41c-b32d5a6b6f5a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf"
Sep 17 22:08:34.351085 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:34.351090    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/135e3bde-367b-460d-b41c-b32d5a6b6f5a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-56c7449565-5kvcf\" (UID: \"135e3bde-367b-460d-b41c-b32d5a6b6f5a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf"
Sep 17 22:08:34.351313 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:34.351119    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/135e3bde-367b-460d-b41c-b32d5a6b6f5a-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-56c7449565-5kvcf\" (UID: \"135e3bde-367b-460d-b41c-b32d5a6b6f5a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf"
Sep 17 22:08:34.351313 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:34.351198    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hckqk\" (UniqueName: \"kubernetes.io/projected/135e3bde-367b-460d-b41c-b32d5a6b6f5a-kube-api-access-hckqk\") pod \"isvc-sklearn-batcher-predictor-56c7449565-5kvcf\" (UID: \"135e3bde-367b-460d-b41c-b32d5a6b6f5a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf"
Sep 17 22:08:34.351313 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:34.351217    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/135e3bde-367b-460d-b41c-b32d5a6b6f5a-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-56c7449565-5kvcf\" (UID: \"135e3bde-367b-460d-b41c-b32d5a6b6f5a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf"
Sep 17 22:08:34.452448 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:34.452411    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/135e3bde-367b-460d-b41c-b32d5a6b6f5a-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-56c7449565-5kvcf\" (UID: \"135e3bde-367b-460d-b41c-b32d5a6b6f5a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf"
Sep 17 22:08:34.452448 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:34.452445    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81109bae-b748-4099-9662-22ce01343cd8-kserve-provision-location\") pod \"isvc-init-fail-e7c5cc-predictor-77d5c4fb69-c4xkh\" (UID: \"81109bae-b748-4099-9662-22ce01343cd8\") " pod="kserve-ci-e2e-test/isvc-init-fail-e7c5cc-predictor-77d5c4fb69-c4xkh"
Sep 17 22:08:34.452647 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:34.452558    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/135e3bde-367b-460d-b41c-b32d5a6b6f5a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-56c7449565-5kvcf\" (UID: \"135e3bde-367b-460d-b41c-b32d5a6b6f5a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf"
Sep 17 22:08:34.452647 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:34.452610    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/135e3bde-367b-460d-b41c-b32d5a6b6f5a-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-56c7449565-5kvcf\" (UID: \"135e3bde-367b-460d-b41c-b32d5a6b6f5a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf"
Sep 17 22:08:34.452741 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:34.452663    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crtqh\" (UniqueName: \"kubernetes.io/projected/81109bae-b748-4099-9662-22ce01343cd8-kube-api-access-crtqh\") pod \"isvc-init-fail-e7c5cc-predictor-77d5c4fb69-c4xkh\" (UID: \"81109bae-b748-4099-9662-22ce01343cd8\") " pod="kserve-ci-e2e-test/isvc-init-fail-e7c5cc-predictor-77d5c4fb69-c4xkh"
Sep 17 22:08:34.452741 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:34.452692    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81109bae-b748-4099-9662-22ce01343cd8-cabundle-cert\") pod \"isvc-init-fail-e7c5cc-predictor-77d5c4fb69-c4xkh\" (UID: \"81109bae-b748-4099-9662-22ce01343cd8\") " pod="kserve-ci-e2e-test/isvc-init-fail-e7c5cc-predictor-77d5c4fb69-c4xkh"
Sep 17 22:08:34.452741 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:34.452724    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81109bae-b748-4099-9662-22ce01343cd8-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-e7c5cc-predictor-77d5c4fb69-c4xkh\" (UID: \"81109bae-b748-4099-9662-22ce01343cd8\") " pod="kserve-ci-e2e-test/isvc-init-fail-e7c5cc-predictor-77d5c4fb69-c4xkh"
Sep 17 22:08:34.452849 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:34.452801    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hckqk\" (UniqueName: \"kubernetes.io/projected/135e3bde-367b-460d-b41c-b32d5a6b6f5a-kube-api-access-hckqk\") pod \"isvc-sklearn-batcher-predictor-56c7449565-5kvcf\" (UID: \"135e3bde-367b-460d-b41c-b32d5a6b6f5a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf"
Sep 17 22:08:34.452849 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:34.452829    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/135e3bde-367b-460d-b41c-b32d5a6b6f5a-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-56c7449565-5kvcf\" (UID: \"135e3bde-367b-460d-b41c-b32d5a6b6f5a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf"
Sep 17 22:08:34.452849 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:34.452839    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/135e3bde-367b-460d-b41c-b32d5a6b6f5a-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-56c7449565-5kvcf\" (UID: \"135e3bde-367b-460d-b41c-b32d5a6b6f5a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf"
Sep 17 22:08:34.452989 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:34.452878    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81109bae-b748-4099-9662-22ce01343cd8-proxy-tls\") pod \"isvc-init-fail-e7c5cc-predictor-77d5c4fb69-c4xkh\" (UID: \"81109bae-b748-4099-9662-22ce01343cd8\") " pod="kserve-ci-e2e-test/isvc-init-fail-e7c5cc-predictor-77d5c4fb69-c4xkh"
Sep 17 22:08:34.453038 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:08:34.453018    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-serving-cert: secret "isvc-sklearn-batcher-predictor-serving-cert" not found
Sep 17 22:08:34.453109 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:08:34.453096    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/135e3bde-367b-460d-b41c-b32d5a6b6f5a-proxy-tls podName:135e3bde-367b-460d-b41c-b32d5a6b6f5a nodeName:}" failed. No retries permitted until 2026-09-17 22:08:34.953077569 +0000 UTC m=+626.121030541 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/135e3bde-367b-460d-b41c-b32d5a6b6f5a-proxy-tls") pod "isvc-sklearn-batcher-predictor-56c7449565-5kvcf" (UID: "135e3bde-367b-460d-b41c-b32d5a6b6f5a") : secret "isvc-sklearn-batcher-predictor-serving-cert" not found
Sep 17 22:08:34.453325 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:34.453303    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/135e3bde-367b-460d-b41c-b32d5a6b6f5a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-56c7449565-5kvcf\" (UID: \"135e3bde-367b-460d-b41c-b32d5a6b6f5a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf"
Sep 17 22:08:34.453388 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:34.453333    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/135e3bde-367b-460d-b41c-b32d5a6b6f5a-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-56c7449565-5kvcf\" (UID: \"135e3bde-367b-460d-b41c-b32d5a6b6f5a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf"
Sep 17 22:08:34.463301 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:34.463247    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hckqk\" (UniqueName: \"kubernetes.io/projected/135e3bde-367b-460d-b41c-b32d5a6b6f5a-kube-api-access-hckqk\") pod \"isvc-sklearn-batcher-predictor-56c7449565-5kvcf\" (UID: \"135e3bde-367b-460d-b41c-b32d5a6b6f5a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf"
Sep 17 22:08:34.554334 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:34.554296    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crtqh\" (UniqueName: \"kubernetes.io/projected/81109bae-b748-4099-9662-22ce01343cd8-kube-api-access-crtqh\") pod \"isvc-init-fail-e7c5cc-predictor-77d5c4fb69-c4xkh\" (UID: \"81109bae-b748-4099-9662-22ce01343cd8\") " pod="kserve-ci-e2e-test/isvc-init-fail-e7c5cc-predictor-77d5c4fb69-c4xkh"
Sep 17 22:08:34.554334 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:34.554334    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81109bae-b748-4099-9662-22ce01343cd8-cabundle-cert\") pod \"isvc-init-fail-e7c5cc-predictor-77d5c4fb69-c4xkh\" (UID: \"81109bae-b748-4099-9662-22ce01343cd8\") " pod="kserve-ci-e2e-test/isvc-init-fail-e7c5cc-predictor-77d5c4fb69-c4xkh"
Sep 17 22:08:34.554571 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:34.554353    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81109bae-b748-4099-9662-22ce01343cd8-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-e7c5cc-predictor-77d5c4fb69-c4xkh\" (UID: \"81109bae-b748-4099-9662-22ce01343cd8\") " pod="kserve-ci-e2e-test/isvc-init-fail-e7c5cc-predictor-77d5c4fb69-c4xkh"
Sep 17 22:08:34.554571 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:34.554409    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81109bae-b748-4099-9662-22ce01343cd8-proxy-tls\") pod \"isvc-init-fail-e7c5cc-predictor-77d5c4fb69-c4xkh\" (UID: \"81109bae-b748-4099-9662-22ce01343cd8\") " pod="kserve-ci-e2e-test/isvc-init-fail-e7c5cc-predictor-77d5c4fb69-c4xkh"
Sep 17 22:08:34.554571 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:34.554444    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81109bae-b748-4099-9662-22ce01343cd8-kserve-provision-location\") pod \"isvc-init-fail-e7c5cc-predictor-77d5c4fb69-c4xkh\" (UID: \"81109bae-b748-4099-9662-22ce01343cd8\") " pod="kserve-ci-e2e-test/isvc-init-fail-e7c5cc-predictor-77d5c4fb69-c4xkh"
Sep 17 22:08:34.554868 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:34.554849    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81109bae-b748-4099-9662-22ce01343cd8-kserve-provision-location\") pod \"isvc-init-fail-e7c5cc-predictor-77d5c4fb69-c4xkh\" (UID: \"81109bae-b748-4099-9662-22ce01343cd8\") " pod="kserve-ci-e2e-test/isvc-init-fail-e7c5cc-predictor-77d5c4fb69-c4xkh"
Sep 17 22:08:34.555173 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:34.555132    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81109bae-b748-4099-9662-22ce01343cd8-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-e7c5cc-predictor-77d5c4fb69-c4xkh\" (UID: \"81109bae-b748-4099-9662-22ce01343cd8\") " pod="kserve-ci-e2e-test/isvc-init-fail-e7c5cc-predictor-77d5c4fb69-c4xkh"
Sep 17 22:08:34.555216 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:34.555132    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81109bae-b748-4099-9662-22ce01343cd8-cabundle-cert\") pod \"isvc-init-fail-e7c5cc-predictor-77d5c4fb69-c4xkh\" (UID: \"81109bae-b748-4099-9662-22ce01343cd8\") " pod="kserve-ci-e2e-test/isvc-init-fail-e7c5cc-predictor-77d5c4fb69-c4xkh"
Sep 17 22:08:34.556931 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:34.556913    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81109bae-b748-4099-9662-22ce01343cd8-proxy-tls\") pod \"isvc-init-fail-e7c5cc-predictor-77d5c4fb69-c4xkh\" (UID: \"81109bae-b748-4099-9662-22ce01343cd8\") " pod="kserve-ci-e2e-test/isvc-init-fail-e7c5cc-predictor-77d5c4fb69-c4xkh"
Sep 17 22:08:34.562702 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:34.562673    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crtqh\" (UniqueName: \"kubernetes.io/projected/81109bae-b748-4099-9662-22ce01343cd8-kube-api-access-crtqh\") pod \"isvc-init-fail-e7c5cc-predictor-77d5c4fb69-c4xkh\" (UID: \"81109bae-b748-4099-9662-22ce01343cd8\") " pod="kserve-ci-e2e-test/isvc-init-fail-e7c5cc-predictor-77d5c4fb69-c4xkh"
Sep 17 22:08:34.620634 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:34.620604    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-e7c5cc-predictor-77d5c4fb69-c4xkh"
Sep 17 22:08:34.893497 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:34.893420    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-e7c5cc-predictor-77d5c4fb69-c4xkh"]
Sep 17 22:08:34.898021 ip-10-0-138-45 kubenswrapper[2563]: W0917 22:08:34.897936    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81109bae_b748_4099_9662_22ce01343cd8.slice/crio-18499e9f197e471f09806c778c3cc51bda0d02b93ffaaf6a762e83ff1dd5c260 WatchSource:0}: Error finding container 18499e9f197e471f09806c778c3cc51bda0d02b93ffaaf6a762e83ff1dd5c260: Status 404 returned error can't find the container with id 18499e9f197e471f09806c778c3cc51bda0d02b93ffaaf6a762e83ff1dd5c260
Sep 17 22:08:34.958207 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:34.958181    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/135e3bde-367b-460d-b41c-b32d5a6b6f5a-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-56c7449565-5kvcf\" (UID: \"135e3bde-367b-460d-b41c-b32d5a6b6f5a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf"
Sep 17 22:08:34.960783 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:34.960758    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/135e3bde-367b-460d-b41c-b32d5a6b6f5a-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-56c7449565-5kvcf\" (UID: \"135e3bde-367b-460d-b41c-b32d5a6b6f5a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf"
Sep 17 22:08:35.121618 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:35.121573    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf"
Sep 17 22:08:35.397291 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:35.397263    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf"]
Sep 17 22:08:35.399010 ip-10-0-138-45 kubenswrapper[2563]: W0917 22:08:35.398976    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod135e3bde_367b_460d_b41c_b32d5a6b6f5a.slice/crio-b673b999fcc46fc89db52d72d628f6602d817bd49ad0760114769d17406427db WatchSource:0}: Error finding container b673b999fcc46fc89db52d72d628f6602d817bd49ad0760114769d17406427db: Status 404 returned error can't find the container with id b673b999fcc46fc89db52d72d628f6602d817bd49ad0760114769d17406427db
Sep 17 22:08:35.477665 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:35.477634    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-e7c5cc-predictor-77d5c4fb69-c4xkh" event={"ID":"81109bae-b748-4099-9662-22ce01343cd8","Type":"ContainerStarted","Data":"02b540cac1e8b96d2c0f867a9c963910d75de8caccacb4c65b8ee8a98a468661"}
Sep 17 22:08:35.477835 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:35.477702    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-e7c5cc-predictor-77d5c4fb69-c4xkh" event={"ID":"81109bae-b748-4099-9662-22ce01343cd8","Type":"ContainerStarted","Data":"18499e9f197e471f09806c778c3cc51bda0d02b93ffaaf6a762e83ff1dd5c260"}
Sep 17 22:08:35.479389 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:35.479360    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf" event={"ID":"135e3bde-367b-460d-b41c-b32d5a6b6f5a","Type":"ContainerStarted","Data":"cb08040a17d5852a1fe16b8d51c4cfcc3d0cdcc8cb093700ff3f2581870666fc"}
Sep 17 22:08:35.479495 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:35.479415    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf" event={"ID":"135e3bde-367b-460d-b41c-b32d5a6b6f5a","Type":"ContainerStarted","Data":"b673b999fcc46fc89db52d72d628f6602d817bd49ad0760114769d17406427db"}
Sep 17 22:08:39.496001 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:39.495916    2563 generic.go:356] "Generic (PLEG): container finished" podID="135e3bde-367b-460d-b41c-b32d5a6b6f5a" containerID="cb08040a17d5852a1fe16b8d51c4cfcc3d0cdcc8cb093700ff3f2581870666fc" exitCode=0
Sep 17 22:08:39.496001 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:39.495972    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf" event={"ID":"135e3bde-367b-460d-b41c-b32d5a6b6f5a","Type":"ContainerDied","Data":"cb08040a17d5852a1fe16b8d51c4cfcc3d0cdcc8cb093700ff3f2581870666fc"}
Sep 17 22:08:41.512861 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:41.512827    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-e7c5cc-predictor-77d5c4fb69-c4xkh_81109bae-b748-4099-9662-22ce01343cd8/storage-initializer/0.log"
Sep 17 22:08:41.513406 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:41.512874    2563 generic.go:356] "Generic (PLEG): container finished" podID="81109bae-b748-4099-9662-22ce01343cd8" containerID="02b540cac1e8b96d2c0f867a9c963910d75de8caccacb4c65b8ee8a98a468661" exitCode=1
Sep 17 22:08:41.513406 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:41.512961    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-e7c5cc-predictor-77d5c4fb69-c4xkh" event={"ID":"81109bae-b748-4099-9662-22ce01343cd8","Type":"ContainerDied","Data":"02b540cac1e8b96d2c0f867a9c963910d75de8caccacb4c65b8ee8a98a468661"}
Sep 17 22:08:42.525154 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:42.525105    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-e7c5cc-predictor-77d5c4fb69-c4xkh_81109bae-b748-4099-9662-22ce01343cd8/storage-initializer/0.log"
Sep 17 22:08:42.525594 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:42.525226    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-e7c5cc-predictor-77d5c4fb69-c4xkh" event={"ID":"81109bae-b748-4099-9662-22ce01343cd8","Type":"ContainerStarted","Data":"6a53138d687fd4ffec08c056fc58c6d0980f3bffb4988ba1b629b047d8264f20"}
Sep 17 22:08:44.017172 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:44.016929    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-e7c5cc-predictor-77d5c4fb69-c4xkh"]
Sep 17 22:08:44.018450 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:44.017922    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-e7c5cc-predictor-77d5c4fb69-c4xkh" podUID="81109bae-b748-4099-9662-22ce01343cd8" containerName="storage-initializer" containerID="cri-o://6a53138d687fd4ffec08c056fc58c6d0980f3bffb4988ba1b629b047d8264f20" gracePeriod=30
Sep 17 22:08:48.556005 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:48.555932    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-e7c5cc-predictor-77d5c4fb69-c4xkh_81109bae-b748-4099-9662-22ce01343cd8/storage-initializer/1.log"
Sep 17 22:08:48.560072 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:48.556426    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-e7c5cc-predictor-77d5c4fb69-c4xkh_81109bae-b748-4099-9662-22ce01343cd8/storage-initializer/0.log"
Sep 17 22:08:48.560072 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:48.556464    2563 generic.go:356] "Generic (PLEG): container finished" podID="81109bae-b748-4099-9662-22ce01343cd8" containerID="6a53138d687fd4ffec08c056fc58c6d0980f3bffb4988ba1b629b047d8264f20" exitCode=1
Sep 17 22:08:48.560072 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:48.556493    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-e7c5cc-predictor-77d5c4fb69-c4xkh" event={"ID":"81109bae-b748-4099-9662-22ce01343cd8","Type":"ContainerDied","Data":"6a53138d687fd4ffec08c056fc58c6d0980f3bffb4988ba1b629b047d8264f20"}
Sep 17 22:08:48.560072 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:48.556557    2563 scope.go:117] "RemoveContainer" containerID="02b540cac1e8b96d2c0f867a9c963910d75de8caccacb4c65b8ee8a98a468661"
Sep 17 22:08:52.429886 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:52.429861    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-e7c5cc-predictor-77d5c4fb69-c4xkh_81109bae-b748-4099-9662-22ce01343cd8/storage-initializer/1.log"
Sep 17 22:08:52.432251 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:52.432234    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-e7c5cc-predictor-77d5c4fb69-c4xkh"
Sep 17 22:08:52.519849 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:52.519825    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81109bae-b748-4099-9662-22ce01343cd8-cabundle-cert\") pod \"81109bae-b748-4099-9662-22ce01343cd8\" (UID: \"81109bae-b748-4099-9662-22ce01343cd8\") "
Sep 17 22:08:52.519971 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:52.519920    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81109bae-b748-4099-9662-22ce01343cd8-kserve-provision-location\") pod \"81109bae-b748-4099-9662-22ce01343cd8\" (UID: \"81109bae-b748-4099-9662-22ce01343cd8\") "
Sep 17 22:08:52.519971 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:52.519944    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81109bae-b748-4099-9662-22ce01343cd8-kube-rbac-proxy-sar-config\") pod \"81109bae-b748-4099-9662-22ce01343cd8\" (UID: \"81109bae-b748-4099-9662-22ce01343cd8\") "
Sep 17 22:08:52.519971 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:52.519966    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81109bae-b748-4099-9662-22ce01343cd8-proxy-tls\") pod \"81109bae-b748-4099-9662-22ce01343cd8\" (UID: \"81109bae-b748-4099-9662-22ce01343cd8\") "
Sep 17 22:08:52.520233 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:52.520010    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-crtqh\" (UniqueName: \"kubernetes.io/projected/81109bae-b748-4099-9662-22ce01343cd8-kube-api-access-crtqh\") pod \"81109bae-b748-4099-9662-22ce01343cd8\" (UID: \"81109bae-b748-4099-9662-22ce01343cd8\") "
Sep 17 22:08:52.520233 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:52.520204    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81109bae-b748-4099-9662-22ce01343cd8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "81109bae-b748-4099-9662-22ce01343cd8" (UID: "81109bae-b748-4099-9662-22ce01343cd8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:08:52.520339 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:52.520274    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81109bae-b748-4099-9662-22ce01343cd8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "81109bae-b748-4099-9662-22ce01343cd8" (UID: "81109bae-b748-4099-9662-22ce01343cd8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:08:52.520379 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:52.520330    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81109bae-b748-4099-9662-22ce01343cd8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "81109bae-b748-4099-9662-22ce01343cd8" (UID: "81109bae-b748-4099-9662-22ce01343cd8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:08:52.522116 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:52.522092    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81109bae-b748-4099-9662-22ce01343cd8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "81109bae-b748-4099-9662-22ce01343cd8" (UID: "81109bae-b748-4099-9662-22ce01343cd8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:08:52.522264 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:52.522240    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81109bae-b748-4099-9662-22ce01343cd8-kube-api-access-crtqh" (OuterVolumeSpecName: "kube-api-access-crtqh") pod "81109bae-b748-4099-9662-22ce01343cd8" (UID: "81109bae-b748-4099-9662-22ce01343cd8"). InnerVolumeSpecName "kube-api-access-crtqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:08:52.576199 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:52.576176    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-e7c5cc-predictor-77d5c4fb69-c4xkh_81109bae-b748-4099-9662-22ce01343cd8/storage-initializer/1.log"
Sep 17 22:08:52.576314 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:52.576295    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-e7c5cc-predictor-77d5c4fb69-c4xkh"
Sep 17 22:08:52.576380 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:52.576315    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-e7c5cc-predictor-77d5c4fb69-c4xkh" event={"ID":"81109bae-b748-4099-9662-22ce01343cd8","Type":"ContainerDied","Data":"18499e9f197e471f09806c778c3cc51bda0d02b93ffaaf6a762e83ff1dd5c260"}
Sep 17 22:08:52.576438 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:52.576385    2563 scope.go:117] "RemoveContainer" containerID="6a53138d687fd4ffec08c056fc58c6d0980f3bffb4988ba1b629b047d8264f20"
Sep 17 22:08:52.578421 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:52.578399    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf" event={"ID":"135e3bde-367b-460d-b41c-b32d5a6b6f5a","Type":"ContainerStarted","Data":"cdcd3284294ba9aa4fca43bfdee23f106435d4c45e2d45039672ec9366a55259"}
Sep 17 22:08:52.612558 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:52.612504    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-e7c5cc-predictor-77d5c4fb69-c4xkh"]
Sep 17 22:08:52.617867 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:52.617841    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-e7c5cc-predictor-77d5c4fb69-c4xkh"]
Sep 17 22:08:52.620794 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:52.620775    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81109bae-b748-4099-9662-22ce01343cd8-kserve-provision-location\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:08:52.620865 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:52.620796    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81109bae-b748-4099-9662-22ce01343cd8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:08:52.620865 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:52.620807    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81109bae-b748-4099-9662-22ce01343cd8-proxy-tls\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:08:52.620865 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:52.620818    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crtqh\" (UniqueName: \"kubernetes.io/projected/81109bae-b748-4099-9662-22ce01343cd8-kube-api-access-crtqh\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:08:52.620865 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:52.620827    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81109bae-b748-4099-9662-22ce01343cd8-cabundle-cert\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:08:52.913692 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:52.913656    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81109bae-b748-4099-9662-22ce01343cd8" path="/var/lib/kubelet/pods/81109bae-b748-4099-9662-22ce01343cd8/volumes"
Sep 17 22:08:55.597243 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:55.597204    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf" event={"ID":"135e3bde-367b-460d-b41c-b32d5a6b6f5a","Type":"ContainerStarted","Data":"68812fea59c62a1f6d2011d23e9ca043efe14784ec288405356fe6af9e32e607"}
Sep 17 22:08:58.611894 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:58.611859    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf" event={"ID":"135e3bde-367b-460d-b41c-b32d5a6b6f5a","Type":"ContainerStarted","Data":"1f8adcebf6c8e513db1e399ac91c805bf1a0728310194cfe1eea66936b296193"}
Sep 17 22:08:58.612413 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:58.611937    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf"
Sep 17 22:08:58.612413 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:58.611975    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf"
Sep 17 22:08:58.613394 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:58.613362    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf" podUID="135e3bde-367b-460d-b41c-b32d5a6b6f5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 22:08:58.632507 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:58.632460    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf" podStartSLOduration=2.920043112 podStartE2EDuration="24.632446739s" totalImagesPullingTime="17.965384511s" totalInitContainerRuntime="3.747019116s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:08:34 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:08:58.630890842 +0000 UTC m=+649.798843824" watchObservedRunningTime="2026-09-17 22:08:58.632446739 +0000 UTC m=+649.800399721"
Sep 17 22:08:59.616525 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:59.616497    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf"
Sep 17 22:08:59.617154 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:59.617101    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf" podUID="135e3bde-367b-460d-b41c-b32d5a6b6f5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 22:08:59.617648 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:59.617626    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf" podUID="135e3bde-367b-460d-b41c-b32d5a6b6f5a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:08:59.621949 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:08:59.621930    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf"
Sep 17 22:09:00.626158 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:09:00.626115    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf" podUID="135e3bde-367b-460d-b41c-b32d5a6b6f5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 22:09:00.626632 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:09:00.626530    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf" podUID="135e3bde-367b-460d-b41c-b32d5a6b6f5a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:09:01.629941 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:09:01.629900    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf" podUID="135e3bde-367b-460d-b41c-b32d5a6b6f5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 22:09:01.630396 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:09:01.630173    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf" podUID="135e3bde-367b-460d-b41c-b32d5a6b6f5a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:09:11.629995 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:09:11.629937    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf" podUID="135e3bde-367b-460d-b41c-b32d5a6b6f5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 22:09:11.630441 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:09:11.630402    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf" podUID="135e3bde-367b-460d-b41c-b32d5a6b6f5a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:09:21.630162 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:09:21.630095    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf" podUID="135e3bde-367b-460d-b41c-b32d5a6b6f5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 22:09:21.630583 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:09:21.630551    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf" podUID="135e3bde-367b-460d-b41c-b32d5a6b6f5a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:09:31.630123 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:09:31.630080    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf" podUID="135e3bde-367b-460d-b41c-b32d5a6b6f5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 22:09:31.630704 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:09:31.630631    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf" podUID="135e3bde-367b-460d-b41c-b32d5a6b6f5a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:09:41.630442 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:09:41.630401    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf" podUID="135e3bde-367b-460d-b41c-b32d5a6b6f5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 22:09:41.630887 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:09:41.630863    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf" podUID="135e3bde-367b-460d-b41c-b32d5a6b6f5a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:09:51.630473 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:09:51.630415    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf" podUID="135e3bde-367b-460d-b41c-b32d5a6b6f5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 22:09:51.630915 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:09:51.630881    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf" podUID="135e3bde-367b-460d-b41c-b32d5a6b6f5a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:10:01.630022 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:01.629924    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf" podUID="135e3bde-367b-460d-b41c-b32d5a6b6f5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 22:10:01.630491 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:01.630469    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf" podUID="135e3bde-367b-460d-b41c-b32d5a6b6f5a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:10:11.631307 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:11.631277    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf"
Sep 17 22:10:11.631850 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:11.631375    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf"
Sep 17 22:10:19.092568 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:19.092536    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf"]
Sep 17 22:10:19.092995 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:19.092782    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf" podUID="135e3bde-367b-460d-b41c-b32d5a6b6f5a" containerName="kserve-container" containerID="cri-o://cdcd3284294ba9aa4fca43bfdee23f106435d4c45e2d45039672ec9366a55259" gracePeriod=30
Sep 17 22:10:19.092995 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:19.092815    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf" podUID="135e3bde-367b-460d-b41c-b32d5a6b6f5a" containerName="agent" containerID="cri-o://1f8adcebf6c8e513db1e399ac91c805bf1a0728310194cfe1eea66936b296193" gracePeriod=30
Sep 17 22:10:19.092995 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:19.092847    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf" podUID="135e3bde-367b-460d-b41c-b32d5a6b6f5a" containerName="kube-rbac-proxy" containerID="cri-o://68812fea59c62a1f6d2011d23e9ca043efe14784ec288405356fe6af9e32e607" gracePeriod=30
Sep 17 22:10:19.223647 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:19.223617    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z"]
Sep 17 22:10:19.223965 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:19.223952    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81109bae-b748-4099-9662-22ce01343cd8" containerName="storage-initializer"
Sep 17 22:10:19.224014 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:19.223967    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="81109bae-b748-4099-9662-22ce01343cd8" containerName="storage-initializer"
Sep 17 22:10:19.224048 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:19.224019    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="81109bae-b748-4099-9662-22ce01343cd8" containerName="storage-initializer"
Sep 17 22:10:19.224093 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:19.224081    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81109bae-b748-4099-9662-22ce01343cd8" containerName="storage-initializer"
Sep 17 22:10:19.224126 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:19.224093    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="81109bae-b748-4099-9662-22ce01343cd8" containerName="storage-initializer"
Sep 17 22:10:19.224176 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:19.224168    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="81109bae-b748-4099-9662-22ce01343cd8" containerName="storage-initializer"
Sep 17 22:10:19.227271 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:19.227254    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z"
Sep 17 22:10:19.229905 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:19.229884    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Sep 17 22:10:19.229969 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:19.229903    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Sep 17 22:10:19.238046 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:19.238020    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z"]
Sep 17 22:10:19.262894 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:19.262863    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41f89d55-e428-405c-87b8-6e24e45490f3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z\" (UID: \"41f89d55-e428-405c-87b8-6e24e45490f3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z"
Sep 17 22:10:19.263061 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:19.262925    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41f89d55-e428-405c-87b8-6e24e45490f3-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z\" (UID: \"41f89d55-e428-405c-87b8-6e24e45490f3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z"
Sep 17 22:10:19.263061 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:19.262958    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41f89d55-e428-405c-87b8-6e24e45490f3-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z\" (UID: \"41f89d55-e428-405c-87b8-6e24e45490f3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z"
Sep 17 22:10:19.263061 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:19.262990    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8kvr\" (UniqueName: \"kubernetes.io/projected/41f89d55-e428-405c-87b8-6e24e45490f3-kube-api-access-k8kvr\") pod \"isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z\" (UID: \"41f89d55-e428-405c-87b8-6e24e45490f3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z"
Sep 17 22:10:19.263173 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:19.263056    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41f89d55-e428-405c-87b8-6e24e45490f3-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z\" (UID: \"41f89d55-e428-405c-87b8-6e24e45490f3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z"
Sep 17 22:10:19.364489 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:19.364406    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8kvr\" (UniqueName: \"kubernetes.io/projected/41f89d55-e428-405c-87b8-6e24e45490f3-kube-api-access-k8kvr\") pod \"isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z\" (UID: \"41f89d55-e428-405c-87b8-6e24e45490f3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z"
Sep 17 22:10:19.364489 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:19.364446    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41f89d55-e428-405c-87b8-6e24e45490f3-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z\" (UID: \"41f89d55-e428-405c-87b8-6e24e45490f3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z"
Sep 17 22:10:19.364489 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:19.364480    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41f89d55-e428-405c-87b8-6e24e45490f3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z\" (UID: \"41f89d55-e428-405c-87b8-6e24e45490f3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z"
Sep 17 22:10:19.364766 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:19.364524    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41f89d55-e428-405c-87b8-6e24e45490f3-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z\" (UID: \"41f89d55-e428-405c-87b8-6e24e45490f3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z"
Sep 17 22:10:19.364766 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:19.364552    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41f89d55-e428-405c-87b8-6e24e45490f3-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z\" (UID: \"41f89d55-e428-405c-87b8-6e24e45490f3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z"
Sep 17 22:10:19.364766 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:10:19.364691    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-serving-cert: secret "isvc-sklearn-batcher-custom-predictor-serving-cert" not found
Sep 17 22:10:19.364902 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:10:19.364787    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41f89d55-e428-405c-87b8-6e24e45490f3-proxy-tls podName:41f89d55-e428-405c-87b8-6e24e45490f3 nodeName:}" failed. No retries permitted until 2026-09-17 22:10:19.864766713 +0000 UTC m=+731.032719682 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/41f89d55-e428-405c-87b8-6e24e45490f3-proxy-tls") pod "isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z" (UID: "41f89d55-e428-405c-87b8-6e24e45490f3") : secret "isvc-sklearn-batcher-custom-predictor-serving-cert" not found
Sep 17 22:10:19.365027 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:19.365007    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41f89d55-e428-405c-87b8-6e24e45490f3-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z\" (UID: \"41f89d55-e428-405c-87b8-6e24e45490f3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z"
Sep 17 22:10:19.365330 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:19.365308    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41f89d55-e428-405c-87b8-6e24e45490f3-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z\" (UID: \"41f89d55-e428-405c-87b8-6e24e45490f3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z"
Sep 17 22:10:19.365370 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:19.365308    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41f89d55-e428-405c-87b8-6e24e45490f3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z\" (UID: \"41f89d55-e428-405c-87b8-6e24e45490f3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z"
Sep 17 22:10:19.376259 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:19.376237    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8kvr\" (UniqueName: \"kubernetes.io/projected/41f89d55-e428-405c-87b8-6e24e45490f3-kube-api-access-k8kvr\") pod \"isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z\" (UID: \"41f89d55-e428-405c-87b8-6e24e45490f3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z"
Sep 17 22:10:19.617085 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:19.616996    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf" podUID="135e3bde-367b-460d-b41c-b32d5a6b6f5a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.24:8643/healthz\": dial tcp 10.133.0.24:8643: connect: connection refused"
Sep 17 22:10:19.868965 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:19.868865    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41f89d55-e428-405c-87b8-6e24e45490f3-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z\" (UID: \"41f89d55-e428-405c-87b8-6e24e45490f3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z"
Sep 17 22:10:19.871388 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:19.871364    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41f89d55-e428-405c-87b8-6e24e45490f3-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z\" (UID: \"41f89d55-e428-405c-87b8-6e24e45490f3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z"
Sep 17 22:10:19.940018 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:19.939985    2563 generic.go:356] "Generic (PLEG): container finished" podID="135e3bde-367b-460d-b41c-b32d5a6b6f5a" containerID="68812fea59c62a1f6d2011d23e9ca043efe14784ec288405356fe6af9e32e607" exitCode=2
Sep 17 22:10:19.940203 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:19.940053    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf" event={"ID":"135e3bde-367b-460d-b41c-b32d5a6b6f5a","Type":"ContainerDied","Data":"68812fea59c62a1f6d2011d23e9ca043efe14784ec288405356fe6af9e32e607"}
Sep 17 22:10:20.137712 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:20.137626    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z"
Sep 17 22:10:20.408052 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:20.408016    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z"]
Sep 17 22:10:20.410224 ip-10-0-138-45 kubenswrapper[2563]: W0917 22:10:20.410194    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f89d55_e428_405c_87b8_6e24e45490f3.slice/crio-5a7d9b039e0be0cfb6ed629289b917ef498f5a50c7bbe129a2e21b4b000a7ea1 WatchSource:0}: Error finding container 5a7d9b039e0be0cfb6ed629289b917ef498f5a50c7bbe129a2e21b4b000a7ea1: Status 404 returned error can't find the container with id 5a7d9b039e0be0cfb6ed629289b917ef498f5a50c7bbe129a2e21b4b000a7ea1
Sep 17 22:10:20.946057 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:20.946016    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z" event={"ID":"41f89d55-e428-405c-87b8-6e24e45490f3","Type":"ContainerStarted","Data":"9e176e61a8f50c72aa869c5d70c4072c05c0e635481bc479432fdc5bc39fd0ca"}
Sep 17 22:10:20.946225 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:20.946100    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z" event={"ID":"41f89d55-e428-405c-87b8-6e24e45490f3","Type":"ContainerStarted","Data":"5a7d9b039e0be0cfb6ed629289b917ef498f5a50c7bbe129a2e21b4b000a7ea1"}
Sep 17 22:10:21.630744 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:21.630701    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf" podUID="135e3bde-367b-460d-b41c-b32d5a6b6f5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 22:10:21.631207 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:21.631101    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf" podUID="135e3bde-367b-460d-b41c-b32d5a6b6f5a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:10:24.617787 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:24.617685    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf" podUID="135e3bde-367b-460d-b41c-b32d5a6b6f5a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.24:8643/healthz\": dial tcp 10.133.0.24:8643: connect: connection refused"
Sep 17 22:10:24.965879 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:24.965848    2563 generic.go:356] "Generic (PLEG): container finished" podID="135e3bde-367b-460d-b41c-b32d5a6b6f5a" containerID="cdcd3284294ba9aa4fca43bfdee23f106435d4c45e2d45039672ec9366a55259" exitCode=0
Sep 17 22:10:24.966064 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:24.965924    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf" event={"ID":"135e3bde-367b-460d-b41c-b32d5a6b6f5a","Type":"ContainerDied","Data":"cdcd3284294ba9aa4fca43bfdee23f106435d4c45e2d45039672ec9366a55259"}
Sep 17 22:10:24.967434 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:24.967414    2563 generic.go:356] "Generic (PLEG): container finished" podID="41f89d55-e428-405c-87b8-6e24e45490f3" containerID="9e176e61a8f50c72aa869c5d70c4072c05c0e635481bc479432fdc5bc39fd0ca" exitCode=0
Sep 17 22:10:24.967539 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:24.967450    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z" event={"ID":"41f89d55-e428-405c-87b8-6e24e45490f3","Type":"ContainerDied","Data":"9e176e61a8f50c72aa869c5d70c4072c05c0e635481bc479432fdc5bc39fd0ca"}
Sep 17 22:10:25.974279 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:25.974239    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z" event={"ID":"41f89d55-e428-405c-87b8-6e24e45490f3","Type":"ContainerStarted","Data":"07e886ae0a379cd846dfa44fae67c1d28c664b7973456550b496878204f0e6c9"}
Sep 17 22:10:25.974665 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:25.974314    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z" event={"ID":"41f89d55-e428-405c-87b8-6e24e45490f3","Type":"ContainerStarted","Data":"cd565f14aaf31de103ece9d09e83ed9d3d745ba0645d303a212b430283903f8f"}
Sep 17 22:10:25.974665 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:25.974339    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z" event={"ID":"41f89d55-e428-405c-87b8-6e24e45490f3","Type":"ContainerStarted","Data":"5fc70943feb2029ecb19c7a60e552dcbeb67ea74d5d857686da50e2fe480ff53"}
Sep 17 22:10:25.974665 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:25.974626    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z"
Sep 17 22:10:25.974839 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:25.974801    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z"
Sep 17 22:10:25.976168 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:25.976128    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z" podUID="41f89d55-e428-405c-87b8-6e24e45490f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:5000: connect: connection refused"
Sep 17 22:10:25.996788 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:25.996742    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z" podStartSLOduration=3.195624892 podStartE2EDuration="6.996723735s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.801098843s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:10:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:10:25.994903754 +0000 UTC m=+737.162856740" watchObservedRunningTime="2026-09-17 22:10:25.996723735 +0000 UTC m=+737.164676718"
Sep 17 22:10:26.978976 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:26.978945    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z"
Sep 17 22:10:26.979464 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:26.979085    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z" podUID="41f89d55-e428-405c-87b8-6e24e45490f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:5000: connect: connection refused"
Sep 17 22:10:26.980305 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:26.980280    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z" podUID="41f89d55-e428-405c-87b8-6e24e45490f3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:10:27.983133 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:27.983092    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z" podUID="41f89d55-e428-405c-87b8-6e24e45490f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:5000: connect: connection refused"
Sep 17 22:10:27.983592 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:27.983560    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z" podUID="41f89d55-e428-405c-87b8-6e24e45490f3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:10:29.617467 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:29.617422    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf" podUID="135e3bde-367b-460d-b41c-b32d5a6b6f5a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.24:8643/healthz\": dial tcp 10.133.0.24:8643: connect: connection refused"
Sep 17 22:10:29.617927 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:29.617547    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf"
Sep 17 22:10:31.630640 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:31.630593    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf" podUID="135e3bde-367b-460d-b41c-b32d5a6b6f5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 22:10:31.631034 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:31.630941    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf" podUID="135e3bde-367b-460d-b41c-b32d5a6b6f5a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:10:32.987593 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:32.987562    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z"
Sep 17 22:10:32.988156 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:32.988111    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z" podUID="41f89d55-e428-405c-87b8-6e24e45490f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:5000: connect: connection refused"
Sep 17 22:10:32.988481 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:32.988460    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z" podUID="41f89d55-e428-405c-87b8-6e24e45490f3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:10:34.617272 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:34.617230    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf" podUID="135e3bde-367b-460d-b41c-b32d5a6b6f5a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.24:8643/healthz\": dial tcp 10.133.0.24:8643: connect: connection refused"
Sep 17 22:10:39.617528 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:39.617489    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf" podUID="135e3bde-367b-460d-b41c-b32d5a6b6f5a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.24:8643/healthz\": dial tcp 10.133.0.24:8643: connect: connection refused"
Sep 17 22:10:41.630319 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:41.630262    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf" podUID="135e3bde-367b-460d-b41c-b32d5a6b6f5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 22:10:41.630731 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:41.630459    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf"
Sep 17 22:10:41.630731 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:41.630646    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf" podUID="135e3bde-367b-460d-b41c-b32d5a6b6f5a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:10:41.630731 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:41.630723    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf"
Sep 17 22:10:42.988224 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:42.988178    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z" podUID="41f89d55-e428-405c-87b8-6e24e45490f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:5000: connect: connection refused"
Sep 17 22:10:42.988766 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:42.988616    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z" podUID="41f89d55-e428-405c-87b8-6e24e45490f3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:10:44.617769 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:44.617729    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf" podUID="135e3bde-367b-460d-b41c-b32d5a6b6f5a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.24:8643/healthz\": dial tcp 10.133.0.24:8643: connect: connection refused"
Sep 17 22:10:49.238057 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:49.238033    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf"
Sep 17 22:10:49.314576 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:49.314541    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/135e3bde-367b-460d-b41c-b32d5a6b6f5a-cabundle-cert\") pod \"135e3bde-367b-460d-b41c-b32d5a6b6f5a\" (UID: \"135e3bde-367b-460d-b41c-b32d5a6b6f5a\") "
Sep 17 22:10:49.314756 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:49.314588    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hckqk\" (UniqueName: \"kubernetes.io/projected/135e3bde-367b-460d-b41c-b32d5a6b6f5a-kube-api-access-hckqk\") pod \"135e3bde-367b-460d-b41c-b32d5a6b6f5a\" (UID: \"135e3bde-367b-460d-b41c-b32d5a6b6f5a\") "
Sep 17 22:10:49.314756 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:49.314619    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/135e3bde-367b-460d-b41c-b32d5a6b6f5a-kube-rbac-proxy-sar-config\") pod \"135e3bde-367b-460d-b41c-b32d5a6b6f5a\" (UID: \"135e3bde-367b-460d-b41c-b32d5a6b6f5a\") "
Sep 17 22:10:49.314756 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:49.314664    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/135e3bde-367b-460d-b41c-b32d5a6b6f5a-proxy-tls\") pod \"135e3bde-367b-460d-b41c-b32d5a6b6f5a\" (UID: \"135e3bde-367b-460d-b41c-b32d5a6b6f5a\") "
Sep 17 22:10:49.314914 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:49.314751    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/135e3bde-367b-460d-b41c-b32d5a6b6f5a-kserve-provision-location\") pod \"135e3bde-367b-460d-b41c-b32d5a6b6f5a\" (UID: \"135e3bde-367b-460d-b41c-b32d5a6b6f5a\") "
Sep 17 22:10:49.314914 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:49.314887    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/135e3bde-367b-460d-b41c-b32d5a6b6f5a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "135e3bde-367b-460d-b41c-b32d5a6b6f5a" (UID: "135e3bde-367b-460d-b41c-b32d5a6b6f5a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:10:49.315059 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:49.315015    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/135e3bde-367b-460d-b41c-b32d5a6b6f5a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "135e3bde-367b-460d-b41c-b32d5a6b6f5a" (UID: "135e3bde-367b-460d-b41c-b32d5a6b6f5a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:10:49.315117 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:49.315074    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/135e3bde-367b-460d-b41c-b32d5a6b6f5a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "135e3bde-367b-460d-b41c-b32d5a6b6f5a" (UID: "135e3bde-367b-460d-b41c-b32d5a6b6f5a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:10:49.315117 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:49.315103    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/135e3bde-367b-460d-b41c-b32d5a6b6f5a-cabundle-cert\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:10:49.315219 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:49.315121    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/135e3bde-367b-460d-b41c-b32d5a6b6f5a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:10:49.316770 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:49.316745    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/135e3bde-367b-460d-b41c-b32d5a6b6f5a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "135e3bde-367b-460d-b41c-b32d5a6b6f5a" (UID: "135e3bde-367b-460d-b41c-b32d5a6b6f5a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:10:49.316899 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:49.316786    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/135e3bde-367b-460d-b41c-b32d5a6b6f5a-kube-api-access-hckqk" (OuterVolumeSpecName: "kube-api-access-hckqk") pod "135e3bde-367b-460d-b41c-b32d5a6b6f5a" (UID: "135e3bde-367b-460d-b41c-b32d5a6b6f5a"). InnerVolumeSpecName "kube-api-access-hckqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:10:49.415844 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:49.415808    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/135e3bde-367b-460d-b41c-b32d5a6b6f5a-proxy-tls\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:10:49.415844 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:49.415843    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/135e3bde-367b-460d-b41c-b32d5a6b6f5a-kserve-provision-location\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:10:49.416079 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:49.415854    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hckqk\" (UniqueName: \"kubernetes.io/projected/135e3bde-367b-460d-b41c-b32d5a6b6f5a-kube-api-access-hckqk\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:10:50.075488 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:50.075455    2563 generic.go:356] "Generic (PLEG): container finished" podID="135e3bde-367b-460d-b41c-b32d5a6b6f5a" containerID="1f8adcebf6c8e513db1e399ac91c805bf1a0728310194cfe1eea66936b296193" exitCode=0
Sep 17 22:10:50.075641 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:50.075525    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf"
Sep 17 22:10:50.075641 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:50.075528    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf" event={"ID":"135e3bde-367b-460d-b41c-b32d5a6b6f5a","Type":"ContainerDied","Data":"1f8adcebf6c8e513db1e399ac91c805bf1a0728310194cfe1eea66936b296193"}
Sep 17 22:10:50.075641 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:50.075592    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf" event={"ID":"135e3bde-367b-460d-b41c-b32d5a6b6f5a","Type":"ContainerDied","Data":"b673b999fcc46fc89db52d72d628f6602d817bd49ad0760114769d17406427db"}
Sep 17 22:10:50.075641 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:50.075623    2563 scope.go:117] "RemoveContainer" containerID="1f8adcebf6c8e513db1e399ac91c805bf1a0728310194cfe1eea66936b296193"
Sep 17 22:10:50.084182 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:50.084159    2563 scope.go:117] "RemoveContainer" containerID="68812fea59c62a1f6d2011d23e9ca043efe14784ec288405356fe6af9e32e607"
Sep 17 22:10:50.091359 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:50.091337    2563 scope.go:117] "RemoveContainer" containerID="cdcd3284294ba9aa4fca43bfdee23f106435d4c45e2d45039672ec9366a55259"
Sep 17 22:10:50.097260 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:50.097236    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf"]
Sep 17 22:10:50.098635 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:50.098480    2563 scope.go:117] "RemoveContainer" containerID="cb08040a17d5852a1fe16b8d51c4cfcc3d0cdcc8cb093700ff3f2581870666fc"
Sep 17 22:10:50.101610 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:50.101585    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-56c7449565-5kvcf"]
Sep 17 22:10:50.105619 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:50.105605    2563 scope.go:117] "RemoveContainer" containerID="1f8adcebf6c8e513db1e399ac91c805bf1a0728310194cfe1eea66936b296193"
Sep 17 22:10:50.105876 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:10:50.105859    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f8adcebf6c8e513db1e399ac91c805bf1a0728310194cfe1eea66936b296193\": container with ID starting with 1f8adcebf6c8e513db1e399ac91c805bf1a0728310194cfe1eea66936b296193 not found: ID does not exist" containerID="1f8adcebf6c8e513db1e399ac91c805bf1a0728310194cfe1eea66936b296193"
Sep 17 22:10:50.105925 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:50.105905    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f8adcebf6c8e513db1e399ac91c805bf1a0728310194cfe1eea66936b296193"} err="failed to get container status \"1f8adcebf6c8e513db1e399ac91c805bf1a0728310194cfe1eea66936b296193\": rpc error: code = NotFound desc = could not find container \"1f8adcebf6c8e513db1e399ac91c805bf1a0728310194cfe1eea66936b296193\": container with ID starting with 1f8adcebf6c8e513db1e399ac91c805bf1a0728310194cfe1eea66936b296193 not found: ID does not exist"
Sep 17 22:10:50.105967 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:50.105925    2563 scope.go:117] "RemoveContainer" containerID="68812fea59c62a1f6d2011d23e9ca043efe14784ec288405356fe6af9e32e607"
Sep 17 22:10:50.106197 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:10:50.106179    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68812fea59c62a1f6d2011d23e9ca043efe14784ec288405356fe6af9e32e607\": container with ID starting with 68812fea59c62a1f6d2011d23e9ca043efe14784ec288405356fe6af9e32e607 not found: ID does not exist" containerID="68812fea59c62a1f6d2011d23e9ca043efe14784ec288405356fe6af9e32e607"
Sep 17 22:10:50.106272 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:50.106206    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68812fea59c62a1f6d2011d23e9ca043efe14784ec288405356fe6af9e32e607"} err="failed to get container status \"68812fea59c62a1f6d2011d23e9ca043efe14784ec288405356fe6af9e32e607\": rpc error: code = NotFound desc = could not find container \"68812fea59c62a1f6d2011d23e9ca043efe14784ec288405356fe6af9e32e607\": container with ID starting with 68812fea59c62a1f6d2011d23e9ca043efe14784ec288405356fe6af9e32e607 not found: ID does not exist"
Sep 17 22:10:50.106272 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:50.106230    2563 scope.go:117] "RemoveContainer" containerID="cdcd3284294ba9aa4fca43bfdee23f106435d4c45e2d45039672ec9366a55259"
Sep 17 22:10:50.106491 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:10:50.106474    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdcd3284294ba9aa4fca43bfdee23f106435d4c45e2d45039672ec9366a55259\": container with ID starting with cdcd3284294ba9aa4fca43bfdee23f106435d4c45e2d45039672ec9366a55259 not found: ID does not exist" containerID="cdcd3284294ba9aa4fca43bfdee23f106435d4c45e2d45039672ec9366a55259"
Sep 17 22:10:50.106526 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:50.106497    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdcd3284294ba9aa4fca43bfdee23f106435d4c45e2d45039672ec9366a55259"} err="failed to get container status \"cdcd3284294ba9aa4fca43bfdee23f106435d4c45e2d45039672ec9366a55259\": rpc error: code = NotFound desc = could not find container \"cdcd3284294ba9aa4fca43bfdee23f106435d4c45e2d45039672ec9366a55259\": container with ID starting with cdcd3284294ba9aa4fca43bfdee23f106435d4c45e2d45039672ec9366a55259 not found: ID does not exist"
Sep 17 22:10:50.106526 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:50.106514    2563 scope.go:117] "RemoveContainer" containerID="cb08040a17d5852a1fe16b8d51c4cfcc3d0cdcc8cb093700ff3f2581870666fc"
Sep 17 22:10:50.106711 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:10:50.106689    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb08040a17d5852a1fe16b8d51c4cfcc3d0cdcc8cb093700ff3f2581870666fc\": container with ID starting with cb08040a17d5852a1fe16b8d51c4cfcc3d0cdcc8cb093700ff3f2581870666fc not found: ID does not exist" containerID="cb08040a17d5852a1fe16b8d51c4cfcc3d0cdcc8cb093700ff3f2581870666fc"
Sep 17 22:10:50.106754 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:50.106718    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb08040a17d5852a1fe16b8d51c4cfcc3d0cdcc8cb093700ff3f2581870666fc"} err="failed to get container status \"cb08040a17d5852a1fe16b8d51c4cfcc3d0cdcc8cb093700ff3f2581870666fc\": rpc error: code = NotFound desc = could not find container \"cb08040a17d5852a1fe16b8d51c4cfcc3d0cdcc8cb093700ff3f2581870666fc\": container with ID starting with cb08040a17d5852a1fe16b8d51c4cfcc3d0cdcc8cb093700ff3f2581870666fc not found: ID does not exist"
Sep 17 22:10:50.915318 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:50.915280    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="135e3bde-367b-460d-b41c-b32d5a6b6f5a" path="/var/lib/kubelet/pods/135e3bde-367b-460d-b41c-b32d5a6b6f5a/volumes"
Sep 17 22:10:52.988183 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:52.988114    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z" podUID="41f89d55-e428-405c-87b8-6e24e45490f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:5000: connect: connection refused"
Sep 17 22:10:52.988681 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:10:52.988641    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z" podUID="41f89d55-e428-405c-87b8-6e24e45490f3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:11:02.989059 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:11:02.989008    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z" podUID="41f89d55-e428-405c-87b8-6e24e45490f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:5000: connect: connection refused"
Sep 17 22:11:02.989518 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:11:02.989493    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z" podUID="41f89d55-e428-405c-87b8-6e24e45490f3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:11:12.988461 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:11:12.988414    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z" podUID="41f89d55-e428-405c-87b8-6e24e45490f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:5000: connect: connection refused"
Sep 17 22:11:12.988882 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:11:12.988786    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z" podUID="41f89d55-e428-405c-87b8-6e24e45490f3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:11:22.988754 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:11:22.988658    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z" podUID="41f89d55-e428-405c-87b8-6e24e45490f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:5000: connect: connection refused"
Sep 17 22:11:22.989379 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:11:22.989020    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z" podUID="41f89d55-e428-405c-87b8-6e24e45490f3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:11:32.988767 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:11:32.988726    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z" podUID="41f89d55-e428-405c-87b8-6e24e45490f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:5000: connect: connection refused"
Sep 17 22:11:32.989254 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:11:32.989226    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z" podUID="41f89d55-e428-405c-87b8-6e24e45490f3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:11:42.988715 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:11:42.988685    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z"
Sep 17 22:11:42.989197 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:11:42.988921    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z"
Sep 17 22:11:54.276110 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:11:54.276073    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z"]
Sep 17 22:11:54.276641 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:11:54.276357    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z" podUID="41f89d55-e428-405c-87b8-6e24e45490f3" containerName="kserve-container" containerID="cri-o://5fc70943feb2029ecb19c7a60e552dcbeb67ea74d5d857686da50e2fe480ff53" gracePeriod=30
Sep 17 22:11:54.276641 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:11:54.276503    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z" podUID="41f89d55-e428-405c-87b8-6e24e45490f3" containerName="kube-rbac-proxy" containerID="cri-o://cd565f14aaf31de103ece9d09e83ed9d3d745ba0645d303a212b430283903f8f" gracePeriod=30
Sep 17 22:11:54.276641 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:11:54.276508    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z" podUID="41f89d55-e428-405c-87b8-6e24e45490f3" containerName="agent" containerID="cri-o://07e886ae0a379cd846dfa44fae67c1d28c664b7973456550b496878204f0e6c9" gracePeriod=30
Sep 17 22:11:55.335882 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:11:55.335848    2563 generic.go:356] "Generic (PLEG): container finished" podID="41f89d55-e428-405c-87b8-6e24e45490f3" containerID="cd565f14aaf31de103ece9d09e83ed9d3d745ba0645d303a212b430283903f8f" exitCode=2
Sep 17 22:11:55.336291 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:11:55.335928    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z" event={"ID":"41f89d55-e428-405c-87b8-6e24e45490f3","Type":"ContainerDied","Data":"cd565f14aaf31de103ece9d09e83ed9d3d745ba0645d303a212b430283903f8f"}
Sep 17 22:11:57.983361 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:11:57.983320    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z" podUID="41f89d55-e428-405c-87b8-6e24e45490f3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Sep 17 22:11:59.354818 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:11:59.354731    2563 generic.go:356] "Generic (PLEG): container finished" podID="41f89d55-e428-405c-87b8-6e24e45490f3" containerID="5fc70943feb2029ecb19c7a60e552dcbeb67ea74d5d857686da50e2fe480ff53" exitCode=0
Sep 17 22:11:59.354818 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:11:59.354802    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z" event={"ID":"41f89d55-e428-405c-87b8-6e24e45490f3","Type":"ContainerDied","Data":"5fc70943feb2029ecb19c7a60e552dcbeb67ea74d5d857686da50e2fe480ff53"}
Sep 17 22:12:02.983637 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:02.983594    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z" podUID="41f89d55-e428-405c-87b8-6e24e45490f3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Sep 17 22:12:02.989075 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:02.989033    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z" podUID="41f89d55-e428-405c-87b8-6e24e45490f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:5000: connect: connection refused"
Sep 17 22:12:02.989372 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:02.989349    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z" podUID="41f89d55-e428-405c-87b8-6e24e45490f3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:12:07.983976 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:07.983933    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z" podUID="41f89d55-e428-405c-87b8-6e24e45490f3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Sep 17 22:12:07.984366 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:07.984067    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z"
Sep 17 22:12:12.984024 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:12.983979    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z" podUID="41f89d55-e428-405c-87b8-6e24e45490f3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Sep 17 22:12:12.988556 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:12.988513    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z" podUID="41f89d55-e428-405c-87b8-6e24e45490f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:5000: connect: connection refused"
Sep 17 22:12:12.988862 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:12.988838    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z" podUID="41f89d55-e428-405c-87b8-6e24e45490f3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:12:14.426435 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:14.426403    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg"]
Sep 17 22:12:14.426977 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:14.426838    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="135e3bde-367b-460d-b41c-b32d5a6b6f5a" containerName="kserve-container"
Sep 17 22:12:14.426977 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:14.426855    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="135e3bde-367b-460d-b41c-b32d5a6b6f5a" containerName="kserve-container"
Sep 17 22:12:14.426977 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:14.426882    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="135e3bde-367b-460d-b41c-b32d5a6b6f5a" containerName="storage-initializer"
Sep 17 22:12:14.426977 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:14.426890    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="135e3bde-367b-460d-b41c-b32d5a6b6f5a" containerName="storage-initializer"
Sep 17 22:12:14.426977 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:14.426905    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="135e3bde-367b-460d-b41c-b32d5a6b6f5a" containerName="kube-rbac-proxy"
Sep 17 22:12:14.426977 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:14.426913    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="135e3bde-367b-460d-b41c-b32d5a6b6f5a" containerName="kube-rbac-proxy"
Sep 17 22:12:14.426977 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:14.426924    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="135e3bde-367b-460d-b41c-b32d5a6b6f5a" containerName="agent"
Sep 17 22:12:14.426977 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:14.426931    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="135e3bde-367b-460d-b41c-b32d5a6b6f5a" containerName="agent"
Sep 17 22:12:14.427413 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:14.427010    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="135e3bde-367b-460d-b41c-b32d5a6b6f5a" containerName="agent"
Sep 17 22:12:14.427413 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:14.427024    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="135e3bde-367b-460d-b41c-b32d5a6b6f5a" containerName="kube-rbac-proxy"
Sep 17 22:12:14.427413 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:14.427035    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="135e3bde-367b-460d-b41c-b32d5a6b6f5a" containerName="kserve-container"
Sep 17 22:12:14.429936 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:14.429914    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg"
Sep 17 22:12:14.432158 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:14.432116    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Sep 17 22:12:14.433050 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:14.433033    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Sep 17 22:12:14.444351 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:14.444329    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg"]
Sep 17 22:12:14.543263 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:14.543216    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73970b41-7022-4c55-b7b8-b4b2f02f7287-proxy-tls\") pod \"isvc-logger-predictor-664ccfb57d-b6tvg\" (UID: \"73970b41-7022-4c55-b7b8-b4b2f02f7287\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg"
Sep 17 22:12:14.543263 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:14.543269    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73970b41-7022-4c55-b7b8-b4b2f02f7287-cabundle-cert\") pod \"isvc-logger-predictor-664ccfb57d-b6tvg\" (UID: \"73970b41-7022-4c55-b7b8-b4b2f02f7287\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg"
Sep 17 22:12:14.543528 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:14.543330    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73970b41-7022-4c55-b7b8-b4b2f02f7287-kserve-provision-location\") pod \"isvc-logger-predictor-664ccfb57d-b6tvg\" (UID: \"73970b41-7022-4c55-b7b8-b4b2f02f7287\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg"
Sep 17 22:12:14.543528 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:14.543379    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbtxr\" (UniqueName: \"kubernetes.io/projected/73970b41-7022-4c55-b7b8-b4b2f02f7287-kube-api-access-lbtxr\") pod \"isvc-logger-predictor-664ccfb57d-b6tvg\" (UID: \"73970b41-7022-4c55-b7b8-b4b2f02f7287\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg"
Sep 17 22:12:14.543528 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:14.543400    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73970b41-7022-4c55-b7b8-b4b2f02f7287-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-664ccfb57d-b6tvg\" (UID: \"73970b41-7022-4c55-b7b8-b4b2f02f7287\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg"
Sep 17 22:12:14.644552 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:14.644498    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbtxr\" (UniqueName: \"kubernetes.io/projected/73970b41-7022-4c55-b7b8-b4b2f02f7287-kube-api-access-lbtxr\") pod \"isvc-logger-predictor-664ccfb57d-b6tvg\" (UID: \"73970b41-7022-4c55-b7b8-b4b2f02f7287\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg"
Sep 17 22:12:14.644552 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:14.644555    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73970b41-7022-4c55-b7b8-b4b2f02f7287-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-664ccfb57d-b6tvg\" (UID: \"73970b41-7022-4c55-b7b8-b4b2f02f7287\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg"
Sep 17 22:12:14.644768 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:14.644644    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73970b41-7022-4c55-b7b8-b4b2f02f7287-proxy-tls\") pod \"isvc-logger-predictor-664ccfb57d-b6tvg\" (UID: \"73970b41-7022-4c55-b7b8-b4b2f02f7287\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg"
Sep 17 22:12:14.644768 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:14.644676    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73970b41-7022-4c55-b7b8-b4b2f02f7287-cabundle-cert\") pod \"isvc-logger-predictor-664ccfb57d-b6tvg\" (UID: \"73970b41-7022-4c55-b7b8-b4b2f02f7287\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg"
Sep 17 22:12:14.644768 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:14.644711    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73970b41-7022-4c55-b7b8-b4b2f02f7287-kserve-provision-location\") pod \"isvc-logger-predictor-664ccfb57d-b6tvg\" (UID: \"73970b41-7022-4c55-b7b8-b4b2f02f7287\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg"
Sep 17 22:12:14.644898 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:12:14.644765    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-predictor-serving-cert: secret "isvc-logger-predictor-serving-cert" not found
Sep 17 22:12:14.644898 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:12:14.644837    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73970b41-7022-4c55-b7b8-b4b2f02f7287-proxy-tls podName:73970b41-7022-4c55-b7b8-b4b2f02f7287 nodeName:}" failed. No retries permitted until 2026-09-17 22:12:15.144816096 +0000 UTC m=+846.312769073 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/73970b41-7022-4c55-b7b8-b4b2f02f7287-proxy-tls") pod "isvc-logger-predictor-664ccfb57d-b6tvg" (UID: "73970b41-7022-4c55-b7b8-b4b2f02f7287") : secret "isvc-logger-predictor-serving-cert" not found
Sep 17 22:12:14.645074 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:14.645059    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73970b41-7022-4c55-b7b8-b4b2f02f7287-kserve-provision-location\") pod \"isvc-logger-predictor-664ccfb57d-b6tvg\" (UID: \"73970b41-7022-4c55-b7b8-b4b2f02f7287\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg"
Sep 17 22:12:14.645363 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:14.645345    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73970b41-7022-4c55-b7b8-b4b2f02f7287-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-664ccfb57d-b6tvg\" (UID: \"73970b41-7022-4c55-b7b8-b4b2f02f7287\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg"
Sep 17 22:12:14.645408 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:14.645344    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73970b41-7022-4c55-b7b8-b4b2f02f7287-cabundle-cert\") pod \"isvc-logger-predictor-664ccfb57d-b6tvg\" (UID: \"73970b41-7022-4c55-b7b8-b4b2f02f7287\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg"
Sep 17 22:12:14.654760 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:14.654737    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbtxr\" (UniqueName: \"kubernetes.io/projected/73970b41-7022-4c55-b7b8-b4b2f02f7287-kube-api-access-lbtxr\") pod \"isvc-logger-predictor-664ccfb57d-b6tvg\" (UID: \"73970b41-7022-4c55-b7b8-b4b2f02f7287\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg"
Sep 17 22:12:15.148764 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:15.148708    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73970b41-7022-4c55-b7b8-b4b2f02f7287-proxy-tls\") pod \"isvc-logger-predictor-664ccfb57d-b6tvg\" (UID: \"73970b41-7022-4c55-b7b8-b4b2f02f7287\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg"
Sep 17 22:12:15.151114 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:15.151084    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73970b41-7022-4c55-b7b8-b4b2f02f7287-proxy-tls\") pod \"isvc-logger-predictor-664ccfb57d-b6tvg\" (UID: \"73970b41-7022-4c55-b7b8-b4b2f02f7287\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg"
Sep 17 22:12:15.339409 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:15.339369    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg"
Sep 17 22:12:15.615374 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:15.615350    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg"]
Sep 17 22:12:15.617809 ip-10-0-138-45 kubenswrapper[2563]: W0917 22:12:15.617778    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73970b41_7022_4c55_b7b8_b4b2f02f7287.slice/crio-003c0cbe4588f1b27736d7911dbfd826f55a45295b6b89a0e9062d9572436c10 WatchSource:0}: Error finding container 003c0cbe4588f1b27736d7911dbfd826f55a45295b6b89a0e9062d9572436c10: Status 404 returned error can't find the container with id 003c0cbe4588f1b27736d7911dbfd826f55a45295b6b89a0e9062d9572436c10
Sep 17 22:12:16.421535 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:16.421498    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg" event={"ID":"73970b41-7022-4c55-b7b8-b4b2f02f7287","Type":"ContainerStarted","Data":"ba05b7880ae41ffd62c493e32b274c1072bfe2fdd2be0bae7c736b9843c16dab"}
Sep 17 22:12:16.421727 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:16.421567    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg" event={"ID":"73970b41-7022-4c55-b7b8-b4b2f02f7287","Type":"ContainerStarted","Data":"003c0cbe4588f1b27736d7911dbfd826f55a45295b6b89a0e9062d9572436c10"}
Sep 17 22:12:17.983782 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:17.983738    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z" podUID="41f89d55-e428-405c-87b8-6e24e45490f3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Sep 17 22:12:20.443569 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:20.443535    2563 generic.go:356] "Generic (PLEG): container finished" podID="73970b41-7022-4c55-b7b8-b4b2f02f7287" containerID="ba05b7880ae41ffd62c493e32b274c1072bfe2fdd2be0bae7c736b9843c16dab" exitCode=0
Sep 17 22:12:20.443960 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:20.443607    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg" event={"ID":"73970b41-7022-4c55-b7b8-b4b2f02f7287","Type":"ContainerDied","Data":"ba05b7880ae41ffd62c493e32b274c1072bfe2fdd2be0bae7c736b9843c16dab"}
Sep 17 22:12:21.449768 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:21.449734    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg" event={"ID":"73970b41-7022-4c55-b7b8-b4b2f02f7287","Type":"ContainerStarted","Data":"2abb59b16b1c0f48cd254c9f3c04590fb6fea5714854b3ac6d5e81d59f6c8202"}
Sep 17 22:12:21.450319 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:21.449809    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg" event={"ID":"73970b41-7022-4c55-b7b8-b4b2f02f7287","Type":"ContainerStarted","Data":"abedd659308ddc7c06f28db4e023a5bcce19f7d495169135777d8ea559fadc8e"}
Sep 17 22:12:21.450319 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:21.449844    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg" event={"ID":"73970b41-7022-4c55-b7b8-b4b2f02f7287","Type":"ContainerStarted","Data":"85cae30350c38c1502b6eba6bcd46117fa6c3f48b72118f4f8befb856e85d186"}
Sep 17 22:12:21.450319 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:21.450099    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg"
Sep 17 22:12:21.471659 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:21.471607    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg" podStartSLOduration=3.678467281 podStartE2EDuration="7.471590427s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.793123146s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:12:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:12:21.469567588 +0000 UTC m=+852.637520596" watchObservedRunningTime="2026-09-17 22:12:21.471590427 +0000 UTC m=+852.639543409"
Sep 17 22:12:22.454804 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:22.454773    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg"
Sep 17 22:12:22.455201 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:22.454880    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg"
Sep 17 22:12:22.456339 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:22.456300    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg" podUID="73970b41-7022-4c55-b7b8-b4b2f02f7287" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 22:12:22.457023 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:22.456999    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg" podUID="73970b41-7022-4c55-b7b8-b4b2f02f7287" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:12:22.983623 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:22.983581    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z" podUID="41f89d55-e428-405c-87b8-6e24e45490f3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Sep 17 22:12:22.988957 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:22.988926    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z" podUID="41f89d55-e428-405c-87b8-6e24e45490f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:5000: connect: connection refused"
Sep 17 22:12:22.989095 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:22.989057    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z"
Sep 17 22:12:22.989283 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:22.989260    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z" podUID="41f89d55-e428-405c-87b8-6e24e45490f3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:12:22.989391 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:22.989358    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z"
Sep 17 22:12:23.458959 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:23.458917    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg" podUID="73970b41-7022-4c55-b7b8-b4b2f02f7287" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 22:12:23.459369 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:23.459319    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg" podUID="73970b41-7022-4c55-b7b8-b4b2f02f7287" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:12:24.421255 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:24.421228    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z"
Sep 17 22:12:24.465593 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:24.465561    2563 generic.go:356] "Generic (PLEG): container finished" podID="41f89d55-e428-405c-87b8-6e24e45490f3" containerID="07e886ae0a379cd846dfa44fae67c1d28c664b7973456550b496878204f0e6c9" exitCode=0
Sep 17 22:12:24.466000 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:24.465639    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z" event={"ID":"41f89d55-e428-405c-87b8-6e24e45490f3","Type":"ContainerDied","Data":"07e886ae0a379cd846dfa44fae67c1d28c664b7973456550b496878204f0e6c9"}
Sep 17 22:12:24.466000 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:24.465653    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z"
Sep 17 22:12:24.466000 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:24.465715    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z" event={"ID":"41f89d55-e428-405c-87b8-6e24e45490f3","Type":"ContainerDied","Data":"5a7d9b039e0be0cfb6ed629289b917ef498f5a50c7bbe129a2e21b4b000a7ea1"}
Sep 17 22:12:24.466000 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:24.465752    2563 scope.go:117] "RemoveContainer" containerID="07e886ae0a379cd846dfa44fae67c1d28c664b7973456550b496878204f0e6c9"
Sep 17 22:12:24.477729 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:24.477697    2563 scope.go:117] "RemoveContainer" containerID="cd565f14aaf31de103ece9d09e83ed9d3d745ba0645d303a212b430283903f8f"
Sep 17 22:12:24.485313 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:24.485299    2563 scope.go:117] "RemoveContainer" containerID="5fc70943feb2029ecb19c7a60e552dcbeb67ea74d5d857686da50e2fe480ff53"
Sep 17 22:12:24.492225 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:24.492205    2563 scope.go:117] "RemoveContainer" containerID="9e176e61a8f50c72aa869c5d70c4072c05c0e635481bc479432fdc5bc39fd0ca"
Sep 17 22:12:24.499269 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:24.499249    2563 scope.go:117] "RemoveContainer" containerID="07e886ae0a379cd846dfa44fae67c1d28c664b7973456550b496878204f0e6c9"
Sep 17 22:12:24.499515 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:12:24.499494    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07e886ae0a379cd846dfa44fae67c1d28c664b7973456550b496878204f0e6c9\": container with ID starting with 07e886ae0a379cd846dfa44fae67c1d28c664b7973456550b496878204f0e6c9 not found: ID does not exist" containerID="07e886ae0a379cd846dfa44fae67c1d28c664b7973456550b496878204f0e6c9"
Sep 17 22:12:24.499561 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:24.499523    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07e886ae0a379cd846dfa44fae67c1d28c664b7973456550b496878204f0e6c9"} err="failed to get container status \"07e886ae0a379cd846dfa44fae67c1d28c664b7973456550b496878204f0e6c9\": rpc error: code = NotFound desc = could not find container \"07e886ae0a379cd846dfa44fae67c1d28c664b7973456550b496878204f0e6c9\": container with ID starting with 07e886ae0a379cd846dfa44fae67c1d28c664b7973456550b496878204f0e6c9 not found: ID does not exist"
Sep 17 22:12:24.499561 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:24.499540    2563 scope.go:117] "RemoveContainer" containerID="cd565f14aaf31de103ece9d09e83ed9d3d745ba0645d303a212b430283903f8f"
Sep 17 22:12:24.499796 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:12:24.499778    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd565f14aaf31de103ece9d09e83ed9d3d745ba0645d303a212b430283903f8f\": container with ID starting with cd565f14aaf31de103ece9d09e83ed9d3d745ba0645d303a212b430283903f8f not found: ID does not exist" containerID="cd565f14aaf31de103ece9d09e83ed9d3d745ba0645d303a212b430283903f8f"
Sep 17 22:12:24.499839 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:24.499802    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd565f14aaf31de103ece9d09e83ed9d3d745ba0645d303a212b430283903f8f"} err="failed to get container status \"cd565f14aaf31de103ece9d09e83ed9d3d745ba0645d303a212b430283903f8f\": rpc error: code = NotFound desc = could not find container \"cd565f14aaf31de103ece9d09e83ed9d3d745ba0645d303a212b430283903f8f\": container with ID starting with cd565f14aaf31de103ece9d09e83ed9d3d745ba0645d303a212b430283903f8f not found: ID does not exist"
Sep 17 22:12:24.499839 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:24.499818    2563 scope.go:117] "RemoveContainer" containerID="5fc70943feb2029ecb19c7a60e552dcbeb67ea74d5d857686da50e2fe480ff53"
Sep 17 22:12:24.500042 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:12:24.500024    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fc70943feb2029ecb19c7a60e552dcbeb67ea74d5d857686da50e2fe480ff53\": container with ID starting with 5fc70943feb2029ecb19c7a60e552dcbeb67ea74d5d857686da50e2fe480ff53 not found: ID does not exist" containerID="5fc70943feb2029ecb19c7a60e552dcbeb67ea74d5d857686da50e2fe480ff53"
Sep 17 22:12:24.500089 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:24.500048    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fc70943feb2029ecb19c7a60e552dcbeb67ea74d5d857686da50e2fe480ff53"} err="failed to get container status \"5fc70943feb2029ecb19c7a60e552dcbeb67ea74d5d857686da50e2fe480ff53\": rpc error: code = NotFound desc = could not find container \"5fc70943feb2029ecb19c7a60e552dcbeb67ea74d5d857686da50e2fe480ff53\": container with ID starting with 5fc70943feb2029ecb19c7a60e552dcbeb67ea74d5d857686da50e2fe480ff53 not found: ID does not exist"
Sep 17 22:12:24.500089 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:24.500062    2563 scope.go:117] "RemoveContainer" containerID="9e176e61a8f50c72aa869c5d70c4072c05c0e635481bc479432fdc5bc39fd0ca"
Sep 17 22:12:24.500301 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:12:24.500285    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e176e61a8f50c72aa869c5d70c4072c05c0e635481bc479432fdc5bc39fd0ca\": container with ID starting with 9e176e61a8f50c72aa869c5d70c4072c05c0e635481bc479432fdc5bc39fd0ca not found: ID does not exist" containerID="9e176e61a8f50c72aa869c5d70c4072c05c0e635481bc479432fdc5bc39fd0ca"
Sep 17 22:12:24.500345 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:24.500305    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e176e61a8f50c72aa869c5d70c4072c05c0e635481bc479432fdc5bc39fd0ca"} err="failed to get container status \"9e176e61a8f50c72aa869c5d70c4072c05c0e635481bc479432fdc5bc39fd0ca\": rpc error: code = NotFound desc = could not find container \"9e176e61a8f50c72aa869c5d70c4072c05c0e635481bc479432fdc5bc39fd0ca\": container with ID starting with 9e176e61a8f50c72aa869c5d70c4072c05c0e635481bc479432fdc5bc39fd0ca not found: ID does not exist"
Sep 17 22:12:24.547289 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:24.547221    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41f89d55-e428-405c-87b8-6e24e45490f3-kserve-provision-location\") pod \"41f89d55-e428-405c-87b8-6e24e45490f3\" (UID: \"41f89d55-e428-405c-87b8-6e24e45490f3\") "
Sep 17 22:12:24.547289 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:24.547260    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41f89d55-e428-405c-87b8-6e24e45490f3-proxy-tls\") pod \"41f89d55-e428-405c-87b8-6e24e45490f3\" (UID: \"41f89d55-e428-405c-87b8-6e24e45490f3\") "
Sep 17 22:12:24.547289 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:24.547279    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k8kvr\" (UniqueName: \"kubernetes.io/projected/41f89d55-e428-405c-87b8-6e24e45490f3-kube-api-access-k8kvr\") pod \"41f89d55-e428-405c-87b8-6e24e45490f3\" (UID: \"41f89d55-e428-405c-87b8-6e24e45490f3\") "
Sep 17 22:12:24.547532 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:24.547324    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41f89d55-e428-405c-87b8-6e24e45490f3-kube-rbac-proxy-sar-config\") pod \"41f89d55-e428-405c-87b8-6e24e45490f3\" (UID: \"41f89d55-e428-405c-87b8-6e24e45490f3\") "
Sep 17 22:12:24.547532 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:24.547424    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41f89d55-e428-405c-87b8-6e24e45490f3-cabundle-cert\") pod \"41f89d55-e428-405c-87b8-6e24e45490f3\" (UID: \"41f89d55-e428-405c-87b8-6e24e45490f3\") "
Sep 17 22:12:24.547642 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:24.547542    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41f89d55-e428-405c-87b8-6e24e45490f3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "41f89d55-e428-405c-87b8-6e24e45490f3" (UID: "41f89d55-e428-405c-87b8-6e24e45490f3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:12:24.547739 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:24.547651    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41f89d55-e428-405c-87b8-6e24e45490f3-kserve-provision-location\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:12:24.547777 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:24.547731    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41f89d55-e428-405c-87b8-6e24e45490f3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "41f89d55-e428-405c-87b8-6e24e45490f3" (UID: "41f89d55-e428-405c-87b8-6e24e45490f3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:12:24.547824 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:24.547804    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41f89d55-e428-405c-87b8-6e24e45490f3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "41f89d55-e428-405c-87b8-6e24e45490f3" (UID: "41f89d55-e428-405c-87b8-6e24e45490f3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:12:24.549374 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:24.549350    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41f89d55-e428-405c-87b8-6e24e45490f3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "41f89d55-e428-405c-87b8-6e24e45490f3" (UID: "41f89d55-e428-405c-87b8-6e24e45490f3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:12:24.549535 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:24.549515    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41f89d55-e428-405c-87b8-6e24e45490f3-kube-api-access-k8kvr" (OuterVolumeSpecName: "kube-api-access-k8kvr") pod "41f89d55-e428-405c-87b8-6e24e45490f3" (UID: "41f89d55-e428-405c-87b8-6e24e45490f3"). InnerVolumeSpecName "kube-api-access-k8kvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:12:24.648393 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:24.648358    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41f89d55-e428-405c-87b8-6e24e45490f3-cabundle-cert\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:12:24.648393 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:24.648386    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41f89d55-e428-405c-87b8-6e24e45490f3-proxy-tls\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:12:24.648393 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:24.648398    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8kvr\" (UniqueName: \"kubernetes.io/projected/41f89d55-e428-405c-87b8-6e24e45490f3-kube-api-access-k8kvr\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:12:24.648636 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:24.648413    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41f89d55-e428-405c-87b8-6e24e45490f3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:12:24.790763 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:24.790731    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z"]
Sep 17 22:12:24.795155 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:24.795118    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5f6888756c-8494z"]
Sep 17 22:12:24.913882 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:24.913852    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41f89d55-e428-405c-87b8-6e24e45490f3" path="/var/lib/kubelet/pods/41f89d55-e428-405c-87b8-6e24e45490f3/volumes"
Sep 17 22:12:28.464243 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:28.464217    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg"
Sep 17 22:12:28.464780 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:28.464758    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg" podUID="73970b41-7022-4c55-b7b8-b4b2f02f7287" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 22:12:28.465111 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:28.465083    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg" podUID="73970b41-7022-4c55-b7b8-b4b2f02f7287" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:12:38.465614 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:38.465568    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg" podUID="73970b41-7022-4c55-b7b8-b4b2f02f7287" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 22:12:38.466065 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:38.465985    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg" podUID="73970b41-7022-4c55-b7b8-b4b2f02f7287" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:12:48.465630 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:48.465586    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg" podUID="73970b41-7022-4c55-b7b8-b4b2f02f7287" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 22:12:48.466203 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:48.466175    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg" podUID="73970b41-7022-4c55-b7b8-b4b2f02f7287" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:12:58.465110 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:58.465012    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg" podUID="73970b41-7022-4c55-b7b8-b4b2f02f7287" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 22:12:58.465555 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:12:58.465531    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg" podUID="73970b41-7022-4c55-b7b8-b4b2f02f7287" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:13:08.465037 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:13:08.464995    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg" podUID="73970b41-7022-4c55-b7b8-b4b2f02f7287" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 22:13:08.465520 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:13:08.465486    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg" podUID="73970b41-7022-4c55-b7b8-b4b2f02f7287" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:13:10.883024 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:13:10.882985    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nrkd5_34b8ef2b-97ba-4562-87ee-8da0cb4604c7/ovn-acl-logging/0.log"
Sep 17 22:13:10.886785 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:13:10.886764    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nrkd5_34b8ef2b-97ba-4562-87ee-8da0cb4604c7/ovn-acl-logging/0.log"
Sep 17 22:13:18.464702 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:13:18.464657    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg" podUID="73970b41-7022-4c55-b7b8-b4b2f02f7287" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 22:13:18.465117 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:13:18.464990    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg" podUID="73970b41-7022-4c55-b7b8-b4b2f02f7287" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:13:28.465040 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:13:28.464996    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg" podUID="73970b41-7022-4c55-b7b8-b4b2f02f7287" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 22:13:28.465507 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:13:28.465486    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg" podUID="73970b41-7022-4c55-b7b8-b4b2f02f7287" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:13:35.909707 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:13:35.909676    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg"
Sep 17 22:13:35.910165 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:13:35.909997    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg"
Sep 17 22:13:49.584843 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:13:49.584807    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg"]
Sep 17 22:13:49.587420 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:13:49.585112    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg" podUID="73970b41-7022-4c55-b7b8-b4b2f02f7287" containerName="kserve-container" containerID="cri-o://85cae30350c38c1502b6eba6bcd46117fa6c3f48b72118f4f8befb856e85d186" gracePeriod=30
Sep 17 22:13:49.587420 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:13:49.585158    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg" podUID="73970b41-7022-4c55-b7b8-b4b2f02f7287" containerName="kube-rbac-proxy" containerID="cri-o://abedd659308ddc7c06f28db4e023a5bcce19f7d495169135777d8ea559fadc8e" gracePeriod=30
Sep 17 22:13:49.587420 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:13:49.585191    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg" podUID="73970b41-7022-4c55-b7b8-b4b2f02f7287" containerName="agent" containerID="cri-o://2abb59b16b1c0f48cd254c9f3c04590fb6fea5714854b3ac6d5e81d59f6c8202" gracePeriod=30
Sep 17 22:13:49.824367 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:13:49.824333    2563 generic.go:356] "Generic (PLEG): container finished" podID="73970b41-7022-4c55-b7b8-b4b2f02f7287" containerID="abedd659308ddc7c06f28db4e023a5bcce19f7d495169135777d8ea559fadc8e" exitCode=2
Sep 17 22:13:49.824545 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:13:49.824411    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg" event={"ID":"73970b41-7022-4c55-b7b8-b4b2f02f7287","Type":"ContainerDied","Data":"abedd659308ddc7c06f28db4e023a5bcce19f7d495169135777d8ea559fadc8e"}
Sep 17 22:13:53.460252 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:13:53.460200    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg" podUID="73970b41-7022-4c55-b7b8-b4b2f02f7287" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Sep 17 22:13:54.847656 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:13:54.847623    2563 generic.go:356] "Generic (PLEG): container finished" podID="73970b41-7022-4c55-b7b8-b4b2f02f7287" containerID="85cae30350c38c1502b6eba6bcd46117fa6c3f48b72118f4f8befb856e85d186" exitCode=0
Sep 17 22:13:54.848024 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:13:54.847695    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg" event={"ID":"73970b41-7022-4c55-b7b8-b4b2f02f7287","Type":"ContainerDied","Data":"85cae30350c38c1502b6eba6bcd46117fa6c3f48b72118f4f8befb856e85d186"}
Sep 17 22:13:55.909670 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:13:55.909622    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg" podUID="73970b41-7022-4c55-b7b8-b4b2f02f7287" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 22:13:55.910089 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:13:55.909899    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg" podUID="73970b41-7022-4c55-b7b8-b4b2f02f7287" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:13:58.459835 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:13:58.459792    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg" podUID="73970b41-7022-4c55-b7b8-b4b2f02f7287" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Sep 17 22:14:03.459465 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:03.459414    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg" podUID="73970b41-7022-4c55-b7b8-b4b2f02f7287" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Sep 17 22:14:03.459851 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:03.459567    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg"
Sep 17 22:14:05.909157 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:05.909103    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg" podUID="73970b41-7022-4c55-b7b8-b4b2f02f7287" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 22:14:05.909556 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:05.909476    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg" podUID="73970b41-7022-4c55-b7b8-b4b2f02f7287" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:14:08.459879 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:08.459836    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg" podUID="73970b41-7022-4c55-b7b8-b4b2f02f7287" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Sep 17 22:14:13.459487 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:13.459448    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg" podUID="73970b41-7022-4c55-b7b8-b4b2f02f7287" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Sep 17 22:14:15.910069 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:15.910013    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg" podUID="73970b41-7022-4c55-b7b8-b4b2f02f7287" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 22:14:15.910716 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:15.910239    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg"
Sep 17 22:14:15.910716 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:15.910352    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg" podUID="73970b41-7022-4c55-b7b8-b4b2f02f7287" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:14:15.910716 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:15.910480    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg"
Sep 17 22:14:16.646964 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:16.646935    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-549r6"]
Sep 17 22:14:16.647265 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:16.647252    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41f89d55-e428-405c-87b8-6e24e45490f3" containerName="kube-rbac-proxy"
Sep 17 22:14:16.647308 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:16.647267    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="41f89d55-e428-405c-87b8-6e24e45490f3" containerName="kube-rbac-proxy"
Sep 17 22:14:16.647308 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:16.647277    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41f89d55-e428-405c-87b8-6e24e45490f3" containerName="kserve-container"
Sep 17 22:14:16.647308 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:16.647282    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="41f89d55-e428-405c-87b8-6e24e45490f3" containerName="kserve-container"
Sep 17 22:14:16.647308 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:16.647288    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41f89d55-e428-405c-87b8-6e24e45490f3" containerName="agent"
Sep 17 22:14:16.647308 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:16.647294    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="41f89d55-e428-405c-87b8-6e24e45490f3" containerName="agent"
Sep 17 22:14:16.647308 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:16.647307    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41f89d55-e428-405c-87b8-6e24e45490f3" containerName="storage-initializer"
Sep 17 22:14:16.647491 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:16.647313    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="41f89d55-e428-405c-87b8-6e24e45490f3" containerName="storage-initializer"
Sep 17 22:14:16.647491 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:16.647361    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="41f89d55-e428-405c-87b8-6e24e45490f3" containerName="kserve-container"
Sep 17 22:14:16.647491 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:16.647368    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="41f89d55-e428-405c-87b8-6e24e45490f3" containerName="agent"
Sep 17 22:14:16.647491 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:16.647376    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="41f89d55-e428-405c-87b8-6e24e45490f3" containerName="kube-rbac-proxy"
Sep 17 22:14:16.650386 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:16.650372    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-549r6"
Sep 17 22:14:16.653301 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:16.653283    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-predictor-serving-cert\""
Sep 17 22:14:16.653426 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:16.653352    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\""
Sep 17 22:14:16.660633 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:16.660611    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-549r6"]
Sep 17 22:14:16.693200 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:16.693180    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9bbe68e7-c0bd-479a-b09f-8d59fb019f37-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-549r6\" (UID: \"9bbe68e7-c0bd-479a-b09f-8d59fb019f37\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-549r6"
Sep 17 22:14:16.693284 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:16.693210    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9bbe68e7-c0bd-479a-b09f-8d59fb019f37-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-549r6\" (UID: \"9bbe68e7-c0bd-479a-b09f-8d59fb019f37\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-549r6"
Sep 17 22:14:16.693284 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:16.693234    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlhj6\" (UniqueName: \"kubernetes.io/projected/9bbe68e7-c0bd-479a-b09f-8d59fb019f37-kube-api-access-dlhj6\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-549r6\" (UID: \"9bbe68e7-c0bd-479a-b09f-8d59fb019f37\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-549r6"
Sep 17 22:14:16.693284 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:16.693260    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9bbe68e7-c0bd-479a-b09f-8d59fb019f37-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-549r6\" (UID: \"9bbe68e7-c0bd-479a-b09f-8d59fb019f37\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-549r6"
Sep 17 22:14:16.693388 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:16.693313    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9bbe68e7-c0bd-479a-b09f-8d59fb019f37-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-549r6\" (UID: \"9bbe68e7-c0bd-479a-b09f-8d59fb019f37\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-549r6"
Sep 17 22:14:16.793837 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:16.793813    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9bbe68e7-c0bd-479a-b09f-8d59fb019f37-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-549r6\" (UID: \"9bbe68e7-c0bd-479a-b09f-8d59fb019f37\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-549r6"
Sep 17 22:14:16.793933 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:16.793843    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9bbe68e7-c0bd-479a-b09f-8d59fb019f37-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-549r6\" (UID: \"9bbe68e7-c0bd-479a-b09f-8d59fb019f37\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-549r6"
Sep 17 22:14:16.793933 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:16.793864    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlhj6\" (UniqueName: \"kubernetes.io/projected/9bbe68e7-c0bd-479a-b09f-8d59fb019f37-kube-api-access-dlhj6\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-549r6\" (UID: \"9bbe68e7-c0bd-479a-b09f-8d59fb019f37\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-549r6"
Sep 17 22:14:16.793933 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:16.793884    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9bbe68e7-c0bd-479a-b09f-8d59fb019f37-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-549r6\" (UID: \"9bbe68e7-c0bd-479a-b09f-8d59fb019f37\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-549r6"
Sep 17 22:14:16.794097 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:16.793934    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9bbe68e7-c0bd-479a-b09f-8d59fb019f37-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-549r6\" (UID: \"9bbe68e7-c0bd-479a-b09f-8d59fb019f37\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-549r6"
Sep 17 22:14:16.794097 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:14:16.794039    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-serving-cert: secret "isvc-predictive-sklearn-v2-predictor-serving-cert" not found
Sep 17 22:14:16.794228 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:14:16.794110    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bbe68e7-c0bd-479a-b09f-8d59fb019f37-proxy-tls podName:9bbe68e7-c0bd-479a-b09f-8d59fb019f37 nodeName:}" failed. No retries permitted until 2026-09-17 22:14:17.294083086 +0000 UTC m=+968.462036048 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9bbe68e7-c0bd-479a-b09f-8d59fb019f37-proxy-tls") pod "isvc-predictive-sklearn-v2-predictor-745f9c8c9f-549r6" (UID: "9bbe68e7-c0bd-479a-b09f-8d59fb019f37") : secret "isvc-predictive-sklearn-v2-predictor-serving-cert" not found
Sep 17 22:14:16.794307 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:16.794288    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9bbe68e7-c0bd-479a-b09f-8d59fb019f37-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-549r6\" (UID: \"9bbe68e7-c0bd-479a-b09f-8d59fb019f37\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-549r6"
Sep 17 22:14:16.794434 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:16.794418    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9bbe68e7-c0bd-479a-b09f-8d59fb019f37-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-549r6\" (UID: \"9bbe68e7-c0bd-479a-b09f-8d59fb019f37\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-549r6"
Sep 17 22:14:16.794496 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:16.794482    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9bbe68e7-c0bd-479a-b09f-8d59fb019f37-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-549r6\" (UID: \"9bbe68e7-c0bd-479a-b09f-8d59fb019f37\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-549r6"
Sep 17 22:14:16.803532 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:16.803506    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlhj6\" (UniqueName: \"kubernetes.io/projected/9bbe68e7-c0bd-479a-b09f-8d59fb019f37-kube-api-access-dlhj6\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-549r6\" (UID: \"9bbe68e7-c0bd-479a-b09f-8d59fb019f37\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-549r6"
Sep 17 22:14:17.297878 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:17.297846    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9bbe68e7-c0bd-479a-b09f-8d59fb019f37-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-549r6\" (UID: \"9bbe68e7-c0bd-479a-b09f-8d59fb019f37\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-549r6"
Sep 17 22:14:17.300130 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:17.300108    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9bbe68e7-c0bd-479a-b09f-8d59fb019f37-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-549r6\" (UID: \"9bbe68e7-c0bd-479a-b09f-8d59fb019f37\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-549r6"
Sep 17 22:14:17.561492 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:17.561406    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-549r6"
Sep 17 22:14:17.828390 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:17.828365    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-549r6"]
Sep 17 22:14:17.830998 ip-10-0-138-45 kubenswrapper[2563]: W0917 22:14:17.830954    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bbe68e7_c0bd_479a_b09f_8d59fb019f37.slice/crio-64068691522ea1acf92a7f1800882d09aef75fb9e8d4106ba7ecf90e6c60c37b WatchSource:0}: Error finding container 64068691522ea1acf92a7f1800882d09aef75fb9e8d4106ba7ecf90e6c60c37b: Status 404 returned error can't find the container with id 64068691522ea1acf92a7f1800882d09aef75fb9e8d4106ba7ecf90e6c60c37b
Sep 17 22:14:17.939535 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:17.939499    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-549r6" event={"ID":"9bbe68e7-c0bd-479a-b09f-8d59fb019f37","Type":"ContainerStarted","Data":"8fc498b73df14b37622f61bb6980cf24a31f3b3d367e0392df57be4a0a217885"}
Sep 17 22:14:17.939643 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:17.939578    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-549r6" event={"ID":"9bbe68e7-c0bd-479a-b09f-8d59fb019f37","Type":"ContainerStarted","Data":"64068691522ea1acf92a7f1800882d09aef75fb9e8d4106ba7ecf90e6c60c37b"}
Sep 17 22:14:18.459338 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:18.459297    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg" podUID="73970b41-7022-4c55-b7b8-b4b2f02f7287" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Sep 17 22:14:19.730405 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:19.730381    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg"
Sep 17 22:14:19.814653 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:19.814625    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73970b41-7022-4c55-b7b8-b4b2f02f7287-kserve-provision-location\") pod \"73970b41-7022-4c55-b7b8-b4b2f02f7287\" (UID: \"73970b41-7022-4c55-b7b8-b4b2f02f7287\") "
Sep 17 22:14:19.814779 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:19.814669    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73970b41-7022-4c55-b7b8-b4b2f02f7287-cabundle-cert\") pod \"73970b41-7022-4c55-b7b8-b4b2f02f7287\" (UID: \"73970b41-7022-4c55-b7b8-b4b2f02f7287\") "
Sep 17 22:14:19.814779 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:19.814718    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73970b41-7022-4c55-b7b8-b4b2f02f7287-proxy-tls\") pod \"73970b41-7022-4c55-b7b8-b4b2f02f7287\" (UID: \"73970b41-7022-4c55-b7b8-b4b2f02f7287\") "
Sep 17 22:14:19.814779 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:19.814736    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73970b41-7022-4c55-b7b8-b4b2f02f7287-kube-rbac-proxy-sar-config\") pod \"73970b41-7022-4c55-b7b8-b4b2f02f7287\" (UID: \"73970b41-7022-4c55-b7b8-b4b2f02f7287\") "
Sep 17 22:14:19.814895 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:19.814838    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lbtxr\" (UniqueName: \"kubernetes.io/projected/73970b41-7022-4c55-b7b8-b4b2f02f7287-kube-api-access-lbtxr\") pod \"73970b41-7022-4c55-b7b8-b4b2f02f7287\" (UID: \"73970b41-7022-4c55-b7b8-b4b2f02f7287\") "
Sep 17 22:14:19.814939 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:19.814914    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73970b41-7022-4c55-b7b8-b4b2f02f7287-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "73970b41-7022-4c55-b7b8-b4b2f02f7287" (UID: "73970b41-7022-4c55-b7b8-b4b2f02f7287"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:14:19.815069 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:19.815054    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73970b41-7022-4c55-b7b8-b4b2f02f7287-kserve-provision-location\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:14:19.815177 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:19.815124    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73970b41-7022-4c55-b7b8-b4b2f02f7287-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "73970b41-7022-4c55-b7b8-b4b2f02f7287" (UID: "73970b41-7022-4c55-b7b8-b4b2f02f7287"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:14:19.815177 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:19.815129    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73970b41-7022-4c55-b7b8-b4b2f02f7287-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "73970b41-7022-4c55-b7b8-b4b2f02f7287" (UID: "73970b41-7022-4c55-b7b8-b4b2f02f7287"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:14:19.816865 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:19.816839    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73970b41-7022-4c55-b7b8-b4b2f02f7287-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "73970b41-7022-4c55-b7b8-b4b2f02f7287" (UID: "73970b41-7022-4c55-b7b8-b4b2f02f7287"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:14:19.816952 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:19.816895    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73970b41-7022-4c55-b7b8-b4b2f02f7287-kube-api-access-lbtxr" (OuterVolumeSpecName: "kube-api-access-lbtxr") pod "73970b41-7022-4c55-b7b8-b4b2f02f7287" (UID: "73970b41-7022-4c55-b7b8-b4b2f02f7287"). InnerVolumeSpecName "kube-api-access-lbtxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:14:19.915940 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:19.915922    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbtxr\" (UniqueName: \"kubernetes.io/projected/73970b41-7022-4c55-b7b8-b4b2f02f7287-kube-api-access-lbtxr\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:14:19.916030 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:19.915940    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73970b41-7022-4c55-b7b8-b4b2f02f7287-cabundle-cert\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:14:19.916030 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:19.915951    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73970b41-7022-4c55-b7b8-b4b2f02f7287-proxy-tls\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:14:19.916030 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:19.915960    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73970b41-7022-4c55-b7b8-b4b2f02f7287-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:14:19.948253 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:19.948223    2563 generic.go:356] "Generic (PLEG): container finished" podID="73970b41-7022-4c55-b7b8-b4b2f02f7287" containerID="2abb59b16b1c0f48cd254c9f3c04590fb6fea5714854b3ac6d5e81d59f6c8202" exitCode=0
Sep 17 22:14:19.948339 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:19.948263    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg" event={"ID":"73970b41-7022-4c55-b7b8-b4b2f02f7287","Type":"ContainerDied","Data":"2abb59b16b1c0f48cd254c9f3c04590fb6fea5714854b3ac6d5e81d59f6c8202"}
Sep 17 22:14:19.948339 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:19.948295    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg"
Sep 17 22:14:19.948339 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:19.948325    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg" event={"ID":"73970b41-7022-4c55-b7b8-b4b2f02f7287","Type":"ContainerDied","Data":"003c0cbe4588f1b27736d7911dbfd826f55a45295b6b89a0e9062d9572436c10"}
Sep 17 22:14:19.948440 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:19.948371    2563 scope.go:117] "RemoveContainer" containerID="2abb59b16b1c0f48cd254c9f3c04590fb6fea5714854b3ac6d5e81d59f6c8202"
Sep 17 22:14:19.957287 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:19.957161    2563 scope.go:117] "RemoveContainer" containerID="abedd659308ddc7c06f28db4e023a5bcce19f7d495169135777d8ea559fadc8e"
Sep 17 22:14:19.964202 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:19.964180    2563 scope.go:117] "RemoveContainer" containerID="85cae30350c38c1502b6eba6bcd46117fa6c3f48b72118f4f8befb856e85d186"
Sep 17 22:14:19.971809 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:19.971783    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg"]
Sep 17 22:14:19.972031 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:19.972015    2563 scope.go:117] "RemoveContainer" containerID="ba05b7880ae41ffd62c493e32b274c1072bfe2fdd2be0bae7c736b9843c16dab"
Sep 17 22:14:19.974680 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:19.974660    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-664ccfb57d-b6tvg"]
Sep 17 22:14:19.979004 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:19.978987    2563 scope.go:117] "RemoveContainer" containerID="2abb59b16b1c0f48cd254c9f3c04590fb6fea5714854b3ac6d5e81d59f6c8202"
Sep 17 22:14:19.979276 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:14:19.979259    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2abb59b16b1c0f48cd254c9f3c04590fb6fea5714854b3ac6d5e81d59f6c8202\": container with ID starting with 2abb59b16b1c0f48cd254c9f3c04590fb6fea5714854b3ac6d5e81d59f6c8202 not found: ID does not exist" containerID="2abb59b16b1c0f48cd254c9f3c04590fb6fea5714854b3ac6d5e81d59f6c8202"
Sep 17 22:14:19.979333 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:19.979284    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2abb59b16b1c0f48cd254c9f3c04590fb6fea5714854b3ac6d5e81d59f6c8202"} err="failed to get container status \"2abb59b16b1c0f48cd254c9f3c04590fb6fea5714854b3ac6d5e81d59f6c8202\": rpc error: code = NotFound desc = could not find container \"2abb59b16b1c0f48cd254c9f3c04590fb6fea5714854b3ac6d5e81d59f6c8202\": container with ID starting with 2abb59b16b1c0f48cd254c9f3c04590fb6fea5714854b3ac6d5e81d59f6c8202 not found: ID does not exist"
Sep 17 22:14:19.979333 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:19.979299    2563 scope.go:117] "RemoveContainer" containerID="abedd659308ddc7c06f28db4e023a5bcce19f7d495169135777d8ea559fadc8e"
Sep 17 22:14:19.979518 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:14:19.979503    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abedd659308ddc7c06f28db4e023a5bcce19f7d495169135777d8ea559fadc8e\": container with ID starting with abedd659308ddc7c06f28db4e023a5bcce19f7d495169135777d8ea559fadc8e not found: ID does not exist" containerID="abedd659308ddc7c06f28db4e023a5bcce19f7d495169135777d8ea559fadc8e"
Sep 17 22:14:19.979562 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:19.979521    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abedd659308ddc7c06f28db4e023a5bcce19f7d495169135777d8ea559fadc8e"} err="failed to get container status \"abedd659308ddc7c06f28db4e023a5bcce19f7d495169135777d8ea559fadc8e\": rpc error: code = NotFound desc = could not find container \"abedd659308ddc7c06f28db4e023a5bcce19f7d495169135777d8ea559fadc8e\": container with ID starting with abedd659308ddc7c06f28db4e023a5bcce19f7d495169135777d8ea559fadc8e not found: ID does not exist"
Sep 17 22:14:19.979562 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:19.979533    2563 scope.go:117] "RemoveContainer" containerID="85cae30350c38c1502b6eba6bcd46117fa6c3f48b72118f4f8befb856e85d186"
Sep 17 22:14:19.979772 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:14:19.979758    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85cae30350c38c1502b6eba6bcd46117fa6c3f48b72118f4f8befb856e85d186\": container with ID starting with 85cae30350c38c1502b6eba6bcd46117fa6c3f48b72118f4f8befb856e85d186 not found: ID does not exist" containerID="85cae30350c38c1502b6eba6bcd46117fa6c3f48b72118f4f8befb856e85d186"
Sep 17 22:14:19.979815 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:19.979775    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85cae30350c38c1502b6eba6bcd46117fa6c3f48b72118f4f8befb856e85d186"} err="failed to get container status \"85cae30350c38c1502b6eba6bcd46117fa6c3f48b72118f4f8befb856e85d186\": rpc error: code = NotFound desc = could not find container \"85cae30350c38c1502b6eba6bcd46117fa6c3f48b72118f4f8befb856e85d186\": container with ID starting with 85cae30350c38c1502b6eba6bcd46117fa6c3f48b72118f4f8befb856e85d186 not found: ID does not exist"
Sep 17 22:14:19.979815 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:19.979787    2563 scope.go:117] "RemoveContainer" containerID="ba05b7880ae41ffd62c493e32b274c1072bfe2fdd2be0bae7c736b9843c16dab"
Sep 17 22:14:19.979979 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:14:19.979963    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba05b7880ae41ffd62c493e32b274c1072bfe2fdd2be0bae7c736b9843c16dab\": container with ID starting with ba05b7880ae41ffd62c493e32b274c1072bfe2fdd2be0bae7c736b9843c16dab not found: ID does not exist" containerID="ba05b7880ae41ffd62c493e32b274c1072bfe2fdd2be0bae7c736b9843c16dab"
Sep 17 22:14:19.980018 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:19.979982    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba05b7880ae41ffd62c493e32b274c1072bfe2fdd2be0bae7c736b9843c16dab"} err="failed to get container status \"ba05b7880ae41ffd62c493e32b274c1072bfe2fdd2be0bae7c736b9843c16dab\": rpc error: code = NotFound desc = could not find container \"ba05b7880ae41ffd62c493e32b274c1072bfe2fdd2be0bae7c736b9843c16dab\": container with ID starting with ba05b7880ae41ffd62c493e32b274c1072bfe2fdd2be0bae7c736b9843c16dab not found: ID does not exist"
Sep 17 22:14:20.913753 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:20.913721    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73970b41-7022-4c55-b7b8-b4b2f02f7287" path="/var/lib/kubelet/pods/73970b41-7022-4c55-b7b8-b4b2f02f7287/volumes"
Sep 17 22:14:21.959620 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:21.959589    2563 generic.go:356] "Generic (PLEG): container finished" podID="9bbe68e7-c0bd-479a-b09f-8d59fb019f37" containerID="8fc498b73df14b37622f61bb6980cf24a31f3b3d367e0392df57be4a0a217885" exitCode=0
Sep 17 22:14:21.959969 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:21.959635    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-549r6" event={"ID":"9bbe68e7-c0bd-479a-b09f-8d59fb019f37","Type":"ContainerDied","Data":"8fc498b73df14b37622f61bb6980cf24a31f3b3d367e0392df57be4a0a217885"}
Sep 17 22:14:21.960948 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:21.960930    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 22:14:45.079625 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:45.079588    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-549r6" event={"ID":"9bbe68e7-c0bd-479a-b09f-8d59fb019f37","Type":"ContainerStarted","Data":"3594b3b64094ca0b6f33a936742dccbcb53352fb9741d836befd700e124abd11"}
Sep 17 22:14:45.080040 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:45.079652    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-549r6" event={"ID":"9bbe68e7-c0bd-479a-b09f-8d59fb019f37","Type":"ContainerStarted","Data":"894ec16b21ab92b1df9be8d5dd46c8b007dde6a330e647f976193b5fba1ce58a"}
Sep 17 22:14:45.080040 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:45.079858    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-549r6"
Sep 17 22:14:45.080040 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:45.080033    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-549r6"
Sep 17 22:14:45.099245 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:45.099192    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-549r6" podStartSLOduration=2.728524751 podStartE2EDuration="29.099177356s" totalImagesPullingTime="22.689567358s" totalInitContainerRuntime="3.681085247s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:14:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:14:45.098440565 +0000 UTC m=+996.266393549" watchObservedRunningTime="2026-09-17 22:14:45.099177356 +0000 UTC m=+996.267130338"
Sep 17 22:14:51.089100 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:14:51.089069    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-549r6"
Sep 17 22:15:21.090800 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:15:21.090761    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-549r6" podUID="9bbe68e7-c0bd-479a-b09f-8d59fb019f37" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.28:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 22:15:31.090286 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:15:31.090244    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-549r6" podUID="9bbe68e7-c0bd-479a-b09f-8d59fb019f37" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.28:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 22:15:41.089929 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:15:41.089892    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-549r6" podUID="9bbe68e7-c0bd-479a-b09f-8d59fb019f37" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.28:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 22:15:51.090006 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:15:51.089965    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-549r6" podUID="9bbe68e7-c0bd-479a-b09f-8d59fb019f37" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.28:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 22:16:01.651672 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:01.651598    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-549r6"
Sep 17 22:16:06.765455 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:06.765418    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-549r6"]
Sep 17 22:16:06.765994 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:06.765710    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-549r6" podUID="9bbe68e7-c0bd-479a-b09f-8d59fb019f37" containerName="kserve-container" containerID="cri-o://894ec16b21ab92b1df9be8d5dd46c8b007dde6a330e647f976193b5fba1ce58a" gracePeriod=30
Sep 17 22:16:06.765994 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:06.765733    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-549r6" podUID="9bbe68e7-c0bd-479a-b09f-8d59fb019f37" containerName="kube-rbac-proxy" containerID="cri-o://3594b3b64094ca0b6f33a936742dccbcb53352fb9741d836befd700e124abd11" gracePeriod=30
Sep 17 22:16:07.392591 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:07.392559    2563 generic.go:356] "Generic (PLEG): container finished" podID="9bbe68e7-c0bd-479a-b09f-8d59fb019f37" containerID="3594b3b64094ca0b6f33a936742dccbcb53352fb9741d836befd700e124abd11" exitCode=2
Sep 17 22:16:07.392752 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:07.392634    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-549r6" event={"ID":"9bbe68e7-c0bd-479a-b09f-8d59fb019f37","Type":"ContainerDied","Data":"3594b3b64094ca0b6f33a936742dccbcb53352fb9741d836befd700e124abd11"}
Sep 17 22:16:11.084550 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:11.084513    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-549r6" podUID="9bbe68e7-c0bd-479a-b09f-8d59fb019f37" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Sep 17 22:16:11.089851 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:11.089823    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-549r6" podUID="9bbe68e7-c0bd-479a-b09f-8d59fb019f37" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.28:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 22:16:11.197883 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:11.197856    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-549r6"
Sep 17 22:16:11.219628 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:11.219602    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9bbe68e7-c0bd-479a-b09f-8d59fb019f37-kserve-provision-location\") pod \"9bbe68e7-c0bd-479a-b09f-8d59fb019f37\" (UID: \"9bbe68e7-c0bd-479a-b09f-8d59fb019f37\") "
Sep 17 22:16:11.219766 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:11.219651    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9bbe68e7-c0bd-479a-b09f-8d59fb019f37-cabundle-cert\") pod \"9bbe68e7-c0bd-479a-b09f-8d59fb019f37\" (UID: \"9bbe68e7-c0bd-479a-b09f-8d59fb019f37\") "
Sep 17 22:16:11.219766 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:11.219696    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9bbe68e7-c0bd-479a-b09f-8d59fb019f37-kube-rbac-proxy-sar-config\") pod \"9bbe68e7-c0bd-479a-b09f-8d59fb019f37\" (UID: \"9bbe68e7-c0bd-479a-b09f-8d59fb019f37\") "
Sep 17 22:16:11.219766 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:11.219718    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dlhj6\" (UniqueName: \"kubernetes.io/projected/9bbe68e7-c0bd-479a-b09f-8d59fb019f37-kube-api-access-dlhj6\") pod \"9bbe68e7-c0bd-479a-b09f-8d59fb019f37\" (UID: \"9bbe68e7-c0bd-479a-b09f-8d59fb019f37\") "
Sep 17 22:16:11.219766 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:11.219758    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9bbe68e7-c0bd-479a-b09f-8d59fb019f37-proxy-tls\") pod \"9bbe68e7-c0bd-479a-b09f-8d59fb019f37\" (UID: \"9bbe68e7-c0bd-479a-b09f-8d59fb019f37\") "
Sep 17 22:16:11.220102 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:11.220010    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bbe68e7-c0bd-479a-b09f-8d59fb019f37-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9bbe68e7-c0bd-479a-b09f-8d59fb019f37" (UID: "9bbe68e7-c0bd-479a-b09f-8d59fb019f37"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:16:11.220739 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:11.220436    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bbe68e7-c0bd-479a-b09f-8d59fb019f37-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9bbe68e7-c0bd-479a-b09f-8d59fb019f37" (UID: "9bbe68e7-c0bd-479a-b09f-8d59fb019f37"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:16:11.220739 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:11.220705    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bbe68e7-c0bd-479a-b09f-8d59fb019f37-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9bbe68e7-c0bd-479a-b09f-8d59fb019f37" (UID: "9bbe68e7-c0bd-479a-b09f-8d59fb019f37"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:16:11.222416 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:11.222382    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bbe68e7-c0bd-479a-b09f-8d59fb019f37-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9bbe68e7-c0bd-479a-b09f-8d59fb019f37" (UID: "9bbe68e7-c0bd-479a-b09f-8d59fb019f37"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:16:11.222544 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:11.222527    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bbe68e7-c0bd-479a-b09f-8d59fb019f37-kube-api-access-dlhj6" (OuterVolumeSpecName: "kube-api-access-dlhj6") pod "9bbe68e7-c0bd-479a-b09f-8d59fb019f37" (UID: "9bbe68e7-c0bd-479a-b09f-8d59fb019f37"). InnerVolumeSpecName "kube-api-access-dlhj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:16:11.320749 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:11.320685    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9bbe68e7-c0bd-479a-b09f-8d59fb019f37-kserve-provision-location\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:16:11.320749 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:11.320706    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9bbe68e7-c0bd-479a-b09f-8d59fb019f37-cabundle-cert\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:16:11.320749 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:11.320717    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9bbe68e7-c0bd-479a-b09f-8d59fb019f37-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:16:11.320749 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:11.320725    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlhj6\" (UniqueName: \"kubernetes.io/projected/9bbe68e7-c0bd-479a-b09f-8d59fb019f37-kube-api-access-dlhj6\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:16:11.320749 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:11.320734    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9bbe68e7-c0bd-479a-b09f-8d59fb019f37-proxy-tls\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:16:11.409623 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:11.409592    2563 generic.go:356] "Generic (PLEG): container finished" podID="9bbe68e7-c0bd-479a-b09f-8d59fb019f37" containerID="894ec16b21ab92b1df9be8d5dd46c8b007dde6a330e647f976193b5fba1ce58a" exitCode=0
Sep 17 22:16:11.409755 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:11.409652    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-549r6" event={"ID":"9bbe68e7-c0bd-479a-b09f-8d59fb019f37","Type":"ContainerDied","Data":"894ec16b21ab92b1df9be8d5dd46c8b007dde6a330e647f976193b5fba1ce58a"}
Sep 17 22:16:11.409755 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:11.409656    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-549r6"
Sep 17 22:16:11.409755 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:11.409712    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-549r6" event={"ID":"9bbe68e7-c0bd-479a-b09f-8d59fb019f37","Type":"ContainerDied","Data":"64068691522ea1acf92a7f1800882d09aef75fb9e8d4106ba7ecf90e6c60c37b"}
Sep 17 22:16:11.409755 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:11.409750    2563 scope.go:117] "RemoveContainer" containerID="3594b3b64094ca0b6f33a936742dccbcb53352fb9741d836befd700e124abd11"
Sep 17 22:16:11.417864 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:11.417849    2563 scope.go:117] "RemoveContainer" containerID="894ec16b21ab92b1df9be8d5dd46c8b007dde6a330e647f976193b5fba1ce58a"
Sep 17 22:16:11.425028 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:11.425012    2563 scope.go:117] "RemoveContainer" containerID="8fc498b73df14b37622f61bb6980cf24a31f3b3d367e0392df57be4a0a217885"
Sep 17 22:16:11.429940 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:11.429920    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-549r6"]
Sep 17 22:16:11.432513 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:11.432491    2563 scope.go:117] "RemoveContainer" containerID="3594b3b64094ca0b6f33a936742dccbcb53352fb9741d836befd700e124abd11"
Sep 17 22:16:11.432853 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:16:11.432830    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3594b3b64094ca0b6f33a936742dccbcb53352fb9741d836befd700e124abd11\": container with ID starting with 3594b3b64094ca0b6f33a936742dccbcb53352fb9741d836befd700e124abd11 not found: ID does not exist" containerID="3594b3b64094ca0b6f33a936742dccbcb53352fb9741d836befd700e124abd11"
Sep 17 22:16:11.432938 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:11.432856    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3594b3b64094ca0b6f33a936742dccbcb53352fb9741d836befd700e124abd11"} err="failed to get container status \"3594b3b64094ca0b6f33a936742dccbcb53352fb9741d836befd700e124abd11\": rpc error: code = NotFound desc = could not find container \"3594b3b64094ca0b6f33a936742dccbcb53352fb9741d836befd700e124abd11\": container with ID starting with 3594b3b64094ca0b6f33a936742dccbcb53352fb9741d836befd700e124abd11 not found: ID does not exist"
Sep 17 22:16:11.432938 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:11.432873    2563 scope.go:117] "RemoveContainer" containerID="894ec16b21ab92b1df9be8d5dd46c8b007dde6a330e647f976193b5fba1ce58a"
Sep 17 22:16:11.433125 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:16:11.433107    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"894ec16b21ab92b1df9be8d5dd46c8b007dde6a330e647f976193b5fba1ce58a\": container with ID starting with 894ec16b21ab92b1df9be8d5dd46c8b007dde6a330e647f976193b5fba1ce58a not found: ID does not exist" containerID="894ec16b21ab92b1df9be8d5dd46c8b007dde6a330e647f976193b5fba1ce58a"
Sep 17 22:16:11.433306 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:11.433134    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"894ec16b21ab92b1df9be8d5dd46c8b007dde6a330e647f976193b5fba1ce58a"} err="failed to get container status \"894ec16b21ab92b1df9be8d5dd46c8b007dde6a330e647f976193b5fba1ce58a\": rpc error: code = NotFound desc = could not find container \"894ec16b21ab92b1df9be8d5dd46c8b007dde6a330e647f976193b5fba1ce58a\": container with ID starting with 894ec16b21ab92b1df9be8d5dd46c8b007dde6a330e647f976193b5fba1ce58a not found: ID does not exist"
Sep 17 22:16:11.433306 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:11.433224    2563 scope.go:117] "RemoveContainer" containerID="8fc498b73df14b37622f61bb6980cf24a31f3b3d367e0392df57be4a0a217885"
Sep 17 22:16:11.433509 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:16:11.433488    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fc498b73df14b37622f61bb6980cf24a31f3b3d367e0392df57be4a0a217885\": container with ID starting with 8fc498b73df14b37622f61bb6980cf24a31f3b3d367e0392df57be4a0a217885 not found: ID does not exist" containerID="8fc498b73df14b37622f61bb6980cf24a31f3b3d367e0392df57be4a0a217885"
Sep 17 22:16:11.433547 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:11.433517    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fc498b73df14b37622f61bb6980cf24a31f3b3d367e0392df57be4a0a217885"} err="failed to get container status \"8fc498b73df14b37622f61bb6980cf24a31f3b3d367e0392df57be4a0a217885\": rpc error: code = NotFound desc = could not find container \"8fc498b73df14b37622f61bb6980cf24a31f3b3d367e0392df57be4a0a217885\": container with ID starting with 8fc498b73df14b37622f61bb6980cf24a31f3b3d367e0392df57be4a0a217885 not found: ID does not exist"
Sep 17 22:16:11.433905 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:11.433891    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-549r6"]
Sep 17 22:16:11.952042 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:11.951974    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zdpmp"]
Sep 17 22:16:11.952277 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:11.952254    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bbe68e7-c0bd-479a-b09f-8d59fb019f37" containerName="kserve-container"
Sep 17 22:16:11.952277 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:11.952278    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bbe68e7-c0bd-479a-b09f-8d59fb019f37" containerName="kserve-container"
Sep 17 22:16:11.952372 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:11.952291    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73970b41-7022-4c55-b7b8-b4b2f02f7287" containerName="agent"
Sep 17 22:16:11.952372 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:11.952298    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="73970b41-7022-4c55-b7b8-b4b2f02f7287" containerName="agent"
Sep 17 22:16:11.952372 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:11.952307    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bbe68e7-c0bd-479a-b09f-8d59fb019f37" containerName="storage-initializer"
Sep 17 22:16:11.952372 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:11.952313    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bbe68e7-c0bd-479a-b09f-8d59fb019f37" containerName="storage-initializer"
Sep 17 22:16:11.952372 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:11.952324    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73970b41-7022-4c55-b7b8-b4b2f02f7287" containerName="kserve-container"
Sep 17 22:16:11.952372 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:11.952329    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="73970b41-7022-4c55-b7b8-b4b2f02f7287" containerName="kserve-container"
Sep 17 22:16:11.952372 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:11.952337    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bbe68e7-c0bd-479a-b09f-8d59fb019f37" containerName="kube-rbac-proxy"
Sep 17 22:16:11.952372 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:11.952342    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bbe68e7-c0bd-479a-b09f-8d59fb019f37" containerName="kube-rbac-proxy"
Sep 17 22:16:11.952372 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:11.952348    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73970b41-7022-4c55-b7b8-b4b2f02f7287" containerName="kube-rbac-proxy"
Sep 17 22:16:11.952372 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:11.952353    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="73970b41-7022-4c55-b7b8-b4b2f02f7287" containerName="kube-rbac-proxy"
Sep 17 22:16:11.952372 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:11.952360    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73970b41-7022-4c55-b7b8-b4b2f02f7287" containerName="storage-initializer"
Sep 17 22:16:11.952372 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:11.952364    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="73970b41-7022-4c55-b7b8-b4b2f02f7287" containerName="storage-initializer"
Sep 17 22:16:11.952956 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:11.952408    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bbe68e7-c0bd-479a-b09f-8d59fb019f37" containerName="kserve-container"
Sep 17 22:16:11.952956 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:11.952417    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="73970b41-7022-4c55-b7b8-b4b2f02f7287" containerName="kube-rbac-proxy"
Sep 17 22:16:11.952956 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:11.952425    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="73970b41-7022-4c55-b7b8-b4b2f02f7287" containerName="kserve-container"
Sep 17 22:16:11.952956 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:11.952430    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="73970b41-7022-4c55-b7b8-b4b2f02f7287" containerName="agent"
Sep 17 22:16:11.952956 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:11.952435    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bbe68e7-c0bd-479a-b09f-8d59fb019f37" containerName="kube-rbac-proxy"
Sep 17 22:16:11.956815 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:11.956797    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zdpmp"
Sep 17 22:16:11.959070 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:11.959047    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 22:16:11.959070 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:11.959047    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 22:16:11.959253 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:11.959090    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-predictor-serving-cert\""
Sep 17 22:16:11.959253 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:11.959249    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:16:11.959473 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:11.959457    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\""
Sep 17 22:16:11.959687 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:11.959667    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 22:16:11.959778 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:11.959735    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v4p4m\""
Sep 17 22:16:11.965247 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:11.965225    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zdpmp"]
Sep 17 22:16:12.025498 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:12.025474    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/333cb4d8-75ac-46ab-af99-793fb0ae133f-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zdpmp\" (UID: \"333cb4d8-75ac-46ab-af99-793fb0ae133f\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zdpmp"
Sep 17 22:16:12.025611 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:12.025523    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/333cb4d8-75ac-46ab-af99-793fb0ae133f-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zdpmp\" (UID: \"333cb4d8-75ac-46ab-af99-793fb0ae133f\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zdpmp"
Sep 17 22:16:12.025611 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:12.025555    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p52gn\" (UniqueName: \"kubernetes.io/projected/333cb4d8-75ac-46ab-af99-793fb0ae133f-kube-api-access-p52gn\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zdpmp\" (UID: \"333cb4d8-75ac-46ab-af99-793fb0ae133f\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zdpmp"
Sep 17 22:16:12.025611 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:12.025588    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/333cb4d8-75ac-46ab-af99-793fb0ae133f-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zdpmp\" (UID: \"333cb4d8-75ac-46ab-af99-793fb0ae133f\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zdpmp"
Sep 17 22:16:12.025753 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:12.025640    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/333cb4d8-75ac-46ab-af99-793fb0ae133f-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zdpmp\" (UID: \"333cb4d8-75ac-46ab-af99-793fb0ae133f\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zdpmp"
Sep 17 22:16:12.126320 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:12.126289    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/333cb4d8-75ac-46ab-af99-793fb0ae133f-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zdpmp\" (UID: \"333cb4d8-75ac-46ab-af99-793fb0ae133f\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zdpmp"
Sep 17 22:16:12.126664 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:12.126335    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/333cb4d8-75ac-46ab-af99-793fb0ae133f-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zdpmp\" (UID: \"333cb4d8-75ac-46ab-af99-793fb0ae133f\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zdpmp"
Sep 17 22:16:12.126664 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:12.126366    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p52gn\" (UniqueName: \"kubernetes.io/projected/333cb4d8-75ac-46ab-af99-793fb0ae133f-kube-api-access-p52gn\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zdpmp\" (UID: \"333cb4d8-75ac-46ab-af99-793fb0ae133f\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zdpmp"
Sep 17 22:16:12.126664 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:12.126400    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/333cb4d8-75ac-46ab-af99-793fb0ae133f-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zdpmp\" (UID: \"333cb4d8-75ac-46ab-af99-793fb0ae133f\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zdpmp"
Sep 17 22:16:12.126664 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:12.126432    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/333cb4d8-75ac-46ab-af99-793fb0ae133f-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zdpmp\" (UID: \"333cb4d8-75ac-46ab-af99-793fb0ae133f\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zdpmp"
Sep 17 22:16:12.126664 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:16:12.126439    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-serving-cert: secret "isvc-predictive-xgboost-v2-predictor-serving-cert" not found
Sep 17 22:16:12.126664 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:16:12.126498    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/333cb4d8-75ac-46ab-af99-793fb0ae133f-proxy-tls podName:333cb4d8-75ac-46ab-af99-793fb0ae133f nodeName:}" failed. No retries permitted until 2026-09-17 22:16:12.626477571 +0000 UTC m=+1083.794430549 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/333cb4d8-75ac-46ab-af99-793fb0ae133f-proxy-tls") pod "isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zdpmp" (UID: "333cb4d8-75ac-46ab-af99-793fb0ae133f") : secret "isvc-predictive-xgboost-v2-predictor-serving-cert" not found
Sep 17 22:16:12.126998 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:12.126784    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/333cb4d8-75ac-46ab-af99-793fb0ae133f-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zdpmp\" (UID: \"333cb4d8-75ac-46ab-af99-793fb0ae133f\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zdpmp"
Sep 17 22:16:12.127070 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:12.127052    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/333cb4d8-75ac-46ab-af99-793fb0ae133f-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zdpmp\" (UID: \"333cb4d8-75ac-46ab-af99-793fb0ae133f\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zdpmp"
Sep 17 22:16:12.127133 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:12.127113    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/333cb4d8-75ac-46ab-af99-793fb0ae133f-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zdpmp\" (UID: \"333cb4d8-75ac-46ab-af99-793fb0ae133f\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zdpmp"
Sep 17 22:16:12.134634 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:12.134614    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p52gn\" (UniqueName: \"kubernetes.io/projected/333cb4d8-75ac-46ab-af99-793fb0ae133f-kube-api-access-p52gn\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zdpmp\" (UID: \"333cb4d8-75ac-46ab-af99-793fb0ae133f\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zdpmp"
Sep 17 22:16:12.631612 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:12.631577    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/333cb4d8-75ac-46ab-af99-793fb0ae133f-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zdpmp\" (UID: \"333cb4d8-75ac-46ab-af99-793fb0ae133f\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zdpmp"
Sep 17 22:16:12.633836 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:12.633813    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/333cb4d8-75ac-46ab-af99-793fb0ae133f-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zdpmp\" (UID: \"333cb4d8-75ac-46ab-af99-793fb0ae133f\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zdpmp"
Sep 17 22:16:12.866693 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:12.866662    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zdpmp"
Sep 17 22:16:12.915897 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:12.915862    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9bbe68e7-c0bd-479a-b09f-8d59fb019f37" path="/var/lib/kubelet/pods/9bbe68e7-c0bd-479a-b09f-8d59fb019f37/volumes"
Sep 17 22:16:13.136683 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:13.136536    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zdpmp"]
Sep 17 22:16:13.139359 ip-10-0-138-45 kubenswrapper[2563]: W0917 22:16:13.139334    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod333cb4d8_75ac_46ab_af99_793fb0ae133f.slice/crio-626f185c8977c460eda66c5d90c9169feb8de14dcb00517da81b6f6ff212cd6e WatchSource:0}: Error finding container 626f185c8977c460eda66c5d90c9169feb8de14dcb00517da81b6f6ff212cd6e: Status 404 returned error can't find the container with id 626f185c8977c460eda66c5d90c9169feb8de14dcb00517da81b6f6ff212cd6e
Sep 17 22:16:13.419057 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:13.419024    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zdpmp" event={"ID":"333cb4d8-75ac-46ab-af99-793fb0ae133f","Type":"ContainerStarted","Data":"7d0a4288c129da2d646cdd48a947b8697ddaf00134180d4415789004920e63e1"}
Sep 17 22:16:13.419231 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:13.419079    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zdpmp" event={"ID":"333cb4d8-75ac-46ab-af99-793fb0ae133f","Type":"ContainerStarted","Data":"626f185c8977c460eda66c5d90c9169feb8de14dcb00517da81b6f6ff212cd6e"}
Sep 17 22:16:17.435589 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:17.435558    2563 generic.go:356] "Generic (PLEG): container finished" podID="333cb4d8-75ac-46ab-af99-793fb0ae133f" containerID="7d0a4288c129da2d646cdd48a947b8697ddaf00134180d4415789004920e63e1" exitCode=0
Sep 17 22:16:17.435943 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:17.435634    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zdpmp" event={"ID":"333cb4d8-75ac-46ab-af99-793fb0ae133f","Type":"ContainerDied","Data":"7d0a4288c129da2d646cdd48a947b8697ddaf00134180d4415789004920e63e1"}
Sep 17 22:16:18.441754 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:18.441722    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zdpmp" event={"ID":"333cb4d8-75ac-46ab-af99-793fb0ae133f","Type":"ContainerStarted","Data":"dde2183ba506d53042518957f811ff1fd97ab9b0f8a0834d276352b4836cab92"}
Sep 17 22:16:18.442123 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:18.441782    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zdpmp" event={"ID":"333cb4d8-75ac-46ab-af99-793fb0ae133f","Type":"ContainerStarted","Data":"31142c5913a718ed3799cbf4436c3d572bc7c493b875dbcd67f9474d83e66696"}
Sep 17 22:16:18.442123 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:18.441977    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zdpmp"
Sep 17 22:16:18.460421 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:18.460378    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zdpmp" podStartSLOduration=3.712438678 podStartE2EDuration="7.460363475s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.747924797s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:16:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:16:18.458803233 +0000 UTC m=+1089.626756216" watchObservedRunningTime="2026-09-17 22:16:18.460363475 +0000 UTC m=+1089.628316457"
Sep 17 22:16:19.446006 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:19.445981    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zdpmp"
Sep 17 22:16:25.454844 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:25.454817    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zdpmp"
Sep 17 22:16:55.456492 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:16:55.456448    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zdpmp" podUID="333cb4d8-75ac-46ab-af99-793fb0ae133f" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.29:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 22:17:05.455729 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:05.455692    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zdpmp" podUID="333cb4d8-75ac-46ab-af99-793fb0ae133f" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.29:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 22:17:15.456036 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:15.455996    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zdpmp" podUID="333cb4d8-75ac-46ab-af99-793fb0ae133f" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.29:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 22:17:25.455928 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:25.455833    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zdpmp" podUID="333cb4d8-75ac-46ab-af99-793fb0ae133f" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.29:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 22:17:27.909494 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:27.909448    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zdpmp" podUID="333cb4d8-75ac-46ab-af99-793fb0ae133f" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.29:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 22:17:38.453865 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:38.453825    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zdpmp"
Sep 17 22:17:42.010924 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:42.010889    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zdpmp"]
Sep 17 22:17:42.011368 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:42.011120    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zdpmp" podUID="333cb4d8-75ac-46ab-af99-793fb0ae133f" containerName="kserve-container" containerID="cri-o://31142c5913a718ed3799cbf4436c3d572bc7c493b875dbcd67f9474d83e66696" gracePeriod=30
Sep 17 22:17:42.011368 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:42.011192    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zdpmp" podUID="333cb4d8-75ac-46ab-af99-793fb0ae133f" containerName="kube-rbac-proxy" containerID="cri-o://dde2183ba506d53042518957f811ff1fd97ab9b0f8a0834d276352b4836cab92" gracePeriod=30
Sep 17 22:17:42.775658 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:42.775626    2563 generic.go:356] "Generic (PLEG): container finished" podID="333cb4d8-75ac-46ab-af99-793fb0ae133f" containerID="dde2183ba506d53042518957f811ff1fd97ab9b0f8a0834d276352b4836cab92" exitCode=2
Sep 17 22:17:42.775839 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:42.775701    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zdpmp" event={"ID":"333cb4d8-75ac-46ab-af99-793fb0ae133f","Type":"ContainerDied","Data":"dde2183ba506d53042518957f811ff1fd97ab9b0f8a0834d276352b4836cab92"}
Sep 17 22:17:45.451016 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:45.450972    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zdpmp" podUID="333cb4d8-75ac-46ab-af99-793fb0ae133f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Sep 17 22:17:47.205064 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:47.205036    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-sn7qh"]
Sep 17 22:17:47.208421 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:47.208403    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-sn7qh"
Sep 17 22:17:47.211613 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:47.211583    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-predictor-serving-cert\""
Sep 17 22:17:47.211613 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:47.211603    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\""
Sep 17 22:17:47.221056 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:47.221031    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-sn7qh"]
Sep 17 22:17:47.297803 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:47.297762    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-sn7qh\" (UID: \"ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-sn7qh"
Sep 17 22:17:47.297975 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:47.297818    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-sn7qh\" (UID: \"ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-sn7qh"
Sep 17 22:17:47.297975 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:47.297874    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-sn7qh\" (UID: \"ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-sn7qh"
Sep 17 22:17:47.297975 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:47.297938    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-sn7qh\" (UID: \"ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-sn7qh"
Sep 17 22:17:47.298182 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:47.297997    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q49p8\" (UniqueName: \"kubernetes.io/projected/ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e-kube-api-access-q49p8\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-sn7qh\" (UID: \"ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-sn7qh"
Sep 17 22:17:47.352549 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:47.352523    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zdpmp"
Sep 17 22:17:47.398902 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:47.398863    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-sn7qh\" (UID: \"ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-sn7qh"
Sep 17 22:17:47.398902 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:47.398899    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-sn7qh\" (UID: \"ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-sn7qh"
Sep 17 22:17:47.399164 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:47.398926    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-sn7qh\" (UID: \"ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-sn7qh"
Sep 17 22:17:47.399164 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:47.398970    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-sn7qh\" (UID: \"ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-sn7qh"
Sep 17 22:17:47.399164 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:47.399012    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q49p8\" (UniqueName: \"kubernetes.io/projected/ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e-kube-api-access-q49p8\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-sn7qh\" (UID: \"ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-sn7qh"
Sep 17 22:17:47.399164 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:17:47.399132    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-serving-cert: secret "isvc-predictive-lightgbm-v2-predictor-serving-cert" not found
Sep 17 22:17:47.399360 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:17:47.399231    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e-proxy-tls podName:ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e nodeName:}" failed. No retries permitted until 2026-09-17 22:17:47.899206083 +0000 UTC m=+1179.067159055 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e-proxy-tls") pod "isvc-predictive-lightgbm-v2-predictor-545b7cc587-sn7qh" (UID: "ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e") : secret "isvc-predictive-lightgbm-v2-predictor-serving-cert" not found
Sep 17 22:17:47.399360 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:47.399307    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-sn7qh\" (UID: \"ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-sn7qh"
Sep 17 22:17:47.399655 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:47.399637    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-sn7qh\" (UID: \"ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-sn7qh"
Sep 17 22:17:47.399704 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:47.399639    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-sn7qh\" (UID: \"ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-sn7qh"
Sep 17 22:17:47.406848 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:47.406829    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q49p8\" (UniqueName: \"kubernetes.io/projected/ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e-kube-api-access-q49p8\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-sn7qh\" (UID: \"ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-sn7qh"
Sep 17 22:17:47.499336 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:47.499302    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p52gn\" (UniqueName: \"kubernetes.io/projected/333cb4d8-75ac-46ab-af99-793fb0ae133f-kube-api-access-p52gn\") pod \"333cb4d8-75ac-46ab-af99-793fb0ae133f\" (UID: \"333cb4d8-75ac-46ab-af99-793fb0ae133f\") "
Sep 17 22:17:47.499515 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:47.499348    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/333cb4d8-75ac-46ab-af99-793fb0ae133f-kube-rbac-proxy-sar-config\") pod \"333cb4d8-75ac-46ab-af99-793fb0ae133f\" (UID: \"333cb4d8-75ac-46ab-af99-793fb0ae133f\") "
Sep 17 22:17:47.499515 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:47.499401    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/333cb4d8-75ac-46ab-af99-793fb0ae133f-kserve-provision-location\") pod \"333cb4d8-75ac-46ab-af99-793fb0ae133f\" (UID: \"333cb4d8-75ac-46ab-af99-793fb0ae133f\") "
Sep 17 22:17:47.499515 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:47.499465    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/333cb4d8-75ac-46ab-af99-793fb0ae133f-proxy-tls\") pod \"333cb4d8-75ac-46ab-af99-793fb0ae133f\" (UID: \"333cb4d8-75ac-46ab-af99-793fb0ae133f\") "
Sep 17 22:17:47.499515 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:47.499500    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/333cb4d8-75ac-46ab-af99-793fb0ae133f-cabundle-cert\") pod \"333cb4d8-75ac-46ab-af99-793fb0ae133f\" (UID: \"333cb4d8-75ac-46ab-af99-793fb0ae133f\") "
Sep 17 22:17:47.499779 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:47.499745    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/333cb4d8-75ac-46ab-af99-793fb0ae133f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "333cb4d8-75ac-46ab-af99-793fb0ae133f" (UID: "333cb4d8-75ac-46ab-af99-793fb0ae133f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:17:47.499918 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:47.499809    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/333cb4d8-75ac-46ab-af99-793fb0ae133f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "333cb4d8-75ac-46ab-af99-793fb0ae133f" (UID: "333cb4d8-75ac-46ab-af99-793fb0ae133f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:17:47.499918 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:47.499871    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/333cb4d8-75ac-46ab-af99-793fb0ae133f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "333cb4d8-75ac-46ab-af99-793fb0ae133f" (UID: "333cb4d8-75ac-46ab-af99-793fb0ae133f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:17:47.501518 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:47.501489    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/333cb4d8-75ac-46ab-af99-793fb0ae133f-kube-api-access-p52gn" (OuterVolumeSpecName: "kube-api-access-p52gn") pod "333cb4d8-75ac-46ab-af99-793fb0ae133f" (UID: "333cb4d8-75ac-46ab-af99-793fb0ae133f"). InnerVolumeSpecName "kube-api-access-p52gn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:17:47.501611 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:47.501525    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/333cb4d8-75ac-46ab-af99-793fb0ae133f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "333cb4d8-75ac-46ab-af99-793fb0ae133f" (UID: "333cb4d8-75ac-46ab-af99-793fb0ae133f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:17:47.600427 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:47.600392    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/333cb4d8-75ac-46ab-af99-793fb0ae133f-cabundle-cert\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:17:47.600427 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:47.600421    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p52gn\" (UniqueName: \"kubernetes.io/projected/333cb4d8-75ac-46ab-af99-793fb0ae133f-kube-api-access-p52gn\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:17:47.600427 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:47.600431    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/333cb4d8-75ac-46ab-af99-793fb0ae133f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:17:47.600677 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:47.600441    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/333cb4d8-75ac-46ab-af99-793fb0ae133f-kserve-provision-location\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:17:47.600677 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:47.600450    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/333cb4d8-75ac-46ab-af99-793fb0ae133f-proxy-tls\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:17:47.798850 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:47.798764    2563 generic.go:356] "Generic (PLEG): container finished" podID="333cb4d8-75ac-46ab-af99-793fb0ae133f" containerID="31142c5913a718ed3799cbf4436c3d572bc7c493b875dbcd67f9474d83e66696" exitCode=0
Sep 17 22:17:47.798850 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:47.798822    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zdpmp"
Sep 17 22:17:47.799033 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:47.798845    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zdpmp" event={"ID":"333cb4d8-75ac-46ab-af99-793fb0ae133f","Type":"ContainerDied","Data":"31142c5913a718ed3799cbf4436c3d572bc7c493b875dbcd67f9474d83e66696"}
Sep 17 22:17:47.799033 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:47.798907    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zdpmp" event={"ID":"333cb4d8-75ac-46ab-af99-793fb0ae133f","Type":"ContainerDied","Data":"626f185c8977c460eda66c5d90c9169feb8de14dcb00517da81b6f6ff212cd6e"}
Sep 17 22:17:47.799033 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:47.798933    2563 scope.go:117] "RemoveContainer" containerID="dde2183ba506d53042518957f811ff1fd97ab9b0f8a0834d276352b4836cab92"
Sep 17 22:17:47.807795 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:47.807776    2563 scope.go:117] "RemoveContainer" containerID="31142c5913a718ed3799cbf4436c3d572bc7c493b875dbcd67f9474d83e66696"
Sep 17 22:17:47.815815 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:47.815800    2563 scope.go:117] "RemoveContainer" containerID="7d0a4288c129da2d646cdd48a947b8697ddaf00134180d4415789004920e63e1"
Sep 17 22:17:47.820459 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:47.820440    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zdpmp"]
Sep 17 22:17:47.823882 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:47.823863    2563 scope.go:117] "RemoveContainer" containerID="dde2183ba506d53042518957f811ff1fd97ab9b0f8a0834d276352b4836cab92"
Sep 17 22:17:47.823942 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:47.823917    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-zdpmp"]
Sep 17 22:17:47.824268 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:17:47.824126    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dde2183ba506d53042518957f811ff1fd97ab9b0f8a0834d276352b4836cab92\": container with ID starting with dde2183ba506d53042518957f811ff1fd97ab9b0f8a0834d276352b4836cab92 not found: ID does not exist" containerID="dde2183ba506d53042518957f811ff1fd97ab9b0f8a0834d276352b4836cab92"
Sep 17 22:17:47.824268 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:47.824218    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dde2183ba506d53042518957f811ff1fd97ab9b0f8a0834d276352b4836cab92"} err="failed to get container status \"dde2183ba506d53042518957f811ff1fd97ab9b0f8a0834d276352b4836cab92\": rpc error: code = NotFound desc = could not find container \"dde2183ba506d53042518957f811ff1fd97ab9b0f8a0834d276352b4836cab92\": container with ID starting with dde2183ba506d53042518957f811ff1fd97ab9b0f8a0834d276352b4836cab92 not found: ID does not exist"
Sep 17 22:17:47.824268 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:47.824241    2563 scope.go:117] "RemoveContainer" containerID="31142c5913a718ed3799cbf4436c3d572bc7c493b875dbcd67f9474d83e66696"
Sep 17 22:17:47.824520 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:17:47.824492    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31142c5913a718ed3799cbf4436c3d572bc7c493b875dbcd67f9474d83e66696\": container with ID starting with 31142c5913a718ed3799cbf4436c3d572bc7c493b875dbcd67f9474d83e66696 not found: ID does not exist" containerID="31142c5913a718ed3799cbf4436c3d572bc7c493b875dbcd67f9474d83e66696"
Sep 17 22:17:47.824569 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:47.824529    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31142c5913a718ed3799cbf4436c3d572bc7c493b875dbcd67f9474d83e66696"} err="failed to get container status \"31142c5913a718ed3799cbf4436c3d572bc7c493b875dbcd67f9474d83e66696\": rpc error: code = NotFound desc = could not find container \"31142c5913a718ed3799cbf4436c3d572bc7c493b875dbcd67f9474d83e66696\": container with ID starting with 31142c5913a718ed3799cbf4436c3d572bc7c493b875dbcd67f9474d83e66696 not found: ID does not exist"
Sep 17 22:17:47.824569 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:47.824550    2563 scope.go:117] "RemoveContainer" containerID="7d0a4288c129da2d646cdd48a947b8697ddaf00134180d4415789004920e63e1"
Sep 17 22:17:47.824796 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:17:47.824781    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d0a4288c129da2d646cdd48a947b8697ddaf00134180d4415789004920e63e1\": container with ID starting with 7d0a4288c129da2d646cdd48a947b8697ddaf00134180d4415789004920e63e1 not found: ID does not exist" containerID="7d0a4288c129da2d646cdd48a947b8697ddaf00134180d4415789004920e63e1"
Sep 17 22:17:47.824843 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:47.824800    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d0a4288c129da2d646cdd48a947b8697ddaf00134180d4415789004920e63e1"} err="failed to get container status \"7d0a4288c129da2d646cdd48a947b8697ddaf00134180d4415789004920e63e1\": rpc error: code = NotFound desc = could not find container \"7d0a4288c129da2d646cdd48a947b8697ddaf00134180d4415789004920e63e1\": container with ID starting with 7d0a4288c129da2d646cdd48a947b8697ddaf00134180d4415789004920e63e1 not found: ID does not exist"
Sep 17 22:17:47.902713 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:47.902681    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-sn7qh\" (UID: \"ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-sn7qh"
Sep 17 22:17:47.905077 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:47.905057    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-sn7qh\" (UID: \"ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-sn7qh"
Sep 17 22:17:48.122844 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:48.122748    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-sn7qh"
Sep 17 22:17:48.398370 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:48.398222    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-sn7qh"]
Sep 17 22:17:48.400927 ip-10-0-138-45 kubenswrapper[2563]: W0917 22:17:48.400897    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae8f85df_f769_4b9c_9e5e_94f2dd1fa63e.slice/crio-7b7d5d99a43c8f24bbdbfc4a28b135bab346baed09b13189bcde2e51d72d8404 WatchSource:0}: Error finding container 7b7d5d99a43c8f24bbdbfc4a28b135bab346baed09b13189bcde2e51d72d8404: Status 404 returned error can't find the container with id 7b7d5d99a43c8f24bbdbfc4a28b135bab346baed09b13189bcde2e51d72d8404
Sep 17 22:17:48.811308 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:48.811262    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-sn7qh" event={"ID":"ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e","Type":"ContainerStarted","Data":"7ea22d3b4cc9764653e9c9668ce65b0717779354da902435a41f761ee15848f7"}
Sep 17 22:17:48.811514 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:48.811367    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-sn7qh" event={"ID":"ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e","Type":"ContainerStarted","Data":"7b7d5d99a43c8f24bbdbfc4a28b135bab346baed09b13189bcde2e51d72d8404"}
Sep 17 22:17:48.914393 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:48.914351    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="333cb4d8-75ac-46ab-af99-793fb0ae133f" path="/var/lib/kubelet/pods/333cb4d8-75ac-46ab-af99-793fb0ae133f/volumes"
Sep 17 22:17:52.829191 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:52.829152    2563 generic.go:356] "Generic (PLEG): container finished" podID="ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e" containerID="7ea22d3b4cc9764653e9c9668ce65b0717779354da902435a41f761ee15848f7" exitCode=0
Sep 17 22:17:52.829568 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:52.829219    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-sn7qh" event={"ID":"ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e","Type":"ContainerDied","Data":"7ea22d3b4cc9764653e9c9668ce65b0717779354da902435a41f761ee15848f7"}
Sep 17 22:17:53.835192 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:53.835156    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-sn7qh" event={"ID":"ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e","Type":"ContainerStarted","Data":"e7c14ae40d1fec1427e95d4b87fde40d7ae39e67e4a019aa2a9638bd152922c4"}
Sep 17 22:17:53.835608 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:53.835252    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-sn7qh" event={"ID":"ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e","Type":"ContainerStarted","Data":"5da0bfc093a3b723747fe527dc282cab1de7cab2a2f02cdbfe8c1050892721ae"}
Sep 17 22:17:53.835608 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:53.835572    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-sn7qh"
Sep 17 22:17:53.855669 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:53.855611    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-sn7qh" podStartSLOduration=3.070572801 podStartE2EDuration="6.855597966s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.785025165s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:17:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:17:53.854580469 +0000 UTC m=+1185.022533455" watchObservedRunningTime="2026-09-17 22:17:53.855597966 +0000 UTC m=+1185.023550948"
Sep 17 22:17:54.839464 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:17:54.839426    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-sn7qh"
Sep 17 22:18:00.848527 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:18:00.848492    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-sn7qh"
Sep 17 22:18:10.911792 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:18:10.911760    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nrkd5_34b8ef2b-97ba-4562-87ee-8da0cb4604c7/ovn-acl-logging/0.log"
Sep 17 22:18:10.912689 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:18:10.912660    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nrkd5_34b8ef2b-97ba-4562-87ee-8da0cb4604c7/ovn-acl-logging/0.log"
Sep 17 22:18:30.850247 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:18:30.850203    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-sn7qh" podUID="ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.30:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 22:18:40.849493 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:18:40.849449    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-sn7qh" podUID="ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.30:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 22:18:50.849973 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:18:50.849931    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-sn7qh" podUID="ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.30:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 22:19:00.849921 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:00.849834    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-sn7qh" podUID="ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.30:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 22:19:10.849722 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:10.849674    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-sn7qh" podUID="ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.30:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 22:19:21.455017 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:21.454980    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-sn7qh"
Sep 17 22:19:27.279563 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:27.279518    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-sn7qh"]
Sep 17 22:19:27.280034 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:27.279750    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-sn7qh" podUID="ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e" containerName="kserve-container" containerID="cri-o://5da0bfc093a3b723747fe527dc282cab1de7cab2a2f02cdbfe8c1050892721ae" gracePeriod=30
Sep 17 22:19:27.280034 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:27.279784    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-sn7qh" podUID="ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e" containerName="kube-rbac-proxy" containerID="cri-o://e7c14ae40d1fec1427e95d4b87fde40d7ae39e67e4a019aa2a9638bd152922c4" gracePeriod=30
Sep 17 22:19:28.218999 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:28.218966    2563 generic.go:356] "Generic (PLEG): container finished" podID="ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e" containerID="e7c14ae40d1fec1427e95d4b87fde40d7ae39e67e4a019aa2a9638bd152922c4" exitCode=2
Sep 17 22:19:28.219206 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:28.219039    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-sn7qh" event={"ID":"ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e","Type":"ContainerDied","Data":"e7c14ae40d1fec1427e95d4b87fde40d7ae39e67e4a019aa2a9638bd152922c4"}
Sep 17 22:19:30.844211 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:30.844171    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-sn7qh" podUID="ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Sep 17 22:19:30.849623 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:30.849591    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-sn7qh" podUID="ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.30:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 22:19:32.937746 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:32.937717    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-sn7qh"
Sep 17 22:19:33.069683 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:33.069576    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e-proxy-tls\") pod \"ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e\" (UID: \"ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e\") "
Sep 17 22:19:33.069683 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:33.069618    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e-kube-rbac-proxy-sar-config\") pod \"ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e\" (UID: \"ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e\") "
Sep 17 22:19:33.069683 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:33.069686    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e-kserve-provision-location\") pod \"ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e\" (UID: \"ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e\") "
Sep 17 22:19:33.069980 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:33.069708    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q49p8\" (UniqueName: \"kubernetes.io/projected/ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e-kube-api-access-q49p8\") pod \"ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e\" (UID: \"ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e\") "
Sep 17 22:19:33.069980 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:33.069725    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e-cabundle-cert\") pod \"ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e\" (UID: \"ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e\") "
Sep 17 22:19:33.070164 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:33.070099    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e" (UID: "ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:19:33.070164 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:33.070128    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e" (UID: "ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:19:33.070318 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:33.070105    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e" (UID: "ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:19:33.071850 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:33.071826    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e-kube-api-access-q49p8" (OuterVolumeSpecName: "kube-api-access-q49p8") pod "ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e" (UID: "ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e"). InnerVolumeSpecName "kube-api-access-q49p8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:19:33.071943 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:33.071875    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e" (UID: "ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:19:33.170441 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:33.170403    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e-kserve-provision-location\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:19:33.170441 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:33.170434    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q49p8\" (UniqueName: \"kubernetes.io/projected/ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e-kube-api-access-q49p8\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:19:33.170441 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:33.170444    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e-cabundle-cert\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:19:33.170441 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:33.170453    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e-proxy-tls\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:19:33.170722 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:33.170464    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:19:33.241456 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:33.241425    2563 generic.go:356] "Generic (PLEG): container finished" podID="ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e" containerID="5da0bfc093a3b723747fe527dc282cab1de7cab2a2f02cdbfe8c1050892721ae" exitCode=0
Sep 17 22:19:33.241456 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:33.241461    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-sn7qh" event={"ID":"ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e","Type":"ContainerDied","Data":"5da0bfc093a3b723747fe527dc282cab1de7cab2a2f02cdbfe8c1050892721ae"}
Sep 17 22:19:33.241719 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:33.241495    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-sn7qh"
Sep 17 22:19:33.241719 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:33.241512    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-sn7qh" event={"ID":"ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e","Type":"ContainerDied","Data":"7b7d5d99a43c8f24bbdbfc4a28b135bab346baed09b13189bcde2e51d72d8404"}
Sep 17 22:19:33.241719 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:33.241540    2563 scope.go:117] "RemoveContainer" containerID="e7c14ae40d1fec1427e95d4b87fde40d7ae39e67e4a019aa2a9638bd152922c4"
Sep 17 22:19:33.250760 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:33.250741    2563 scope.go:117] "RemoveContainer" containerID="5da0bfc093a3b723747fe527dc282cab1de7cab2a2f02cdbfe8c1050892721ae"
Sep 17 22:19:33.258689 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:33.258673    2563 scope.go:117] "RemoveContainer" containerID="7ea22d3b4cc9764653e9c9668ce65b0717779354da902435a41f761ee15848f7"
Sep 17 22:19:33.262269 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:33.262243    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-sn7qh"]
Sep 17 22:19:33.266453 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:33.266429    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-sn7qh"]
Sep 17 22:19:33.266607 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:33.266593    2563 scope.go:117] "RemoveContainer" containerID="e7c14ae40d1fec1427e95d4b87fde40d7ae39e67e4a019aa2a9638bd152922c4"
Sep 17 22:19:33.266890 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:19:33.266874    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7c14ae40d1fec1427e95d4b87fde40d7ae39e67e4a019aa2a9638bd152922c4\": container with ID starting with e7c14ae40d1fec1427e95d4b87fde40d7ae39e67e4a019aa2a9638bd152922c4 not found: ID does not exist" containerID="e7c14ae40d1fec1427e95d4b87fde40d7ae39e67e4a019aa2a9638bd152922c4"
Sep 17 22:19:33.266929 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:33.266900    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7c14ae40d1fec1427e95d4b87fde40d7ae39e67e4a019aa2a9638bd152922c4"} err="failed to get container status \"e7c14ae40d1fec1427e95d4b87fde40d7ae39e67e4a019aa2a9638bd152922c4\": rpc error: code = NotFound desc = could not find container \"e7c14ae40d1fec1427e95d4b87fde40d7ae39e67e4a019aa2a9638bd152922c4\": container with ID starting with e7c14ae40d1fec1427e95d4b87fde40d7ae39e67e4a019aa2a9638bd152922c4 not found: ID does not exist"
Sep 17 22:19:33.266929 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:33.266918    2563 scope.go:117] "RemoveContainer" containerID="5da0bfc093a3b723747fe527dc282cab1de7cab2a2f02cdbfe8c1050892721ae"
Sep 17 22:19:33.267194 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:19:33.267177    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5da0bfc093a3b723747fe527dc282cab1de7cab2a2f02cdbfe8c1050892721ae\": container with ID starting with 5da0bfc093a3b723747fe527dc282cab1de7cab2a2f02cdbfe8c1050892721ae not found: ID does not exist" containerID="5da0bfc093a3b723747fe527dc282cab1de7cab2a2f02cdbfe8c1050892721ae"
Sep 17 22:19:33.267246 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:33.267199    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5da0bfc093a3b723747fe527dc282cab1de7cab2a2f02cdbfe8c1050892721ae"} err="failed to get container status \"5da0bfc093a3b723747fe527dc282cab1de7cab2a2f02cdbfe8c1050892721ae\": rpc error: code = NotFound desc = could not find container \"5da0bfc093a3b723747fe527dc282cab1de7cab2a2f02cdbfe8c1050892721ae\": container with ID starting with 5da0bfc093a3b723747fe527dc282cab1de7cab2a2f02cdbfe8c1050892721ae not found: ID does not exist"
Sep 17 22:19:33.267246 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:33.267213    2563 scope.go:117] "RemoveContainer" containerID="7ea22d3b4cc9764653e9c9668ce65b0717779354da902435a41f761ee15848f7"
Sep 17 22:19:33.267468 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:19:33.267450    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ea22d3b4cc9764653e9c9668ce65b0717779354da902435a41f761ee15848f7\": container with ID starting with 7ea22d3b4cc9764653e9c9668ce65b0717779354da902435a41f761ee15848f7 not found: ID does not exist" containerID="7ea22d3b4cc9764653e9c9668ce65b0717779354da902435a41f761ee15848f7"
Sep 17 22:19:33.267505 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:33.267474    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ea22d3b4cc9764653e9c9668ce65b0717779354da902435a41f761ee15848f7"} err="failed to get container status \"7ea22d3b4cc9764653e9c9668ce65b0717779354da902435a41f761ee15848f7\": rpc error: code = NotFound desc = could not find container \"7ea22d3b4cc9764653e9c9668ce65b0717779354da902435a41f761ee15848f7\": container with ID starting with 7ea22d3b4cc9764653e9c9668ce65b0717779354da902435a41f761ee15848f7 not found: ID does not exist"
Sep 17 22:19:34.914024 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:34.913989    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e" path="/var/lib/kubelet/pods/ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e/volumes"
Sep 17 22:19:50.031693 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:50.031659    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/curl-metrics-79b34d4b"]
Sep 17 22:19:50.032179 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:50.031932    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="333cb4d8-75ac-46ab-af99-793fb0ae133f" containerName="kube-rbac-proxy"
Sep 17 22:19:50.032179 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:50.031943    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="333cb4d8-75ac-46ab-af99-793fb0ae133f" containerName="kube-rbac-proxy"
Sep 17 22:19:50.032179 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:50.031956    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e" containerName="storage-initializer"
Sep 17 22:19:50.032179 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:50.031962    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e" containerName="storage-initializer"
Sep 17 22:19:50.032179 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:50.031977    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e" containerName="kube-rbac-proxy"
Sep 17 22:19:50.032179 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:50.031982    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e" containerName="kube-rbac-proxy"
Sep 17 22:19:50.032179 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:50.031991    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="333cb4d8-75ac-46ab-af99-793fb0ae133f" containerName="storage-initializer"
Sep 17 22:19:50.032179 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:50.031996    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="333cb4d8-75ac-46ab-af99-793fb0ae133f" containerName="storage-initializer"
Sep 17 22:19:50.032179 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:50.032005    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="333cb4d8-75ac-46ab-af99-793fb0ae133f" containerName="kserve-container"
Sep 17 22:19:50.032179 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:50.032010    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="333cb4d8-75ac-46ab-af99-793fb0ae133f" containerName="kserve-container"
Sep 17 22:19:50.032179 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:50.032016    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e" containerName="kserve-container"
Sep 17 22:19:50.032179 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:50.032021    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e" containerName="kserve-container"
Sep 17 22:19:50.032179 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:50.032066    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e" containerName="kube-rbac-proxy"
Sep 17 22:19:50.032179 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:50.032073    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="333cb4d8-75ac-46ab-af99-793fb0ae133f" containerName="kserve-container"
Sep 17 22:19:50.032179 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:50.032078    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae8f85df-f769-4b9c-9e5e-94f2dd1fa63e" containerName="kserve-container"
Sep 17 22:19:50.032179 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:50.032084    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="333cb4d8-75ac-46ab-af99-793fb0ae133f" containerName="kube-rbac-proxy"
Sep 17 22:19:50.036726 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:50.036701    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-79b34d4b"
Sep 17 22:19:50.038916 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:50.038893    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-ndvdn\""
Sep 17 22:19:50.039041 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:50.038894    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 22:19:50.040177 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:50.039658    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 22:19:50.041865 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:50.041837    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-79b34d4b"]
Sep 17 22:19:50.214498 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:50.214452    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kchzc\" (UniqueName: \"kubernetes.io/projected/744db51e-a88e-4f44-8f2e-7be116689b9f-kube-api-access-kchzc\") pod \"curl-metrics-79b34d4b\" (UID: \"744db51e-a88e-4f44-8f2e-7be116689b9f\") " pod="kserve/curl-metrics-79b34d4b"
Sep 17 22:19:50.315527 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:50.315439    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kchzc\" (UniqueName: \"kubernetes.io/projected/744db51e-a88e-4f44-8f2e-7be116689b9f-kube-api-access-kchzc\") pod \"curl-metrics-79b34d4b\" (UID: \"744db51e-a88e-4f44-8f2e-7be116689b9f\") " pod="kserve/curl-metrics-79b34d4b"
Sep 17 22:19:50.324507 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:50.324479    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kchzc\" (UniqueName: \"kubernetes.io/projected/744db51e-a88e-4f44-8f2e-7be116689b9f-kube-api-access-kchzc\") pod \"curl-metrics-79b34d4b\" (UID: \"744db51e-a88e-4f44-8f2e-7be116689b9f\") " pod="kserve/curl-metrics-79b34d4b"
Sep 17 22:19:50.355047 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:50.355013    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-79b34d4b"
Sep 17 22:19:50.625678 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:50.625603    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-79b34d4b"]
Sep 17 22:19:50.629127 ip-10-0-138-45 kubenswrapper[2563]: W0917 22:19:50.629104    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod744db51e_a88e_4f44_8f2e_7be116689b9f.slice/crio-73322565e5d8444b5548736e39bf3037d923683dee1124862993dba9cb2ffda1 WatchSource:0}: Error finding container 73322565e5d8444b5548736e39bf3037d923683dee1124862993dba9cb2ffda1: Status 404 returned error can't find the container with id 73322565e5d8444b5548736e39bf3037d923683dee1124862993dba9cb2ffda1
Sep 17 22:19:50.631432 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:50.631412    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 22:19:51.317232 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:51.317156    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-79b34d4b" event={"ID":"744db51e-a88e-4f44-8f2e-7be116689b9f","Type":"ContainerStarted","Data":"73322565e5d8444b5548736e39bf3037d923683dee1124862993dba9cb2ffda1"}
Sep 17 22:19:52.321962 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:52.321922    2563 generic.go:356] "Generic (PLEG): container finished" podID="744db51e-a88e-4f44-8f2e-7be116689b9f" containerID="e06fdc914d78ff13fc5b777b14903e2521093e594a98d7447c0e82dffb69b7d4" exitCode=0
Sep 17 22:19:52.322402 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:52.322011    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-79b34d4b" event={"ID":"744db51e-a88e-4f44-8f2e-7be116689b9f","Type":"ContainerDied","Data":"e06fdc914d78ff13fc5b777b14903e2521093e594a98d7447c0e82dffb69b7d4"}
Sep 17 22:19:53.450554 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:53.450530    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-79b34d4b"
Sep 17 22:19:53.484953 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:53.484919    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_curl-metrics-79b34d4b_744db51e-a88e-4f44-8f2e-7be116689b9f/curl-metrics-79b34d4b/0.log"
Sep 17 22:19:53.509921 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:53.509890    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/curl-metrics-79b34d4b"]
Sep 17 22:19:53.513578 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:53.513554    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/curl-metrics-79b34d4b"]
Sep 17 22:19:53.543526 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:53.543499    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kchzc\" (UniqueName: \"kubernetes.io/projected/744db51e-a88e-4f44-8f2e-7be116689b9f-kube-api-access-kchzc\") pod \"744db51e-a88e-4f44-8f2e-7be116689b9f\" (UID: \"744db51e-a88e-4f44-8f2e-7be116689b9f\") "
Sep 17 22:19:53.545560 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:53.545534    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/744db51e-a88e-4f44-8f2e-7be116689b9f-kube-api-access-kchzc" (OuterVolumeSpecName: "kube-api-access-kchzc") pod "744db51e-a88e-4f44-8f2e-7be116689b9f" (UID: "744db51e-a88e-4f44-8f2e-7be116689b9f"). InnerVolumeSpecName "kube-api-access-kchzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:19:53.644834 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:53.644752    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kchzc\" (UniqueName: \"kubernetes.io/projected/744db51e-a88e-4f44-8f2e-7be116689b9f-kube-api-access-kchzc\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:19:54.331453 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:54.331426    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-79b34d4b"
Sep 17 22:19:54.331622 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:54.331474    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73322565e5d8444b5548736e39bf3037d923683dee1124862993dba9cb2ffda1"
Sep 17 22:19:54.340684 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:19:54.340661    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"curl-metrics-79b34d4b\" is forbidden: User \"system:node:ip-10-0-138-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kserve\": no relationship found between node 'ip-10-0-138-45.ec2.internal' and this object" podUID="744db51e-a88e-4f44-8f2e-7be116689b9f" pod="kserve/curl-metrics-79b34d4b"
Sep 17 22:19:54.913212 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:54.913177    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="744db51e-a88e-4f44-8f2e-7be116689b9f" path="/var/lib/kubelet/pods/744db51e-a88e-4f44-8f2e-7be116689b9f/volumes"
Sep 17 22:19:58.917578 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:58.917547    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/curl-metrics-29bfa30a"]
Sep 17 22:19:58.918061 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:58.917782    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="744db51e-a88e-4f44-8f2e-7be116689b9f" containerName="curl-metrics-79b34d4b"
Sep 17 22:19:58.918061 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:58.917793    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="744db51e-a88e-4f44-8f2e-7be116689b9f" containerName="curl-metrics-79b34d4b"
Sep 17 22:19:58.918061 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:58.917853    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="744db51e-a88e-4f44-8f2e-7be116689b9f" containerName="curl-metrics-79b34d4b"
Sep 17 22:19:58.921051 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:58.921030    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-29bfa30a"]
Sep 17 22:19:58.921176 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:58.921106    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-29bfa30a"
Sep 17 22:19:58.923779 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:58.923758    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 22:19:58.923912 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:58.923760    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 22:19:58.924586 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:58.924568    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-ndvdn\""
Sep 17 22:19:59.088110 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:59.088077    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4fj5\" (UniqueName: \"kubernetes.io/projected/99738559-ef81-4cbe-90f8-6eeed2f58d45-kube-api-access-h4fj5\") pod \"curl-metrics-29bfa30a\" (UID: \"99738559-ef81-4cbe-90f8-6eeed2f58d45\") " pod="kserve/curl-metrics-29bfa30a"
Sep 17 22:19:59.189451 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:59.189359    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4fj5\" (UniqueName: \"kubernetes.io/projected/99738559-ef81-4cbe-90f8-6eeed2f58d45-kube-api-access-h4fj5\") pod \"curl-metrics-29bfa30a\" (UID: \"99738559-ef81-4cbe-90f8-6eeed2f58d45\") " pod="kserve/curl-metrics-29bfa30a"
Sep 17 22:19:59.197373 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:59.197351    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4fj5\" (UniqueName: \"kubernetes.io/projected/99738559-ef81-4cbe-90f8-6eeed2f58d45-kube-api-access-h4fj5\") pod \"curl-metrics-29bfa30a\" (UID: \"99738559-ef81-4cbe-90f8-6eeed2f58d45\") " pod="kserve/curl-metrics-29bfa30a"
Sep 17 22:19:59.241404 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:59.241376    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-29bfa30a"
Sep 17 22:19:59.506018 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:19:59.505942    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-29bfa30a"]
Sep 17 22:19:59.509655 ip-10-0-138-45 kubenswrapper[2563]: W0917 22:19:59.509626    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99738559_ef81_4cbe_90f8_6eeed2f58d45.slice/crio-e33befd632894f52c0f7123cb901fc24631779c8543c90756924b6ce366d38f6 WatchSource:0}: Error finding container e33befd632894f52c0f7123cb901fc24631779c8543c90756924b6ce366d38f6: Status 404 returned error can't find the container with id e33befd632894f52c0f7123cb901fc24631779c8543c90756924b6ce366d38f6
Sep 17 22:20:00.356364 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:20:00.356330    2563 generic.go:356] "Generic (PLEG): container finished" podID="99738559-ef81-4cbe-90f8-6eeed2f58d45" containerID="4df06c8e1c550f70796ce1a66d0d44394b5b893dc189f924bbad5f909b63712a" exitCode=0
Sep 17 22:20:00.356776 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:20:00.356386    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-29bfa30a" event={"ID":"99738559-ef81-4cbe-90f8-6eeed2f58d45","Type":"ContainerDied","Data":"4df06c8e1c550f70796ce1a66d0d44394b5b893dc189f924bbad5f909b63712a"}
Sep 17 22:20:00.356776 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:20:00.356415    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-29bfa30a" event={"ID":"99738559-ef81-4cbe-90f8-6eeed2f58d45","Type":"ContainerStarted","Data":"e33befd632894f52c0f7123cb901fc24631779c8543c90756924b6ce366d38f6"}
Sep 17 22:20:01.489009 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:20:01.488980    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-29bfa30a"
Sep 17 22:20:01.506544 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:20:01.506513    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h4fj5\" (UniqueName: \"kubernetes.io/projected/99738559-ef81-4cbe-90f8-6eeed2f58d45-kube-api-access-h4fj5\") pod \"99738559-ef81-4cbe-90f8-6eeed2f58d45\" (UID: \"99738559-ef81-4cbe-90f8-6eeed2f58d45\") "
Sep 17 22:20:01.508545 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:20:01.508512    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99738559-ef81-4cbe-90f8-6eeed2f58d45-kube-api-access-h4fj5" (OuterVolumeSpecName: "kube-api-access-h4fj5") pod "99738559-ef81-4cbe-90f8-6eeed2f58d45" (UID: "99738559-ef81-4cbe-90f8-6eeed2f58d45"). InnerVolumeSpecName "kube-api-access-h4fj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:20:01.510156 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:20:01.510112    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_curl-metrics-29bfa30a_99738559-ef81-4cbe-90f8-6eeed2f58d45/curl-metrics-29bfa30a/0.log"
Sep 17 22:20:01.535735 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:20:01.535710    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/curl-metrics-29bfa30a"]
Sep 17 22:20:01.540294 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:20:01.540270    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/curl-metrics-29bfa30a"]
Sep 17 22:20:01.607827 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:20:01.607798    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4fj5\" (UniqueName: \"kubernetes.io/projected/99738559-ef81-4cbe-90f8-6eeed2f58d45-kube-api-access-h4fj5\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:20:02.365520 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:20:02.365494    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-29bfa30a"
Sep 17 22:20:02.365699 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:20:02.365527    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e33befd632894f52c0f7123cb901fc24631779c8543c90756924b6ce366d38f6"
Sep 17 22:20:02.374929 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:20:02.374902    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"curl-metrics-29bfa30a\" is forbidden: User \"system:node:ip-10-0-138-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kserve\": no relationship found between node 'ip-10-0-138-45.ec2.internal' and this object" podUID="99738559-ef81-4cbe-90f8-6eeed2f58d45" pod="kserve/curl-metrics-29bfa30a"
Sep 17 22:20:02.915607 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:20:02.915573    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99738559-ef81-4cbe-90f8-6eeed2f58d45" path="/var/lib/kubelet/pods/99738559-ef81-4cbe-90f8-6eeed2f58d45/volumes"
Sep 17 22:20:06.915531 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:20:06.915493    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-p9vh9"]
Sep 17 22:20:06.915995 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:20:06.915878    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99738559-ef81-4cbe-90f8-6eeed2f58d45" containerName="curl-metrics-29bfa30a"
Sep 17 22:20:06.915995 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:20:06.915895    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="99738559-ef81-4cbe-90f8-6eeed2f58d45" containerName="curl-metrics-29bfa30a"
Sep 17 22:20:06.915995 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:20:06.915972    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="99738559-ef81-4cbe-90f8-6eeed2f58d45" containerName="curl-metrics-29bfa30a"
Sep 17 22:20:06.919210 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:20:06.919188    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-p9vh9"
Sep 17 22:20:06.921651 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:20:06.921627    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 22:20:06.921816 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:20:06.921789    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-predictor-serving-cert\""
Sep 17 22:20:06.921875 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:20:06.921839    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:20:06.921913 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:20:06.921792    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v4p4m\""
Sep 17 22:20:06.922096 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:20:06.922075    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-kube-rbac-proxy-sar-config\""
Sep 17 22:20:06.922398 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:20:06.922381    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 22:20:06.922489 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:20:06.922441    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 22:20:06.926887 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:20:06.926863    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-p9vh9"]
Sep 17 22:20:06.947914 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:20:06.947869    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2907464-0041-49ef-a661-7372a6ecd32e-cabundle-cert\") pod \"isvc-sklearn-predictor-56c47844df-p9vh9\" (UID: \"d2907464-0041-49ef-a661-7372a6ecd32e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-p9vh9"
Sep 17 22:20:06.948292 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:20:06.948267    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r687s\" (UniqueName: \"kubernetes.io/projected/d2907464-0041-49ef-a661-7372a6ecd32e-kube-api-access-r687s\") pod \"isvc-sklearn-predictor-56c47844df-p9vh9\" (UID: \"d2907464-0041-49ef-a661-7372a6ecd32e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-p9vh9"
Sep 17 22:20:06.948426 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:20:06.948404    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2907464-0041-49ef-a661-7372a6ecd32e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-56c47844df-p9vh9\" (UID: \"d2907464-0041-49ef-a661-7372a6ecd32e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-p9vh9"
Sep 17 22:20:06.948585 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:20:06.948562    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2907464-0041-49ef-a661-7372a6ecd32e-proxy-tls\") pod \"isvc-sklearn-predictor-56c47844df-p9vh9\" (UID: \"d2907464-0041-49ef-a661-7372a6ecd32e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-p9vh9"
Sep 17 22:20:06.948664 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:20:06.948600    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2907464-0041-49ef-a661-7372a6ecd32e-kserve-provision-location\") pod \"isvc-sklearn-predictor-56c47844df-p9vh9\" (UID: \"d2907464-0041-49ef-a661-7372a6ecd32e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-p9vh9"
Sep 17 22:20:07.049535 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:20:07.049502    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2907464-0041-49ef-a661-7372a6ecd32e-cabundle-cert\") pod \"isvc-sklearn-predictor-56c47844df-p9vh9\" (UID: \"d2907464-0041-49ef-a661-7372a6ecd32e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-p9vh9"
Sep 17 22:20:07.049707 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:20:07.049552    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r687s\" (UniqueName: \"kubernetes.io/projected/d2907464-0041-49ef-a661-7372a6ecd32e-kube-api-access-r687s\") pod \"isvc-sklearn-predictor-56c47844df-p9vh9\" (UID: \"d2907464-0041-49ef-a661-7372a6ecd32e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-p9vh9"
Sep 17 22:20:07.049707 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:20:07.049577    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2907464-0041-49ef-a661-7372a6ecd32e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-56c47844df-p9vh9\" (UID: \"d2907464-0041-49ef-a661-7372a6ecd32e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-p9vh9"
Sep 17 22:20:07.049707 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:20:07.049595    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2907464-0041-49ef-a661-7372a6ecd32e-proxy-tls\") pod \"isvc-sklearn-predictor-56c47844df-p9vh9\" (UID: \"d2907464-0041-49ef-a661-7372a6ecd32e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-p9vh9"
Sep 17 22:20:07.049707 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:20:07.049610    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2907464-0041-49ef-a661-7372a6ecd32e-kserve-provision-location\") pod \"isvc-sklearn-predictor-56c47844df-p9vh9\" (UID: \"d2907464-0041-49ef-a661-7372a6ecd32e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-p9vh9"
Sep 17 22:20:07.049997 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:20:07.049766    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-predictor-serving-cert: secret "isvc-sklearn-predictor-serving-cert" not found
Sep 17 22:20:07.049997 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:20:07.049853    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2907464-0041-49ef-a661-7372a6ecd32e-proxy-tls podName:d2907464-0041-49ef-a661-7372a6ecd32e nodeName:}" failed. No retries permitted until 2026-09-17 22:20:07.54982997 +0000 UTC m=+1318.717782946 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d2907464-0041-49ef-a661-7372a6ecd32e-proxy-tls") pod "isvc-sklearn-predictor-56c47844df-p9vh9" (UID: "d2907464-0041-49ef-a661-7372a6ecd32e") : secret "isvc-sklearn-predictor-serving-cert" not found
Sep 17 22:20:07.050347 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:20:07.050327    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2907464-0041-49ef-a661-7372a6ecd32e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-56c47844df-p9vh9\" (UID: \"d2907464-0041-49ef-a661-7372a6ecd32e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-p9vh9"
Sep 17 22:20:07.050403 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:20:07.050328    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2907464-0041-49ef-a661-7372a6ecd32e-cabundle-cert\") pod \"isvc-sklearn-predictor-56c47844df-p9vh9\" (UID: \"d2907464-0041-49ef-a661-7372a6ecd32e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-p9vh9"
Sep 17 22:20:07.050403 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:20:07.050333    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2907464-0041-49ef-a661-7372a6ecd32e-kserve-provision-location\") pod \"isvc-sklearn-predictor-56c47844df-p9vh9\" (UID: \"d2907464-0041-49ef-a661-7372a6ecd32e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-p9vh9"
Sep 17 22:20:07.058346 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:20:07.058324    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r687s\" (UniqueName: \"kubernetes.io/projected/d2907464-0041-49ef-a661-7372a6ecd32e-kube-api-access-r687s\") pod \"isvc-sklearn-predictor-56c47844df-p9vh9\" (UID: \"d2907464-0041-49ef-a661-7372a6ecd32e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-p9vh9"
Sep 17 22:20:07.553893 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:20:07.553857    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2907464-0041-49ef-a661-7372a6ecd32e-proxy-tls\") pod \"isvc-sklearn-predictor-56c47844df-p9vh9\" (UID: \"d2907464-0041-49ef-a661-7372a6ecd32e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-p9vh9"
Sep 17 22:20:07.556129 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:20:07.556104    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2907464-0041-49ef-a661-7372a6ecd32e-proxy-tls\") pod \"isvc-sklearn-predictor-56c47844df-p9vh9\" (UID: \"d2907464-0041-49ef-a661-7372a6ecd32e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-p9vh9"
Sep 17 22:20:07.829909 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:20:07.829810    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-p9vh9"
Sep 17 22:20:08.100687 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:20:08.100661    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-p9vh9"]
Sep 17 22:20:08.102939 ip-10-0-138-45 kubenswrapper[2563]: W0917 22:20:08.102912    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2907464_0041_49ef_a661_7372a6ecd32e.slice/crio-5045df2a7ac338e5c0586b7ef15195676cc7fbf9678b7fba38f2446b2cc1ec31 WatchSource:0}: Error finding container 5045df2a7ac338e5c0586b7ef15195676cc7fbf9678b7fba38f2446b2cc1ec31: Status 404 returned error can't find the container with id 5045df2a7ac338e5c0586b7ef15195676cc7fbf9678b7fba38f2446b2cc1ec31
Sep 17 22:20:08.391703 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:20:08.391620    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-p9vh9" event={"ID":"d2907464-0041-49ef-a661-7372a6ecd32e","Type":"ContainerStarted","Data":"e2b78d2f2842f79a0201fb862faa18c8360dfd043a16f5e9094da9b37e816b04"}
Sep 17 22:20:08.391849 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:20:08.391707    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-p9vh9" event={"ID":"d2907464-0041-49ef-a661-7372a6ecd32e","Type":"ContainerStarted","Data":"5045df2a7ac338e5c0586b7ef15195676cc7fbf9678b7fba38f2446b2cc1ec31"}
Sep 17 22:20:12.411254 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:20:12.411224    2563 generic.go:356] "Generic (PLEG): container finished" podID="d2907464-0041-49ef-a661-7372a6ecd32e" containerID="e2b78d2f2842f79a0201fb862faa18c8360dfd043a16f5e9094da9b37e816b04" exitCode=0
Sep 17 22:20:12.411602 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:20:12.411294    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-p9vh9" event={"ID":"d2907464-0041-49ef-a661-7372a6ecd32e","Type":"ContainerDied","Data":"e2b78d2f2842f79a0201fb862faa18c8360dfd043a16f5e9094da9b37e816b04"}
Sep 17 22:20:13.417221 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:20:13.417184    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-p9vh9" event={"ID":"d2907464-0041-49ef-a661-7372a6ecd32e","Type":"ContainerStarted","Data":"5fe4750642abb32046aebed4895d920f6d8313482b051ad202e4229e63a0dd0f"}
Sep 17 22:20:13.417605 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:20:13.417246    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-p9vh9" event={"ID":"d2907464-0041-49ef-a661-7372a6ecd32e","Type":"ContainerStarted","Data":"c0fea075ff01bfbb7462a61cad89e6aeaa16415bdd13753de17e8020cb9a672a"}
Sep 17 22:20:13.417605 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:20:13.417526    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-p9vh9"
Sep 17 22:20:13.417677 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:20:13.417660    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-p9vh9"
Sep 17 22:20:13.418977 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:20:13.418952    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-p9vh9" podUID="d2907464-0041-49ef-a661-7372a6ecd32e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 22:20:13.437437 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:20:13.437385    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-p9vh9" podStartSLOduration=3.675709951 podStartE2EDuration="7.437369815s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.761659864s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:20:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:20:13.435685553 +0000 UTC m=+1324.603638546" watchObservedRunningTime="2026-09-17 22:20:13.437369815 +0000 UTC m=+1324.605322850"
Sep 17 22:20:14.421058 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:20:14.421018    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-p9vh9" podUID="d2907464-0041-49ef-a661-7372a6ecd32e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 22:20:19.425128 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:20:19.425097    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-p9vh9"
Sep 17 22:20:19.425731 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:20:19.425698    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-p9vh9" podUID="d2907464-0041-49ef-a661-7372a6ecd32e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 22:20:29.426582 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:20:29.426486    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-p9vh9" podUID="d2907464-0041-49ef-a661-7372a6ecd32e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 22:20:39.425999 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:20:39.425960    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-p9vh9" podUID="d2907464-0041-49ef-a661-7372a6ecd32e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 22:20:49.425693 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:20:49.425650    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-p9vh9" podUID="d2907464-0041-49ef-a661-7372a6ecd32e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 22:20:59.426369 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:20:59.426326    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-p9vh9" podUID="d2907464-0041-49ef-a661-7372a6ecd32e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 22:21:09.425805 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:09.425762    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-p9vh9" podUID="d2907464-0041-49ef-a661-7372a6ecd32e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 22:21:19.425983 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:19.425945    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-p9vh9" podUID="d2907464-0041-49ef-a661-7372a6ecd32e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 22:21:20.911684 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:20.911643    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-p9vh9" podUID="d2907464-0041-49ef-a661-7372a6ecd32e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 22:21:30.912994 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:30.912966    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-p9vh9"
Sep 17 22:21:36.973551 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:36.973518    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-p9vh9"]
Sep 17 22:21:36.974114 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:36.973821    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-p9vh9" podUID="d2907464-0041-49ef-a661-7372a6ecd32e" containerName="kserve-container" containerID="cri-o://c0fea075ff01bfbb7462a61cad89e6aeaa16415bdd13753de17e8020cb9a672a" gracePeriod=30
Sep 17 22:21:36.974114 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:36.973874    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-p9vh9" podUID="d2907464-0041-49ef-a661-7372a6ecd32e" containerName="kube-rbac-proxy" containerID="cri-o://5fe4750642abb32046aebed4895d920f6d8313482b051ad202e4229e63a0dd0f" gracePeriod=30
Sep 17 22:21:37.763919 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:37.763888    2563 generic.go:356] "Generic (PLEG): container finished" podID="d2907464-0041-49ef-a661-7372a6ecd32e" containerID="5fe4750642abb32046aebed4895d920f6d8313482b051ad202e4229e63a0dd0f" exitCode=2
Sep 17 22:21:37.763919 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:37.763930    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-p9vh9" event={"ID":"d2907464-0041-49ef-a661-7372a6ecd32e","Type":"ContainerDied","Data":"5fe4750642abb32046aebed4895d920f6d8313482b051ad202e4229e63a0dd0f"}
Sep 17 22:21:39.421950 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:39.421907    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-p9vh9" podUID="d2907464-0041-49ef-a661-7372a6ecd32e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Sep 17 22:21:40.911686 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:40.911648    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-p9vh9" podUID="d2907464-0041-49ef-a661-7372a6ecd32e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 22:21:41.717066 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:41.717042    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-p9vh9"
Sep 17 22:21:41.781858 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:41.781825    2563 generic.go:356] "Generic (PLEG): container finished" podID="d2907464-0041-49ef-a661-7372a6ecd32e" containerID="c0fea075ff01bfbb7462a61cad89e6aeaa16415bdd13753de17e8020cb9a672a" exitCode=0
Sep 17 22:21:41.782047 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:41.781884    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-p9vh9"
Sep 17 22:21:41.782047 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:41.781894    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-p9vh9" event={"ID":"d2907464-0041-49ef-a661-7372a6ecd32e","Type":"ContainerDied","Data":"c0fea075ff01bfbb7462a61cad89e6aeaa16415bdd13753de17e8020cb9a672a"}
Sep 17 22:21:41.782047 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:41.781943    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-p9vh9" event={"ID":"d2907464-0041-49ef-a661-7372a6ecd32e","Type":"ContainerDied","Data":"5045df2a7ac338e5c0586b7ef15195676cc7fbf9678b7fba38f2446b2cc1ec31"}
Sep 17 22:21:41.782047 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:41.781968    2563 scope.go:117] "RemoveContainer" containerID="5fe4750642abb32046aebed4895d920f6d8313482b051ad202e4229e63a0dd0f"
Sep 17 22:21:41.789894 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:41.789877    2563 scope.go:117] "RemoveContainer" containerID="c0fea075ff01bfbb7462a61cad89e6aeaa16415bdd13753de17e8020cb9a672a"
Sep 17 22:21:41.796979 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:41.796960    2563 scope.go:117] "RemoveContainer" containerID="e2b78d2f2842f79a0201fb862faa18c8360dfd043a16f5e9094da9b37e816b04"
Sep 17 22:21:41.804206 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:41.804189    2563 scope.go:117] "RemoveContainer" containerID="5fe4750642abb32046aebed4895d920f6d8313482b051ad202e4229e63a0dd0f"
Sep 17 22:21:41.804479 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:21:41.804461    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fe4750642abb32046aebed4895d920f6d8313482b051ad202e4229e63a0dd0f\": container with ID starting with 5fe4750642abb32046aebed4895d920f6d8313482b051ad202e4229e63a0dd0f not found: ID does not exist" containerID="5fe4750642abb32046aebed4895d920f6d8313482b051ad202e4229e63a0dd0f"
Sep 17 22:21:41.804541 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:41.804496    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fe4750642abb32046aebed4895d920f6d8313482b051ad202e4229e63a0dd0f"} err="failed to get container status \"5fe4750642abb32046aebed4895d920f6d8313482b051ad202e4229e63a0dd0f\": rpc error: code = NotFound desc = could not find container \"5fe4750642abb32046aebed4895d920f6d8313482b051ad202e4229e63a0dd0f\": container with ID starting with 5fe4750642abb32046aebed4895d920f6d8313482b051ad202e4229e63a0dd0f not found: ID does not exist"
Sep 17 22:21:41.804541 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:41.804516    2563 scope.go:117] "RemoveContainer" containerID="c0fea075ff01bfbb7462a61cad89e6aeaa16415bdd13753de17e8020cb9a672a"
Sep 17 22:21:41.804797 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:21:41.804777    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0fea075ff01bfbb7462a61cad89e6aeaa16415bdd13753de17e8020cb9a672a\": container with ID starting with c0fea075ff01bfbb7462a61cad89e6aeaa16415bdd13753de17e8020cb9a672a not found: ID does not exist" containerID="c0fea075ff01bfbb7462a61cad89e6aeaa16415bdd13753de17e8020cb9a672a"
Sep 17 22:21:41.804851 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:41.804802    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0fea075ff01bfbb7462a61cad89e6aeaa16415bdd13753de17e8020cb9a672a"} err="failed to get container status \"c0fea075ff01bfbb7462a61cad89e6aeaa16415bdd13753de17e8020cb9a672a\": rpc error: code = NotFound desc = could not find container \"c0fea075ff01bfbb7462a61cad89e6aeaa16415bdd13753de17e8020cb9a672a\": container with ID starting with c0fea075ff01bfbb7462a61cad89e6aeaa16415bdd13753de17e8020cb9a672a not found: ID does not exist"
Sep 17 22:21:41.804851 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:41.804817    2563 scope.go:117] "RemoveContainer" containerID="e2b78d2f2842f79a0201fb862faa18c8360dfd043a16f5e9094da9b37e816b04"
Sep 17 22:21:41.805048 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:21:41.805029    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2b78d2f2842f79a0201fb862faa18c8360dfd043a16f5e9094da9b37e816b04\": container with ID starting with e2b78d2f2842f79a0201fb862faa18c8360dfd043a16f5e9094da9b37e816b04 not found: ID does not exist" containerID="e2b78d2f2842f79a0201fb862faa18c8360dfd043a16f5e9094da9b37e816b04"
Sep 17 22:21:41.805088 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:41.805052    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2b78d2f2842f79a0201fb862faa18c8360dfd043a16f5e9094da9b37e816b04"} err="failed to get container status \"e2b78d2f2842f79a0201fb862faa18c8360dfd043a16f5e9094da9b37e816b04\": rpc error: code = NotFound desc = could not find container \"e2b78d2f2842f79a0201fb862faa18c8360dfd043a16f5e9094da9b37e816b04\": container with ID starting with e2b78d2f2842f79a0201fb862faa18c8360dfd043a16f5e9094da9b37e816b04 not found: ID does not exist"
Sep 17 22:21:41.839440 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:41.839358    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2907464-0041-49ef-a661-7372a6ecd32e-proxy-tls\") pod \"d2907464-0041-49ef-a661-7372a6ecd32e\" (UID: \"d2907464-0041-49ef-a661-7372a6ecd32e\") "
Sep 17 22:21:41.839440 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:41.839398    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2907464-0041-49ef-a661-7372a6ecd32e-cabundle-cert\") pod \"d2907464-0041-49ef-a661-7372a6ecd32e\" (UID: \"d2907464-0041-49ef-a661-7372a6ecd32e\") "
Sep 17 22:21:41.839611 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:41.839462    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r687s\" (UniqueName: \"kubernetes.io/projected/d2907464-0041-49ef-a661-7372a6ecd32e-kube-api-access-r687s\") pod \"d2907464-0041-49ef-a661-7372a6ecd32e\" (UID: \"d2907464-0041-49ef-a661-7372a6ecd32e\") "
Sep 17 22:21:41.839611 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:41.839492    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2907464-0041-49ef-a661-7372a6ecd32e-kube-rbac-proxy-sar-config\") pod \"d2907464-0041-49ef-a661-7372a6ecd32e\" (UID: \"d2907464-0041-49ef-a661-7372a6ecd32e\") "
Sep 17 22:21:41.839611 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:41.839520    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2907464-0041-49ef-a661-7372a6ecd32e-kserve-provision-location\") pod \"d2907464-0041-49ef-a661-7372a6ecd32e\" (UID: \"d2907464-0041-49ef-a661-7372a6ecd32e\") "
Sep 17 22:21:41.839862 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:41.839840    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2907464-0041-49ef-a661-7372a6ecd32e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d2907464-0041-49ef-a661-7372a6ecd32e" (UID: "d2907464-0041-49ef-a661-7372a6ecd32e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:21:41.839929 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:41.839899    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2907464-0041-49ef-a661-7372a6ecd32e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d2907464-0041-49ef-a661-7372a6ecd32e" (UID: "d2907464-0041-49ef-a661-7372a6ecd32e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:21:41.839929 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:41.839912    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2907464-0041-49ef-a661-7372a6ecd32e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d2907464-0041-49ef-a661-7372a6ecd32e" (UID: "d2907464-0041-49ef-a661-7372a6ecd32e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:21:41.841626 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:41.841605    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2907464-0041-49ef-a661-7372a6ecd32e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d2907464-0041-49ef-a661-7372a6ecd32e" (UID: "d2907464-0041-49ef-a661-7372a6ecd32e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:21:41.841707 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:41.841659    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2907464-0041-49ef-a661-7372a6ecd32e-kube-api-access-r687s" (OuterVolumeSpecName: "kube-api-access-r687s") pod "d2907464-0041-49ef-a661-7372a6ecd32e" (UID: "d2907464-0041-49ef-a661-7372a6ecd32e"). InnerVolumeSpecName "kube-api-access-r687s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:21:41.940728 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:41.940689    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2907464-0041-49ef-a661-7372a6ecd32e-proxy-tls\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:21:41.940728 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:41.940720    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2907464-0041-49ef-a661-7372a6ecd32e-cabundle-cert\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:21:41.940728 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:41.940729    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r687s\" (UniqueName: \"kubernetes.io/projected/d2907464-0041-49ef-a661-7372a6ecd32e-kube-api-access-r687s\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:21:41.940728 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:41.940738    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2907464-0041-49ef-a661-7372a6ecd32e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:21:41.941247 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:41.940747    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2907464-0041-49ef-a661-7372a6ecd32e-kserve-provision-location\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:21:42.104397 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:42.104366    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-p9vh9"]
Sep 17 22:21:42.110521 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:42.110490    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-56c47844df-p9vh9"]
Sep 17 22:21:42.173959 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:42.173918    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-mbtbz"]
Sep 17 22:21:42.174237 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:42.174225    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2907464-0041-49ef-a661-7372a6ecd32e" containerName="storage-initializer"
Sep 17 22:21:42.174288 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:42.174239    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2907464-0041-49ef-a661-7372a6ecd32e" containerName="storage-initializer"
Sep 17 22:21:42.174288 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:42.174247    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2907464-0041-49ef-a661-7372a6ecd32e" containerName="kube-rbac-proxy"
Sep 17 22:21:42.174288 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:42.174253    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2907464-0041-49ef-a661-7372a6ecd32e" containerName="kube-rbac-proxy"
Sep 17 22:21:42.174288 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:42.174264    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2907464-0041-49ef-a661-7372a6ecd32e" containerName="kserve-container"
Sep 17 22:21:42.174288 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:42.174269    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2907464-0041-49ef-a661-7372a6ecd32e" containerName="kserve-container"
Sep 17 22:21:42.174464 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:42.174323    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2907464-0041-49ef-a661-7372a6ecd32e" containerName="kube-rbac-proxy"
Sep 17 22:21:42.174464 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:42.174330    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2907464-0041-49ef-a661-7372a6ecd32e" containerName="kserve-container"
Sep 17 22:21:42.178714 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:42.178697    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-mbtbz"
Sep 17 22:21:42.181272 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:42.181246    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 22:21:42.181272 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:42.181261    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-predictor-serving-cert\""
Sep 17 22:21:42.181427 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:42.181311    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:21:42.181531 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:42.181517    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 22:21:42.181631 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:42.181608    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 22:21:42.181767 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:42.181731    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v4p4m\""
Sep 17 22:21:42.181767 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:42.181736    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 17 22:21:42.188918 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:42.188897    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-mbtbz"]
Sep 17 22:21:42.344547 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:42.344446    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd11069c-0ab0-454a-a5df-bbc6503f7631-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-mbtbz\" (UID: \"fd11069c-0ab0-454a-a5df-bbc6503f7631\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-mbtbz"
Sep 17 22:21:42.344547 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:42.344483    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fd11069c-0ab0-454a-a5df-bbc6503f7631-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-mbtbz\" (UID: \"fd11069c-0ab0-454a-a5df-bbc6503f7631\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-mbtbz"
Sep 17 22:21:42.344753 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:42.344575    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6nd5\" (UniqueName: \"kubernetes.io/projected/fd11069c-0ab0-454a-a5df-bbc6503f7631-kube-api-access-w6nd5\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-mbtbz\" (UID: \"fd11069c-0ab0-454a-a5df-bbc6503f7631\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-mbtbz"
Sep 17 22:21:42.344753 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:42.344661    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd11069c-0ab0-454a-a5df-bbc6503f7631-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-mbtbz\" (UID: \"fd11069c-0ab0-454a-a5df-bbc6503f7631\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-mbtbz"
Sep 17 22:21:42.344753 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:42.344680    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd11069c-0ab0-454a-a5df-bbc6503f7631-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-mbtbz\" (UID: \"fd11069c-0ab0-454a-a5df-bbc6503f7631\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-mbtbz"
Sep 17 22:21:42.445077 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:42.445040    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd11069c-0ab0-454a-a5df-bbc6503f7631-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-mbtbz\" (UID: \"fd11069c-0ab0-454a-a5df-bbc6503f7631\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-mbtbz"
Sep 17 22:21:42.445077 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:42.445075    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fd11069c-0ab0-454a-a5df-bbc6503f7631-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-mbtbz\" (UID: \"fd11069c-0ab0-454a-a5df-bbc6503f7631\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-mbtbz"
Sep 17 22:21:42.445368 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:42.445102    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6nd5\" (UniqueName: \"kubernetes.io/projected/fd11069c-0ab0-454a-a5df-bbc6503f7631-kube-api-access-w6nd5\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-mbtbz\" (UID: \"fd11069c-0ab0-454a-a5df-bbc6503f7631\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-mbtbz"
Sep 17 22:21:42.445368 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:42.445189    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd11069c-0ab0-454a-a5df-bbc6503f7631-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-mbtbz\" (UID: \"fd11069c-0ab0-454a-a5df-bbc6503f7631\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-mbtbz"
Sep 17 22:21:42.445483 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:42.445372    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd11069c-0ab0-454a-a5df-bbc6503f7631-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-mbtbz\" (UID: \"fd11069c-0ab0-454a-a5df-bbc6503f7631\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-mbtbz"
Sep 17 22:21:42.445755 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:42.445732    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd11069c-0ab0-454a-a5df-bbc6503f7631-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-mbtbz\" (UID: \"fd11069c-0ab0-454a-a5df-bbc6503f7631\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-mbtbz"
Sep 17 22:21:42.445844 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:42.445830    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fd11069c-0ab0-454a-a5df-bbc6503f7631-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-mbtbz\" (UID: \"fd11069c-0ab0-454a-a5df-bbc6503f7631\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-mbtbz"
Sep 17 22:21:42.445947 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:42.445929    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd11069c-0ab0-454a-a5df-bbc6503f7631-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-mbtbz\" (UID: \"fd11069c-0ab0-454a-a5df-bbc6503f7631\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-mbtbz"
Sep 17 22:21:42.447667 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:42.447648    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd11069c-0ab0-454a-a5df-bbc6503f7631-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-mbtbz\" (UID: \"fd11069c-0ab0-454a-a5df-bbc6503f7631\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-mbtbz"
Sep 17 22:21:42.453982 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:42.453959    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6nd5\" (UniqueName: \"kubernetes.io/projected/fd11069c-0ab0-454a-a5df-bbc6503f7631-kube-api-access-w6nd5\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-mbtbz\" (UID: \"fd11069c-0ab0-454a-a5df-bbc6503f7631\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-mbtbz"
Sep 17 22:21:42.489589 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:42.489555    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-mbtbz"
Sep 17 22:21:42.759666 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:42.759470    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-mbtbz"]
Sep 17 22:21:42.762191 ip-10-0-138-45 kubenswrapper[2563]: W0917 22:21:42.762156    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd11069c_0ab0_454a_a5df_bbc6503f7631.slice/crio-4ad66908088955f168ffc72abbb6fb41e36e77dd874a95c119578ff5cc82bdb3 WatchSource:0}: Error finding container 4ad66908088955f168ffc72abbb6fb41e36e77dd874a95c119578ff5cc82bdb3: Status 404 returned error can't find the container with id 4ad66908088955f168ffc72abbb6fb41e36e77dd874a95c119578ff5cc82bdb3
Sep 17 22:21:42.787149 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:42.787106    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-mbtbz" event={"ID":"fd11069c-0ab0-454a-a5df-bbc6503f7631","Type":"ContainerStarted","Data":"4ad66908088955f168ffc72abbb6fb41e36e77dd874a95c119578ff5cc82bdb3"}
Sep 17 22:21:42.914495 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:42.914457    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2907464-0041-49ef-a661-7372a6ecd32e" path="/var/lib/kubelet/pods/d2907464-0041-49ef-a661-7372a6ecd32e/volumes"
Sep 17 22:21:43.793730 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:43.793695    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-mbtbz" event={"ID":"fd11069c-0ab0-454a-a5df-bbc6503f7631","Type":"ContainerStarted","Data":"e856cb4d06c94e1b193d23a0a0c82b28ea08f66cfbf2865f9b0a1cccfc7f5d40"}
Sep 17 22:21:46.807244 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:46.807212    2563 generic.go:356] "Generic (PLEG): container finished" podID="fd11069c-0ab0-454a-a5df-bbc6503f7631" containerID="e856cb4d06c94e1b193d23a0a0c82b28ea08f66cfbf2865f9b0a1cccfc7f5d40" exitCode=0
Sep 17 22:21:46.807642 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:21:46.807260    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-mbtbz" event={"ID":"fd11069c-0ab0-454a-a5df-bbc6503f7631","Type":"ContainerDied","Data":"e856cb4d06c94e1b193d23a0a0c82b28ea08f66cfbf2865f9b0a1cccfc7f5d40"}
Sep 17 22:23:49.579602 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:23:49.579575    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nrkd5_34b8ef2b-97ba-4562-87ee-8da0cb4604c7/ovn-acl-logging/0.log"
Sep 17 22:23:49.580157 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:23:49.579575    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nrkd5_34b8ef2b-97ba-4562-87ee-8da0cb4604c7/ovn-acl-logging/0.log"
Sep 17 22:23:50.379422 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:23:50.379391    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-mbtbz" event={"ID":"fd11069c-0ab0-454a-a5df-bbc6503f7631","Type":"ContainerStarted","Data":"f4d8c9ddb1701783da748134282eefd4291bd79577a625c28293c4975774fc49"}
Sep 17 22:23:50.379594 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:23:50.379456    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-mbtbz" event={"ID":"fd11069c-0ab0-454a-a5df-bbc6503f7631","Type":"ContainerStarted","Data":"eef2164942f12ef42107f11bb0e211ba6f62605dacf4b1e035c9bff4467fcead"}
Sep 17 22:23:51.383793 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:23:51.383752    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-mbtbz"
Sep 17 22:23:51.406545 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:23:51.383990    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-mbtbz"
Sep 17 22:23:51.407158 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:23:51.407097    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-mbtbz" podStartSLOduration=2.124980321 podStartE2EDuration="2m9.407081208s" totalImagesPullingTime="2m3.366193098s" totalInitContainerRuntime="3.915907789s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:21:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:23:51.404813214 +0000 UTC m=+1542.572766201" watchObservedRunningTime="2026-09-17 22:23:51.407081208 +0000 UTC m=+1542.575034203"
Sep 17 22:23:57.393765 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:23:57.393735    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-mbtbz"
Sep 17 22:24:27.453306 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:27.453267    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-mbtbz" podUID="fd11069c-0ab0-454a-a5df-bbc6503f7631" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 17 22:24:37.396269 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:37.396237    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-mbtbz"
Sep 17 22:24:42.321085 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:42.321047    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-mbtbz"]
Sep 17 22:24:42.321559 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:42.321385    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-mbtbz" podUID="fd11069c-0ab0-454a-a5df-bbc6503f7631" containerName="kserve-container" containerID="cri-o://eef2164942f12ef42107f11bb0e211ba6f62605dacf4b1e035c9bff4467fcead" gracePeriod=30
Sep 17 22:24:42.321559 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:42.321484    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-mbtbz" podUID="fd11069c-0ab0-454a-a5df-bbc6503f7631" containerName="kube-rbac-proxy" containerID="cri-o://f4d8c9ddb1701783da748134282eefd4291bd79577a625c28293c4975774fc49" gracePeriod=30
Sep 17 22:24:42.388712 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:42.388676    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-mbtbz" podUID="fd11069c-0ab0-454a-a5df-bbc6503f7631" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Sep 17 22:24:42.587832 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:42.587744    2563 generic.go:356] "Generic (PLEG): container finished" podID="fd11069c-0ab0-454a-a5df-bbc6503f7631" containerID="f4d8c9ddb1701783da748134282eefd4291bd79577a625c28293c4975774fc49" exitCode=2
Sep 17 22:24:42.587832 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:42.587791    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-mbtbz" event={"ID":"fd11069c-0ab0-454a-a5df-bbc6503f7631","Type":"ContainerDied","Data":"f4d8c9ddb1701783da748134282eefd4291bd79577a625c28293c4975774fc49"}
Sep 17 22:24:45.663339 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:45.663300    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-b8xrk"]
Sep 17 22:24:45.666888 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:45.666864    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-b8xrk"
Sep 17 22:24:45.669188 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:45.669165    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Sep 17 22:24:45.669442 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:45.669429    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 17 22:24:45.673659 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:45.673607    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-5nv57"]
Sep 17 22:24:45.676705 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:45.676691    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-5nv57"
Sep 17 22:24:45.678663 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:45.678638    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 17 22:24:45.686905 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:45.686878    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-b8xrk"]
Sep 17 22:24:45.689190 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:45.689168    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-5nv57"]
Sep 17 22:24:45.691389 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:45.691366    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6b825c2a-99e0-4496-809a-64043cc86dd1-proxy-tls\") pod \"isvc-canary-create-predictor-5f845bd94d-5nv57\" (UID: \"6b825c2a-99e0-4496-809a-64043cc86dd1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-5nv57"
Sep 17 22:24:45.691473 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:45.691429    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6b825c2a-99e0-4496-809a-64043cc86dd1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5f845bd94d-5nv57\" (UID: \"6b825c2a-99e0-4496-809a-64043cc86dd1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-5nv57"
Sep 17 22:24:45.691533 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:45.691499    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/832cb065-cc5e-42e2-829c-8eaadd78e6df-proxy-tls\") pod \"isvc-canary-create-v2-predictor-79d6f5c6d9-b8xrk\" (UID: \"832cb065-cc5e-42e2-829c-8eaadd78e6df\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-b8xrk"
Sep 17 22:24:45.691597 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:45.691567    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v95m\" (UniqueName: \"kubernetes.io/projected/832cb065-cc5e-42e2-829c-8eaadd78e6df-kube-api-access-6v95m\") pod \"isvc-canary-create-v2-predictor-79d6f5c6d9-b8xrk\" (UID: \"832cb065-cc5e-42e2-829c-8eaadd78e6df\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-b8xrk"
Sep 17 22:24:45.691653 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:45.691605    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6b825c2a-99e0-4496-809a-64043cc86dd1-cabundle-cert\") pod \"isvc-canary-create-predictor-5f845bd94d-5nv57\" (UID: \"6b825c2a-99e0-4496-809a-64043cc86dd1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-5nv57"
Sep 17 22:24:45.691703 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:45.691652    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/832cb065-cc5e-42e2-829c-8eaadd78e6df-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-79d6f5c6d9-b8xrk\" (UID: \"832cb065-cc5e-42e2-829c-8eaadd78e6df\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-b8xrk"
Sep 17 22:24:45.691703 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:45.691683    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/832cb065-cc5e-42e2-829c-8eaadd78e6df-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-79d6f5c6d9-b8xrk\" (UID: \"832cb065-cc5e-42e2-829c-8eaadd78e6df\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-b8xrk"
Sep 17 22:24:45.691786 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:45.691721    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6b825c2a-99e0-4496-809a-64043cc86dd1-kserve-provision-location\") pod \"isvc-canary-create-predictor-5f845bd94d-5nv57\" (UID: \"6b825c2a-99e0-4496-809a-64043cc86dd1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-5nv57"
Sep 17 22:24:45.691820 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:45.691792    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/832cb065-cc5e-42e2-829c-8eaadd78e6df-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-79d6f5c6d9-b8xrk\" (UID: \"832cb065-cc5e-42e2-829c-8eaadd78e6df\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-b8xrk"
Sep 17 22:24:45.691867 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:45.691823    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96rst\" (UniqueName: \"kubernetes.io/projected/6b825c2a-99e0-4496-809a-64043cc86dd1-kube-api-access-96rst\") pod \"isvc-canary-create-predictor-5f845bd94d-5nv57\" (UID: \"6b825c2a-99e0-4496-809a-64043cc86dd1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-5nv57"
Sep 17 22:24:45.792396 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:45.792359    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6b825c2a-99e0-4496-809a-64043cc86dd1-proxy-tls\") pod \"isvc-canary-create-predictor-5f845bd94d-5nv57\" (UID: \"6b825c2a-99e0-4496-809a-64043cc86dd1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-5nv57"
Sep 17 22:24:45.792563 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:45.792414    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6b825c2a-99e0-4496-809a-64043cc86dd1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5f845bd94d-5nv57\" (UID: \"6b825c2a-99e0-4496-809a-64043cc86dd1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-5nv57"
Sep 17 22:24:45.792563 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:45.792508    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/832cb065-cc5e-42e2-829c-8eaadd78e6df-proxy-tls\") pod \"isvc-canary-create-v2-predictor-79d6f5c6d9-b8xrk\" (UID: \"832cb065-cc5e-42e2-829c-8eaadd78e6df\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-b8xrk"
Sep 17 22:24:45.792563 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:45.792534    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v95m\" (UniqueName: \"kubernetes.io/projected/832cb065-cc5e-42e2-829c-8eaadd78e6df-kube-api-access-6v95m\") pod \"isvc-canary-create-v2-predictor-79d6f5c6d9-b8xrk\" (UID: \"832cb065-cc5e-42e2-829c-8eaadd78e6df\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-b8xrk"
Sep 17 22:24:45.792563 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:45.792561    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6b825c2a-99e0-4496-809a-64043cc86dd1-cabundle-cert\") pod \"isvc-canary-create-predictor-5f845bd94d-5nv57\" (UID: \"6b825c2a-99e0-4496-809a-64043cc86dd1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-5nv57"
Sep 17 22:24:45.792744 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:24:45.792620    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-v2-predictor-serving-cert: secret "isvc-canary-create-v2-predictor-serving-cert" not found
Sep 17 22:24:45.792744 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:24:45.792692    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/832cb065-cc5e-42e2-829c-8eaadd78e6df-proxy-tls podName:832cb065-cc5e-42e2-829c-8eaadd78e6df nodeName:}" failed. No retries permitted until 2026-09-17 22:24:46.292671757 +0000 UTC m=+1597.460624729 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/832cb065-cc5e-42e2-829c-8eaadd78e6df-proxy-tls") pod "isvc-canary-create-v2-predictor-79d6f5c6d9-b8xrk" (UID: "832cb065-cc5e-42e2-829c-8eaadd78e6df") : secret "isvc-canary-create-v2-predictor-serving-cert" not found
Sep 17 22:24:45.792744 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:45.792686    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/832cb065-cc5e-42e2-829c-8eaadd78e6df-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-79d6f5c6d9-b8xrk\" (UID: \"832cb065-cc5e-42e2-829c-8eaadd78e6df\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-b8xrk"
Sep 17 22:24:45.792744 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:45.792724    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/832cb065-cc5e-42e2-829c-8eaadd78e6df-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-79d6f5c6d9-b8xrk\" (UID: \"832cb065-cc5e-42e2-829c-8eaadd78e6df\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-b8xrk"
Sep 17 22:24:45.792957 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:45.792760    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6b825c2a-99e0-4496-809a-64043cc86dd1-kserve-provision-location\") pod \"isvc-canary-create-predictor-5f845bd94d-5nv57\" (UID: \"6b825c2a-99e0-4496-809a-64043cc86dd1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-5nv57"
Sep 17 22:24:45.792957 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:45.792841    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/832cb065-cc5e-42e2-829c-8eaadd78e6df-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-79d6f5c6d9-b8xrk\" (UID: \"832cb065-cc5e-42e2-829c-8eaadd78e6df\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-b8xrk"
Sep 17 22:24:45.792957 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:45.792875    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96rst\" (UniqueName: \"kubernetes.io/projected/6b825c2a-99e0-4496-809a-64043cc86dd1-kube-api-access-96rst\") pod \"isvc-canary-create-predictor-5f845bd94d-5nv57\" (UID: \"6b825c2a-99e0-4496-809a-64043cc86dd1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-5nv57"
Sep 17 22:24:45.793285 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:45.793259    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6b825c2a-99e0-4496-809a-64043cc86dd1-kserve-provision-location\") pod \"isvc-canary-create-predictor-5f845bd94d-5nv57\" (UID: \"6b825c2a-99e0-4496-809a-64043cc86dd1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-5nv57"
Sep 17 22:24:45.793386 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:45.793306    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6b825c2a-99e0-4496-809a-64043cc86dd1-cabundle-cert\") pod \"isvc-canary-create-predictor-5f845bd94d-5nv57\" (UID: \"6b825c2a-99e0-4496-809a-64043cc86dd1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-5nv57"
Sep 17 22:24:45.793386 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:45.793357    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6b825c2a-99e0-4496-809a-64043cc86dd1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5f845bd94d-5nv57\" (UID: \"6b825c2a-99e0-4496-809a-64043cc86dd1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-5nv57"
Sep 17 22:24:45.793508 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:45.793480    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/832cb065-cc5e-42e2-829c-8eaadd78e6df-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-79d6f5c6d9-b8xrk\" (UID: \"832cb065-cc5e-42e2-829c-8eaadd78e6df\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-b8xrk"
Sep 17 22:24:45.793566 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:45.793537    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/832cb065-cc5e-42e2-829c-8eaadd78e6df-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-79d6f5c6d9-b8xrk\" (UID: \"832cb065-cc5e-42e2-829c-8eaadd78e6df\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-b8xrk"
Sep 17 22:24:45.793634 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:45.793608    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/832cb065-cc5e-42e2-829c-8eaadd78e6df-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-79d6f5c6d9-b8xrk\" (UID: \"832cb065-cc5e-42e2-829c-8eaadd78e6df\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-b8xrk"
Sep 17 22:24:45.794995 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:45.794972    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6b825c2a-99e0-4496-809a-64043cc86dd1-proxy-tls\") pod \"isvc-canary-create-predictor-5f845bd94d-5nv57\" (UID: \"6b825c2a-99e0-4496-809a-64043cc86dd1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-5nv57"
Sep 17 22:24:45.804215 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:45.804195    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v95m\" (UniqueName: \"kubernetes.io/projected/832cb065-cc5e-42e2-829c-8eaadd78e6df-kube-api-access-6v95m\") pod \"isvc-canary-create-v2-predictor-79d6f5c6d9-b8xrk\" (UID: \"832cb065-cc5e-42e2-829c-8eaadd78e6df\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-b8xrk"
Sep 17 22:24:45.804304 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:45.804237    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96rst\" (UniqueName: \"kubernetes.io/projected/6b825c2a-99e0-4496-809a-64043cc86dd1-kube-api-access-96rst\") pod \"isvc-canary-create-predictor-5f845bd94d-5nv57\" (UID: \"6b825c2a-99e0-4496-809a-64043cc86dd1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-5nv57"
Sep 17 22:24:45.989538 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:45.989452    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-5nv57"
Sep 17 22:24:46.263909 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:46.263883    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-5nv57"]
Sep 17 22:24:46.266445 ip-10-0-138-45 kubenswrapper[2563]: W0917 22:24:46.266417    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b825c2a_99e0_4496_809a_64043cc86dd1.slice/crio-b693ac70a61e2ed20c830bb3e9536beffba7fddcdf9397fd9cd900b9006a1d03 WatchSource:0}: Error finding container b693ac70a61e2ed20c830bb3e9536beffba7fddcdf9397fd9cd900b9006a1d03: Status 404 returned error can't find the container with id b693ac70a61e2ed20c830bb3e9536beffba7fddcdf9397fd9cd900b9006a1d03
Sep 17 22:24:46.295758 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:46.295736    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/832cb065-cc5e-42e2-829c-8eaadd78e6df-proxy-tls\") pod \"isvc-canary-create-v2-predictor-79d6f5c6d9-b8xrk\" (UID: \"832cb065-cc5e-42e2-829c-8eaadd78e6df\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-b8xrk"
Sep 17 22:24:46.297868 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:46.297845    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/832cb065-cc5e-42e2-829c-8eaadd78e6df-proxy-tls\") pod \"isvc-canary-create-v2-predictor-79d6f5c6d9-b8xrk\" (UID: \"832cb065-cc5e-42e2-829c-8eaadd78e6df\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-b8xrk"
Sep 17 22:24:46.577392 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:46.577297    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-b8xrk"
Sep 17 22:24:46.606537 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:46.606499    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-5nv57" event={"ID":"6b825c2a-99e0-4496-809a-64043cc86dd1","Type":"ContainerStarted","Data":"09c820ea54f2f965f0460866a402ec90fe155431276bdbfaa80519d400e7b071"}
Sep 17 22:24:46.606680 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:46.606581    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-5nv57" event={"ID":"6b825c2a-99e0-4496-809a-64043cc86dd1","Type":"ContainerStarted","Data":"b693ac70a61e2ed20c830bb3e9536beffba7fddcdf9397fd9cd900b9006a1d03"}
Sep 17 22:24:46.856475 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:46.856452    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-b8xrk"]
Sep 17 22:24:46.858659 ip-10-0-138-45 kubenswrapper[2563]: W0917 22:24:46.858630    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod832cb065_cc5e_42e2_829c_8eaadd78e6df.slice/crio-3e666c8ab6781e797af66202d4600facb86a6767ed16476f078458adb5eefea5 WatchSource:0}: Error finding container 3e666c8ab6781e797af66202d4600facb86a6767ed16476f078458adb5eefea5: Status 404 returned error can't find the container with id 3e666c8ab6781e797af66202d4600facb86a6767ed16476f078458adb5eefea5
Sep 17 22:24:47.389220 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:47.389170    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-mbtbz" podUID="fd11069c-0ab0-454a-a5df-bbc6503f7631" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Sep 17 22:24:47.394431 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:47.394398    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-mbtbz" podUID="fd11069c-0ab0-454a-a5df-bbc6503f7631" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.34:8080/v2/models/sklearn-v2-mlserver/ready\": dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 22:24:47.488136 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:47.488105    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-rfvmh"]
Sep 17 22:24:47.491551 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:47.491536    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-rfvmh"
Sep 17 22:24:47.494101 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:47.494079    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-predictor-serving-cert\""
Sep 17 22:24:47.494816 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:47.494799    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\""
Sep 17 22:24:47.503598 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:47.503569    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f94ae26-e178-4f13-946c-b6d07f52be8a-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-56ccc8b76f-rfvmh\" (UID: \"8f94ae26-e178-4f13-946c-b6d07f52be8a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-rfvmh"
Sep 17 22:24:47.503598 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:47.503597    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f94ae26-e178-4f13-946c-b6d07f52be8a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-56ccc8b76f-rfvmh\" (UID: \"8f94ae26-e178-4f13-946c-b6d07f52be8a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-rfvmh"
Sep 17 22:24:47.503751 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:47.503635    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42x2f\" (UniqueName: \"kubernetes.io/projected/8f94ae26-e178-4f13-946c-b6d07f52be8a-kube-api-access-42x2f\") pod \"isvc-sklearn-runtime-predictor-56ccc8b76f-rfvmh\" (UID: \"8f94ae26-e178-4f13-946c-b6d07f52be8a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-rfvmh"
Sep 17 22:24:47.503751 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:47.503689    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f94ae26-e178-4f13-946c-b6d07f52be8a-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-56ccc8b76f-rfvmh\" (UID: \"8f94ae26-e178-4f13-946c-b6d07f52be8a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-rfvmh"
Sep 17 22:24:47.503751 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:47.503693    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-rfvmh"]
Sep 17 22:24:47.503751 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:47.503714    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f94ae26-e178-4f13-946c-b6d07f52be8a-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-56ccc8b76f-rfvmh\" (UID: \"8f94ae26-e178-4f13-946c-b6d07f52be8a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-rfvmh"
Sep 17 22:24:47.604228 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:47.604192    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f94ae26-e178-4f13-946c-b6d07f52be8a-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-56ccc8b76f-rfvmh\" (UID: \"8f94ae26-e178-4f13-946c-b6d07f52be8a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-rfvmh"
Sep 17 22:24:47.604228 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:47.604233    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f94ae26-e178-4f13-946c-b6d07f52be8a-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-56ccc8b76f-rfvmh\" (UID: \"8f94ae26-e178-4f13-946c-b6d07f52be8a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-rfvmh"
Sep 17 22:24:47.604448 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:47.604273    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f94ae26-e178-4f13-946c-b6d07f52be8a-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-56ccc8b76f-rfvmh\" (UID: \"8f94ae26-e178-4f13-946c-b6d07f52be8a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-rfvmh"
Sep 17 22:24:47.604448 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:47.604289    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f94ae26-e178-4f13-946c-b6d07f52be8a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-56ccc8b76f-rfvmh\" (UID: \"8f94ae26-e178-4f13-946c-b6d07f52be8a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-rfvmh"
Sep 17 22:24:47.604448 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:47.604313    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42x2f\" (UniqueName: \"kubernetes.io/projected/8f94ae26-e178-4f13-946c-b6d07f52be8a-kube-api-access-42x2f\") pod \"isvc-sklearn-runtime-predictor-56ccc8b76f-rfvmh\" (UID: \"8f94ae26-e178-4f13-946c-b6d07f52be8a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-rfvmh"
Sep 17 22:24:47.604672 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:47.604650    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f94ae26-e178-4f13-946c-b6d07f52be8a-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-56ccc8b76f-rfvmh\" (UID: \"8f94ae26-e178-4f13-946c-b6d07f52be8a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-rfvmh"
Sep 17 22:24:47.604933 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:47.604914    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f94ae26-e178-4f13-946c-b6d07f52be8a-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-56ccc8b76f-rfvmh\" (UID: \"8f94ae26-e178-4f13-946c-b6d07f52be8a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-rfvmh"
Sep 17 22:24:47.604973 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:47.604962    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f94ae26-e178-4f13-946c-b6d07f52be8a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-56ccc8b76f-rfvmh\" (UID: \"8f94ae26-e178-4f13-946c-b6d07f52be8a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-rfvmh"
Sep 17 22:24:47.606630 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:47.606614    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f94ae26-e178-4f13-946c-b6d07f52be8a-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-56ccc8b76f-rfvmh\" (UID: \"8f94ae26-e178-4f13-946c-b6d07f52be8a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-rfvmh"
Sep 17 22:24:47.612244 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:47.612222    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42x2f\" (UniqueName: \"kubernetes.io/projected/8f94ae26-e178-4f13-946c-b6d07f52be8a-kube-api-access-42x2f\") pod \"isvc-sklearn-runtime-predictor-56ccc8b76f-rfvmh\" (UID: \"8f94ae26-e178-4f13-946c-b6d07f52be8a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-rfvmh"
Sep 17 22:24:47.612719 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:47.612689    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-b8xrk" event={"ID":"832cb065-cc5e-42e2-829c-8eaadd78e6df","Type":"ContainerStarted","Data":"f1e98bf825c49ec2c3bf1c161b63890d7a0077c78b29932d26177738a7a9852f"}
Sep 17 22:24:47.612831 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:47.612753    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-b8xrk" event={"ID":"832cb065-cc5e-42e2-829c-8eaadd78e6df","Type":"ContainerStarted","Data":"3e666c8ab6781e797af66202d4600facb86a6767ed16476f078458adb5eefea5"}
Sep 17 22:24:47.804336 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:47.804286    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-rfvmh"
Sep 17 22:24:48.077888 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:48.077863    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-rfvmh"]
Sep 17 22:24:48.080300 ip-10-0-138-45 kubenswrapper[2563]: W0917 22:24:48.080273    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f94ae26_e178_4f13_946c_b6d07f52be8a.slice/crio-f30a508a1325b52be4922462ce052ae1995132305de533742e8247271a1a54fd WatchSource:0}: Error finding container f30a508a1325b52be4922462ce052ae1995132305de533742e8247271a1a54fd: Status 404 returned error can't find the container with id f30a508a1325b52be4922462ce052ae1995132305de533742e8247271a1a54fd
Sep 17 22:24:48.618999 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:48.618961    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-rfvmh" event={"ID":"8f94ae26-e178-4f13-946c-b6d07f52be8a","Type":"ContainerStarted","Data":"619e905c12f513e5dda2d56ee1da2b7e0f24f25d82682a4c5bcf8679a49b9035"}
Sep 17 22:24:48.619183 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:48.619026    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-rfvmh" event={"ID":"8f94ae26-e178-4f13-946c-b6d07f52be8a","Type":"ContainerStarted","Data":"f30a508a1325b52be4922462ce052ae1995132305de533742e8247271a1a54fd"}
Sep 17 22:24:49.557609 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:49.557588    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-mbtbz"
Sep 17 22:24:49.620156 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:49.620066    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd11069c-0ab0-454a-a5df-bbc6503f7631-kube-rbac-proxy-sar-config\") pod \"fd11069c-0ab0-454a-a5df-bbc6503f7631\" (UID: \"fd11069c-0ab0-454a-a5df-bbc6503f7631\") "
Sep 17 22:24:49.620156 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:49.620132    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd11069c-0ab0-454a-a5df-bbc6503f7631-proxy-tls\") pod \"fd11069c-0ab0-454a-a5df-bbc6503f7631\" (UID: \"fd11069c-0ab0-454a-a5df-bbc6503f7631\") "
Sep 17 22:24:49.620389 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:49.620198    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd11069c-0ab0-454a-a5df-bbc6503f7631-kserve-provision-location\") pod \"fd11069c-0ab0-454a-a5df-bbc6503f7631\" (UID: \"fd11069c-0ab0-454a-a5df-bbc6503f7631\") "
Sep 17 22:24:49.620389 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:49.620216    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w6nd5\" (UniqueName: \"kubernetes.io/projected/fd11069c-0ab0-454a-a5df-bbc6503f7631-kube-api-access-w6nd5\") pod \"fd11069c-0ab0-454a-a5df-bbc6503f7631\" (UID: \"fd11069c-0ab0-454a-a5df-bbc6503f7631\") "
Sep 17 22:24:49.620389 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:49.620235    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fd11069c-0ab0-454a-a5df-bbc6503f7631-cabundle-cert\") pod \"fd11069c-0ab0-454a-a5df-bbc6503f7631\" (UID: \"fd11069c-0ab0-454a-a5df-bbc6503f7631\") "
Sep 17 22:24:49.620554 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:49.620507    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd11069c-0ab0-454a-a5df-bbc6503f7631-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fd11069c-0ab0-454a-a5df-bbc6503f7631" (UID: "fd11069c-0ab0-454a-a5df-bbc6503f7631"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:24:49.620638 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:49.620576    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd11069c-0ab0-454a-a5df-bbc6503f7631-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fd11069c-0ab0-454a-a5df-bbc6503f7631" (UID: "fd11069c-0ab0-454a-a5df-bbc6503f7631"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:24:49.620764 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:49.620733    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd11069c-0ab0-454a-a5df-bbc6503f7631-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fd11069c-0ab0-454a-a5df-bbc6503f7631" (UID: "fd11069c-0ab0-454a-a5df-bbc6503f7631"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:24:49.622375 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:49.622354    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd11069c-0ab0-454a-a5df-bbc6503f7631-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fd11069c-0ab0-454a-a5df-bbc6503f7631" (UID: "fd11069c-0ab0-454a-a5df-bbc6503f7631"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:24:49.622518 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:49.622503    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd11069c-0ab0-454a-a5df-bbc6503f7631-kube-api-access-w6nd5" (OuterVolumeSpecName: "kube-api-access-w6nd5") pod "fd11069c-0ab0-454a-a5df-bbc6503f7631" (UID: "fd11069c-0ab0-454a-a5df-bbc6503f7631"). InnerVolumeSpecName "kube-api-access-w6nd5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:24:49.626159 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:49.626116    2563 generic.go:356] "Generic (PLEG): container finished" podID="fd11069c-0ab0-454a-a5df-bbc6503f7631" containerID="eef2164942f12ef42107f11bb0e211ba6f62605dacf4b1e035c9bff4467fcead" exitCode=0
Sep 17 22:24:49.626264 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:49.626183    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-mbtbz" event={"ID":"fd11069c-0ab0-454a-a5df-bbc6503f7631","Type":"ContainerDied","Data":"eef2164942f12ef42107f11bb0e211ba6f62605dacf4b1e035c9bff4467fcead"}
Sep 17 22:24:49.626264 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:49.626225    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-mbtbz"
Sep 17 22:24:49.626264 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:49.626261    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-mbtbz" event={"ID":"fd11069c-0ab0-454a-a5df-bbc6503f7631","Type":"ContainerDied","Data":"4ad66908088955f168ffc72abbb6fb41e36e77dd874a95c119578ff5cc82bdb3"}
Sep 17 22:24:49.626625 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:49.626290    2563 scope.go:117] "RemoveContainer" containerID="f4d8c9ddb1701783da748134282eefd4291bd79577a625c28293c4975774fc49"
Sep 17 22:24:49.637319 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:49.637303    2563 scope.go:117] "RemoveContainer" containerID="eef2164942f12ef42107f11bb0e211ba6f62605dacf4b1e035c9bff4467fcead"
Sep 17 22:24:49.644526 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:49.644507    2563 scope.go:117] "RemoveContainer" containerID="e856cb4d06c94e1b193d23a0a0c82b28ea08f66cfbf2865f9b0a1cccfc7f5d40"
Sep 17 22:24:49.648605 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:49.648577    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-mbtbz"]
Sep 17 22:24:49.651986 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:49.651962    2563 scope.go:117] "RemoveContainer" containerID="f4d8c9ddb1701783da748134282eefd4291bd79577a625c28293c4975774fc49"
Sep 17 22:24:49.652296 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:24:49.652277    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4d8c9ddb1701783da748134282eefd4291bd79577a625c28293c4975774fc49\": container with ID starting with f4d8c9ddb1701783da748134282eefd4291bd79577a625c28293c4975774fc49 not found: ID does not exist" containerID="f4d8c9ddb1701783da748134282eefd4291bd79577a625c28293c4975774fc49"
Sep 17 22:24:49.652370 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:49.652310    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4d8c9ddb1701783da748134282eefd4291bd79577a625c28293c4975774fc49"} err="failed to get container status \"f4d8c9ddb1701783da748134282eefd4291bd79577a625c28293c4975774fc49\": rpc error: code = NotFound desc = could not find container \"f4d8c9ddb1701783da748134282eefd4291bd79577a625c28293c4975774fc49\": container with ID starting with f4d8c9ddb1701783da748134282eefd4291bd79577a625c28293c4975774fc49 not found: ID does not exist"
Sep 17 22:24:49.652370 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:49.652330    2563 scope.go:117] "RemoveContainer" containerID="eef2164942f12ef42107f11bb0e211ba6f62605dacf4b1e035c9bff4467fcead"
Sep 17 22:24:49.652370 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:49.652284    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-mbtbz"]
Sep 17 22:24:49.652624 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:24:49.652608    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eef2164942f12ef42107f11bb0e211ba6f62605dacf4b1e035c9bff4467fcead\": container with ID starting with eef2164942f12ef42107f11bb0e211ba6f62605dacf4b1e035c9bff4467fcead not found: ID does not exist" containerID="eef2164942f12ef42107f11bb0e211ba6f62605dacf4b1e035c9bff4467fcead"
Sep 17 22:24:49.652675 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:49.652629    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eef2164942f12ef42107f11bb0e211ba6f62605dacf4b1e035c9bff4467fcead"} err="failed to get container status \"eef2164942f12ef42107f11bb0e211ba6f62605dacf4b1e035c9bff4467fcead\": rpc error: code = NotFound desc = could not find container \"eef2164942f12ef42107f11bb0e211ba6f62605dacf4b1e035c9bff4467fcead\": container with ID starting with eef2164942f12ef42107f11bb0e211ba6f62605dacf4b1e035c9bff4467fcead not found: ID does not exist"
Sep 17 22:24:49.652675 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:49.652644    2563 scope.go:117] "RemoveContainer" containerID="e856cb4d06c94e1b193d23a0a0c82b28ea08f66cfbf2865f9b0a1cccfc7f5d40"
Sep 17 22:24:49.652903 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:24:49.652887    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e856cb4d06c94e1b193d23a0a0c82b28ea08f66cfbf2865f9b0a1cccfc7f5d40\": container with ID starting with e856cb4d06c94e1b193d23a0a0c82b28ea08f66cfbf2865f9b0a1cccfc7f5d40 not found: ID does not exist" containerID="e856cb4d06c94e1b193d23a0a0c82b28ea08f66cfbf2865f9b0a1cccfc7f5d40"
Sep 17 22:24:49.652944 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:49.652908    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e856cb4d06c94e1b193d23a0a0c82b28ea08f66cfbf2865f9b0a1cccfc7f5d40"} err="failed to get container status \"e856cb4d06c94e1b193d23a0a0c82b28ea08f66cfbf2865f9b0a1cccfc7f5d40\": rpc error: code = NotFound desc = could not find container \"e856cb4d06c94e1b193d23a0a0c82b28ea08f66cfbf2865f9b0a1cccfc7f5d40\": container with ID starting with e856cb4d06c94e1b193d23a0a0c82b28ea08f66cfbf2865f9b0a1cccfc7f5d40 not found: ID does not exist"
Sep 17 22:24:49.721102 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:49.721077    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd11069c-0ab0-454a-a5df-bbc6503f7631-proxy-tls\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:24:49.721102 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:49.721102    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd11069c-0ab0-454a-a5df-bbc6503f7631-kserve-provision-location\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:24:49.721334 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:49.721117    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6nd5\" (UniqueName: \"kubernetes.io/projected/fd11069c-0ab0-454a-a5df-bbc6503f7631-kube-api-access-w6nd5\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:24:49.721334 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:49.721130    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fd11069c-0ab0-454a-a5df-bbc6503f7631-cabundle-cert\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:24:49.721334 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:49.721166    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd11069c-0ab0-454a-a5df-bbc6503f7631-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:24:50.631687 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:50.631613    2563 generic.go:356] "Generic (PLEG): container finished" podID="832cb065-cc5e-42e2-829c-8eaadd78e6df" containerID="f1e98bf825c49ec2c3bf1c161b63890d7a0077c78b29932d26177738a7a9852f" exitCode=0
Sep 17 22:24:50.632030 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:50.631689    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-b8xrk" event={"ID":"832cb065-cc5e-42e2-829c-8eaadd78e6df","Type":"ContainerDied","Data":"f1e98bf825c49ec2c3bf1c161b63890d7a0077c78b29932d26177738a7a9852f"}
Sep 17 22:24:50.633292 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:50.633271    2563 generic.go:356] "Generic (PLEG): container finished" podID="6b825c2a-99e0-4496-809a-64043cc86dd1" containerID="09c820ea54f2f965f0460866a402ec90fe155431276bdbfaa80519d400e7b071" exitCode=0
Sep 17 22:24:50.633369 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:50.633346    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-5nv57" event={"ID":"6b825c2a-99e0-4496-809a-64043cc86dd1","Type":"ContainerDied","Data":"09c820ea54f2f965f0460866a402ec90fe155431276bdbfaa80519d400e7b071"}
Sep 17 22:24:50.913446 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:50.913407    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd11069c-0ab0-454a-a5df-bbc6503f7631" path="/var/lib/kubelet/pods/fd11069c-0ab0-454a-a5df-bbc6503f7631/volumes"
Sep 17 22:24:51.640426 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:51.640393    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-b8xrk" event={"ID":"832cb065-cc5e-42e2-829c-8eaadd78e6df","Type":"ContainerStarted","Data":"dff64a8bd3579bba16a7b891d769dc2f56bb2f85d4b34af1a98bc74e9be8af49"}
Sep 17 22:24:51.640838 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:51.640453    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-b8xrk" event={"ID":"832cb065-cc5e-42e2-829c-8eaadd78e6df","Type":"ContainerStarted","Data":"33797cec4a4a4f22e349d3e6be4d20bdf4eae7dae665e8afc1ebf1099019344d"}
Sep 17 22:24:51.640838 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:51.640752    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-b8xrk"
Sep 17 22:24:51.640975 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:51.640897    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-b8xrk"
Sep 17 22:24:51.642235 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:51.642199    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-b8xrk" podUID="832cb065-cc5e-42e2-829c-8eaadd78e6df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 22:24:51.642661 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:51.642646    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-5nv57" event={"ID":"6b825c2a-99e0-4496-809a-64043cc86dd1","Type":"ContainerStarted","Data":"8a19361bf824e926d6376f8dacafc01631275550df7a1b5a3c314e6b5b17500c"}
Sep 17 22:24:51.642709 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:51.642686    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-5nv57" event={"ID":"6b825c2a-99e0-4496-809a-64043cc86dd1","Type":"ContainerStarted","Data":"2d5e82eb33cd325f02b5445224c555e4eef0d4d7d6709244346dd1d534cbda74"}
Sep 17 22:24:51.642990 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:51.642974    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-5nv57"
Sep 17 22:24:51.643024 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:51.643016    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-5nv57"
Sep 17 22:24:51.643960 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:51.643940    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-5nv57" podUID="6b825c2a-99e0-4496-809a-64043cc86dd1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 22:24:51.659550 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:51.659505    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-b8xrk" podStartSLOduration=3.047345777 podStartE2EDuration="6.659490644s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.612144867s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:24:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:24:51.658397664 +0000 UTC m=+1602.826350650" watchObservedRunningTime="2026-09-17 22:24:51.659490644 +0000 UTC m=+1602.827443627"
Sep 17 22:24:51.676920 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:51.676876    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-5nv57" podStartSLOduration=2.999550751 podStartE2EDuration="6.676861954s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.677311203s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:24:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:24:51.675223731 +0000 UTC m=+1602.843176714" watchObservedRunningTime="2026-09-17 22:24:51.676861954 +0000 UTC m=+1602.844814936"
Sep 17 22:24:52.652210 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:52.652152    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-5nv57" podUID="6b825c2a-99e0-4496-809a-64043cc86dd1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 22:24:52.652675 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:52.652327    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-b8xrk" podUID="832cb065-cc5e-42e2-829c-8eaadd78e6df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 22:24:54.662661 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:54.662629    2563 generic.go:356] "Generic (PLEG): container finished" podID="8f94ae26-e178-4f13-946c-b6d07f52be8a" containerID="619e905c12f513e5dda2d56ee1da2b7e0f24f25d82682a4c5bcf8679a49b9035" exitCode=0
Sep 17 22:24:54.663035 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:54.662708    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-rfvmh" event={"ID":"8f94ae26-e178-4f13-946c-b6d07f52be8a","Type":"ContainerDied","Data":"619e905c12f513e5dda2d56ee1da2b7e0f24f25d82682a4c5bcf8679a49b9035"}
Sep 17 22:24:55.669997 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:55.669963    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-rfvmh" event={"ID":"8f94ae26-e178-4f13-946c-b6d07f52be8a","Type":"ContainerStarted","Data":"2eb0015c84bbdb01d462e3db1373446fcffd44831793c199373279408ffab4c6"}
Sep 17 22:24:55.670407 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:55.670030    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-rfvmh" event={"ID":"8f94ae26-e178-4f13-946c-b6d07f52be8a","Type":"ContainerStarted","Data":"e4ba50a885b507c4a938298b069a5a923c548a27a387daf8d5df2b77c5f0d573"}
Sep 17 22:24:55.670407 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:55.670315    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-rfvmh"
Sep 17 22:24:55.670407 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:55.670367    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-rfvmh"
Sep 17 22:24:55.671455 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:55.671432    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-rfvmh" podUID="8f94ae26-e178-4f13-946c-b6d07f52be8a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 22:24:55.689224 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:55.689016    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-rfvmh" podStartSLOduration=2.766779571 podStartE2EDuration="8.688998012s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.922218441s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:24:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:24:55.688975432 +0000 UTC m=+1606.856928417" watchObservedRunningTime="2026-09-17 22:24:55.688998012 +0000 UTC m=+1606.856950996"
Sep 17 22:24:56.675932 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:56.675894    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-rfvmh" podUID="8f94ae26-e178-4f13-946c-b6d07f52be8a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 22:24:57.656647 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:57.656616    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-5nv57"
Sep 17 22:24:57.657101 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:57.657081    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-b8xrk"
Sep 17 22:24:57.657225 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:57.657172    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-5nv57" podUID="6b825c2a-99e0-4496-809a-64043cc86dd1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 22:24:57.657591 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:24:57.657568    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-b8xrk" podUID="832cb065-cc5e-42e2-829c-8eaadd78e6df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 22:25:01.681235 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:01.681196    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-rfvmh"
Sep 17 22:25:01.681917 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:01.681884    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-rfvmh" podUID="8f94ae26-e178-4f13-946c-b6d07f52be8a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 22:25:07.657163 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:07.657102    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-5nv57" podUID="6b825c2a-99e0-4496-809a-64043cc86dd1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 22:25:07.657739 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:07.657669    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-b8xrk" podUID="832cb065-cc5e-42e2-829c-8eaadd78e6df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 22:25:11.682342 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:11.682315    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-rfvmh"
Sep 17 22:25:17.657929 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:17.657891    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-5nv57" podUID="6b825c2a-99e0-4496-809a-64043cc86dd1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 22:25:17.658414 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:17.657890    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-b8xrk" podUID="832cb065-cc5e-42e2-829c-8eaadd78e6df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 22:25:19.421081 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:19.421044    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-rfvmh"]
Sep 17 22:25:19.421471 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:19.421315    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-rfvmh" podUID="8f94ae26-e178-4f13-946c-b6d07f52be8a" containerName="kserve-container" containerID="cri-o://e4ba50a885b507c4a938298b069a5a923c548a27a387daf8d5df2b77c5f0d573" gracePeriod=30
Sep 17 22:25:19.421471 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:19.421355    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-rfvmh" podUID="8f94ae26-e178-4f13-946c-b6d07f52be8a" containerName="kube-rbac-proxy" containerID="cri-o://2eb0015c84bbdb01d462e3db1373446fcffd44831793c199373279408ffab4c6" gracePeriod=30
Sep 17 22:25:19.782796 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:19.782710    2563 generic.go:356] "Generic (PLEG): container finished" podID="8f94ae26-e178-4f13-946c-b6d07f52be8a" containerID="2eb0015c84bbdb01d462e3db1373446fcffd44831793c199373279408ffab4c6" exitCode=2
Sep 17 22:25:19.782935 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:19.782793    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-rfvmh" event={"ID":"8f94ae26-e178-4f13-946c-b6d07f52be8a","Type":"ContainerDied","Data":"2eb0015c84bbdb01d462e3db1373446fcffd44831793c199373279408ffab4c6"}
Sep 17 22:25:20.358877 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:20.358853    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-rfvmh"
Sep 17 22:25:20.486001 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:20.485914    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f94ae26-e178-4f13-946c-b6d07f52be8a-kserve-provision-location\") pod \"8f94ae26-e178-4f13-946c-b6d07f52be8a\" (UID: \"8f94ae26-e178-4f13-946c-b6d07f52be8a\") "
Sep 17 22:25:20.486001 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:20.485980    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-42x2f\" (UniqueName: \"kubernetes.io/projected/8f94ae26-e178-4f13-946c-b6d07f52be8a-kube-api-access-42x2f\") pod \"8f94ae26-e178-4f13-946c-b6d07f52be8a\" (UID: \"8f94ae26-e178-4f13-946c-b6d07f52be8a\") "
Sep 17 22:25:20.486520 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:20.486046    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f94ae26-e178-4f13-946c-b6d07f52be8a-kube-rbac-proxy-sar-config\") pod \"8f94ae26-e178-4f13-946c-b6d07f52be8a\" (UID: \"8f94ae26-e178-4f13-946c-b6d07f52be8a\") "
Sep 17 22:25:20.486520 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:20.486105    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f94ae26-e178-4f13-946c-b6d07f52be8a-proxy-tls\") pod \"8f94ae26-e178-4f13-946c-b6d07f52be8a\" (UID: \"8f94ae26-e178-4f13-946c-b6d07f52be8a\") "
Sep 17 22:25:20.486520 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:20.486174    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f94ae26-e178-4f13-946c-b6d07f52be8a-cabundle-cert\") pod \"8f94ae26-e178-4f13-946c-b6d07f52be8a\" (UID: \"8f94ae26-e178-4f13-946c-b6d07f52be8a\") "
Sep 17 22:25:20.486520 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:20.486493    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f94ae26-e178-4f13-946c-b6d07f52be8a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8f94ae26-e178-4f13-946c-b6d07f52be8a" (UID: "8f94ae26-e178-4f13-946c-b6d07f52be8a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:25:20.486691 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:20.486604    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f94ae26-e178-4f13-946c-b6d07f52be8a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8f94ae26-e178-4f13-946c-b6d07f52be8a" (UID: "8f94ae26-e178-4f13-946c-b6d07f52be8a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:25:20.488166 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:20.488124    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f94ae26-e178-4f13-946c-b6d07f52be8a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8f94ae26-e178-4f13-946c-b6d07f52be8a" (UID: "8f94ae26-e178-4f13-946c-b6d07f52be8a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:25:20.488308 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:20.488213    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f94ae26-e178-4f13-946c-b6d07f52be8a-kube-api-access-42x2f" (OuterVolumeSpecName: "kube-api-access-42x2f") pod "8f94ae26-e178-4f13-946c-b6d07f52be8a" (UID: "8f94ae26-e178-4f13-946c-b6d07f52be8a"). InnerVolumeSpecName "kube-api-access-42x2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:25:20.514178 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:20.514130    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f94ae26-e178-4f13-946c-b6d07f52be8a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8f94ae26-e178-4f13-946c-b6d07f52be8a" (UID: "8f94ae26-e178-4f13-946c-b6d07f52be8a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:25:20.587500 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:20.587461    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f94ae26-e178-4f13-946c-b6d07f52be8a-proxy-tls\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:25:20.587500 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:20.587491    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f94ae26-e178-4f13-946c-b6d07f52be8a-cabundle-cert\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:25:20.587500 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:20.587500    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f94ae26-e178-4f13-946c-b6d07f52be8a-kserve-provision-location\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:25:20.587500 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:20.587509    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42x2f\" (UniqueName: \"kubernetes.io/projected/8f94ae26-e178-4f13-946c-b6d07f52be8a-kube-api-access-42x2f\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:25:20.587748 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:20.587518    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f94ae26-e178-4f13-946c-b6d07f52be8a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:25:20.789223 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:20.789127    2563 generic.go:356] "Generic (PLEG): container finished" podID="8f94ae26-e178-4f13-946c-b6d07f52be8a" containerID="e4ba50a885b507c4a938298b069a5a923c548a27a387daf8d5df2b77c5f0d573" exitCode=0
Sep 17 22:25:20.789223 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:20.789185    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-rfvmh" event={"ID":"8f94ae26-e178-4f13-946c-b6d07f52be8a","Type":"ContainerDied","Data":"e4ba50a885b507c4a938298b069a5a923c548a27a387daf8d5df2b77c5f0d573"}
Sep 17 22:25:20.789223 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:20.789209    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-rfvmh"
Sep 17 22:25:20.789444 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:20.789251    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-rfvmh" event={"ID":"8f94ae26-e178-4f13-946c-b6d07f52be8a","Type":"ContainerDied","Data":"f30a508a1325b52be4922462ce052ae1995132305de533742e8247271a1a54fd"}
Sep 17 22:25:20.789444 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:20.789280    2563 scope.go:117] "RemoveContainer" containerID="2eb0015c84bbdb01d462e3db1373446fcffd44831793c199373279408ffab4c6"
Sep 17 22:25:20.798400 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:20.798383    2563 scope.go:117] "RemoveContainer" containerID="e4ba50a885b507c4a938298b069a5a923c548a27a387daf8d5df2b77c5f0d573"
Sep 17 22:25:20.805960 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:20.805938    2563 scope.go:117] "RemoveContainer" containerID="619e905c12f513e5dda2d56ee1da2b7e0f24f25d82682a4c5bcf8679a49b9035"
Sep 17 22:25:20.809777 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:20.809756    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-rfvmh"]
Sep 17 22:25:20.813670 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:20.813653    2563 scope.go:117] "RemoveContainer" containerID="2eb0015c84bbdb01d462e3db1373446fcffd44831793c199373279408ffab4c6"
Sep 17 22:25:20.813955 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:20.813937    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56ccc8b76f-rfvmh"]
Sep 17 22:25:20.813999 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:25:20.813940    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2eb0015c84bbdb01d462e3db1373446fcffd44831793c199373279408ffab4c6\": container with ID starting with 2eb0015c84bbdb01d462e3db1373446fcffd44831793c199373279408ffab4c6 not found: ID does not exist" containerID="2eb0015c84bbdb01d462e3db1373446fcffd44831793c199373279408ffab4c6"
Sep 17 22:25:20.814040 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:20.813993    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2eb0015c84bbdb01d462e3db1373446fcffd44831793c199373279408ffab4c6"} err="failed to get container status \"2eb0015c84bbdb01d462e3db1373446fcffd44831793c199373279408ffab4c6\": rpc error: code = NotFound desc = could not find container \"2eb0015c84bbdb01d462e3db1373446fcffd44831793c199373279408ffab4c6\": container with ID starting with 2eb0015c84bbdb01d462e3db1373446fcffd44831793c199373279408ffab4c6 not found: ID does not exist"
Sep 17 22:25:20.814040 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:20.814009    2563 scope.go:117] "RemoveContainer" containerID="e4ba50a885b507c4a938298b069a5a923c548a27a387daf8d5df2b77c5f0d573"
Sep 17 22:25:20.814236 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:25:20.814221    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4ba50a885b507c4a938298b069a5a923c548a27a387daf8d5df2b77c5f0d573\": container with ID starting with e4ba50a885b507c4a938298b069a5a923c548a27a387daf8d5df2b77c5f0d573 not found: ID does not exist" containerID="e4ba50a885b507c4a938298b069a5a923c548a27a387daf8d5df2b77c5f0d573"
Sep 17 22:25:20.814288 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:20.814243    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4ba50a885b507c4a938298b069a5a923c548a27a387daf8d5df2b77c5f0d573"} err="failed to get container status \"e4ba50a885b507c4a938298b069a5a923c548a27a387daf8d5df2b77c5f0d573\": rpc error: code = NotFound desc = could not find container \"e4ba50a885b507c4a938298b069a5a923c548a27a387daf8d5df2b77c5f0d573\": container with ID starting with e4ba50a885b507c4a938298b069a5a923c548a27a387daf8d5df2b77c5f0d573 not found: ID does not exist"
Sep 17 22:25:20.814288 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:20.814257    2563 scope.go:117] "RemoveContainer" containerID="619e905c12f513e5dda2d56ee1da2b7e0f24f25d82682a4c5bcf8679a49b9035"
Sep 17 22:25:20.814491 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:25:20.814476    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"619e905c12f513e5dda2d56ee1da2b7e0f24f25d82682a4c5bcf8679a49b9035\": container with ID starting with 619e905c12f513e5dda2d56ee1da2b7e0f24f25d82682a4c5bcf8679a49b9035 not found: ID does not exist" containerID="619e905c12f513e5dda2d56ee1da2b7e0f24f25d82682a4c5bcf8679a49b9035"
Sep 17 22:25:20.814541 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:20.814493    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"619e905c12f513e5dda2d56ee1da2b7e0f24f25d82682a4c5bcf8679a49b9035"} err="failed to get container status \"619e905c12f513e5dda2d56ee1da2b7e0f24f25d82682a4c5bcf8679a49b9035\": rpc error: code = NotFound desc = could not find container \"619e905c12f513e5dda2d56ee1da2b7e0f24f25d82682a4c5bcf8679a49b9035\": container with ID starting with 619e905c12f513e5dda2d56ee1da2b7e0f24f25d82682a4c5bcf8679a49b9035 not found: ID does not exist"
Sep 17 22:25:20.913970 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:20.913938    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f94ae26-e178-4f13-946c-b6d07f52be8a" path="/var/lib/kubelet/pods/8f94ae26-e178-4f13-946c-b6d07f52be8a/volumes"
Sep 17 22:25:24.597888 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:24.597851    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6n4k"]
Sep 17 22:25:24.598263 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:24.598178    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f94ae26-e178-4f13-946c-b6d07f52be8a" containerName="kserve-container"
Sep 17 22:25:24.598263 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:24.598195    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f94ae26-e178-4f13-946c-b6d07f52be8a" containerName="kserve-container"
Sep 17 22:25:24.598263 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:24.598208    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f94ae26-e178-4f13-946c-b6d07f52be8a" containerName="kube-rbac-proxy"
Sep 17 22:25:24.598263 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:24.598214    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f94ae26-e178-4f13-946c-b6d07f52be8a" containerName="kube-rbac-proxy"
Sep 17 22:25:24.598263 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:24.598224    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f94ae26-e178-4f13-946c-b6d07f52be8a" containerName="storage-initializer"
Sep 17 22:25:24.598263 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:24.598230    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f94ae26-e178-4f13-946c-b6d07f52be8a" containerName="storage-initializer"
Sep 17 22:25:24.598263 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:24.598237    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd11069c-0ab0-454a-a5df-bbc6503f7631" containerName="kserve-container"
Sep 17 22:25:24.598263 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:24.598242    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd11069c-0ab0-454a-a5df-bbc6503f7631" containerName="kserve-container"
Sep 17 22:25:24.598263 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:24.598249    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd11069c-0ab0-454a-a5df-bbc6503f7631" containerName="storage-initializer"
Sep 17 22:25:24.598263 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:24.598254    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd11069c-0ab0-454a-a5df-bbc6503f7631" containerName="storage-initializer"
Sep 17 22:25:24.598263 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:24.598263    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd11069c-0ab0-454a-a5df-bbc6503f7631" containerName="kube-rbac-proxy"
Sep 17 22:25:24.598592 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:24.598271    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd11069c-0ab0-454a-a5df-bbc6503f7631" containerName="kube-rbac-proxy"
Sep 17 22:25:24.598592 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:24.598320    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd11069c-0ab0-454a-a5df-bbc6503f7631" containerName="kube-rbac-proxy"
Sep 17 22:25:24.598592 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:24.598328    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f94ae26-e178-4f13-946c-b6d07f52be8a" containerName="kserve-container"
Sep 17 22:25:24.598592 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:24.598336    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f94ae26-e178-4f13-946c-b6d07f52be8a" containerName="kube-rbac-proxy"
Sep 17 22:25:24.598592 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:24.598347    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd11069c-0ab0-454a-a5df-bbc6503f7631" containerName="kserve-container"
Sep 17 22:25:24.603050 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:24.603034    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6n4k"
Sep 17 22:25:24.609243 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:24.609102    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-predictor-serving-cert\""
Sep 17 22:25:24.609342 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:24.609237    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 17 22:25:24.614356 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:24.614333    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6n4k"]
Sep 17 22:25:24.727414 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:24.727374    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71d8784e-cc84-4506-8d86-5a71d2262920-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6n4k\" (UID: \"71d8784e-cc84-4506-8d86-5a71d2262920\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6n4k"
Sep 17 22:25:24.727602 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:24.727431    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5c5v\" (UniqueName: \"kubernetes.io/projected/71d8784e-cc84-4506-8d86-5a71d2262920-kube-api-access-q5c5v\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6n4k\" (UID: \"71d8784e-cc84-4506-8d86-5a71d2262920\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6n4k"
Sep 17 22:25:24.727602 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:24.727472    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71d8784e-cc84-4506-8d86-5a71d2262920-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6n4k\" (UID: \"71d8784e-cc84-4506-8d86-5a71d2262920\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6n4k"
Sep 17 22:25:24.727602 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:24.727503    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71d8784e-cc84-4506-8d86-5a71d2262920-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6n4k\" (UID: \"71d8784e-cc84-4506-8d86-5a71d2262920\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6n4k"
Sep 17 22:25:24.727602 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:24.727517    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71d8784e-cc84-4506-8d86-5a71d2262920-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6n4k\" (UID: \"71d8784e-cc84-4506-8d86-5a71d2262920\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6n4k"
Sep 17 22:25:24.828345 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:24.828302    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71d8784e-cc84-4506-8d86-5a71d2262920-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6n4k\" (UID: \"71d8784e-cc84-4506-8d86-5a71d2262920\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6n4k"
Sep 17 22:25:24.828500 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:24.828363    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5c5v\" (UniqueName: \"kubernetes.io/projected/71d8784e-cc84-4506-8d86-5a71d2262920-kube-api-access-q5c5v\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6n4k\" (UID: \"71d8784e-cc84-4506-8d86-5a71d2262920\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6n4k"
Sep 17 22:25:24.828500 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:24.828391    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71d8784e-cc84-4506-8d86-5a71d2262920-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6n4k\" (UID: \"71d8784e-cc84-4506-8d86-5a71d2262920\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6n4k"
Sep 17 22:25:24.828500 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:24.828415    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71d8784e-cc84-4506-8d86-5a71d2262920-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6n4k\" (UID: \"71d8784e-cc84-4506-8d86-5a71d2262920\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6n4k"
Sep 17 22:25:24.828500 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:24.828429    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71d8784e-cc84-4506-8d86-5a71d2262920-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6n4k\" (UID: \"71d8784e-cc84-4506-8d86-5a71d2262920\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6n4k"
Sep 17 22:25:24.828759 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:25:24.828532    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-serving-cert: secret "isvc-sklearn-v2-runtime-predictor-serving-cert" not found
Sep 17 22:25:24.828759 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:25:24.828626    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71d8784e-cc84-4506-8d86-5a71d2262920-proxy-tls podName:71d8784e-cc84-4506-8d86-5a71d2262920 nodeName:}" failed. No retries permitted until 2026-09-17 22:25:25.328606385 +0000 UTC m=+1636.496559347 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/71d8784e-cc84-4506-8d86-5a71d2262920-proxy-tls") pod "isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6n4k" (UID: "71d8784e-cc84-4506-8d86-5a71d2262920") : secret "isvc-sklearn-v2-runtime-predictor-serving-cert" not found
Sep 17 22:25:24.828877 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:24.828843    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71d8784e-cc84-4506-8d86-5a71d2262920-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6n4k\" (UID: \"71d8784e-cc84-4506-8d86-5a71d2262920\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6n4k"
Sep 17 22:25:24.829024 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:24.829005    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71d8784e-cc84-4506-8d86-5a71d2262920-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6n4k\" (UID: \"71d8784e-cc84-4506-8d86-5a71d2262920\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6n4k"
Sep 17 22:25:24.829062 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:24.829012    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71d8784e-cc84-4506-8d86-5a71d2262920-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6n4k\" (UID: \"71d8784e-cc84-4506-8d86-5a71d2262920\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6n4k"
Sep 17 22:25:24.836839 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:24.836818    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5c5v\" (UniqueName: \"kubernetes.io/projected/71d8784e-cc84-4506-8d86-5a71d2262920-kube-api-access-q5c5v\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6n4k\" (UID: \"71d8784e-cc84-4506-8d86-5a71d2262920\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6n4k"
Sep 17 22:25:25.331577 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:25.331536    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71d8784e-cc84-4506-8d86-5a71d2262920-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6n4k\" (UID: \"71d8784e-cc84-4506-8d86-5a71d2262920\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6n4k"
Sep 17 22:25:25.334033 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:25.334004    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71d8784e-cc84-4506-8d86-5a71d2262920-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6n4k\" (UID: \"71d8784e-cc84-4506-8d86-5a71d2262920\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6n4k"
Sep 17 22:25:25.513211 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:25.513174    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6n4k"
Sep 17 22:25:25.785330 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:25.785306    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6n4k"]
Sep 17 22:25:25.787596 ip-10-0-138-45 kubenswrapper[2563]: W0917 22:25:25.787564    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71d8784e_cc84_4506_8d86_5a71d2262920.slice/crio-1c7d05d420dbbbf434209415ec26d598f0f0672290c14c1359057dcb448c4dde WatchSource:0}: Error finding container 1c7d05d420dbbbf434209415ec26d598f0f0672290c14c1359057dcb448c4dde: Status 404 returned error can't find the container with id 1c7d05d420dbbbf434209415ec26d598f0f0672290c14c1359057dcb448c4dde
Sep 17 22:25:25.818057 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:25.818035    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6n4k" event={"ID":"71d8784e-cc84-4506-8d86-5a71d2262920","Type":"ContainerStarted","Data":"1c7d05d420dbbbf434209415ec26d598f0f0672290c14c1359057dcb448c4dde"}
Sep 17 22:25:26.824969 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:26.824929    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6n4k" event={"ID":"71d8784e-cc84-4506-8d86-5a71d2262920","Type":"ContainerStarted","Data":"fe6d02baaae62193bb70aedb47a138fc28e9316597a0d6ce15cfd89cea107c4a"}
Sep 17 22:25:27.657409 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:27.657366    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-5nv57" podUID="6b825c2a-99e0-4496-809a-64043cc86dd1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 22:25:27.657601 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:27.657526    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-b8xrk" podUID="832cb065-cc5e-42e2-829c-8eaadd78e6df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 22:25:29.839291 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:29.839202    2563 generic.go:356] "Generic (PLEG): container finished" podID="71d8784e-cc84-4506-8d86-5a71d2262920" containerID="fe6d02baaae62193bb70aedb47a138fc28e9316597a0d6ce15cfd89cea107c4a" exitCode=0
Sep 17 22:25:29.839291 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:29.839279    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6n4k" event={"ID":"71d8784e-cc84-4506-8d86-5a71d2262920","Type":"ContainerDied","Data":"fe6d02baaae62193bb70aedb47a138fc28e9316597a0d6ce15cfd89cea107c4a"}
Sep 17 22:25:30.846820 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:30.846785    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6n4k" event={"ID":"71d8784e-cc84-4506-8d86-5a71d2262920","Type":"ContainerStarted","Data":"60456f96af3099c8b1575a284d6590123bfed5fe213d25ff13bc6bbe83f1bb40"}
Sep 17 22:25:30.847287 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:30.846858    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6n4k" event={"ID":"71d8784e-cc84-4506-8d86-5a71d2262920","Type":"ContainerStarted","Data":"b70cbe6460afb573831989ecf55e29cbf1b249808ec5ba21d859945d57dccd43"}
Sep 17 22:25:30.847287 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:30.847065    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6n4k"
Sep 17 22:25:30.867339 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:30.867285    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6n4k" podStartSLOduration=3.166300421 podStartE2EDuration="6.867266902s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.700966481s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:25:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:25:30.865785829 +0000 UTC m=+1642.033738847" watchObservedRunningTime="2026-09-17 22:25:30.867266902 +0000 UTC m=+1642.035219883"
Sep 17 22:25:31.859538 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:31.859465    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6n4k"
Sep 17 22:25:37.658063 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:37.658017    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-b8xrk" podUID="832cb065-cc5e-42e2-829c-8eaadd78e6df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 22:25:37.658453 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:37.658025    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-5nv57" podUID="6b825c2a-99e0-4496-809a-64043cc86dd1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 22:25:37.865069 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:37.865042    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6n4k"
Sep 17 22:25:47.657180 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:47.657113    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-5nv57" podUID="6b825c2a-99e0-4496-809a-64043cc86dd1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 22:25:47.657790 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:47.657538    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-b8xrk" podUID="832cb065-cc5e-42e2-829c-8eaadd78e6df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 22:25:57.657403 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:57.657352    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-5nv57" podUID="6b825c2a-99e0-4496-809a-64043cc86dd1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 22:25:57.657927 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:25:57.657518    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-b8xrk" podUID="832cb065-cc5e-42e2-829c-8eaadd78e6df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 22:26:07.657761 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:07.657731    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-5nv57"
Sep 17 22:26:07.658305 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:07.657969    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-b8xrk" podUID="832cb065-cc5e-42e2-829c-8eaadd78e6df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 22:26:07.952303 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:07.952199    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6n4k" podUID="71d8784e-cc84-4506-8d86-5a71d2262920" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 17 22:26:11.019784 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:11.019748    2563 scope.go:117] "RemoveContainer" containerID="4df06c8e1c550f70796ce1a66d0d44394b5b893dc189f924bbad5f909b63712a"
Sep 17 22:26:11.027397 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:11.027375    2563 scope.go:117] "RemoveContainer" containerID="e06fdc914d78ff13fc5b777b14903e2521093e594a98d7447c0e82dffb69b7d4"
Sep 17 22:26:17.658336 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:17.658300    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-b8xrk"
Sep 17 22:26:17.868116 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:17.868085    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6n4k"
Sep 17 22:26:24.659093 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:24.659011    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6n4k"]
Sep 17 22:26:24.659625 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:24.659276    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6n4k" podUID="71d8784e-cc84-4506-8d86-5a71d2262920" containerName="kserve-container" containerID="cri-o://b70cbe6460afb573831989ecf55e29cbf1b249808ec5ba21d859945d57dccd43" gracePeriod=30
Sep 17 22:26:24.659625 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:24.659307    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6n4k" podUID="71d8784e-cc84-4506-8d86-5a71d2262920" containerName="kube-rbac-proxy" containerID="cri-o://60456f96af3099c8b1575a284d6590123bfed5fe213d25ff13bc6bbe83f1bb40" gracePeriod=30
Sep 17 22:26:25.100054 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:25.099967    2563 generic.go:356] "Generic (PLEG): container finished" podID="71d8784e-cc84-4506-8d86-5a71d2262920" containerID="60456f96af3099c8b1575a284d6590123bfed5fe213d25ff13bc6bbe83f1bb40" exitCode=2
Sep 17 22:26:25.100054 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:25.100041    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6n4k" event={"ID":"71d8784e-cc84-4506-8d86-5a71d2262920","Type":"ContainerDied","Data":"60456f96af3099c8b1575a284d6590123bfed5fe213d25ff13bc6bbe83f1bb40"}
Sep 17 22:26:25.717638 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:25.717597    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-5nv57"]
Sep 17 22:26:25.718031 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:25.717892    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-5nv57" podUID="6b825c2a-99e0-4496-809a-64043cc86dd1" containerName="kserve-container" containerID="cri-o://2d5e82eb33cd325f02b5445224c555e4eef0d4d7d6709244346dd1d534cbda74" gracePeriod=30
Sep 17 22:26:25.718472 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:25.718266    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-5nv57" podUID="6b825c2a-99e0-4496-809a-64043cc86dd1" containerName="kube-rbac-proxy" containerID="cri-o://8a19361bf824e926d6376f8dacafc01631275550df7a1b5a3c314e6b5b17500c" gracePeriod=30
Sep 17 22:26:25.721577 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:25.721552    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-b8xrk"]
Sep 17 22:26:25.722000 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:25.721971    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-b8xrk" podUID="832cb065-cc5e-42e2-829c-8eaadd78e6df" containerName="kserve-container" containerID="cri-o://33797cec4a4a4f22e349d3e6be4d20bdf4eae7dae665e8afc1ebf1099019344d" gracePeriod=30
Sep 17 22:26:25.722116 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:25.722052    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-b8xrk" podUID="832cb065-cc5e-42e2-829c-8eaadd78e6df" containerName="kube-rbac-proxy" containerID="cri-o://dff64a8bd3579bba16a7b891d769dc2f56bb2f85d4b34af1a98bc74e9be8af49" gracePeriod=30
Sep 17 22:26:26.106673 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:26.106585    2563 generic.go:356] "Generic (PLEG): container finished" podID="6b825c2a-99e0-4496-809a-64043cc86dd1" containerID="8a19361bf824e926d6376f8dacafc01631275550df7a1b5a3c314e6b5b17500c" exitCode=2
Sep 17 22:26:26.106673 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:26.106657    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-5nv57" event={"ID":"6b825c2a-99e0-4496-809a-64043cc86dd1","Type":"ContainerDied","Data":"8a19361bf824e926d6376f8dacafc01631275550df7a1b5a3c314e6b5b17500c"}
Sep 17 22:26:26.108733 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:26.108710    2563 generic.go:356] "Generic (PLEG): container finished" podID="832cb065-cc5e-42e2-829c-8eaadd78e6df" containerID="dff64a8bd3579bba16a7b891d769dc2f56bb2f85d4b34af1a98bc74e9be8af49" exitCode=2
Sep 17 22:26:26.108868 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:26.108741    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-b8xrk" event={"ID":"832cb065-cc5e-42e2-829c-8eaadd78e6df","Type":"ContainerDied","Data":"dff64a8bd3579bba16a7b891d769dc2f56bb2f85d4b34af1a98bc74e9be8af49"}
Sep 17 22:26:27.652994 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:27.652942    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-5nv57" podUID="6b825c2a-99e0-4496-809a-64043cc86dd1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Sep 17 22:26:27.653503 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:27.652951    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-b8xrk" podUID="832cb065-cc5e-42e2-829c-8eaadd78e6df" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Sep 17 22:26:27.657392 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:27.657351    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-5nv57" podUID="6b825c2a-99e0-4496-809a-64043cc86dd1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 22:26:27.657594 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:27.657568    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-b8xrk" podUID="832cb065-cc5e-42e2-829c-8eaadd78e6df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 22:26:27.860077 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:27.860033    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6n4k" podUID="71d8784e-cc84-4506-8d86-5a71d2262920" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.38:8643/healthz\": dial tcp 10.133.0.38:8643: connect: connection refused"
Sep 17 22:26:30.869785 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:30.869757    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-5nv57"
Sep 17 22:26:30.888482 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:30.888456    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6b825c2a-99e0-4496-809a-64043cc86dd1-kserve-provision-location\") pod \"6b825c2a-99e0-4496-809a-64043cc86dd1\" (UID: \"6b825c2a-99e0-4496-809a-64043cc86dd1\") "
Sep 17 22:26:30.888652 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:30.888554    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6b825c2a-99e0-4496-809a-64043cc86dd1-cabundle-cert\") pod \"6b825c2a-99e0-4496-809a-64043cc86dd1\" (UID: \"6b825c2a-99e0-4496-809a-64043cc86dd1\") "
Sep 17 22:26:30.888652 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:30.888584    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-96rst\" (UniqueName: \"kubernetes.io/projected/6b825c2a-99e0-4496-809a-64043cc86dd1-kube-api-access-96rst\") pod \"6b825c2a-99e0-4496-809a-64043cc86dd1\" (UID: \"6b825c2a-99e0-4496-809a-64043cc86dd1\") "
Sep 17 22:26:30.888652 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:30.888626    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6b825c2a-99e0-4496-809a-64043cc86dd1-kube-rbac-proxy-sar-config\") pod \"6b825c2a-99e0-4496-809a-64043cc86dd1\" (UID: \"6b825c2a-99e0-4496-809a-64043cc86dd1\") "
Sep 17 22:26:30.888840 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:30.888655    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6b825c2a-99e0-4496-809a-64043cc86dd1-proxy-tls\") pod \"6b825c2a-99e0-4496-809a-64043cc86dd1\" (UID: \"6b825c2a-99e0-4496-809a-64043cc86dd1\") "
Sep 17 22:26:30.888840 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:30.888779    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b825c2a-99e0-4496-809a-64043cc86dd1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6b825c2a-99e0-4496-809a-64043cc86dd1" (UID: "6b825c2a-99e0-4496-809a-64043cc86dd1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:26:30.888979 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:30.888959    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6b825c2a-99e0-4496-809a-64043cc86dd1-kserve-provision-location\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:26:30.889108 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:30.889028    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b825c2a-99e0-4496-809a-64043cc86dd1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6b825c2a-99e0-4496-809a-64043cc86dd1" (UID: "6b825c2a-99e0-4496-809a-64043cc86dd1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:26:30.889284 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:30.889168    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b825c2a-99e0-4496-809a-64043cc86dd1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6b825c2a-99e0-4496-809a-64043cc86dd1" (UID: "6b825c2a-99e0-4496-809a-64043cc86dd1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:26:30.891110 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:30.891083    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b825c2a-99e0-4496-809a-64043cc86dd1-kube-api-access-96rst" (OuterVolumeSpecName: "kube-api-access-96rst") pod "6b825c2a-99e0-4496-809a-64043cc86dd1" (UID: "6b825c2a-99e0-4496-809a-64043cc86dd1"). InnerVolumeSpecName "kube-api-access-96rst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:26:30.891448 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:30.891310    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b825c2a-99e0-4496-809a-64043cc86dd1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6b825c2a-99e0-4496-809a-64043cc86dd1" (UID: "6b825c2a-99e0-4496-809a-64043cc86dd1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:26:30.967806 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:30.967772    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-m22tp"]
Sep 17 22:26:30.968084 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:30.968073    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b825c2a-99e0-4496-809a-64043cc86dd1" containerName="kube-rbac-proxy"
Sep 17 22:26:30.968128 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:30.968086    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b825c2a-99e0-4496-809a-64043cc86dd1" containerName="kube-rbac-proxy"
Sep 17 22:26:30.968128 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:30.968103    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b825c2a-99e0-4496-809a-64043cc86dd1" containerName="kserve-container"
Sep 17 22:26:30.968128 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:30.968108    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b825c2a-99e0-4496-809a-64043cc86dd1" containerName="kserve-container"
Sep 17 22:26:30.968128 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:30.968121    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b825c2a-99e0-4496-809a-64043cc86dd1" containerName="storage-initializer"
Sep 17 22:26:30.968128 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:30.968127    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b825c2a-99e0-4496-809a-64043cc86dd1" containerName="storage-initializer"
Sep 17 22:26:30.968314 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:30.968191    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b825c2a-99e0-4496-809a-64043cc86dd1" containerName="kserve-container"
Sep 17 22:26:30.968314 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:30.968201    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b825c2a-99e0-4496-809a-64043cc86dd1" containerName="kube-rbac-proxy"
Sep 17 22:26:30.971468 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:30.971452    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-m22tp"
Sep 17 22:26:30.973981 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:30.973962    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 17 22:26:30.973981 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:30.973976    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 17 22:26:30.980679 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:30.980657    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-m22tp"]
Sep 17 22:26:30.989377 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:30.989358    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5ed96d8-a48a-444d-bcbe-6c31417af8f4-cabundle-cert\") pod \"isvc-canary-promote-predictor-567686cc76-m22tp\" (UID: \"c5ed96d8-a48a-444d-bcbe-6c31417af8f4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-m22tp"
Sep 17 22:26:30.989467 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:30.989393    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5ed96d8-a48a-444d-bcbe-6c31417af8f4-proxy-tls\") pod \"isvc-canary-promote-predictor-567686cc76-m22tp\" (UID: \"c5ed96d8-a48a-444d-bcbe-6c31417af8f4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-m22tp"
Sep 17 22:26:30.989507 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:30.989480    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5ed96d8-a48a-444d-bcbe-6c31417af8f4-kserve-provision-location\") pod \"isvc-canary-promote-predictor-567686cc76-m22tp\" (UID: \"c5ed96d8-a48a-444d-bcbe-6c31417af8f4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-m22tp"
Sep 17 22:26:30.989548 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:30.989533    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c5ed96d8-a48a-444d-bcbe-6c31417af8f4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-567686cc76-m22tp\" (UID: \"c5ed96d8-a48a-444d-bcbe-6c31417af8f4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-m22tp"
Sep 17 22:26:30.989583 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:30.989552    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f9g7\" (UniqueName: \"kubernetes.io/projected/c5ed96d8-a48a-444d-bcbe-6c31417af8f4-kube-api-access-5f9g7\") pod \"isvc-canary-promote-predictor-567686cc76-m22tp\" (UID: \"c5ed96d8-a48a-444d-bcbe-6c31417af8f4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-m22tp"
Sep 17 22:26:30.989618 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:30.989584    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6b825c2a-99e0-4496-809a-64043cc86dd1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:26:30.989618 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:30.989598    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6b825c2a-99e0-4496-809a-64043cc86dd1-proxy-tls\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:26:30.989618 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:30.989608    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6b825c2a-99e0-4496-809a-64043cc86dd1-cabundle-cert\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:26:30.989618 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:30.989616    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96rst\" (UniqueName: \"kubernetes.io/projected/6b825c2a-99e0-4496-809a-64043cc86dd1-kube-api-access-96rst\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:26:31.089962 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:31.089927    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5ed96d8-a48a-444d-bcbe-6c31417af8f4-cabundle-cert\") pod \"isvc-canary-promote-predictor-567686cc76-m22tp\" (UID: \"c5ed96d8-a48a-444d-bcbe-6c31417af8f4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-m22tp"
Sep 17 22:26:31.090178 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:31.089976    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5ed96d8-a48a-444d-bcbe-6c31417af8f4-proxy-tls\") pod \"isvc-canary-promote-predictor-567686cc76-m22tp\" (UID: \"c5ed96d8-a48a-444d-bcbe-6c31417af8f4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-m22tp"
Sep 17 22:26:31.090178 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:31.090016    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5ed96d8-a48a-444d-bcbe-6c31417af8f4-kserve-provision-location\") pod \"isvc-canary-promote-predictor-567686cc76-m22tp\" (UID: \"c5ed96d8-a48a-444d-bcbe-6c31417af8f4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-m22tp"
Sep 17 22:26:31.090178 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:31.090042    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c5ed96d8-a48a-444d-bcbe-6c31417af8f4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-567686cc76-m22tp\" (UID: \"c5ed96d8-a48a-444d-bcbe-6c31417af8f4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-m22tp"
Sep 17 22:26:31.090178 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:31.090061    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f9g7\" (UniqueName: \"kubernetes.io/projected/c5ed96d8-a48a-444d-bcbe-6c31417af8f4-kube-api-access-5f9g7\") pod \"isvc-canary-promote-predictor-567686cc76-m22tp\" (UID: \"c5ed96d8-a48a-444d-bcbe-6c31417af8f4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-m22tp"
Sep 17 22:26:31.090490 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:31.090467    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5ed96d8-a48a-444d-bcbe-6c31417af8f4-kserve-provision-location\") pod \"isvc-canary-promote-predictor-567686cc76-m22tp\" (UID: \"c5ed96d8-a48a-444d-bcbe-6c31417af8f4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-m22tp"
Sep 17 22:26:31.090718 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:31.090701    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5ed96d8-a48a-444d-bcbe-6c31417af8f4-cabundle-cert\") pod \"isvc-canary-promote-predictor-567686cc76-m22tp\" (UID: \"c5ed96d8-a48a-444d-bcbe-6c31417af8f4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-m22tp"
Sep 17 22:26:31.090758 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:31.090731    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c5ed96d8-a48a-444d-bcbe-6c31417af8f4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-567686cc76-m22tp\" (UID: \"c5ed96d8-a48a-444d-bcbe-6c31417af8f4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-m22tp"
Sep 17 22:26:31.092461 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:31.092443    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5ed96d8-a48a-444d-bcbe-6c31417af8f4-proxy-tls\") pod \"isvc-canary-promote-predictor-567686cc76-m22tp\" (UID: \"c5ed96d8-a48a-444d-bcbe-6c31417af8f4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-m22tp"
Sep 17 22:26:31.097842 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:31.097820    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f9g7\" (UniqueName: \"kubernetes.io/projected/c5ed96d8-a48a-444d-bcbe-6c31417af8f4-kube-api-access-5f9g7\") pod \"isvc-canary-promote-predictor-567686cc76-m22tp\" (UID: \"c5ed96d8-a48a-444d-bcbe-6c31417af8f4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-m22tp"
Sep 17 22:26:31.132465 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:31.132436    2563 generic.go:356] "Generic (PLEG): container finished" podID="6b825c2a-99e0-4496-809a-64043cc86dd1" containerID="2d5e82eb33cd325f02b5445224c555e4eef0d4d7d6709244346dd1d534cbda74" exitCode=0
Sep 17 22:26:31.132610 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:31.132476    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-5nv57" event={"ID":"6b825c2a-99e0-4496-809a-64043cc86dd1","Type":"ContainerDied","Data":"2d5e82eb33cd325f02b5445224c555e4eef0d4d7d6709244346dd1d534cbda74"}
Sep 17 22:26:31.132610 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:31.132499    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-5nv57"
Sep 17 22:26:31.132610 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:31.132545    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-5nv57" event={"ID":"6b825c2a-99e0-4496-809a-64043cc86dd1","Type":"ContainerDied","Data":"b693ac70a61e2ed20c830bb3e9536beffba7fddcdf9397fd9cd900b9006a1d03"}
Sep 17 22:26:31.132610 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:31.132587    2563 scope.go:117] "RemoveContainer" containerID="8a19361bf824e926d6376f8dacafc01631275550df7a1b5a3c314e6b5b17500c"
Sep 17 22:26:31.140804 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:31.140786    2563 scope.go:117] "RemoveContainer" containerID="2d5e82eb33cd325f02b5445224c555e4eef0d4d7d6709244346dd1d534cbda74"
Sep 17 22:26:31.148931 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:31.148913    2563 scope.go:117] "RemoveContainer" containerID="09c820ea54f2f965f0460866a402ec90fe155431276bdbfaa80519d400e7b071"
Sep 17 22:26:31.149076 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:31.149056    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-5nv57"]
Sep 17 22:26:31.152922 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:31.152888    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-5nv57"]
Sep 17 22:26:31.157373 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:31.157349    2563 scope.go:117] "RemoveContainer" containerID="8a19361bf824e926d6376f8dacafc01631275550df7a1b5a3c314e6b5b17500c"
Sep 17 22:26:31.157708 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:26:31.157681    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a19361bf824e926d6376f8dacafc01631275550df7a1b5a3c314e6b5b17500c\": container with ID starting with 8a19361bf824e926d6376f8dacafc01631275550df7a1b5a3c314e6b5b17500c not found: ID does not exist" containerID="8a19361bf824e926d6376f8dacafc01631275550df7a1b5a3c314e6b5b17500c"
Sep 17 22:26:31.157783 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:31.157717    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a19361bf824e926d6376f8dacafc01631275550df7a1b5a3c314e6b5b17500c"} err="failed to get container status \"8a19361bf824e926d6376f8dacafc01631275550df7a1b5a3c314e6b5b17500c\": rpc error: code = NotFound desc = could not find container \"8a19361bf824e926d6376f8dacafc01631275550df7a1b5a3c314e6b5b17500c\": container with ID starting with 8a19361bf824e926d6376f8dacafc01631275550df7a1b5a3c314e6b5b17500c not found: ID does not exist"
Sep 17 22:26:31.157783 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:31.157737    2563 scope.go:117] "RemoveContainer" containerID="2d5e82eb33cd325f02b5445224c555e4eef0d4d7d6709244346dd1d534cbda74"
Sep 17 22:26:31.158006 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:26:31.157988    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d5e82eb33cd325f02b5445224c555e4eef0d4d7d6709244346dd1d534cbda74\": container with ID starting with 2d5e82eb33cd325f02b5445224c555e4eef0d4d7d6709244346dd1d534cbda74 not found: ID does not exist" containerID="2d5e82eb33cd325f02b5445224c555e4eef0d4d7d6709244346dd1d534cbda74"
Sep 17 22:26:31.158071 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:31.158021    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d5e82eb33cd325f02b5445224c555e4eef0d4d7d6709244346dd1d534cbda74"} err="failed to get container status \"2d5e82eb33cd325f02b5445224c555e4eef0d4d7d6709244346dd1d534cbda74\": rpc error: code = NotFound desc = could not find container \"2d5e82eb33cd325f02b5445224c555e4eef0d4d7d6709244346dd1d534cbda74\": container with ID starting with 2d5e82eb33cd325f02b5445224c555e4eef0d4d7d6709244346dd1d534cbda74 not found: ID does not exist"
Sep 17 22:26:31.158071 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:31.158035    2563 scope.go:117] "RemoveContainer" containerID="09c820ea54f2f965f0460866a402ec90fe155431276bdbfaa80519d400e7b071"
Sep 17 22:26:31.158301 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:26:31.158284    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09c820ea54f2f965f0460866a402ec90fe155431276bdbfaa80519d400e7b071\": container with ID starting with 09c820ea54f2f965f0460866a402ec90fe155431276bdbfaa80519d400e7b071 not found: ID does not exist" containerID="09c820ea54f2f965f0460866a402ec90fe155431276bdbfaa80519d400e7b071"
Sep 17 22:26:31.158378 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:31.158314    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09c820ea54f2f965f0460866a402ec90fe155431276bdbfaa80519d400e7b071"} err="failed to get container status \"09c820ea54f2f965f0460866a402ec90fe155431276bdbfaa80519d400e7b071\": rpc error: code = NotFound desc = could not find container \"09c820ea54f2f965f0460866a402ec90fe155431276bdbfaa80519d400e7b071\": container with ID starting with 09c820ea54f2f965f0460866a402ec90fe155431276bdbfaa80519d400e7b071 not found: ID does not exist"
Sep 17 22:26:31.282276 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:31.282185    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-m22tp"
Sep 17 22:26:31.549428 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:31.549402    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-b8xrk"
Sep 17 22:26:31.581711 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:31.581687    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-m22tp"]
Sep 17 22:26:31.584121 ip-10-0-138-45 kubenswrapper[2563]: W0917 22:26:31.584094    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5ed96d8_a48a_444d_bcbe_6c31417af8f4.slice/crio-0d4e31f228d0e8f21a501e7014c900a7e223465772e17c8566d1cb56c11a6354 WatchSource:0}: Error finding container 0d4e31f228d0e8f21a501e7014c900a7e223465772e17c8566d1cb56c11a6354: Status 404 returned error can't find the container with id 0d4e31f228d0e8f21a501e7014c900a7e223465772e17c8566d1cb56c11a6354
Sep 17 22:26:31.593029 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:31.593005    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/832cb065-cc5e-42e2-829c-8eaadd78e6df-proxy-tls\") pod \"832cb065-cc5e-42e2-829c-8eaadd78e6df\" (UID: \"832cb065-cc5e-42e2-829c-8eaadd78e6df\") "
Sep 17 22:26:31.594988 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:31.594967    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/832cb065-cc5e-42e2-829c-8eaadd78e6df-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "832cb065-cc5e-42e2-829c-8eaadd78e6df" (UID: "832cb065-cc5e-42e2-829c-8eaadd78e6df"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:26:31.693972 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:31.693940    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/832cb065-cc5e-42e2-829c-8eaadd78e6df-kserve-provision-location\") pod \"832cb065-cc5e-42e2-829c-8eaadd78e6df\" (UID: \"832cb065-cc5e-42e2-829c-8eaadd78e6df\") "
Sep 17 22:26:31.694161 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:31.693999    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6v95m\" (UniqueName: \"kubernetes.io/projected/832cb065-cc5e-42e2-829c-8eaadd78e6df-kube-api-access-6v95m\") pod \"832cb065-cc5e-42e2-829c-8eaadd78e6df\" (UID: \"832cb065-cc5e-42e2-829c-8eaadd78e6df\") "
Sep 17 22:26:31.694161 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:31.694029    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/832cb065-cc5e-42e2-829c-8eaadd78e6df-cabundle-cert\") pod \"832cb065-cc5e-42e2-829c-8eaadd78e6df\" (UID: \"832cb065-cc5e-42e2-829c-8eaadd78e6df\") "
Sep 17 22:26:31.694161 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:31.694071    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/832cb065-cc5e-42e2-829c-8eaadd78e6df-kube-rbac-proxy-sar-config\") pod \"832cb065-cc5e-42e2-829c-8eaadd78e6df\" (UID: \"832cb065-cc5e-42e2-829c-8eaadd78e6df\") "
Sep 17 22:26:31.694341 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:31.694291    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/832cb065-cc5e-42e2-829c-8eaadd78e6df-proxy-tls\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:26:31.694400 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:31.694307    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/832cb065-cc5e-42e2-829c-8eaadd78e6df-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "832cb065-cc5e-42e2-829c-8eaadd78e6df" (UID: "832cb065-cc5e-42e2-829c-8eaadd78e6df"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:26:31.694455 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:31.694414    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/832cb065-cc5e-42e2-829c-8eaadd78e6df-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "832cb065-cc5e-42e2-829c-8eaadd78e6df" (UID: "832cb065-cc5e-42e2-829c-8eaadd78e6df"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:26:31.694500 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:31.694478    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/832cb065-cc5e-42e2-829c-8eaadd78e6df-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "832cb065-cc5e-42e2-829c-8eaadd78e6df" (UID: "832cb065-cc5e-42e2-829c-8eaadd78e6df"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:26:31.696115 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:31.696098    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/832cb065-cc5e-42e2-829c-8eaadd78e6df-kube-api-access-6v95m" (OuterVolumeSpecName: "kube-api-access-6v95m") pod "832cb065-cc5e-42e2-829c-8eaadd78e6df" (UID: "832cb065-cc5e-42e2-829c-8eaadd78e6df"). InnerVolumeSpecName "kube-api-access-6v95m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:26:31.794945 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:31.794855    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/832cb065-cc5e-42e2-829c-8eaadd78e6df-kserve-provision-location\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:26:31.794945 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:31.794886    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v95m\" (UniqueName: \"kubernetes.io/projected/832cb065-cc5e-42e2-829c-8eaadd78e6df-kube-api-access-6v95m\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:26:31.794945 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:31.794899    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/832cb065-cc5e-42e2-829c-8eaadd78e6df-cabundle-cert\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:26:31.794945 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:31.794910    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/832cb065-cc5e-42e2-829c-8eaadd78e6df-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:26:32.140166 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:32.140053    2563 generic.go:356] "Generic (PLEG): container finished" podID="832cb065-cc5e-42e2-829c-8eaadd78e6df" containerID="33797cec4a4a4f22e349d3e6be4d20bdf4eae7dae665e8afc1ebf1099019344d" exitCode=0
Sep 17 22:26:32.140166 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:32.140125    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-b8xrk"
Sep 17 22:26:32.140166 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:32.140158    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-b8xrk" event={"ID":"832cb065-cc5e-42e2-829c-8eaadd78e6df","Type":"ContainerDied","Data":"33797cec4a4a4f22e349d3e6be4d20bdf4eae7dae665e8afc1ebf1099019344d"}
Sep 17 22:26:32.140754 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:32.140302    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-b8xrk" event={"ID":"832cb065-cc5e-42e2-829c-8eaadd78e6df","Type":"ContainerDied","Data":"3e666c8ab6781e797af66202d4600facb86a6767ed16476f078458adb5eefea5"}
Sep 17 22:26:32.140754 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:32.140326    2563 scope.go:117] "RemoveContainer" containerID="dff64a8bd3579bba16a7b891d769dc2f56bb2f85d4b34af1a98bc74e9be8af49"
Sep 17 22:26:32.152304 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:32.152235    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-m22tp" event={"ID":"c5ed96d8-a48a-444d-bcbe-6c31417af8f4","Type":"ContainerStarted","Data":"d89c4584caf84e01ce5c11e576b6ed8cd00192a80e5afab95ac28784a482eac5"}
Sep 17 22:26:32.152448 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:32.152338    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-m22tp" event={"ID":"c5ed96d8-a48a-444d-bcbe-6c31417af8f4","Type":"ContainerStarted","Data":"0d4e31f228d0e8f21a501e7014c900a7e223465772e17c8566d1cb56c11a6354"}
Sep 17 22:26:32.157498 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:32.157479    2563 scope.go:117] "RemoveContainer" containerID="33797cec4a4a4f22e349d3e6be4d20bdf4eae7dae665e8afc1ebf1099019344d"
Sep 17 22:26:32.166855 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:32.166833    2563 scope.go:117] "RemoveContainer" containerID="f1e98bf825c49ec2c3bf1c161b63890d7a0077c78b29932d26177738a7a9852f"
Sep 17 22:26:32.175793 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:32.175768    2563 scope.go:117] "RemoveContainer" containerID="dff64a8bd3579bba16a7b891d769dc2f56bb2f85d4b34af1a98bc74e9be8af49"
Sep 17 22:26:32.176108 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:26:32.176090    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dff64a8bd3579bba16a7b891d769dc2f56bb2f85d4b34af1a98bc74e9be8af49\": container with ID starting with dff64a8bd3579bba16a7b891d769dc2f56bb2f85d4b34af1a98bc74e9be8af49 not found: ID does not exist" containerID="dff64a8bd3579bba16a7b891d769dc2f56bb2f85d4b34af1a98bc74e9be8af49"
Sep 17 22:26:32.176179 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:32.176117    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dff64a8bd3579bba16a7b891d769dc2f56bb2f85d4b34af1a98bc74e9be8af49"} err="failed to get container status \"dff64a8bd3579bba16a7b891d769dc2f56bb2f85d4b34af1a98bc74e9be8af49\": rpc error: code = NotFound desc = could not find container \"dff64a8bd3579bba16a7b891d769dc2f56bb2f85d4b34af1a98bc74e9be8af49\": container with ID starting with dff64a8bd3579bba16a7b891d769dc2f56bb2f85d4b34af1a98bc74e9be8af49 not found: ID does not exist"
Sep 17 22:26:32.176179 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:32.176134    2563 scope.go:117] "RemoveContainer" containerID="33797cec4a4a4f22e349d3e6be4d20bdf4eae7dae665e8afc1ebf1099019344d"
Sep 17 22:26:32.176401 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:26:32.176387    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33797cec4a4a4f22e349d3e6be4d20bdf4eae7dae665e8afc1ebf1099019344d\": container with ID starting with 33797cec4a4a4f22e349d3e6be4d20bdf4eae7dae665e8afc1ebf1099019344d not found: ID does not exist" containerID="33797cec4a4a4f22e349d3e6be4d20bdf4eae7dae665e8afc1ebf1099019344d"
Sep 17 22:26:32.176441 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:32.176404    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33797cec4a4a4f22e349d3e6be4d20bdf4eae7dae665e8afc1ebf1099019344d"} err="failed to get container status \"33797cec4a4a4f22e349d3e6be4d20bdf4eae7dae665e8afc1ebf1099019344d\": rpc error: code = NotFound desc = could not find container \"33797cec4a4a4f22e349d3e6be4d20bdf4eae7dae665e8afc1ebf1099019344d\": container with ID starting with 33797cec4a4a4f22e349d3e6be4d20bdf4eae7dae665e8afc1ebf1099019344d not found: ID does not exist"
Sep 17 22:26:32.176441 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:32.176419    2563 scope.go:117] "RemoveContainer" containerID="f1e98bf825c49ec2c3bf1c161b63890d7a0077c78b29932d26177738a7a9852f"
Sep 17 22:26:32.176684 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:26:32.176669    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1e98bf825c49ec2c3bf1c161b63890d7a0077c78b29932d26177738a7a9852f\": container with ID starting with f1e98bf825c49ec2c3bf1c161b63890d7a0077c78b29932d26177738a7a9852f not found: ID does not exist" containerID="f1e98bf825c49ec2c3bf1c161b63890d7a0077c78b29932d26177738a7a9852f"
Sep 17 22:26:32.176732 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:32.176688    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1e98bf825c49ec2c3bf1c161b63890d7a0077c78b29932d26177738a7a9852f"} err="failed to get container status \"f1e98bf825c49ec2c3bf1c161b63890d7a0077c78b29932d26177738a7a9852f\": rpc error: code = NotFound desc = could not find container \"f1e98bf825c49ec2c3bf1c161b63890d7a0077c78b29932d26177738a7a9852f\": container with ID starting with f1e98bf825c49ec2c3bf1c161b63890d7a0077c78b29932d26177738a7a9852f not found: ID does not exist"
Sep 17 22:26:32.184388 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:32.184364    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-b8xrk"]
Sep 17 22:26:32.187656 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:32.187635    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-b8xrk"]
Sep 17 22:26:32.859842 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:32.859804    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6n4k" podUID="71d8784e-cc84-4506-8d86-5a71d2262920" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.38:8643/healthz\": dial tcp 10.133.0.38:8643: connect: connection refused"
Sep 17 22:26:32.916134 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:32.916100    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b825c2a-99e0-4496-809a-64043cc86dd1" path="/var/lib/kubelet/pods/6b825c2a-99e0-4496-809a-64043cc86dd1/volumes"
Sep 17 22:26:32.916626 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:32.916612    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="832cb065-cc5e-42e2-829c-8eaadd78e6df" path="/var/lib/kubelet/pods/832cb065-cc5e-42e2-829c-8eaadd78e6df/volumes"
Sep 17 22:26:33.112437 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:33.112383    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6n4k"
Sep 17 22:26:33.159360 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:33.159325    2563 generic.go:356] "Generic (PLEG): container finished" podID="71d8784e-cc84-4506-8d86-5a71d2262920" containerID="b70cbe6460afb573831989ecf55e29cbf1b249808ec5ba21d859945d57dccd43" exitCode=0
Sep 17 22:26:33.159756 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:33.159385    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6n4k"
Sep 17 22:26:33.159756 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:33.159410    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6n4k" event={"ID":"71d8784e-cc84-4506-8d86-5a71d2262920","Type":"ContainerDied","Data":"b70cbe6460afb573831989ecf55e29cbf1b249808ec5ba21d859945d57dccd43"}
Sep 17 22:26:33.159756 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:33.159470    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6n4k" event={"ID":"71d8784e-cc84-4506-8d86-5a71d2262920","Type":"ContainerDied","Data":"1c7d05d420dbbbf434209415ec26d598f0f0672290c14c1359057dcb448c4dde"}
Sep 17 22:26:33.159756 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:33.159495    2563 scope.go:117] "RemoveContainer" containerID="60456f96af3099c8b1575a284d6590123bfed5fe213d25ff13bc6bbe83f1bb40"
Sep 17 22:26:33.167925 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:33.167906    2563 scope.go:117] "RemoveContainer" containerID="b70cbe6460afb573831989ecf55e29cbf1b249808ec5ba21d859945d57dccd43"
Sep 17 22:26:33.176124 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:33.176104    2563 scope.go:117] "RemoveContainer" containerID="fe6d02baaae62193bb70aedb47a138fc28e9316597a0d6ce15cfd89cea107c4a"
Sep 17 22:26:33.183470 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:33.183452    2563 scope.go:117] "RemoveContainer" containerID="60456f96af3099c8b1575a284d6590123bfed5fe213d25ff13bc6bbe83f1bb40"
Sep 17 22:26:33.183739 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:26:33.183720    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60456f96af3099c8b1575a284d6590123bfed5fe213d25ff13bc6bbe83f1bb40\": container with ID starting with 60456f96af3099c8b1575a284d6590123bfed5fe213d25ff13bc6bbe83f1bb40 not found: ID does not exist" containerID="60456f96af3099c8b1575a284d6590123bfed5fe213d25ff13bc6bbe83f1bb40"
Sep 17 22:26:33.183782 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:33.183752    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60456f96af3099c8b1575a284d6590123bfed5fe213d25ff13bc6bbe83f1bb40"} err="failed to get container status \"60456f96af3099c8b1575a284d6590123bfed5fe213d25ff13bc6bbe83f1bb40\": rpc error: code = NotFound desc = could not find container \"60456f96af3099c8b1575a284d6590123bfed5fe213d25ff13bc6bbe83f1bb40\": container with ID starting with 60456f96af3099c8b1575a284d6590123bfed5fe213d25ff13bc6bbe83f1bb40 not found: ID does not exist"
Sep 17 22:26:33.183782 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:33.183770    2563 scope.go:117] "RemoveContainer" containerID="b70cbe6460afb573831989ecf55e29cbf1b249808ec5ba21d859945d57dccd43"
Sep 17 22:26:33.184027 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:26:33.184014    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b70cbe6460afb573831989ecf55e29cbf1b249808ec5ba21d859945d57dccd43\": container with ID starting with b70cbe6460afb573831989ecf55e29cbf1b249808ec5ba21d859945d57dccd43 not found: ID does not exist" containerID="b70cbe6460afb573831989ecf55e29cbf1b249808ec5ba21d859945d57dccd43"
Sep 17 22:26:33.184075 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:33.184031    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b70cbe6460afb573831989ecf55e29cbf1b249808ec5ba21d859945d57dccd43"} err="failed to get container status \"b70cbe6460afb573831989ecf55e29cbf1b249808ec5ba21d859945d57dccd43\": rpc error: code = NotFound desc = could not find container \"b70cbe6460afb573831989ecf55e29cbf1b249808ec5ba21d859945d57dccd43\": container with ID starting with b70cbe6460afb573831989ecf55e29cbf1b249808ec5ba21d859945d57dccd43 not found: ID does not exist"
Sep 17 22:26:33.184075 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:33.184042    2563 scope.go:117] "RemoveContainer" containerID="fe6d02baaae62193bb70aedb47a138fc28e9316597a0d6ce15cfd89cea107c4a"
Sep 17 22:26:33.184299 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:26:33.184279    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe6d02baaae62193bb70aedb47a138fc28e9316597a0d6ce15cfd89cea107c4a\": container with ID starting with fe6d02baaae62193bb70aedb47a138fc28e9316597a0d6ce15cfd89cea107c4a not found: ID does not exist" containerID="fe6d02baaae62193bb70aedb47a138fc28e9316597a0d6ce15cfd89cea107c4a"
Sep 17 22:26:33.184342 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:33.184303    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe6d02baaae62193bb70aedb47a138fc28e9316597a0d6ce15cfd89cea107c4a"} err="failed to get container status \"fe6d02baaae62193bb70aedb47a138fc28e9316597a0d6ce15cfd89cea107c4a\": rpc error: code = NotFound desc = could not find container \"fe6d02baaae62193bb70aedb47a138fc28e9316597a0d6ce15cfd89cea107c4a\": container with ID starting with fe6d02baaae62193bb70aedb47a138fc28e9316597a0d6ce15cfd89cea107c4a not found: ID does not exist"
Sep 17 22:26:33.307316 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:33.307277    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71d8784e-cc84-4506-8d86-5a71d2262920-proxy-tls\") pod \"71d8784e-cc84-4506-8d86-5a71d2262920\" (UID: \"71d8784e-cc84-4506-8d86-5a71d2262920\") "
Sep 17 22:26:33.307477 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:33.307323    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71d8784e-cc84-4506-8d86-5a71d2262920-cabundle-cert\") pod \"71d8784e-cc84-4506-8d86-5a71d2262920\" (UID: \"71d8784e-cc84-4506-8d86-5a71d2262920\") "
Sep 17 22:26:33.307477 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:33.307369    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71d8784e-cc84-4506-8d86-5a71d2262920-kube-rbac-proxy-sar-config\") pod \"71d8784e-cc84-4506-8d86-5a71d2262920\" (UID: \"71d8784e-cc84-4506-8d86-5a71d2262920\") "
Sep 17 22:26:33.307477 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:33.307392    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q5c5v\" (UniqueName: \"kubernetes.io/projected/71d8784e-cc84-4506-8d86-5a71d2262920-kube-api-access-q5c5v\") pod \"71d8784e-cc84-4506-8d86-5a71d2262920\" (UID: \"71d8784e-cc84-4506-8d86-5a71d2262920\") "
Sep 17 22:26:33.307477 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:33.307420    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71d8784e-cc84-4506-8d86-5a71d2262920-kserve-provision-location\") pod \"71d8784e-cc84-4506-8d86-5a71d2262920\" (UID: \"71d8784e-cc84-4506-8d86-5a71d2262920\") "
Sep 17 22:26:33.307825 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:33.307795    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71d8784e-cc84-4506-8d86-5a71d2262920-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "71d8784e-cc84-4506-8d86-5a71d2262920" (UID: "71d8784e-cc84-4506-8d86-5a71d2262920"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:26:33.307887 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:33.307806    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71d8784e-cc84-4506-8d86-5a71d2262920-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "71d8784e-cc84-4506-8d86-5a71d2262920" (UID: "71d8784e-cc84-4506-8d86-5a71d2262920"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:26:33.307887 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:33.307804    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71d8784e-cc84-4506-8d86-5a71d2262920-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "71d8784e-cc84-4506-8d86-5a71d2262920" (UID: "71d8784e-cc84-4506-8d86-5a71d2262920"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:26:33.309608 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:33.309587    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71d8784e-cc84-4506-8d86-5a71d2262920-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "71d8784e-cc84-4506-8d86-5a71d2262920" (UID: "71d8784e-cc84-4506-8d86-5a71d2262920"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:26:33.309799 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:33.309777    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71d8784e-cc84-4506-8d86-5a71d2262920-kube-api-access-q5c5v" (OuterVolumeSpecName: "kube-api-access-q5c5v") pod "71d8784e-cc84-4506-8d86-5a71d2262920" (UID: "71d8784e-cc84-4506-8d86-5a71d2262920"). InnerVolumeSpecName "kube-api-access-q5c5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:26:33.408261 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:33.408223    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71d8784e-cc84-4506-8d86-5a71d2262920-proxy-tls\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:26:33.408261 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:33.408257    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71d8784e-cc84-4506-8d86-5a71d2262920-cabundle-cert\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:26:33.408465 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:33.408270    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71d8784e-cc84-4506-8d86-5a71d2262920-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:26:33.408465 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:33.408283    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5c5v\" (UniqueName: \"kubernetes.io/projected/71d8784e-cc84-4506-8d86-5a71d2262920-kube-api-access-q5c5v\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:26:33.408465 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:33.408295    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71d8784e-cc84-4506-8d86-5a71d2262920-kserve-provision-location\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:26:33.481652 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:33.481620    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6n4k"]
Sep 17 22:26:33.484777 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:33.484756    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6n4k"]
Sep 17 22:26:34.913273 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:34.913238    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71d8784e-cc84-4506-8d86-5a71d2262920" path="/var/lib/kubelet/pods/71d8784e-cc84-4506-8d86-5a71d2262920/volumes"
Sep 17 22:26:36.173771 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:36.173736    2563 generic.go:356] "Generic (PLEG): container finished" podID="c5ed96d8-a48a-444d-bcbe-6c31417af8f4" containerID="d89c4584caf84e01ce5c11e576b6ed8cd00192a80e5afab95ac28784a482eac5" exitCode=0
Sep 17 22:26:36.174165 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:36.173788    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-m22tp" event={"ID":"c5ed96d8-a48a-444d-bcbe-6c31417af8f4","Type":"ContainerDied","Data":"d89c4584caf84e01ce5c11e576b6ed8cd00192a80e5afab95ac28784a482eac5"}
Sep 17 22:26:37.180896 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:37.180858    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-m22tp" event={"ID":"c5ed96d8-a48a-444d-bcbe-6c31417af8f4","Type":"ContainerStarted","Data":"16f28f641bfcf900685fe2a3cb451c890b1a35f3c31cca0adc19d6d0f1352366"}
Sep 17 22:26:37.181283 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:37.180920    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-m22tp" event={"ID":"c5ed96d8-a48a-444d-bcbe-6c31417af8f4","Type":"ContainerStarted","Data":"82d7d00cbe58df167aa12579621116abc38b853d0e50cf62d596c3ce29d3491a"}
Sep 17 22:26:37.181283 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:37.181104    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-m22tp"
Sep 17 22:26:37.200322 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:37.200269    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-m22tp" podStartSLOduration=3.460611243 podStartE2EDuration="7.200251805s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.739640562s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:26:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:26:37.19977858 +0000 UTC m=+1708.367731557" watchObservedRunningTime="2026-09-17 22:26:37.200251805 +0000 UTC m=+1708.368204789"
Sep 17 22:26:38.186771 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:38.186739    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-m22tp"
Sep 17 22:26:38.188001 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:38.187971    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-m22tp" podUID="c5ed96d8-a48a-444d-bcbe-6c31417af8f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 22:26:39.190822 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:39.190785    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-m22tp" podUID="c5ed96d8-a48a-444d-bcbe-6c31417af8f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 22:26:44.195622 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:44.195595    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-m22tp"
Sep 17 22:26:44.196245 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:44.196216    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-m22tp" podUID="c5ed96d8-a48a-444d-bcbe-6c31417af8f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 22:26:54.196291 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:26:54.196252    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-m22tp" podUID="c5ed96d8-a48a-444d-bcbe-6c31417af8f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 22:27:04.196872 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:27:04.196826    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-m22tp" podUID="c5ed96d8-a48a-444d-bcbe-6c31417af8f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 22:27:14.196811 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:27:14.196766    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-m22tp" podUID="c5ed96d8-a48a-444d-bcbe-6c31417af8f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 22:27:24.196998 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:27:24.196952    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-m22tp" podUID="c5ed96d8-a48a-444d-bcbe-6c31417af8f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 22:27:34.196725 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:27:34.196682    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-m22tp" podUID="c5ed96d8-a48a-444d-bcbe-6c31417af8f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 22:27:44.196692 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:27:44.196656    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-m22tp" podUID="c5ed96d8-a48a-444d-bcbe-6c31417af8f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 22:27:54.197380 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:27:54.197304    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-m22tp"
Sep 17 22:27:54.585960 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:27:54.585861    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-m22tp"]
Sep 17 22:27:54.586200 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:27:54.586173    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-m22tp" podUID="c5ed96d8-a48a-444d-bcbe-6c31417af8f4" containerName="kserve-container" containerID="cri-o://82d7d00cbe58df167aa12579621116abc38b853d0e50cf62d596c3ce29d3491a" gracePeriod=30
Sep 17 22:27:54.586322 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:27:54.586177    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-m22tp" podUID="c5ed96d8-a48a-444d-bcbe-6c31417af8f4" containerName="kube-rbac-proxy" containerID="cri-o://16f28f641bfcf900685fe2a3cb451c890b1a35f3c31cca0adc19d6d0f1352366" gracePeriod=30
Sep 17 22:27:55.500695 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:27:55.500657    2563 generic.go:356] "Generic (PLEG): container finished" podID="c5ed96d8-a48a-444d-bcbe-6c31417af8f4" containerID="16f28f641bfcf900685fe2a3cb451c890b1a35f3c31cca0adc19d6d0f1352366" exitCode=2
Sep 17 22:27:55.501074 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:27:55.500732    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-m22tp" event={"ID":"c5ed96d8-a48a-444d-bcbe-6c31417af8f4","Type":"ContainerDied","Data":"16f28f641bfcf900685fe2a3cb451c890b1a35f3c31cca0adc19d6d0f1352366"}
Sep 17 22:27:59.192013 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:27:59.191963    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-m22tp" podUID="c5ed96d8-a48a-444d-bcbe-6c31417af8f4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.39:8643/healthz\": dial tcp 10.133.0.39:8643: connect: connection refused"
Sep 17 22:27:59.319317 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:27:59.319292    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-m22tp"
Sep 17 22:27:59.420280 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:27:59.420250    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c5ed96d8-a48a-444d-bcbe-6c31417af8f4-kube-rbac-proxy-sar-config\") pod \"c5ed96d8-a48a-444d-bcbe-6c31417af8f4\" (UID: \"c5ed96d8-a48a-444d-bcbe-6c31417af8f4\") "
Sep 17 22:27:59.420449 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:27:59.420318    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5ed96d8-a48a-444d-bcbe-6c31417af8f4-kserve-provision-location\") pod \"c5ed96d8-a48a-444d-bcbe-6c31417af8f4\" (UID: \"c5ed96d8-a48a-444d-bcbe-6c31417af8f4\") "
Sep 17 22:27:59.420449 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:27:59.420368    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5ed96d8-a48a-444d-bcbe-6c31417af8f4-cabundle-cert\") pod \"c5ed96d8-a48a-444d-bcbe-6c31417af8f4\" (UID: \"c5ed96d8-a48a-444d-bcbe-6c31417af8f4\") "
Sep 17 22:27:59.420449 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:27:59.420386    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5ed96d8-a48a-444d-bcbe-6c31417af8f4-proxy-tls\") pod \"c5ed96d8-a48a-444d-bcbe-6c31417af8f4\" (UID: \"c5ed96d8-a48a-444d-bcbe-6c31417af8f4\") "
Sep 17 22:27:59.420449 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:27:59.420402    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5f9g7\" (UniqueName: \"kubernetes.io/projected/c5ed96d8-a48a-444d-bcbe-6c31417af8f4-kube-api-access-5f9g7\") pod \"c5ed96d8-a48a-444d-bcbe-6c31417af8f4\" (UID: \"c5ed96d8-a48a-444d-bcbe-6c31417af8f4\") "
Sep 17 22:27:59.420686 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:27:59.420649    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5ed96d8-a48a-444d-bcbe-6c31417af8f4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c5ed96d8-a48a-444d-bcbe-6c31417af8f4" (UID: "c5ed96d8-a48a-444d-bcbe-6c31417af8f4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:27:59.420740 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:27:59.420688    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5ed96d8-a48a-444d-bcbe-6c31417af8f4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c5ed96d8-a48a-444d-bcbe-6c31417af8f4" (UID: "c5ed96d8-a48a-444d-bcbe-6c31417af8f4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:27:59.420797 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:27:59.420736    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5ed96d8-a48a-444d-bcbe-6c31417af8f4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c5ed96d8-a48a-444d-bcbe-6c31417af8f4" (UID: "c5ed96d8-a48a-444d-bcbe-6c31417af8f4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:27:59.422571 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:27:59.422542    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5ed96d8-a48a-444d-bcbe-6c31417af8f4-kube-api-access-5f9g7" (OuterVolumeSpecName: "kube-api-access-5f9g7") pod "c5ed96d8-a48a-444d-bcbe-6c31417af8f4" (UID: "c5ed96d8-a48a-444d-bcbe-6c31417af8f4"). InnerVolumeSpecName "kube-api-access-5f9g7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:27:59.422675 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:27:59.422575    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5ed96d8-a48a-444d-bcbe-6c31417af8f4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c5ed96d8-a48a-444d-bcbe-6c31417af8f4" (UID: "c5ed96d8-a48a-444d-bcbe-6c31417af8f4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:27:59.520049 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:27:59.520015    2563 generic.go:356] "Generic (PLEG): container finished" podID="c5ed96d8-a48a-444d-bcbe-6c31417af8f4" containerID="82d7d00cbe58df167aa12579621116abc38b853d0e50cf62d596c3ce29d3491a" exitCode=0
Sep 17 22:27:59.520239 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:27:59.520075    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-m22tp"
Sep 17 22:27:59.520239 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:27:59.520096    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-m22tp" event={"ID":"c5ed96d8-a48a-444d-bcbe-6c31417af8f4","Type":"ContainerDied","Data":"82d7d00cbe58df167aa12579621116abc38b853d0e50cf62d596c3ce29d3491a"}
Sep 17 22:27:59.520239 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:27:59.520172    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-m22tp" event={"ID":"c5ed96d8-a48a-444d-bcbe-6c31417af8f4","Type":"ContainerDied","Data":"0d4e31f228d0e8f21a501e7014c900a7e223465772e17c8566d1cb56c11a6354"}
Sep 17 22:27:59.520239 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:27:59.520200    2563 scope.go:117] "RemoveContainer" containerID="16f28f641bfcf900685fe2a3cb451c890b1a35f3c31cca0adc19d6d0f1352366"
Sep 17 22:27:59.520920 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:27:59.520900    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5ed96d8-a48a-444d-bcbe-6c31417af8f4-cabundle-cert\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:27:59.520920 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:27:59.520919    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5ed96d8-a48a-444d-bcbe-6c31417af8f4-proxy-tls\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:27:59.521062 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:27:59.520927    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5f9g7\" (UniqueName: \"kubernetes.io/projected/c5ed96d8-a48a-444d-bcbe-6c31417af8f4-kube-api-access-5f9g7\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:27:59.521062 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:27:59.520936    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c5ed96d8-a48a-444d-bcbe-6c31417af8f4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:27:59.521062 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:27:59.520945    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5ed96d8-a48a-444d-bcbe-6c31417af8f4-kserve-provision-location\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:27:59.528773 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:27:59.528757    2563 scope.go:117] "RemoveContainer" containerID="82d7d00cbe58df167aa12579621116abc38b853d0e50cf62d596c3ce29d3491a"
Sep 17 22:27:59.535682 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:27:59.535662    2563 scope.go:117] "RemoveContainer" containerID="d89c4584caf84e01ce5c11e576b6ed8cd00192a80e5afab95ac28784a482eac5"
Sep 17 22:27:59.540480 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:27:59.540460    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-m22tp"]
Sep 17 22:27:59.542897 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:27:59.542885    2563 scope.go:117] "RemoveContainer" containerID="16f28f641bfcf900685fe2a3cb451c890b1a35f3c31cca0adc19d6d0f1352366"
Sep 17 22:27:59.543156 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:27:59.543128    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16f28f641bfcf900685fe2a3cb451c890b1a35f3c31cca0adc19d6d0f1352366\": container with ID starting with 16f28f641bfcf900685fe2a3cb451c890b1a35f3c31cca0adc19d6d0f1352366 not found: ID does not exist" containerID="16f28f641bfcf900685fe2a3cb451c890b1a35f3c31cca0adc19d6d0f1352366"
Sep 17 22:27:59.543206 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:27:59.543163    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16f28f641bfcf900685fe2a3cb451c890b1a35f3c31cca0adc19d6d0f1352366"} err="failed to get container status \"16f28f641bfcf900685fe2a3cb451c890b1a35f3c31cca0adc19d6d0f1352366\": rpc error: code = NotFound desc = could not find container \"16f28f641bfcf900685fe2a3cb451c890b1a35f3c31cca0adc19d6d0f1352366\": container with ID starting with 16f28f641bfcf900685fe2a3cb451c890b1a35f3c31cca0adc19d6d0f1352366 not found: ID does not exist"
Sep 17 22:27:59.543206 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:27:59.543178    2563 scope.go:117] "RemoveContainer" containerID="82d7d00cbe58df167aa12579621116abc38b853d0e50cf62d596c3ce29d3491a"
Sep 17 22:27:59.543411 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:27:59.543392    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82d7d00cbe58df167aa12579621116abc38b853d0e50cf62d596c3ce29d3491a\": container with ID starting with 82d7d00cbe58df167aa12579621116abc38b853d0e50cf62d596c3ce29d3491a not found: ID does not exist" containerID="82d7d00cbe58df167aa12579621116abc38b853d0e50cf62d596c3ce29d3491a"
Sep 17 22:27:59.543457 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:27:59.543418    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82d7d00cbe58df167aa12579621116abc38b853d0e50cf62d596c3ce29d3491a"} err="failed to get container status \"82d7d00cbe58df167aa12579621116abc38b853d0e50cf62d596c3ce29d3491a\": rpc error: code = NotFound desc = could not find container \"82d7d00cbe58df167aa12579621116abc38b853d0e50cf62d596c3ce29d3491a\": container with ID starting with 82d7d00cbe58df167aa12579621116abc38b853d0e50cf62d596c3ce29d3491a not found: ID does not exist"
Sep 17 22:27:59.543457 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:27:59.543434    2563 scope.go:117] "RemoveContainer" containerID="d89c4584caf84e01ce5c11e576b6ed8cd00192a80e5afab95ac28784a482eac5"
Sep 17 22:27:59.543646 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:27:59.543630    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d89c4584caf84e01ce5c11e576b6ed8cd00192a80e5afab95ac28784a482eac5\": container with ID starting with d89c4584caf84e01ce5c11e576b6ed8cd00192a80e5afab95ac28784a482eac5 not found: ID does not exist" containerID="d89c4584caf84e01ce5c11e576b6ed8cd00192a80e5afab95ac28784a482eac5"
Sep 17 22:27:59.543696 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:27:59.543655    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d89c4584caf84e01ce5c11e576b6ed8cd00192a80e5afab95ac28784a482eac5"} err="failed to get container status \"d89c4584caf84e01ce5c11e576b6ed8cd00192a80e5afab95ac28784a482eac5\": rpc error: code = NotFound desc = could not find container \"d89c4584caf84e01ce5c11e576b6ed8cd00192a80e5afab95ac28784a482eac5\": container with ID starting with d89c4584caf84e01ce5c11e576b6ed8cd00192a80e5afab95ac28784a482eac5 not found: ID does not exist"
Sep 17 22:27:59.554448 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:27:59.554427    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-m22tp"]
Sep 17 22:28:00.913513 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:00.913469    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5ed96d8-a48a-444d-bcbe-6c31417af8f4" path="/var/lib/kubelet/pods/c5ed96d8-a48a-444d-bcbe-6c31417af8f4/volumes"
Sep 17 22:28:01.027046 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:01.027016    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-2tf77"]
Sep 17 22:28:01.027331 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:01.027318    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="832cb065-cc5e-42e2-829c-8eaadd78e6df" containerName="storage-initializer"
Sep 17 22:28:01.027378 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:01.027333    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="832cb065-cc5e-42e2-829c-8eaadd78e6df" containerName="storage-initializer"
Sep 17 22:28:01.027378 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:01.027341    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71d8784e-cc84-4506-8d86-5a71d2262920" containerName="storage-initializer"
Sep 17 22:28:01.027378 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:01.027346    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="71d8784e-cc84-4506-8d86-5a71d2262920" containerName="storage-initializer"
Sep 17 22:28:01.027378 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:01.027354    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71d8784e-cc84-4506-8d86-5a71d2262920" containerName="kube-rbac-proxy"
Sep 17 22:28:01.027378 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:01.027360    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="71d8784e-cc84-4506-8d86-5a71d2262920" containerName="kube-rbac-proxy"
Sep 17 22:28:01.027378 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:01.027372    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71d8784e-cc84-4506-8d86-5a71d2262920" containerName="kserve-container"
Sep 17 22:28:01.027378 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:01.027378    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="71d8784e-cc84-4506-8d86-5a71d2262920" containerName="kserve-container"
Sep 17 22:28:01.027596 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:01.027383    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5ed96d8-a48a-444d-bcbe-6c31417af8f4" containerName="kube-rbac-proxy"
Sep 17 22:28:01.027596 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:01.027389    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5ed96d8-a48a-444d-bcbe-6c31417af8f4" containerName="kube-rbac-proxy"
Sep 17 22:28:01.027596 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:01.027401    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="832cb065-cc5e-42e2-829c-8eaadd78e6df" containerName="kube-rbac-proxy"
Sep 17 22:28:01.027596 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:01.027407    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="832cb065-cc5e-42e2-829c-8eaadd78e6df" containerName="kube-rbac-proxy"
Sep 17 22:28:01.027596 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:01.027413    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5ed96d8-a48a-444d-bcbe-6c31417af8f4" containerName="kserve-container"
Sep 17 22:28:01.027596 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:01.027418    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5ed96d8-a48a-444d-bcbe-6c31417af8f4" containerName="kserve-container"
Sep 17 22:28:01.027596 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:01.027425    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5ed96d8-a48a-444d-bcbe-6c31417af8f4" containerName="storage-initializer"
Sep 17 22:28:01.027596 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:01.027430    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5ed96d8-a48a-444d-bcbe-6c31417af8f4" containerName="storage-initializer"
Sep 17 22:28:01.027596 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:01.027436    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="832cb065-cc5e-42e2-829c-8eaadd78e6df" containerName="kserve-container"
Sep 17 22:28:01.027596 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:01.027441    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="832cb065-cc5e-42e2-829c-8eaadd78e6df" containerName="kserve-container"
Sep 17 22:28:01.027596 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:01.027483    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5ed96d8-a48a-444d-bcbe-6c31417af8f4" containerName="kube-rbac-proxy"
Sep 17 22:28:01.027596 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:01.027492    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="71d8784e-cc84-4506-8d86-5a71d2262920" containerName="kube-rbac-proxy"
Sep 17 22:28:01.027596 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:01.027498    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5ed96d8-a48a-444d-bcbe-6c31417af8f4" containerName="kserve-container"
Sep 17 22:28:01.027596 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:01.027504    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="71d8784e-cc84-4506-8d86-5a71d2262920" containerName="kserve-container"
Sep 17 22:28:01.027596 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:01.027510    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="832cb065-cc5e-42e2-829c-8eaadd78e6df" containerName="kube-rbac-proxy"
Sep 17 22:28:01.027596 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:01.027517    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="832cb065-cc5e-42e2-829c-8eaadd78e6df" containerName="kserve-container"
Sep 17 22:28:01.032001 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:01.031984    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-2tf77"
Sep 17 22:28:01.034777 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:01.034749    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 17 22:28:01.034916 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:01.034779    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 22:28:01.034916 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:01.034803    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 17 22:28:01.034916 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:01.034900    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 22:28:01.035703 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:01.035677    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 22:28:01.035969 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:01.035952    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:28:01.036048 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:01.036032    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v4p4m\""
Sep 17 22:28:01.040351 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:01.040330    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-2tf77"]
Sep 17 22:28:01.135046 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:01.135009    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f6e235a-5213-42cd-9cf4-b82db69c0a30-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-2tf77\" (UID: \"4f6e235a-5213-42cd-9cf4-b82db69c0a30\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-2tf77"
Sep 17 22:28:01.135238 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:01.135057    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f6e235a-5213-42cd-9cf4-b82db69c0a30-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-2tf77\" (UID: \"4f6e235a-5213-42cd-9cf4-b82db69c0a30\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-2tf77"
Sep 17 22:28:01.135238 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:01.135123    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s74pg\" (UniqueName: \"kubernetes.io/projected/4f6e235a-5213-42cd-9cf4-b82db69c0a30-kube-api-access-s74pg\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-2tf77\" (UID: \"4f6e235a-5213-42cd-9cf4-b82db69c0a30\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-2tf77"
Sep 17 22:28:01.135238 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:01.135211    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f6e235a-5213-42cd-9cf4-b82db69c0a30-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-2tf77\" (UID: \"4f6e235a-5213-42cd-9cf4-b82db69c0a30\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-2tf77"
Sep 17 22:28:01.135358 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:01.135257    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f6e235a-5213-42cd-9cf4-b82db69c0a30-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-2tf77\" (UID: \"4f6e235a-5213-42cd-9cf4-b82db69c0a30\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-2tf77"
Sep 17 22:28:01.236719 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:01.236616    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f6e235a-5213-42cd-9cf4-b82db69c0a30-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-2tf77\" (UID: \"4f6e235a-5213-42cd-9cf4-b82db69c0a30\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-2tf77"
Sep 17 22:28:01.236719 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:01.236676    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f6e235a-5213-42cd-9cf4-b82db69c0a30-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-2tf77\" (UID: \"4f6e235a-5213-42cd-9cf4-b82db69c0a30\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-2tf77"
Sep 17 22:28:01.236719 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:01.236701    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f6e235a-5213-42cd-9cf4-b82db69c0a30-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-2tf77\" (UID: \"4f6e235a-5213-42cd-9cf4-b82db69c0a30\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-2tf77"
Sep 17 22:28:01.236719 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:01.236730    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f6e235a-5213-42cd-9cf4-b82db69c0a30-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-2tf77\" (UID: \"4f6e235a-5213-42cd-9cf4-b82db69c0a30\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-2tf77"
Sep 17 22:28:01.237096 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:01.236751    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s74pg\" (UniqueName: \"kubernetes.io/projected/4f6e235a-5213-42cd-9cf4-b82db69c0a30-kube-api-access-s74pg\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-2tf77\" (UID: \"4f6e235a-5213-42cd-9cf4-b82db69c0a30\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-2tf77"
Sep 17 22:28:01.237096 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:01.237038    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f6e235a-5213-42cd-9cf4-b82db69c0a30-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-2tf77\" (UID: \"4f6e235a-5213-42cd-9cf4-b82db69c0a30\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-2tf77"
Sep 17 22:28:01.237329 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:01.237309    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f6e235a-5213-42cd-9cf4-b82db69c0a30-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-2tf77\" (UID: \"4f6e235a-5213-42cd-9cf4-b82db69c0a30\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-2tf77"
Sep 17 22:28:01.237674 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:01.237653    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f6e235a-5213-42cd-9cf4-b82db69c0a30-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-2tf77\" (UID: \"4f6e235a-5213-42cd-9cf4-b82db69c0a30\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-2tf77"
Sep 17 22:28:01.239306 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:01.239288    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f6e235a-5213-42cd-9cf4-b82db69c0a30-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-2tf77\" (UID: \"4f6e235a-5213-42cd-9cf4-b82db69c0a30\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-2tf77"
Sep 17 22:28:01.244953 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:01.244929    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s74pg\" (UniqueName: \"kubernetes.io/projected/4f6e235a-5213-42cd-9cf4-b82db69c0a30-kube-api-access-s74pg\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-2tf77\" (UID: \"4f6e235a-5213-42cd-9cf4-b82db69c0a30\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-2tf77"
Sep 17 22:28:01.342757 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:01.342712    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-2tf77"
Sep 17 22:28:01.618076 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:01.618050    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-2tf77"]
Sep 17 22:28:01.620250 ip-10-0-138-45 kubenswrapper[2563]: W0917 22:28:01.620220    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f6e235a_5213_42cd_9cf4_b82db69c0a30.slice/crio-6b4b346f90f3cdde1011b6fcc96b47add1fac0a6ea6567ee8ca8014f20a163de WatchSource:0}: Error finding container 6b4b346f90f3cdde1011b6fcc96b47add1fac0a6ea6567ee8ca8014f20a163de: Status 404 returned error can't find the container with id 6b4b346f90f3cdde1011b6fcc96b47add1fac0a6ea6567ee8ca8014f20a163de
Sep 17 22:28:02.537666 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:02.537628    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-2tf77" event={"ID":"4f6e235a-5213-42cd-9cf4-b82db69c0a30","Type":"ContainerStarted","Data":"6a4c49e28ba78a5b6a067f75976db32a5b4d9c260a213ffd4c564be0fa6621ee"}
Sep 17 22:28:02.538040 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:02.537727    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-2tf77" event={"ID":"4f6e235a-5213-42cd-9cf4-b82db69c0a30","Type":"ContainerStarted","Data":"6b4b346f90f3cdde1011b6fcc96b47add1fac0a6ea6567ee8ca8014f20a163de"}
Sep 17 22:28:05.551377 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:05.551346    2563 generic.go:356] "Generic (PLEG): container finished" podID="4f6e235a-5213-42cd-9cf4-b82db69c0a30" containerID="6a4c49e28ba78a5b6a067f75976db32a5b4d9c260a213ffd4c564be0fa6621ee" exitCode=0
Sep 17 22:28:05.551709 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:05.551382    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-2tf77" event={"ID":"4f6e235a-5213-42cd-9cf4-b82db69c0a30","Type":"ContainerDied","Data":"6a4c49e28ba78a5b6a067f75976db32a5b4d9c260a213ffd4c564be0fa6621ee"}
Sep 17 22:28:06.558396 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:06.558360    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-2tf77" event={"ID":"4f6e235a-5213-42cd-9cf4-b82db69c0a30","Type":"ContainerStarted","Data":"8e711c318b71906e4c14cdba8ae93185beae3a7e863c89fdfb1796667441a2b2"}
Sep 17 22:28:06.558858 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:06.558433    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-2tf77" event={"ID":"4f6e235a-5213-42cd-9cf4-b82db69c0a30","Type":"ContainerStarted","Data":"ac8d373473a364a3496bb94c9661e16a6af1cb22253d18f8b96666acd0a3c759"}
Sep 17 22:28:06.558858 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:06.558659    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-2tf77"
Sep 17 22:28:06.578733 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:06.578684    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-2tf77" podStartSLOduration=1.845878704 podStartE2EDuration="5.5786689s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.732790196s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:28:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:28:06.576422975 +0000 UTC m=+1797.744375959" watchObservedRunningTime="2026-09-17 22:28:06.5786689 +0000 UTC m=+1797.746621927"
Sep 17 22:28:07.562529 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:07.562497    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-2tf77"
Sep 17 22:28:07.563709 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:07.563679    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-2tf77" podUID="4f6e235a-5213-42cd-9cf4-b82db69c0a30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 22:28:08.566490 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:08.566449    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-2tf77" podUID="4f6e235a-5213-42cd-9cf4-b82db69c0a30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 22:28:13.570945 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:13.570911    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-2tf77"
Sep 17 22:28:13.571410 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:13.571384    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-2tf77" podUID="4f6e235a-5213-42cd-9cf4-b82db69c0a30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 22:28:23.571420 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:23.571381    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-2tf77" podUID="4f6e235a-5213-42cd-9cf4-b82db69c0a30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 22:28:33.571583 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:33.571543    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-2tf77" podUID="4f6e235a-5213-42cd-9cf4-b82db69c0a30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 22:28:43.571387 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:43.571352    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-2tf77" podUID="4f6e235a-5213-42cd-9cf4-b82db69c0a30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 22:28:49.605908 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:49.605880    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nrkd5_34b8ef2b-97ba-4562-87ee-8da0cb4604c7/ovn-acl-logging/0.log"
Sep 17 22:28:49.606840 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:49.606818    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nrkd5_34b8ef2b-97ba-4562-87ee-8da0cb4604c7/ovn-acl-logging/0.log"
Sep 17 22:28:53.571781 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:28:53.571741    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-2tf77" podUID="4f6e235a-5213-42cd-9cf4-b82db69c0a30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 22:29:03.572172 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:03.572118    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-2tf77" podUID="4f6e235a-5213-42cd-9cf4-b82db69c0a30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 22:29:13.571876 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:13.571827    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-2tf77" podUID="4f6e235a-5213-42cd-9cf4-b82db69c0a30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 22:29:20.913094 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:20.913067    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-2tf77"
Sep 17 22:29:31.212246 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:31.212162    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-2tf77"]
Sep 17 22:29:31.212696 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:31.212464    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-2tf77" podUID="4f6e235a-5213-42cd-9cf4-b82db69c0a30" containerName="kserve-container" containerID="cri-o://ac8d373473a364a3496bb94c9661e16a6af1cb22253d18f8b96666acd0a3c759" gracePeriod=30
Sep 17 22:29:31.212696 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:31.212506    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-2tf77" podUID="4f6e235a-5213-42cd-9cf4-b82db69c0a30" containerName="kube-rbac-proxy" containerID="cri-o://8e711c318b71906e4c14cdba8ae93185beae3a7e863c89fdfb1796667441a2b2" gracePeriod=30
Sep 17 22:29:31.906157 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:31.906119    2563 generic.go:356] "Generic (PLEG): container finished" podID="4f6e235a-5213-42cd-9cf4-b82db69c0a30" containerID="8e711c318b71906e4c14cdba8ae93185beae3a7e863c89fdfb1796667441a2b2" exitCode=2
Sep 17 22:29:31.906328 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:31.906194    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-2tf77" event={"ID":"4f6e235a-5213-42cd-9cf4-b82db69c0a30","Type":"ContainerDied","Data":"8e711c318b71906e4c14cdba8ae93185beae3a7e863c89fdfb1796667441a2b2"}
Sep 17 22:29:33.567430 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:33.567392    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-2tf77" podUID="4f6e235a-5213-42cd-9cf4-b82db69c0a30" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
Sep 17 22:29:36.039269 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.039247    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-2tf77"
Sep 17 22:29:36.109135 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.109100    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s74pg\" (UniqueName: \"kubernetes.io/projected/4f6e235a-5213-42cd-9cf4-b82db69c0a30-kube-api-access-s74pg\") pod \"4f6e235a-5213-42cd-9cf4-b82db69c0a30\" (UID: \"4f6e235a-5213-42cd-9cf4-b82db69c0a30\") "
Sep 17 22:29:36.109340 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.109169    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f6e235a-5213-42cd-9cf4-b82db69c0a30-kserve-provision-location\") pod \"4f6e235a-5213-42cd-9cf4-b82db69c0a30\" (UID: \"4f6e235a-5213-42cd-9cf4-b82db69c0a30\") "
Sep 17 22:29:36.109340 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.109269    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f6e235a-5213-42cd-9cf4-b82db69c0a30-kube-rbac-proxy-sar-config\") pod \"4f6e235a-5213-42cd-9cf4-b82db69c0a30\" (UID: \"4f6e235a-5213-42cd-9cf4-b82db69c0a30\") "
Sep 17 22:29:36.109340 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.109306    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f6e235a-5213-42cd-9cf4-b82db69c0a30-proxy-tls\") pod \"4f6e235a-5213-42cd-9cf4-b82db69c0a30\" (UID: \"4f6e235a-5213-42cd-9cf4-b82db69c0a30\") "
Sep 17 22:29:36.109524 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.109363    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f6e235a-5213-42cd-9cf4-b82db69c0a30-cabundle-cert\") pod \"4f6e235a-5213-42cd-9cf4-b82db69c0a30\" (UID: \"4f6e235a-5213-42cd-9cf4-b82db69c0a30\") "
Sep 17 22:29:36.109524 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.109484    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f6e235a-5213-42cd-9cf4-b82db69c0a30-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4f6e235a-5213-42cd-9cf4-b82db69c0a30" (UID: "4f6e235a-5213-42cd-9cf4-b82db69c0a30"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:29:36.109598 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.109579    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f6e235a-5213-42cd-9cf4-b82db69c0a30-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4f6e235a-5213-42cd-9cf4-b82db69c0a30" (UID: "4f6e235a-5213-42cd-9cf4-b82db69c0a30"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:29:36.109649 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.109605    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f6e235a-5213-42cd-9cf4-b82db69c0a30-kserve-provision-location\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:29:36.109758 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.109732    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f6e235a-5213-42cd-9cf4-b82db69c0a30-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4f6e235a-5213-42cd-9cf4-b82db69c0a30" (UID: "4f6e235a-5213-42cd-9cf4-b82db69c0a30"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:29:36.111328 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.111276    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f6e235a-5213-42cd-9cf4-b82db69c0a30-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4f6e235a-5213-42cd-9cf4-b82db69c0a30" (UID: "4f6e235a-5213-42cd-9cf4-b82db69c0a30"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:29:36.111328 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.111293    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f6e235a-5213-42cd-9cf4-b82db69c0a30-kube-api-access-s74pg" (OuterVolumeSpecName: "kube-api-access-s74pg") pod "4f6e235a-5213-42cd-9cf4-b82db69c0a30" (UID: "4f6e235a-5213-42cd-9cf4-b82db69c0a30"). InnerVolumeSpecName "kube-api-access-s74pg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:29:36.210643 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.210609    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f6e235a-5213-42cd-9cf4-b82db69c0a30-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:29:36.210643 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.210636    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f6e235a-5213-42cd-9cf4-b82db69c0a30-proxy-tls\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:29:36.210643 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.210646    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f6e235a-5213-42cd-9cf4-b82db69c0a30-cabundle-cert\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:29:36.210643 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.210654    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s74pg\" (UniqueName: \"kubernetes.io/projected/4f6e235a-5213-42cd-9cf4-b82db69c0a30-kube-api-access-s74pg\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:29:36.390158 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.390061    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-vx4g7"]
Sep 17 22:29:36.390386 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.390372    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f6e235a-5213-42cd-9cf4-b82db69c0a30" containerName="kserve-container"
Sep 17 22:29:36.390427 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.390388    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f6e235a-5213-42cd-9cf4-b82db69c0a30" containerName="kserve-container"
Sep 17 22:29:36.390427 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.390396    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f6e235a-5213-42cd-9cf4-b82db69c0a30" containerName="storage-initializer"
Sep 17 22:29:36.390427 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.390402    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f6e235a-5213-42cd-9cf4-b82db69c0a30" containerName="storage-initializer"
Sep 17 22:29:36.390427 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.390415    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f6e235a-5213-42cd-9cf4-b82db69c0a30" containerName="kube-rbac-proxy"
Sep 17 22:29:36.390427 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.390420    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f6e235a-5213-42cd-9cf4-b82db69c0a30" containerName="kube-rbac-proxy"
Sep 17 22:29:36.390573 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.390464    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f6e235a-5213-42cd-9cf4-b82db69c0a30" containerName="kserve-container"
Sep 17 22:29:36.390573 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.390471    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f6e235a-5213-42cd-9cf4-b82db69c0a30" containerName="kube-rbac-proxy"
Sep 17 22:29:36.393590 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.393572    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-vx4g7"
Sep 17 22:29:36.395647 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.395632    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Sep 17 22:29:36.395768 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.395751    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 17 22:29:36.403584 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.403562    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-vx4g7"]
Sep 17 22:29:36.443213 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.443184    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-jmrl6"]
Sep 17 22:29:36.446385 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.446369    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-jmrl6"
Sep 17 22:29:36.448371 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.448355    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 17 22:29:36.458033 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.458009    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-jmrl6"]
Sep 17 22:29:36.513369 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.513329    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/689c1955-688f-4f55-a8e6-822ac7b779b8-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-76cbc5464-jmrl6\" (UID: \"689c1955-688f-4f55-a8e6-822ac7b779b8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-jmrl6"
Sep 17 22:29:36.513531 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.513378    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/364afdeb-f401-4a87-956c-fa21679ae4e7-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-78968789f7-vx4g7\" (UID: \"364afdeb-f401-4a87-956c-fa21679ae4e7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-vx4g7"
Sep 17 22:29:36.513531 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.513408    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqhg4\" (UniqueName: \"kubernetes.io/projected/689c1955-688f-4f55-a8e6-822ac7b779b8-kube-api-access-tqhg4\") pod \"isvc-canary-rollback-predictor-76cbc5464-jmrl6\" (UID: \"689c1955-688f-4f55-a8e6-822ac7b779b8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-jmrl6"
Sep 17 22:29:36.513531 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.513434    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/364afdeb-f401-4a87-956c-fa21679ae4e7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-78968789f7-vx4g7\" (UID: \"364afdeb-f401-4a87-956c-fa21679ae4e7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-vx4g7"
Sep 17 22:29:36.513531 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.513464    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/689c1955-688f-4f55-a8e6-822ac7b779b8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-76cbc5464-jmrl6\" (UID: \"689c1955-688f-4f55-a8e6-822ac7b779b8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-jmrl6"
Sep 17 22:29:36.513673 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.513532    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/689c1955-688f-4f55-a8e6-822ac7b779b8-proxy-tls\") pod \"isvc-canary-rollback-predictor-76cbc5464-jmrl6\" (UID: \"689c1955-688f-4f55-a8e6-822ac7b779b8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-jmrl6"
Sep 17 22:29:36.513673 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.513596    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/689c1955-688f-4f55-a8e6-822ac7b779b8-cabundle-cert\") pod \"isvc-canary-rollback-predictor-76cbc5464-jmrl6\" (UID: \"689c1955-688f-4f55-a8e6-822ac7b779b8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-jmrl6"
Sep 17 22:29:36.513673 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.513625    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2nlf\" (UniqueName: \"kubernetes.io/projected/364afdeb-f401-4a87-956c-fa21679ae4e7-kube-api-access-g2nlf\") pod \"isvc-canary-rollback-v2-predictor-78968789f7-vx4g7\" (UID: \"364afdeb-f401-4a87-956c-fa21679ae4e7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-vx4g7"
Sep 17 22:29:36.513673 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.513652    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/364afdeb-f401-4a87-956c-fa21679ae4e7-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-78968789f7-vx4g7\" (UID: \"364afdeb-f401-4a87-956c-fa21679ae4e7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-vx4g7"
Sep 17 22:29:36.513673 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.513667    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/364afdeb-f401-4a87-956c-fa21679ae4e7-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-78968789f7-vx4g7\" (UID: \"364afdeb-f401-4a87-956c-fa21679ae4e7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-vx4g7"
Sep 17 22:29:36.614801 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.614751    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/689c1955-688f-4f55-a8e6-822ac7b779b8-cabundle-cert\") pod \"isvc-canary-rollback-predictor-76cbc5464-jmrl6\" (UID: \"689c1955-688f-4f55-a8e6-822ac7b779b8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-jmrl6"
Sep 17 22:29:36.614801 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.614812    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2nlf\" (UniqueName: \"kubernetes.io/projected/364afdeb-f401-4a87-956c-fa21679ae4e7-kube-api-access-g2nlf\") pod \"isvc-canary-rollback-v2-predictor-78968789f7-vx4g7\" (UID: \"364afdeb-f401-4a87-956c-fa21679ae4e7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-vx4g7"
Sep 17 22:29:36.615060 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.614856    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/364afdeb-f401-4a87-956c-fa21679ae4e7-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-78968789f7-vx4g7\" (UID: \"364afdeb-f401-4a87-956c-fa21679ae4e7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-vx4g7"
Sep 17 22:29:36.615060 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.614885    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/364afdeb-f401-4a87-956c-fa21679ae4e7-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-78968789f7-vx4g7\" (UID: \"364afdeb-f401-4a87-956c-fa21679ae4e7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-vx4g7"
Sep 17 22:29:36.615060 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.614913    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/689c1955-688f-4f55-a8e6-822ac7b779b8-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-76cbc5464-jmrl6\" (UID: \"689c1955-688f-4f55-a8e6-822ac7b779b8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-jmrl6"
Sep 17 22:29:36.615060 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.614928    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/364afdeb-f401-4a87-956c-fa21679ae4e7-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-78968789f7-vx4g7\" (UID: \"364afdeb-f401-4a87-956c-fa21679ae4e7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-vx4g7"
Sep 17 22:29:36.615060 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.614946    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqhg4\" (UniqueName: \"kubernetes.io/projected/689c1955-688f-4f55-a8e6-822ac7b779b8-kube-api-access-tqhg4\") pod \"isvc-canary-rollback-predictor-76cbc5464-jmrl6\" (UID: \"689c1955-688f-4f55-a8e6-822ac7b779b8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-jmrl6"
Sep 17 22:29:36.615060 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.614980    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/364afdeb-f401-4a87-956c-fa21679ae4e7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-78968789f7-vx4g7\" (UID: \"364afdeb-f401-4a87-956c-fa21679ae4e7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-vx4g7"
Sep 17 22:29:36.615060 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.615026    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/689c1955-688f-4f55-a8e6-822ac7b779b8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-76cbc5464-jmrl6\" (UID: \"689c1955-688f-4f55-a8e6-822ac7b779b8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-jmrl6"
Sep 17 22:29:36.615060 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.615053    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/689c1955-688f-4f55-a8e6-822ac7b779b8-proxy-tls\") pod \"isvc-canary-rollback-predictor-76cbc5464-jmrl6\" (UID: \"689c1955-688f-4f55-a8e6-822ac7b779b8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-jmrl6"
Sep 17 22:29:36.615470 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:29:36.615110    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-serving-cert: secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Sep 17 22:29:36.615470 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:29:36.615205    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/364afdeb-f401-4a87-956c-fa21679ae4e7-proxy-tls podName:364afdeb-f401-4a87-956c-fa21679ae4e7 nodeName:}" failed. No retries permitted until 2026-09-17 22:29:37.115182813 +0000 UTC m=+1888.283135787 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/364afdeb-f401-4a87-956c-fa21679ae4e7-proxy-tls") pod "isvc-canary-rollback-v2-predictor-78968789f7-vx4g7" (UID: "364afdeb-f401-4a87-956c-fa21679ae4e7") : secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Sep 17 22:29:36.615571 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.615503    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/364afdeb-f401-4a87-956c-fa21679ae4e7-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-78968789f7-vx4g7\" (UID: \"364afdeb-f401-4a87-956c-fa21679ae4e7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-vx4g7"
Sep 17 22:29:36.615571 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.615513    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/689c1955-688f-4f55-a8e6-822ac7b779b8-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-76cbc5464-jmrl6\" (UID: \"689c1955-688f-4f55-a8e6-822ac7b779b8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-jmrl6"
Sep 17 22:29:36.615794 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.615771    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/364afdeb-f401-4a87-956c-fa21679ae4e7-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-78968789f7-vx4g7\" (UID: \"364afdeb-f401-4a87-956c-fa21679ae4e7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-vx4g7"
Sep 17 22:29:36.615877 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.615856    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/364afdeb-f401-4a87-956c-fa21679ae4e7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-78968789f7-vx4g7\" (UID: \"364afdeb-f401-4a87-956c-fa21679ae4e7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-vx4g7"
Sep 17 22:29:36.615919 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.615881    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/689c1955-688f-4f55-a8e6-822ac7b779b8-cabundle-cert\") pod \"isvc-canary-rollback-predictor-76cbc5464-jmrl6\" (UID: \"689c1955-688f-4f55-a8e6-822ac7b779b8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-jmrl6"
Sep 17 22:29:36.615919 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.615906    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/689c1955-688f-4f55-a8e6-822ac7b779b8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-76cbc5464-jmrl6\" (UID: \"689c1955-688f-4f55-a8e6-822ac7b779b8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-jmrl6"
Sep 17 22:29:36.617586 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.617569    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/689c1955-688f-4f55-a8e6-822ac7b779b8-proxy-tls\") pod \"isvc-canary-rollback-predictor-76cbc5464-jmrl6\" (UID: \"689c1955-688f-4f55-a8e6-822ac7b779b8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-jmrl6"
Sep 17 22:29:36.622775 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.622754    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2nlf\" (UniqueName: \"kubernetes.io/projected/364afdeb-f401-4a87-956c-fa21679ae4e7-kube-api-access-g2nlf\") pod \"isvc-canary-rollback-v2-predictor-78968789f7-vx4g7\" (UID: \"364afdeb-f401-4a87-956c-fa21679ae4e7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-vx4g7"
Sep 17 22:29:36.622975 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.622956    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqhg4\" (UniqueName: \"kubernetes.io/projected/689c1955-688f-4f55-a8e6-822ac7b779b8-kube-api-access-tqhg4\") pod \"isvc-canary-rollback-predictor-76cbc5464-jmrl6\" (UID: \"689c1955-688f-4f55-a8e6-822ac7b779b8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-jmrl6"
Sep 17 22:29:36.756117 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.756087    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-jmrl6"
Sep 17 22:29:36.928497 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.928460    2563 generic.go:356] "Generic (PLEG): container finished" podID="4f6e235a-5213-42cd-9cf4-b82db69c0a30" containerID="ac8d373473a364a3496bb94c9661e16a6af1cb22253d18f8b96666acd0a3c759" exitCode=0
Sep 17 22:29:36.928633 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.928559    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-2tf77"
Sep 17 22:29:36.928633 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.928555    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-2tf77" event={"ID":"4f6e235a-5213-42cd-9cf4-b82db69c0a30","Type":"ContainerDied","Data":"ac8d373473a364a3496bb94c9661e16a6af1cb22253d18f8b96666acd0a3c759"}
Sep 17 22:29:36.928633 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.928628    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-2tf77" event={"ID":"4f6e235a-5213-42cd-9cf4-b82db69c0a30","Type":"ContainerDied","Data":"6b4b346f90f3cdde1011b6fcc96b47add1fac0a6ea6567ee8ca8014f20a163de"}
Sep 17 22:29:36.928742 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.928666    2563 scope.go:117] "RemoveContainer" containerID="8e711c318b71906e4c14cdba8ae93185beae3a7e863c89fdfb1796667441a2b2"
Sep 17 22:29:36.936590 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.936573    2563 scope.go:117] "RemoveContainer" containerID="ac8d373473a364a3496bb94c9661e16a6af1cb22253d18f8b96666acd0a3c759"
Sep 17 22:29:36.943969 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.943950    2563 scope.go:117] "RemoveContainer" containerID="6a4c49e28ba78a5b6a067f75976db32a5b4d9c260a213ffd4c564be0fa6621ee"
Sep 17 22:29:36.944329 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.944308    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-2tf77"]
Sep 17 22:29:36.947815 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.947794    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-2tf77"]
Sep 17 22:29:36.951234 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.951220    2563 scope.go:117] "RemoveContainer" containerID="8e711c318b71906e4c14cdba8ae93185beae3a7e863c89fdfb1796667441a2b2"
Sep 17 22:29:36.951483 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:29:36.951466    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e711c318b71906e4c14cdba8ae93185beae3a7e863c89fdfb1796667441a2b2\": container with ID starting with 8e711c318b71906e4c14cdba8ae93185beae3a7e863c89fdfb1796667441a2b2 not found: ID does not exist" containerID="8e711c318b71906e4c14cdba8ae93185beae3a7e863c89fdfb1796667441a2b2"
Sep 17 22:29:36.951531 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.951490    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e711c318b71906e4c14cdba8ae93185beae3a7e863c89fdfb1796667441a2b2"} err="failed to get container status \"8e711c318b71906e4c14cdba8ae93185beae3a7e863c89fdfb1796667441a2b2\": rpc error: code = NotFound desc = could not find container \"8e711c318b71906e4c14cdba8ae93185beae3a7e863c89fdfb1796667441a2b2\": container with ID starting with 8e711c318b71906e4c14cdba8ae93185beae3a7e863c89fdfb1796667441a2b2 not found: ID does not exist"
Sep 17 22:29:36.951531 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.951506    2563 scope.go:117] "RemoveContainer" containerID="ac8d373473a364a3496bb94c9661e16a6af1cb22253d18f8b96666acd0a3c759"
Sep 17 22:29:36.951718 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:29:36.951702    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac8d373473a364a3496bb94c9661e16a6af1cb22253d18f8b96666acd0a3c759\": container with ID starting with ac8d373473a364a3496bb94c9661e16a6af1cb22253d18f8b96666acd0a3c759 not found: ID does not exist" containerID="ac8d373473a364a3496bb94c9661e16a6af1cb22253d18f8b96666acd0a3c759"
Sep 17 22:29:36.951756 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.951721    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac8d373473a364a3496bb94c9661e16a6af1cb22253d18f8b96666acd0a3c759"} err="failed to get container status \"ac8d373473a364a3496bb94c9661e16a6af1cb22253d18f8b96666acd0a3c759\": rpc error: code = NotFound desc = could not find container \"ac8d373473a364a3496bb94c9661e16a6af1cb22253d18f8b96666acd0a3c759\": container with ID starting with ac8d373473a364a3496bb94c9661e16a6af1cb22253d18f8b96666acd0a3c759 not found: ID does not exist"
Sep 17 22:29:36.951756 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.951733    2563 scope.go:117] "RemoveContainer" containerID="6a4c49e28ba78a5b6a067f75976db32a5b4d9c260a213ffd4c564be0fa6621ee"
Sep 17 22:29:36.951912 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:29:36.951898    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a4c49e28ba78a5b6a067f75976db32a5b4d9c260a213ffd4c564be0fa6621ee\": container with ID starting with 6a4c49e28ba78a5b6a067f75976db32a5b4d9c260a213ffd4c564be0fa6621ee not found: ID does not exist" containerID="6a4c49e28ba78a5b6a067f75976db32a5b4d9c260a213ffd4c564be0fa6621ee"
Sep 17 22:29:36.951955 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:36.951915    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a4c49e28ba78a5b6a067f75976db32a5b4d9c260a213ffd4c564be0fa6621ee"} err="failed to get container status \"6a4c49e28ba78a5b6a067f75976db32a5b4d9c260a213ffd4c564be0fa6621ee\": rpc error: code = NotFound desc = could not find container \"6a4c49e28ba78a5b6a067f75976db32a5b4d9c260a213ffd4c564be0fa6621ee\": container with ID starting with 6a4c49e28ba78a5b6a067f75976db32a5b4d9c260a213ffd4c564be0fa6621ee not found: ID does not exist"
Sep 17 22:29:37.025259 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:37.025236    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-jmrl6"]
Sep 17 22:29:37.027421 ip-10-0-138-45 kubenswrapper[2563]: W0917 22:29:37.027394    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod689c1955_688f_4f55_a8e6_822ac7b779b8.slice/crio-5cc1bae6194084bfed49ac34ba467a94548cf559cddd2d35cfd0477fd87384f2 WatchSource:0}: Error finding container 5cc1bae6194084bfed49ac34ba467a94548cf559cddd2d35cfd0477fd87384f2: Status 404 returned error can't find the container with id 5cc1bae6194084bfed49ac34ba467a94548cf559cddd2d35cfd0477fd87384f2
Sep 17 22:29:37.118500 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:37.118462    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/364afdeb-f401-4a87-956c-fa21679ae4e7-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-78968789f7-vx4g7\" (UID: \"364afdeb-f401-4a87-956c-fa21679ae4e7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-vx4g7"
Sep 17 22:29:37.120855 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:37.120832    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/364afdeb-f401-4a87-956c-fa21679ae4e7-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-78968789f7-vx4g7\" (UID: \"364afdeb-f401-4a87-956c-fa21679ae4e7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-vx4g7"
Sep 17 22:29:37.303816 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:37.303737    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-vx4g7"
Sep 17 22:29:37.575746 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:37.575659    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-vx4g7"]
Sep 17 22:29:37.579277 ip-10-0-138-45 kubenswrapper[2563]: W0917 22:29:37.579248    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod364afdeb_f401_4a87_956c_fa21679ae4e7.slice/crio-c81bff994360bc7969960482bf3d81d14998e8e21e181632b79496507bcb8519 WatchSource:0}: Error finding container c81bff994360bc7969960482bf3d81d14998e8e21e181632b79496507bcb8519: Status 404 returned error can't find the container with id c81bff994360bc7969960482bf3d81d14998e8e21e181632b79496507bcb8519
Sep 17 22:29:37.936019 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:37.935976    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-vx4g7" event={"ID":"364afdeb-f401-4a87-956c-fa21679ae4e7","Type":"ContainerStarted","Data":"26c2d4a8d996c407cc70da2a6d1e84f88ad3031fe0a91e459fb573bf28923544"}
Sep 17 22:29:37.936223 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:37.936050    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-vx4g7" event={"ID":"364afdeb-f401-4a87-956c-fa21679ae4e7","Type":"ContainerStarted","Data":"c81bff994360bc7969960482bf3d81d14998e8e21e181632b79496507bcb8519"}
Sep 17 22:29:37.937489 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:37.937466    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-jmrl6" event={"ID":"689c1955-688f-4f55-a8e6-822ac7b779b8","Type":"ContainerStarted","Data":"ed8465235da98e94991a2a0daf3623fd666ddfbcf1be8aa369cdde44614758f9"}
Sep 17 22:29:37.937593 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:37.937507    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-jmrl6" event={"ID":"689c1955-688f-4f55-a8e6-822ac7b779b8","Type":"ContainerStarted","Data":"5cc1bae6194084bfed49ac34ba467a94548cf559cddd2d35cfd0477fd87384f2"}
Sep 17 22:29:38.914011 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:38.913968    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f6e235a-5213-42cd-9cf4-b82db69c0a30" path="/var/lib/kubelet/pods/4f6e235a-5213-42cd-9cf4-b82db69c0a30/volumes"
Sep 17 22:29:40.952106 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:40.952071    2563 generic.go:356] "Generic (PLEG): container finished" podID="689c1955-688f-4f55-a8e6-822ac7b779b8" containerID="ed8465235da98e94991a2a0daf3623fd666ddfbcf1be8aa369cdde44614758f9" exitCode=0
Sep 17 22:29:40.952525 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:40.952124    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-jmrl6" event={"ID":"689c1955-688f-4f55-a8e6-822ac7b779b8","Type":"ContainerDied","Data":"ed8465235da98e94991a2a0daf3623fd666ddfbcf1be8aa369cdde44614758f9"}
Sep 17 22:29:41.959557 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:41.959518    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-jmrl6" event={"ID":"689c1955-688f-4f55-a8e6-822ac7b779b8","Type":"ContainerStarted","Data":"ede27372fe035b4859582536e4edf4533196afe709a8cf673687e3401eae32a8"}
Sep 17 22:29:41.959984 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:41.959591    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-jmrl6" event={"ID":"689c1955-688f-4f55-a8e6-822ac7b779b8","Type":"ContainerStarted","Data":"1f6d62480def39ed2e9e0486365cb2912101b3ac018eb0cbb73dac69a23eea36"}
Sep 17 22:29:41.959984 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:41.959813    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-jmrl6"
Sep 17 22:29:41.961229 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:41.961204    2563 generic.go:356] "Generic (PLEG): container finished" podID="364afdeb-f401-4a87-956c-fa21679ae4e7" containerID="26c2d4a8d996c407cc70da2a6d1e84f88ad3031fe0a91e459fb573bf28923544" exitCode=0
Sep 17 22:29:41.961332 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:41.961234    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-vx4g7" event={"ID":"364afdeb-f401-4a87-956c-fa21679ae4e7","Type":"ContainerDied","Data":"26c2d4a8d996c407cc70da2a6d1e84f88ad3031fe0a91e459fb573bf28923544"}
Sep 17 22:29:41.979731 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:41.979678    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-jmrl6" podStartSLOduration=2.29681417 podStartE2EDuration="5.979659293s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.682845123s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:29:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:29:41.978359134 +0000 UTC m=+1893.146312138" watchObservedRunningTime="2026-09-17 22:29:41.979659293 +0000 UTC m=+1893.147612277"
Sep 17 22:29:42.967206 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:42.967170    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-vx4g7" event={"ID":"364afdeb-f401-4a87-956c-fa21679ae4e7","Type":"ContainerStarted","Data":"66bec810c8716459b8031fac8261b5f541942047bcbb4729c91c29a18c9fa375"}
Sep 17 22:29:42.967576 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:42.967231    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-vx4g7" event={"ID":"364afdeb-f401-4a87-956c-fa21679ae4e7","Type":"ContainerStarted","Data":"60704a27e7138bb309b80e2955ac3627c14de6adddb66d5638ee033755b84dd0"}
Sep 17 22:29:42.967576 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:42.967481    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-jmrl6"
Sep 17 22:29:42.967685 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:42.967599    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-vx4g7"
Sep 17 22:29:42.968893 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:42.968864    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-jmrl6" podUID="689c1955-688f-4f55-a8e6-822ac7b779b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 17 22:29:42.989838 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:42.989794    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-vx4g7" podStartSLOduration=3.031865912 podStartE2EDuration="6.98977933s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.957913418s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:29:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:29:42.988256735 +0000 UTC m=+1894.156209718" watchObservedRunningTime="2026-09-17 22:29:42.98977933 +0000 UTC m=+1894.157732312"
Sep 17 22:29:43.971375 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:43.971332    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-jmrl6" podUID="689c1955-688f-4f55-a8e6-822ac7b779b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 17 22:29:43.971780 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:43.971505    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-vx4g7"
Sep 17 22:29:43.972540 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:43.972494    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-vx4g7" podUID="364afdeb-f401-4a87-956c-fa21679ae4e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 22:29:44.977621 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:44.977583    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-vx4g7" podUID="364afdeb-f401-4a87-956c-fa21679ae4e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 22:29:48.975947 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:48.975919    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-jmrl6"
Sep 17 22:29:48.976602 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:48.976573    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-jmrl6" podUID="689c1955-688f-4f55-a8e6-822ac7b779b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 17 22:29:49.981881 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:49.981857    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-vx4g7"
Sep 17 22:29:49.982383 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:49.982358    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-vx4g7" podUID="364afdeb-f401-4a87-956c-fa21679ae4e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 22:29:58.976662 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:58.976623    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-jmrl6" podUID="689c1955-688f-4f55-a8e6-822ac7b779b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 17 22:29:59.983335 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:29:59.983295    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-vx4g7" podUID="364afdeb-f401-4a87-956c-fa21679ae4e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 22:30:08.977026 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:30:08.976985    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-jmrl6" podUID="689c1955-688f-4f55-a8e6-822ac7b779b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 17 22:30:09.983198 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:30:09.983133    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-vx4g7" podUID="364afdeb-f401-4a87-956c-fa21679ae4e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 22:30:18.976669 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:30:18.976628    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-jmrl6" podUID="689c1955-688f-4f55-a8e6-822ac7b779b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 17 22:30:19.982981 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:30:19.982939    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-vx4g7" podUID="364afdeb-f401-4a87-956c-fa21679ae4e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 22:30:28.976769 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:30:28.976719    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-jmrl6" podUID="689c1955-688f-4f55-a8e6-822ac7b779b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 17 22:30:29.982323 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:30:29.982283    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-vx4g7" podUID="364afdeb-f401-4a87-956c-fa21679ae4e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 22:30:38.976942 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:30:38.976901    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-jmrl6" podUID="689c1955-688f-4f55-a8e6-822ac7b779b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 17 22:30:39.983188 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:30:39.983134    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-vx4g7" podUID="364afdeb-f401-4a87-956c-fa21679ae4e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 22:30:48.976659 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:30:48.976621    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-jmrl6" podUID="689c1955-688f-4f55-a8e6-822ac7b779b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 17 22:30:49.982970 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:30:49.982932    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-vx4g7" podUID="364afdeb-f401-4a87-956c-fa21679ae4e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 22:30:56.913922 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:30:56.913839    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-vx4g7"
Sep 17 22:30:57.909316 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:30:57.909286    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-jmrl6"
Sep 17 22:31:06.367402 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:06.367365    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-vx4g7"]
Sep 17 22:31:06.367821 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:06.367591    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-vx4g7" podUID="364afdeb-f401-4a87-956c-fa21679ae4e7" containerName="kserve-container" containerID="cri-o://60704a27e7138bb309b80e2955ac3627c14de6adddb66d5638ee033755b84dd0" gracePeriod=30
Sep 17 22:31:06.367821 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:06.367603    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-vx4g7" podUID="364afdeb-f401-4a87-956c-fa21679ae4e7" containerName="kube-rbac-proxy" containerID="cri-o://66bec810c8716459b8031fac8261b5f541942047bcbb4729c91c29a18c9fa375" gracePeriod=30
Sep 17 22:31:06.910014 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:06.909973    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-vx4g7" podUID="364afdeb-f401-4a87-956c-fa21679ae4e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 22:31:07.315396 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:07.315316    2563 generic.go:356] "Generic (PLEG): container finished" podID="364afdeb-f401-4a87-956c-fa21679ae4e7" containerID="66bec810c8716459b8031fac8261b5f541942047bcbb4729c91c29a18c9fa375" exitCode=2
Sep 17 22:31:07.315531 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:07.315385    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-vx4g7" event={"ID":"364afdeb-f401-4a87-956c-fa21679ae4e7","Type":"ContainerDied","Data":"66bec810c8716459b8031fac8261b5f541942047bcbb4729c91c29a18c9fa375"}
Sep 17 22:31:09.978754 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:09.978714    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-vx4g7" podUID="364afdeb-f401-4a87-956c-fa21679ae4e7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.41:8643/healthz\": dial tcp 10.133.0.41:8643: connect: connection refused"
Sep 17 22:31:11.103270 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:11.103246    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-vx4g7"
Sep 17 22:31:11.216303 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:11.216224    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/364afdeb-f401-4a87-956c-fa21679ae4e7-cabundle-cert\") pod \"364afdeb-f401-4a87-956c-fa21679ae4e7\" (UID: \"364afdeb-f401-4a87-956c-fa21679ae4e7\") "
Sep 17 22:31:11.216303 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:11.216287    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g2nlf\" (UniqueName: \"kubernetes.io/projected/364afdeb-f401-4a87-956c-fa21679ae4e7-kube-api-access-g2nlf\") pod \"364afdeb-f401-4a87-956c-fa21679ae4e7\" (UID: \"364afdeb-f401-4a87-956c-fa21679ae4e7\") "
Sep 17 22:31:11.216540 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:11.216386    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/364afdeb-f401-4a87-956c-fa21679ae4e7-kserve-provision-location\") pod \"364afdeb-f401-4a87-956c-fa21679ae4e7\" (UID: \"364afdeb-f401-4a87-956c-fa21679ae4e7\") "
Sep 17 22:31:11.216540 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:11.216431    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/364afdeb-f401-4a87-956c-fa21679ae4e7-kube-rbac-proxy-sar-config\") pod \"364afdeb-f401-4a87-956c-fa21679ae4e7\" (UID: \"364afdeb-f401-4a87-956c-fa21679ae4e7\") "
Sep 17 22:31:11.216540 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:11.216474    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/364afdeb-f401-4a87-956c-fa21679ae4e7-proxy-tls\") pod \"364afdeb-f401-4a87-956c-fa21679ae4e7\" (UID: \"364afdeb-f401-4a87-956c-fa21679ae4e7\") "
Sep 17 22:31:11.216712 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:11.216640    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/364afdeb-f401-4a87-956c-fa21679ae4e7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "364afdeb-f401-4a87-956c-fa21679ae4e7" (UID: "364afdeb-f401-4a87-956c-fa21679ae4e7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:31:11.216771 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:11.216739    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/364afdeb-f401-4a87-956c-fa21679ae4e7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "364afdeb-f401-4a87-956c-fa21679ae4e7" (UID: "364afdeb-f401-4a87-956c-fa21679ae4e7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:31:11.216827 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:11.216788    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/364afdeb-f401-4a87-956c-fa21679ae4e7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "364afdeb-f401-4a87-956c-fa21679ae4e7" (UID: "364afdeb-f401-4a87-956c-fa21679ae4e7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:31:11.216914 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:11.216898    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/364afdeb-f401-4a87-956c-fa21679ae4e7-cabundle-cert\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:31:11.216974 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:11.216920    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/364afdeb-f401-4a87-956c-fa21679ae4e7-kserve-provision-location\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:31:11.216974 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:11.216951    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/364afdeb-f401-4a87-956c-fa21679ae4e7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:31:11.218623 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:11.218599    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/364afdeb-f401-4a87-956c-fa21679ae4e7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "364afdeb-f401-4a87-956c-fa21679ae4e7" (UID: "364afdeb-f401-4a87-956c-fa21679ae4e7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:31:11.218987 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:11.218953    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/364afdeb-f401-4a87-956c-fa21679ae4e7-kube-api-access-g2nlf" (OuterVolumeSpecName: "kube-api-access-g2nlf") pod "364afdeb-f401-4a87-956c-fa21679ae4e7" (UID: "364afdeb-f401-4a87-956c-fa21679ae4e7"). InnerVolumeSpecName "kube-api-access-g2nlf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:31:11.317545 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:11.317509    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/364afdeb-f401-4a87-956c-fa21679ae4e7-proxy-tls\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:31:11.317545 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:11.317537    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2nlf\" (UniqueName: \"kubernetes.io/projected/364afdeb-f401-4a87-956c-fa21679ae4e7-kube-api-access-g2nlf\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:31:11.331326 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:11.331293    2563 generic.go:356] "Generic (PLEG): container finished" podID="364afdeb-f401-4a87-956c-fa21679ae4e7" containerID="60704a27e7138bb309b80e2955ac3627c14de6adddb66d5638ee033755b84dd0" exitCode=0
Sep 17 22:31:11.331451 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:11.331351    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-vx4g7"
Sep 17 22:31:11.331451 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:11.331376    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-vx4g7" event={"ID":"364afdeb-f401-4a87-956c-fa21679ae4e7","Type":"ContainerDied","Data":"60704a27e7138bb309b80e2955ac3627c14de6adddb66d5638ee033755b84dd0"}
Sep 17 22:31:11.331451 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:11.331434    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-vx4g7" event={"ID":"364afdeb-f401-4a87-956c-fa21679ae4e7","Type":"ContainerDied","Data":"c81bff994360bc7969960482bf3d81d14998e8e21e181632b79496507bcb8519"}
Sep 17 22:31:11.331556 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:11.331459    2563 scope.go:117] "RemoveContainer" containerID="66bec810c8716459b8031fac8261b5f541942047bcbb4729c91c29a18c9fa375"
Sep 17 22:31:11.339841 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:11.339825    2563 scope.go:117] "RemoveContainer" containerID="60704a27e7138bb309b80e2955ac3627c14de6adddb66d5638ee033755b84dd0"
Sep 17 22:31:11.347052 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:11.347033    2563 scope.go:117] "RemoveContainer" containerID="26c2d4a8d996c407cc70da2a6d1e84f88ad3031fe0a91e459fb573bf28923544"
Sep 17 22:31:11.352866 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:11.352847    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-vx4g7"]
Sep 17 22:31:11.354153 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:11.354118    2563 scope.go:117] "RemoveContainer" containerID="66bec810c8716459b8031fac8261b5f541942047bcbb4729c91c29a18c9fa375"
Sep 17 22:31:11.354444 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:31:11.354427    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66bec810c8716459b8031fac8261b5f541942047bcbb4729c91c29a18c9fa375\": container with ID starting with 66bec810c8716459b8031fac8261b5f541942047bcbb4729c91c29a18c9fa375 not found: ID does not exist" containerID="66bec810c8716459b8031fac8261b5f541942047bcbb4729c91c29a18c9fa375"
Sep 17 22:31:11.354495 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:11.354453    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66bec810c8716459b8031fac8261b5f541942047bcbb4729c91c29a18c9fa375"} err="failed to get container status \"66bec810c8716459b8031fac8261b5f541942047bcbb4729c91c29a18c9fa375\": rpc error: code = NotFound desc = could not find container \"66bec810c8716459b8031fac8261b5f541942047bcbb4729c91c29a18c9fa375\": container with ID starting with 66bec810c8716459b8031fac8261b5f541942047bcbb4729c91c29a18c9fa375 not found: ID does not exist"
Sep 17 22:31:11.354495 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:11.354475    2563 scope.go:117] "RemoveContainer" containerID="60704a27e7138bb309b80e2955ac3627c14de6adddb66d5638ee033755b84dd0"
Sep 17 22:31:11.354714 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:31:11.354693    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60704a27e7138bb309b80e2955ac3627c14de6adddb66d5638ee033755b84dd0\": container with ID starting with 60704a27e7138bb309b80e2955ac3627c14de6adddb66d5638ee033755b84dd0 not found: ID does not exist" containerID="60704a27e7138bb309b80e2955ac3627c14de6adddb66d5638ee033755b84dd0"
Sep 17 22:31:11.354758 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:11.354718    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60704a27e7138bb309b80e2955ac3627c14de6adddb66d5638ee033755b84dd0"} err="failed to get container status \"60704a27e7138bb309b80e2955ac3627c14de6adddb66d5638ee033755b84dd0\": rpc error: code = NotFound desc = could not find container \"60704a27e7138bb309b80e2955ac3627c14de6adddb66d5638ee033755b84dd0\": container with ID starting with 60704a27e7138bb309b80e2955ac3627c14de6adddb66d5638ee033755b84dd0 not found: ID does not exist"
Sep 17 22:31:11.354758 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:11.354730    2563 scope.go:117] "RemoveContainer" containerID="26c2d4a8d996c407cc70da2a6d1e84f88ad3031fe0a91e459fb573bf28923544"
Sep 17 22:31:11.354960 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:31:11.354942    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26c2d4a8d996c407cc70da2a6d1e84f88ad3031fe0a91e459fb573bf28923544\": container with ID starting with 26c2d4a8d996c407cc70da2a6d1e84f88ad3031fe0a91e459fb573bf28923544 not found: ID does not exist" containerID="26c2d4a8d996c407cc70da2a6d1e84f88ad3031fe0a91e459fb573bf28923544"
Sep 17 22:31:11.354996 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:11.354968    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26c2d4a8d996c407cc70da2a6d1e84f88ad3031fe0a91e459fb573bf28923544"} err="failed to get container status \"26c2d4a8d996c407cc70da2a6d1e84f88ad3031fe0a91e459fb573bf28923544\": rpc error: code = NotFound desc = could not find container \"26c2d4a8d996c407cc70da2a6d1e84f88ad3031fe0a91e459fb573bf28923544\": container with ID starting with 26c2d4a8d996c407cc70da2a6d1e84f88ad3031fe0a91e459fb573bf28923544 not found: ID does not exist"
Sep 17 22:31:11.357688 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:11.357666    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-vx4g7"]
Sep 17 22:31:12.914492 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:12.914459    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="364afdeb-f401-4a87-956c-fa21679ae4e7" path="/var/lib/kubelet/pods/364afdeb-f401-4a87-956c-fa21679ae4e7/volumes"
Sep 17 22:31:14.152225 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:14.152189    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-22h6f"]
Sep 17 22:31:14.152713 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:14.152692    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="364afdeb-f401-4a87-956c-fa21679ae4e7" containerName="kserve-container"
Sep 17 22:31:14.152786 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:14.152713    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="364afdeb-f401-4a87-956c-fa21679ae4e7" containerName="kserve-container"
Sep 17 22:31:14.152786 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:14.152736    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="364afdeb-f401-4a87-956c-fa21679ae4e7" containerName="storage-initializer"
Sep 17 22:31:14.152786 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:14.152746    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="364afdeb-f401-4a87-956c-fa21679ae4e7" containerName="storage-initializer"
Sep 17 22:31:14.152786 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:14.152762    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="364afdeb-f401-4a87-956c-fa21679ae4e7" containerName="kube-rbac-proxy"
Sep 17 22:31:14.152786 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:14.152772    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="364afdeb-f401-4a87-956c-fa21679ae4e7" containerName="kube-rbac-proxy"
Sep 17 22:31:14.153042 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:14.152847    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="364afdeb-f401-4a87-956c-fa21679ae4e7" containerName="kserve-container"
Sep 17 22:31:14.153042 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:14.152860    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="364afdeb-f401-4a87-956c-fa21679ae4e7" containerName="kube-rbac-proxy"
Sep 17 22:31:14.157341 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:14.157321    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-22h6f"
Sep 17 22:31:14.159617 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:14.159601    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-predictor-serving-cert\""
Sep 17 22:31:14.159712 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:14.159651    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-kube-rbac-proxy-sar-config\""
Sep 17 22:31:14.165590 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:14.165570    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-22h6f"]
Sep 17 22:31:14.243692 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:14.243662    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvbbb\" (UniqueName: \"kubernetes.io/projected/e92ff6c4-7ef8-4500-a219-6d0e61773381-kube-api-access-xvbbb\") pod \"isvc-triton-predictor-5d7db58676-22h6f\" (UID: \"e92ff6c4-7ef8-4500-a219-6d0e61773381\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-22h6f"
Sep 17 22:31:14.243847 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:14.243736    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e92ff6c4-7ef8-4500-a219-6d0e61773381-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-22h6f\" (UID: \"e92ff6c4-7ef8-4500-a219-6d0e61773381\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-22h6f"
Sep 17 22:31:14.243847 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:14.243756    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e92ff6c4-7ef8-4500-a219-6d0e61773381-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-22h6f\" (UID: \"e92ff6c4-7ef8-4500-a219-6d0e61773381\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-22h6f"
Sep 17 22:31:14.243847 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:14.243774    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e92ff6c4-7ef8-4500-a219-6d0e61773381-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-22h6f\" (UID: \"e92ff6c4-7ef8-4500-a219-6d0e61773381\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-22h6f"
Sep 17 22:31:14.243847 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:14.243793    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e92ff6c4-7ef8-4500-a219-6d0e61773381-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-22h6f\" (UID: \"e92ff6c4-7ef8-4500-a219-6d0e61773381\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-22h6f"
Sep 17 22:31:14.347695 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:14.347666    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e92ff6c4-7ef8-4500-a219-6d0e61773381-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-22h6f\" (UID: \"e92ff6c4-7ef8-4500-a219-6d0e61773381\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-22h6f"
Sep 17 22:31:14.347857 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:14.347708    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e92ff6c4-7ef8-4500-a219-6d0e61773381-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-22h6f\" (UID: \"e92ff6c4-7ef8-4500-a219-6d0e61773381\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-22h6f"
Sep 17 22:31:14.347857 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:14.347758    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvbbb\" (UniqueName: \"kubernetes.io/projected/e92ff6c4-7ef8-4500-a219-6d0e61773381-kube-api-access-xvbbb\") pod \"isvc-triton-predictor-5d7db58676-22h6f\" (UID: \"e92ff6c4-7ef8-4500-a219-6d0e61773381\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-22h6f"
Sep 17 22:31:14.347857 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:14.347844    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e92ff6c4-7ef8-4500-a219-6d0e61773381-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-22h6f\" (UID: \"e92ff6c4-7ef8-4500-a219-6d0e61773381\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-22h6f"
Sep 17 22:31:14.348015 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:14.347866    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e92ff6c4-7ef8-4500-a219-6d0e61773381-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-22h6f\" (UID: \"e92ff6c4-7ef8-4500-a219-6d0e61773381\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-22h6f"
Sep 17 22:31:14.348015 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:31:14.347966    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-triton-predictor-serving-cert: secret "isvc-triton-predictor-serving-cert" not found
Sep 17 22:31:14.348127 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:31:14.348021    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e92ff6c4-7ef8-4500-a219-6d0e61773381-proxy-tls podName:e92ff6c4-7ef8-4500-a219-6d0e61773381 nodeName:}" failed. No retries permitted until 2026-09-17 22:31:14.848001992 +0000 UTC m=+1986.015954956 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e92ff6c4-7ef8-4500-a219-6d0e61773381-proxy-tls") pod "isvc-triton-predictor-5d7db58676-22h6f" (UID: "e92ff6c4-7ef8-4500-a219-6d0e61773381") : secret "isvc-triton-predictor-serving-cert" not found
Sep 17 22:31:14.348221 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:14.348134    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e92ff6c4-7ef8-4500-a219-6d0e61773381-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-22h6f\" (UID: \"e92ff6c4-7ef8-4500-a219-6d0e61773381\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-22h6f"
Sep 17 22:31:14.348518 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:14.348472    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e92ff6c4-7ef8-4500-a219-6d0e61773381-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-22h6f\" (UID: \"e92ff6c4-7ef8-4500-a219-6d0e61773381\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-22h6f"
Sep 17 22:31:14.348629 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:14.348561    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e92ff6c4-7ef8-4500-a219-6d0e61773381-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-22h6f\" (UID: \"e92ff6c4-7ef8-4500-a219-6d0e61773381\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-22h6f"
Sep 17 22:31:14.358561 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:14.358541    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvbbb\" (UniqueName: \"kubernetes.io/projected/e92ff6c4-7ef8-4500-a219-6d0e61773381-kube-api-access-xvbbb\") pod \"isvc-triton-predictor-5d7db58676-22h6f\" (UID: \"e92ff6c4-7ef8-4500-a219-6d0e61773381\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-22h6f"
Sep 17 22:31:14.851103 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:14.851061    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e92ff6c4-7ef8-4500-a219-6d0e61773381-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-22h6f\" (UID: \"e92ff6c4-7ef8-4500-a219-6d0e61773381\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-22h6f"
Sep 17 22:31:14.853567 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:14.853547    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e92ff6c4-7ef8-4500-a219-6d0e61773381-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-22h6f\" (UID: \"e92ff6c4-7ef8-4500-a219-6d0e61773381\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-22h6f"
Sep 17 22:31:15.067397 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:15.067358    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-22h6f"
Sep 17 22:31:15.338664 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:15.338621    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-22h6f"]
Sep 17 22:31:15.341252 ip-10-0-138-45 kubenswrapper[2563]: W0917 22:31:15.341220    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode92ff6c4_7ef8_4500_a219_6d0e61773381.slice/crio-9fc9489cb4bc2ddb7c42a6d580ecc25b3f03c55dae955cbdcd2be7200d91d4dd WatchSource:0}: Error finding container 9fc9489cb4bc2ddb7c42a6d580ecc25b3f03c55dae955cbdcd2be7200d91d4dd: Status 404 returned error can't find the container with id 9fc9489cb4bc2ddb7c42a6d580ecc25b3f03c55dae955cbdcd2be7200d91d4dd
Sep 17 22:31:15.353789 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:15.353761    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-22h6f" event={"ID":"e92ff6c4-7ef8-4500-a219-6d0e61773381","Type":"ContainerStarted","Data":"9fc9489cb4bc2ddb7c42a6d580ecc25b3f03c55dae955cbdcd2be7200d91d4dd"}
Sep 17 22:31:16.359680 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:16.359648    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-22h6f" event={"ID":"e92ff6c4-7ef8-4500-a219-6d0e61773381","Type":"ContainerStarted","Data":"18313776ddc40fb9ccb7aba1cf592b971eb089c950ca3596d18694313fe64545"}
Sep 17 22:31:19.374578 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:19.374496    2563 generic.go:356] "Generic (PLEG): container finished" podID="e92ff6c4-7ef8-4500-a219-6d0e61773381" containerID="18313776ddc40fb9ccb7aba1cf592b971eb089c950ca3596d18694313fe64545" exitCode=0
Sep 17 22:31:19.374578 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:19.374566    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-22h6f" event={"ID":"e92ff6c4-7ef8-4500-a219-6d0e61773381","Type":"ContainerDied","Data":"18313776ddc40fb9ccb7aba1cf592b971eb089c950ca3596d18694313fe64545"}
Sep 17 22:31:19.375708 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:31:19.375694    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 22:32:36.569440 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:32:36.569399    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-jmrl6"]
Sep 17 22:32:36.570188 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:32:36.569689    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-jmrl6" podUID="689c1955-688f-4f55-a8e6-822ac7b779b8" containerName="kserve-container" containerID="cri-o://1f6d62480def39ed2e9e0486365cb2912101b3ac018eb0cbb73dac69a23eea36" gracePeriod=30
Sep 17 22:32:36.570188 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:32:36.569910    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-jmrl6" podUID="689c1955-688f-4f55-a8e6-822ac7b779b8" containerName="kube-rbac-proxy" containerID="cri-o://ede27372fe035b4859582536e4edf4533196afe709a8cf673687e3401eae32a8" gracePeriod=30
Sep 17 22:32:37.811425 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:32:37.811388    2563 generic.go:356] "Generic (PLEG): container finished" podID="689c1955-688f-4f55-a8e6-822ac7b779b8" containerID="ede27372fe035b4859582536e4edf4533196afe709a8cf673687e3401eae32a8" exitCode=2
Sep 17 22:32:37.811875 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:32:37.811478    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-jmrl6" event={"ID":"689c1955-688f-4f55-a8e6-822ac7b779b8","Type":"ContainerDied","Data":"ede27372fe035b4859582536e4edf4533196afe709a8cf673687e3401eae32a8"}
Sep 17 22:32:37.909159 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:32:37.909107    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-jmrl6" podUID="689c1955-688f-4f55-a8e6-822ac7b779b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 17 22:32:38.971645 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:32:38.971597    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-jmrl6" podUID="689c1955-688f-4f55-a8e6-822ac7b779b8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.42:8643/healthz\": dial tcp 10.133.0.42:8643: connect: connection refused"
Sep 17 22:32:41.798929 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:32:41.798889    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-v7q8v"]
Sep 17 22:32:41.869836 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:32:41.869803    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-v7q8v"]
Sep 17 22:32:41.870035 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:32:41.869936    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-v7q8v"
Sep 17 22:32:41.872271 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:32:41.872245    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 17 22:32:41.872399 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:32:41.872254    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 17 22:32:41.959534 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:32:41.959498    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7652705-54c0-4597-90a4-c3b0762276f6-cabundle-cert\") pod \"isvc-canary-stop-predictor-646f5bdbcb-v7q8v\" (UID: \"f7652705-54c0-4597-90a4-c3b0762276f6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-v7q8v"
Sep 17 22:32:41.959690 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:32:41.959610    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7652705-54c0-4597-90a4-c3b0762276f6-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-646f5bdbcb-v7q8v\" (UID: \"f7652705-54c0-4597-90a4-c3b0762276f6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-v7q8v"
Sep 17 22:32:41.959690 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:32:41.959651    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtwfw\" (UniqueName: \"kubernetes.io/projected/f7652705-54c0-4597-90a4-c3b0762276f6-kube-api-access-vtwfw\") pod \"isvc-canary-stop-predictor-646f5bdbcb-v7q8v\" (UID: \"f7652705-54c0-4597-90a4-c3b0762276f6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-v7q8v"
Sep 17 22:32:41.959807 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:32:41.959688    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7652705-54c0-4597-90a4-c3b0762276f6-proxy-tls\") pod \"isvc-canary-stop-predictor-646f5bdbcb-v7q8v\" (UID: \"f7652705-54c0-4597-90a4-c3b0762276f6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-v7q8v"
Sep 17 22:32:41.959807 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:32:41.959731    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7652705-54c0-4597-90a4-c3b0762276f6-kserve-provision-location\") pod \"isvc-canary-stop-predictor-646f5bdbcb-v7q8v\" (UID: \"f7652705-54c0-4597-90a4-c3b0762276f6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-v7q8v"
Sep 17 22:32:42.060867 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:32:42.060782    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7652705-54c0-4597-90a4-c3b0762276f6-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-646f5bdbcb-v7q8v\" (UID: \"f7652705-54c0-4597-90a4-c3b0762276f6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-v7q8v"
Sep 17 22:32:42.060867 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:32:42.060824    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtwfw\" (UniqueName: \"kubernetes.io/projected/f7652705-54c0-4597-90a4-c3b0762276f6-kube-api-access-vtwfw\") pod \"isvc-canary-stop-predictor-646f5bdbcb-v7q8v\" (UID: \"f7652705-54c0-4597-90a4-c3b0762276f6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-v7q8v"
Sep 17 22:32:42.061121 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:32:42.060959    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7652705-54c0-4597-90a4-c3b0762276f6-proxy-tls\") pod \"isvc-canary-stop-predictor-646f5bdbcb-v7q8v\" (UID: \"f7652705-54c0-4597-90a4-c3b0762276f6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-v7q8v"
Sep 17 22:32:42.061121 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:32:42.061022    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7652705-54c0-4597-90a4-c3b0762276f6-kserve-provision-location\") pod \"isvc-canary-stop-predictor-646f5bdbcb-v7q8v\" (UID: \"f7652705-54c0-4597-90a4-c3b0762276f6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-v7q8v"
Sep 17 22:32:42.061121 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:32:42.061060    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7652705-54c0-4597-90a4-c3b0762276f6-cabundle-cert\") pod \"isvc-canary-stop-predictor-646f5bdbcb-v7q8v\" (UID: \"f7652705-54c0-4597-90a4-c3b0762276f6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-v7q8v"
Sep 17 22:32:42.061453 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:32:42.061421    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7652705-54c0-4597-90a4-c3b0762276f6-kserve-provision-location\") pod \"isvc-canary-stop-predictor-646f5bdbcb-v7q8v\" (UID: \"f7652705-54c0-4597-90a4-c3b0762276f6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-v7q8v"
Sep 17 22:32:42.061612 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:32:42.061588    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7652705-54c0-4597-90a4-c3b0762276f6-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-646f5bdbcb-v7q8v\" (UID: \"f7652705-54c0-4597-90a4-c3b0762276f6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-v7q8v"
Sep 17 22:32:42.061744 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:32:42.061721    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7652705-54c0-4597-90a4-c3b0762276f6-cabundle-cert\") pod \"isvc-canary-stop-predictor-646f5bdbcb-v7q8v\" (UID: \"f7652705-54c0-4597-90a4-c3b0762276f6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-v7q8v"
Sep 17 22:32:42.063896 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:32:42.063871    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7652705-54c0-4597-90a4-c3b0762276f6-proxy-tls\") pod \"isvc-canary-stop-predictor-646f5bdbcb-v7q8v\" (UID: \"f7652705-54c0-4597-90a4-c3b0762276f6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-v7q8v"
Sep 17 22:32:42.069289 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:32:42.069264    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtwfw\" (UniqueName: \"kubernetes.io/projected/f7652705-54c0-4597-90a4-c3b0762276f6-kube-api-access-vtwfw\") pod \"isvc-canary-stop-predictor-646f5bdbcb-v7q8v\" (UID: \"f7652705-54c0-4597-90a4-c3b0762276f6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-v7q8v"
Sep 17 22:32:42.182978 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:32:42.182940    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-v7q8v"
Sep 17 22:32:42.835668 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:32:42.835582    2563 generic.go:356] "Generic (PLEG): container finished" podID="689c1955-688f-4f55-a8e6-822ac7b779b8" containerID="1f6d62480def39ed2e9e0486365cb2912101b3ac018eb0cbb73dac69a23eea36" exitCode=0
Sep 17 22:32:42.835668 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:32:42.835639    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-jmrl6" event={"ID":"689c1955-688f-4f55-a8e6-822ac7b779b8","Type":"ContainerDied","Data":"1f6d62480def39ed2e9e0486365cb2912101b3ac018eb0cbb73dac69a23eea36"}
Sep 17 22:32:43.706669 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:32:43.706632    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-jmrl6"
Sep 17 22:32:43.776409 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:32:43.776374    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/689c1955-688f-4f55-a8e6-822ac7b779b8-kserve-provision-location\") pod \"689c1955-688f-4f55-a8e6-822ac7b779b8\" (UID: \"689c1955-688f-4f55-a8e6-822ac7b779b8\") "
Sep 17 22:32:43.776615 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:32:43.776439    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/689c1955-688f-4f55-a8e6-822ac7b779b8-proxy-tls\") pod \"689c1955-688f-4f55-a8e6-822ac7b779b8\" (UID: \"689c1955-688f-4f55-a8e6-822ac7b779b8\") "
Sep 17 22:32:43.776615 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:32:43.776487    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tqhg4\" (UniqueName: \"kubernetes.io/projected/689c1955-688f-4f55-a8e6-822ac7b779b8-kube-api-access-tqhg4\") pod \"689c1955-688f-4f55-a8e6-822ac7b779b8\" (UID: \"689c1955-688f-4f55-a8e6-822ac7b779b8\") "
Sep 17 22:32:43.776615 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:32:43.776508    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/689c1955-688f-4f55-a8e6-822ac7b779b8-kube-rbac-proxy-sar-config\") pod \"689c1955-688f-4f55-a8e6-822ac7b779b8\" (UID: \"689c1955-688f-4f55-a8e6-822ac7b779b8\") "
Sep 17 22:32:43.776615 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:32:43.776528    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/689c1955-688f-4f55-a8e6-822ac7b779b8-cabundle-cert\") pod \"689c1955-688f-4f55-a8e6-822ac7b779b8\" (UID: \"689c1955-688f-4f55-a8e6-822ac7b779b8\") "
Sep 17 22:32:43.776846 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:32:43.776775    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/689c1955-688f-4f55-a8e6-822ac7b779b8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "689c1955-688f-4f55-a8e6-822ac7b779b8" (UID: "689c1955-688f-4f55-a8e6-822ac7b779b8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:32:43.776846 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:32:43.776831    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/689c1955-688f-4f55-a8e6-822ac7b779b8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "689c1955-688f-4f55-a8e6-822ac7b779b8" (UID: "689c1955-688f-4f55-a8e6-822ac7b779b8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:32:43.776942 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:32:43.776907    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/689c1955-688f-4f55-a8e6-822ac7b779b8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "689c1955-688f-4f55-a8e6-822ac7b779b8" (UID: "689c1955-688f-4f55-a8e6-822ac7b779b8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:32:43.778656 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:32:43.778630    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/689c1955-688f-4f55-a8e6-822ac7b779b8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "689c1955-688f-4f55-a8e6-822ac7b779b8" (UID: "689c1955-688f-4f55-a8e6-822ac7b779b8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:32:43.778747 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:32:43.778717    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/689c1955-688f-4f55-a8e6-822ac7b779b8-kube-api-access-tqhg4" (OuterVolumeSpecName: "kube-api-access-tqhg4") pod "689c1955-688f-4f55-a8e6-822ac7b779b8" (UID: "689c1955-688f-4f55-a8e6-822ac7b779b8"). InnerVolumeSpecName "kube-api-access-tqhg4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:32:43.847724 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:32:43.846467    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-jmrl6" event={"ID":"689c1955-688f-4f55-a8e6-822ac7b779b8","Type":"ContainerDied","Data":"5cc1bae6194084bfed49ac34ba467a94548cf559cddd2d35cfd0477fd87384f2"}
Sep 17 22:32:43.847724 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:32:43.846547    2563 scope.go:117] "RemoveContainer" containerID="ede27372fe035b4859582536e4edf4533196afe709a8cf673687e3401eae32a8"
Sep 17 22:32:43.847724 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:32:43.846732    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-jmrl6"
Sep 17 22:32:43.873621 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:32:43.873586    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-jmrl6"]
Sep 17 22:32:43.877557 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:32:43.877524    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/689c1955-688f-4f55-a8e6-822ac7b779b8-kserve-provision-location\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:32:43.877557 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:32:43.877559    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/689c1955-688f-4f55-a8e6-822ac7b779b8-proxy-tls\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:32:43.877762 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:32:43.877575    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqhg4\" (UniqueName: \"kubernetes.io/projected/689c1955-688f-4f55-a8e6-822ac7b779b8-kube-api-access-tqhg4\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:32:43.877762 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:32:43.877585    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/689c1955-688f-4f55-a8e6-822ac7b779b8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:32:43.877762 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:32:43.877593    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/689c1955-688f-4f55-a8e6-822ac7b779b8-cabundle-cert\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:32:43.879658 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:32:43.879628    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-jmrl6"]
Sep 17 22:32:43.936578 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:32:43.936552    2563 scope.go:117] "RemoveContainer" containerID="1f6d62480def39ed2e9e0486365cb2912101b3ac018eb0cbb73dac69a23eea36"
Sep 17 22:32:44.062402 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:32:44.062375    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-v7q8v"]
Sep 17 22:32:44.915094 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:32:44.915051    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="689c1955-688f-4f55-a8e6-822ac7b779b8" path="/var/lib/kubelet/pods/689c1955-688f-4f55-a8e6-822ac7b779b8/volumes"
Sep 17 22:33:11.817244 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:33:11.816881    2563 scope.go:117] "RemoveContainer" containerID="ed8465235da98e94991a2a0daf3623fd666ddfbcf1be8aa369cdde44614758f9"
Sep 17 22:33:11.818215 ip-10-0-138-45 kubenswrapper[2563]: W0917 22:33:11.818184    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7652705_54c0_4597_90a4_c3b0762276f6.slice/crio-71e8166989d2a62a4508155809e4bcd52ce7dc4efccaa5c46a08aa7a013ba664 WatchSource:0}: Error finding container 71e8166989d2a62a4508155809e4bcd52ce7dc4efccaa5c46a08aa7a013ba664: Status 404 returned error can't find the container with id 71e8166989d2a62a4508155809e4bcd52ce7dc4efccaa5c46a08aa7a013ba664
Sep 17 22:33:11.976343 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:33:11.976306    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-v7q8v" event={"ID":"f7652705-54c0-4597-90a4-c3b0762276f6","Type":"ContainerStarted","Data":"71e8166989d2a62a4508155809e4bcd52ce7dc4efccaa5c46a08aa7a013ba664"}
Sep 17 22:33:12.983755 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:33:12.983708    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-v7q8v" event={"ID":"f7652705-54c0-4597-90a4-c3b0762276f6","Type":"ContainerStarted","Data":"ac41a9c4209a73272145488825cce0c05eb4e85292c47c77f55346e5d5720afc"}
Sep 17 22:33:13.995809 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:33:13.995771    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-22h6f" event={"ID":"e92ff6c4-7ef8-4500-a219-6d0e61773381","Type":"ContainerStarted","Data":"2d4d07869e25ffa5488a4a9bed2ae56220a2531e56f0c39eceb298561d38bf00"}
Sep 17 22:33:13.996343 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:33:13.995855    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-22h6f" event={"ID":"e92ff6c4-7ef8-4500-a219-6d0e61773381","Type":"ContainerStarted","Data":"834d7170fb560f59de410bfc3ff84f0e761d43989d617932c264941583499782"}
Sep 17 22:33:13.996343 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:33:13.996029    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-22h6f"
Sep 17 22:33:13.996343 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:33:13.996081    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-22h6f"
Sep 17 22:33:13.997369 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:33:13.997345    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-22h6f" podUID="e92ff6c4-7ef8-4500-a219-6d0e61773381" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 22:33:14.014780 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:33:14.014736    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-22h6f" podStartSLOduration=2.3352728320000002 podStartE2EDuration="2m0.014723054s" totalImagesPullingTime="1m53.960823266s" totalInitContainerRuntime="3.718626956s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:31:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:33:14.01367596 +0000 UTC m=+2105.181628944" watchObservedRunningTime="2026-09-17 22:33:14.014723054 +0000 UTC m=+2105.182676037"
Sep 17 22:33:15.000624 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:33:15.000586    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-22h6f" podUID="e92ff6c4-7ef8-4500-a219-6d0e61773381" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 22:33:17.009659 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:33:17.009577    2563 generic.go:356] "Generic (PLEG): container finished" podID="f7652705-54c0-4597-90a4-c3b0762276f6" containerID="ac41a9c4209a73272145488825cce0c05eb4e85292c47c77f55346e5d5720afc" exitCode=0
Sep 17 22:33:17.010067 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:33:17.009651    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-v7q8v" event={"ID":"f7652705-54c0-4597-90a4-c3b0762276f6","Type":"ContainerDied","Data":"ac41a9c4209a73272145488825cce0c05eb4e85292c47c77f55346e5d5720afc"}
Sep 17 22:33:18.015338 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:33:18.015305    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-v7q8v" event={"ID":"f7652705-54c0-4597-90a4-c3b0762276f6","Type":"ContainerStarted","Data":"49948b26f99e89335c0bcba4acc98c5402a9f83b4f87c280ae668c35adfb72db"}
Sep 17 22:33:18.015718 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:33:18.015367    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-v7q8v" event={"ID":"f7652705-54c0-4597-90a4-c3b0762276f6","Type":"ContainerStarted","Data":"18ab2591c1f017003e1413cdb572f8e4072cf2fda8d7c92797a0a56ddf5bcb2e"}
Sep 17 22:33:18.015718 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:33:18.015577    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-v7q8v"
Sep 17 22:33:18.033976 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:33:18.033921    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-v7q8v" podStartSLOduration=32.828571632 podStartE2EDuration="37.033905401s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.205333769s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:32:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:33:18.033189619 +0000 UTC m=+2109.201142599" watchObservedRunningTime="2026-09-17 22:33:18.033905401 +0000 UTC m=+2109.201858385"
Sep 17 22:33:19.019855 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:33:19.019822    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-v7q8v"
Sep 17 22:33:19.021273 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:33:19.021244    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-v7q8v" podUID="f7652705-54c0-4597-90a4-c3b0762276f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 17 22:33:20.005262 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:33:20.005234    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-22h6f"
Sep 17 22:33:20.005775 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:33:20.005748    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-22h6f" podUID="e92ff6c4-7ef8-4500-a219-6d0e61773381" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 22:33:20.024333 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:33:20.024288    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-v7q8v" podUID="f7652705-54c0-4597-90a4-c3b0762276f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 17 22:33:25.028523 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:33:25.028493    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-v7q8v"
Sep 17 22:33:25.029012 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:33:25.028987    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-v7q8v" podUID="f7652705-54c0-4597-90a4-c3b0762276f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 17 22:33:30.006042 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:33:30.006009    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-22h6f"
Sep 17 22:33:35.029855 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:33:35.029814    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-v7q8v" podUID="f7652705-54c0-4597-90a4-c3b0762276f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 17 22:33:35.964224 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:33:35.964190    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-22h6f"]
Sep 17 22:33:35.964513 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:33:35.964469    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-22h6f" podUID="e92ff6c4-7ef8-4500-a219-6d0e61773381" containerName="kserve-container" containerID="cri-o://834d7170fb560f59de410bfc3ff84f0e761d43989d617932c264941583499782" gracePeriod=30
Sep 17 22:33:35.964636 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:33:35.964505    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-22h6f" podUID="e92ff6c4-7ef8-4500-a219-6d0e61773381" containerName="kube-rbac-proxy" containerID="cri-o://2d4d07869e25ffa5488a4a9bed2ae56220a2531e56f0c39eceb298561d38bf00" gracePeriod=30
Sep 17 22:33:36.090462 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:33:36.090429    2563 generic.go:356] "Generic (PLEG): container finished" podID="e92ff6c4-7ef8-4500-a219-6d0e61773381" containerID="2d4d07869e25ffa5488a4a9bed2ae56220a2531e56f0c39eceb298561d38bf00" exitCode=2
Sep 17 22:33:36.090838 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:33:36.090481    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-22h6f" event={"ID":"e92ff6c4-7ef8-4500-a219-6d0e61773381","Type":"ContainerDied","Data":"2d4d07869e25ffa5488a4a9bed2ae56220a2531e56f0c39eceb298561d38bf00"}
Sep 17 22:33:38.834517 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:33:38.834481    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-22h6f"
Sep 17 22:33:38.857199 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:33:38.857170    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e92ff6c4-7ef8-4500-a219-6d0e61773381-kserve-provision-location\") pod \"e92ff6c4-7ef8-4500-a219-6d0e61773381\" (UID: \"e92ff6c4-7ef8-4500-a219-6d0e61773381\") "
Sep 17 22:33:38.857373 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:33:38.857244    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e92ff6c4-7ef8-4500-a219-6d0e61773381-proxy-tls\") pod \"e92ff6c4-7ef8-4500-a219-6d0e61773381\" (UID: \"e92ff6c4-7ef8-4500-a219-6d0e61773381\") "
Sep 17 22:33:38.857373 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:33:38.857268    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xvbbb\" (UniqueName: \"kubernetes.io/projected/e92ff6c4-7ef8-4500-a219-6d0e61773381-kube-api-access-xvbbb\") pod \"e92ff6c4-7ef8-4500-a219-6d0e61773381\" (UID: \"e92ff6c4-7ef8-4500-a219-6d0e61773381\") "
Sep 17 22:33:38.857373 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:33:38.857314    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e92ff6c4-7ef8-4500-a219-6d0e61773381-kube-rbac-proxy-sar-config\") pod \"e92ff6c4-7ef8-4500-a219-6d0e61773381\" (UID: \"e92ff6c4-7ef8-4500-a219-6d0e61773381\") "
Sep 17 22:33:38.857555 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:33:38.857402    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e92ff6c4-7ef8-4500-a219-6d0e61773381-cabundle-cert\") pod \"e92ff6c4-7ef8-4500-a219-6d0e61773381\" (UID: \"e92ff6c4-7ef8-4500-a219-6d0e61773381\") "
Sep 17 22:33:38.857646 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:33:38.857617    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e92ff6c4-7ef8-4500-a219-6d0e61773381-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e92ff6c4-7ef8-4500-a219-6d0e61773381" (UID: "e92ff6c4-7ef8-4500-a219-6d0e61773381"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:33:38.857721 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:33:38.857640    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e92ff6c4-7ef8-4500-a219-6d0e61773381-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e92ff6c4-7ef8-4500-a219-6d0e61773381" (UID: "e92ff6c4-7ef8-4500-a219-6d0e61773381"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:33:38.857905 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:33:38.857886    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e92ff6c4-7ef8-4500-a219-6d0e61773381-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e92ff6c4-7ef8-4500-a219-6d0e61773381" (UID: "e92ff6c4-7ef8-4500-a219-6d0e61773381"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:33:38.859926 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:33:38.859905    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e92ff6c4-7ef8-4500-a219-6d0e61773381-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e92ff6c4-7ef8-4500-a219-6d0e61773381" (UID: "e92ff6c4-7ef8-4500-a219-6d0e61773381"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:33:38.860014 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:33:38.859975    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e92ff6c4-7ef8-4500-a219-6d0e61773381-kube-api-access-xvbbb" (OuterVolumeSpecName: "kube-api-access-xvbbb") pod "e92ff6c4-7ef8-4500-a219-6d0e61773381" (UID: "e92ff6c4-7ef8-4500-a219-6d0e61773381"). InnerVolumeSpecName "kube-api-access-xvbbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:33:38.958373 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:33:38.958291    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e92ff6c4-7ef8-4500-a219-6d0e61773381-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:33:38.958373 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:33:38.958320    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e92ff6c4-7ef8-4500-a219-6d0e61773381-cabundle-cert\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:33:38.958373 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:33:38.958333    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e92ff6c4-7ef8-4500-a219-6d0e61773381-kserve-provision-location\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:33:38.958373 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:33:38.958346    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e92ff6c4-7ef8-4500-a219-6d0e61773381-proxy-tls\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:33:38.958373 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:33:38.958360    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvbbb\" (UniqueName: \"kubernetes.io/projected/e92ff6c4-7ef8-4500-a219-6d0e61773381-kube-api-access-xvbbb\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:33:39.104368 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:33:39.104333    2563 generic.go:356] "Generic (PLEG): container finished" podID="e92ff6c4-7ef8-4500-a219-6d0e61773381" containerID="834d7170fb560f59de410bfc3ff84f0e761d43989d617932c264941583499782" exitCode=0
Sep 17 22:33:39.104541 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:33:39.104385    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-22h6f"
Sep 17 22:33:39.104541 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:33:39.104421    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-22h6f" event={"ID":"e92ff6c4-7ef8-4500-a219-6d0e61773381","Type":"ContainerDied","Data":"834d7170fb560f59de410bfc3ff84f0e761d43989d617932c264941583499782"}
Sep 17 22:33:39.104541 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:33:39.104490    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-22h6f" event={"ID":"e92ff6c4-7ef8-4500-a219-6d0e61773381","Type":"ContainerDied","Data":"9fc9489cb4bc2ddb7c42a6d580ecc25b3f03c55dae955cbdcd2be7200d91d4dd"}
Sep 17 22:33:39.104541 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:33:39.104517    2563 scope.go:117] "RemoveContainer" containerID="2d4d07869e25ffa5488a4a9bed2ae56220a2531e56f0c39eceb298561d38bf00"
Sep 17 22:33:39.112917 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:33:39.112890    2563 scope.go:117] "RemoveContainer" containerID="834d7170fb560f59de410bfc3ff84f0e761d43989d617932c264941583499782"
Sep 17 22:33:39.120847 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:33:39.120827    2563 scope.go:117] "RemoveContainer" containerID="18313776ddc40fb9ccb7aba1cf592b971eb089c950ca3596d18694313fe64545"
Sep 17 22:33:39.121569 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:33:39.121548    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-22h6f"]
Sep 17 22:33:39.124059 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:33:39.124039    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-22h6f"]
Sep 17 22:33:39.128040 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:33:39.128023    2563 scope.go:117] "RemoveContainer" containerID="2d4d07869e25ffa5488a4a9bed2ae56220a2531e56f0c39eceb298561d38bf00"
Sep 17 22:33:39.128387 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:33:39.128368    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d4d07869e25ffa5488a4a9bed2ae56220a2531e56f0c39eceb298561d38bf00\": container with ID starting with 2d4d07869e25ffa5488a4a9bed2ae56220a2531e56f0c39eceb298561d38bf00 not found: ID does not exist" containerID="2d4d07869e25ffa5488a4a9bed2ae56220a2531e56f0c39eceb298561d38bf00"
Sep 17 22:33:39.128469 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:33:39.128398    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d4d07869e25ffa5488a4a9bed2ae56220a2531e56f0c39eceb298561d38bf00"} err="failed to get container status \"2d4d07869e25ffa5488a4a9bed2ae56220a2531e56f0c39eceb298561d38bf00\": rpc error: code = NotFound desc = could not find container \"2d4d07869e25ffa5488a4a9bed2ae56220a2531e56f0c39eceb298561d38bf00\": container with ID starting with 2d4d07869e25ffa5488a4a9bed2ae56220a2531e56f0c39eceb298561d38bf00 not found: ID does not exist"
Sep 17 22:33:39.128469 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:33:39.128418    2563 scope.go:117] "RemoveContainer" containerID="834d7170fb560f59de410bfc3ff84f0e761d43989d617932c264941583499782"
Sep 17 22:33:39.128675 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:33:39.128657    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"834d7170fb560f59de410bfc3ff84f0e761d43989d617932c264941583499782\": container with ID starting with 834d7170fb560f59de410bfc3ff84f0e761d43989d617932c264941583499782 not found: ID does not exist" containerID="834d7170fb560f59de410bfc3ff84f0e761d43989d617932c264941583499782"
Sep 17 22:33:39.128749 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:33:39.128687    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"834d7170fb560f59de410bfc3ff84f0e761d43989d617932c264941583499782"} err="failed to get container status \"834d7170fb560f59de410bfc3ff84f0e761d43989d617932c264941583499782\": rpc error: code = NotFound desc = could not find container \"834d7170fb560f59de410bfc3ff84f0e761d43989d617932c264941583499782\": container with ID starting with 834d7170fb560f59de410bfc3ff84f0e761d43989d617932c264941583499782 not found: ID does not exist"
Sep 17 22:33:39.128749 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:33:39.128711    2563 scope.go:117] "RemoveContainer" containerID="18313776ddc40fb9ccb7aba1cf592b971eb089c950ca3596d18694313fe64545"
Sep 17 22:33:39.128966 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:33:39.128950    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18313776ddc40fb9ccb7aba1cf592b971eb089c950ca3596d18694313fe64545\": container with ID starting with 18313776ddc40fb9ccb7aba1cf592b971eb089c950ca3596d18694313fe64545 not found: ID does not exist" containerID="18313776ddc40fb9ccb7aba1cf592b971eb089c950ca3596d18694313fe64545"
Sep 17 22:33:39.129002 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:33:39.128973    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18313776ddc40fb9ccb7aba1cf592b971eb089c950ca3596d18694313fe64545"} err="failed to get container status \"18313776ddc40fb9ccb7aba1cf592b971eb089c950ca3596d18694313fe64545\": rpc error: code = NotFound desc = could not find container \"18313776ddc40fb9ccb7aba1cf592b971eb089c950ca3596d18694313fe64545\": container with ID starting with 18313776ddc40fb9ccb7aba1cf592b971eb089c950ca3596d18694313fe64545 not found: ID does not exist"
Sep 17 22:33:40.914226 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:33:40.914189    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e92ff6c4-7ef8-4500-a219-6d0e61773381" path="/var/lib/kubelet/pods/e92ff6c4-7ef8-4500-a219-6d0e61773381/volumes"
Sep 17 22:33:45.029356 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:33:45.029319    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-v7q8v" podUID="f7652705-54c0-4597-90a4-c3b0762276f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 17 22:33:49.631695 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:33:49.631659    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nrkd5_34b8ef2b-97ba-4562-87ee-8da0cb4604c7/ovn-acl-logging/0.log"
Sep 17 22:33:49.633333 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:33:49.633296    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nrkd5_34b8ef2b-97ba-4562-87ee-8da0cb4604c7/ovn-acl-logging/0.log"
Sep 17 22:33:55.029803 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:33:55.029756    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-v7q8v" podUID="f7652705-54c0-4597-90a4-c3b0762276f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 17 22:34:05.029681 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:34:05.029642    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-v7q8v" podUID="f7652705-54c0-4597-90a4-c3b0762276f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 17 22:34:13.415666 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:34:13.415627    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-v7q8v"]
Sep 17 22:34:13.416091 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:34:13.415870    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-v7q8v" podUID="f7652705-54c0-4597-90a4-c3b0762276f6" containerName="kserve-container" containerID="cri-o://18ab2591c1f017003e1413cdb572f8e4072cf2fda8d7c92797a0a56ddf5bcb2e" gracePeriod=30
Sep 17 22:34:13.416091 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:34:13.415912    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-v7q8v" podUID="f7652705-54c0-4597-90a4-c3b0762276f6" containerName="kube-rbac-proxy" containerID="cri-o://49948b26f99e89335c0bcba4acc98c5402a9f83b4f87c280ae668c35adfb72db" gracePeriod=30
Sep 17 22:34:13.416261 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:34:13.416165    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-v7q8v" podUID="f7652705-54c0-4597-90a4-c3b0762276f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 17 22:34:14.253012 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:34:14.252977    2563 generic.go:356] "Generic (PLEG): container finished" podID="f7652705-54c0-4597-90a4-c3b0762276f6" containerID="49948b26f99e89335c0bcba4acc98c5402a9f83b4f87c280ae668c35adfb72db" exitCode=2
Sep 17 22:34:14.253012 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:34:14.253018    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-v7q8v" event={"ID":"f7652705-54c0-4597-90a4-c3b0762276f6","Type":"ContainerDied","Data":"49948b26f99e89335c0bcba4acc98c5402a9f83b4f87c280ae668c35adfb72db"}
Sep 17 22:34:15.025083 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:34:15.025036    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-v7q8v" podUID="f7652705-54c0-4597-90a4-c3b0762276f6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.44:8643/healthz\": dial tcp 10.133.0.44:8643: connect: connection refused"
Sep 17 22:34:20.025171 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:34:20.025114    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-v7q8v" podUID="f7652705-54c0-4597-90a4-c3b0762276f6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.44:8643/healthz\": dial tcp 10.133.0.44:8643: connect: connection refused"
Sep 17 22:34:23.416383 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:34:23.416337    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-v7q8v" podUID="f7652705-54c0-4597-90a4-c3b0762276f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 17 22:34:25.024480 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:34:25.024433    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-v7q8v" podUID="f7652705-54c0-4597-90a4-c3b0762276f6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.44:8643/healthz\": dial tcp 10.133.0.44:8643: connect: connection refused"
Sep 17 22:34:25.024948 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:34:25.024579    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-v7q8v"
Sep 17 22:34:30.025326 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:34:30.025285    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-v7q8v" podUID="f7652705-54c0-4597-90a4-c3b0762276f6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.44:8643/healthz\": dial tcp 10.133.0.44:8643: connect: connection refused"
Sep 17 22:34:33.417892 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:34:33.417864    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-v7q8v"
Sep 17 22:34:35.025474 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:34:35.025436    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-v7q8v" podUID="f7652705-54c0-4597-90a4-c3b0762276f6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.44:8643/healthz\": dial tcp 10.133.0.44:8643: connect: connection refused"
Sep 17 22:34:40.024635 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:34:40.024595    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-v7q8v" podUID="f7652705-54c0-4597-90a4-c3b0762276f6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.44:8643/healthz\": dial tcp 10.133.0.44:8643: connect: connection refused"
Sep 17 22:34:44.060931 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:34:44.060906    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-v7q8v"
Sep 17 22:34:44.112629 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:34:44.112598    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7652705-54c0-4597-90a4-c3b0762276f6-proxy-tls\") pod \"f7652705-54c0-4597-90a4-c3b0762276f6\" (UID: \"f7652705-54c0-4597-90a4-c3b0762276f6\") "
Sep 17 22:34:44.112629 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:34:44.112632    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7652705-54c0-4597-90a4-c3b0762276f6-cabundle-cert\") pod \"f7652705-54c0-4597-90a4-c3b0762276f6\" (UID: \"f7652705-54c0-4597-90a4-c3b0762276f6\") "
Sep 17 22:34:44.112828 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:34:44.112670    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7652705-54c0-4597-90a4-c3b0762276f6-kserve-provision-location\") pod \"f7652705-54c0-4597-90a4-c3b0762276f6\" (UID: \"f7652705-54c0-4597-90a4-c3b0762276f6\") "
Sep 17 22:34:44.112917 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:34:44.112897    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7652705-54c0-4597-90a4-c3b0762276f6-kube-rbac-proxy-sar-config\") pod \"f7652705-54c0-4597-90a4-c3b0762276f6\" (UID: \"f7652705-54c0-4597-90a4-c3b0762276f6\") "
Sep 17 22:34:44.112984 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:34:44.112970    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vtwfw\" (UniqueName: \"kubernetes.io/projected/f7652705-54c0-4597-90a4-c3b0762276f6-kube-api-access-vtwfw\") pod \"f7652705-54c0-4597-90a4-c3b0762276f6\" (UID: \"f7652705-54c0-4597-90a4-c3b0762276f6\") "
Sep 17 22:34:44.113101 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:34:44.112968    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7652705-54c0-4597-90a4-c3b0762276f6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f7652705-54c0-4597-90a4-c3b0762276f6" (UID: "f7652705-54c0-4597-90a4-c3b0762276f6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:34:44.113101 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:34:44.113002    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7652705-54c0-4597-90a4-c3b0762276f6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f7652705-54c0-4597-90a4-c3b0762276f6" (UID: "f7652705-54c0-4597-90a4-c3b0762276f6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:34:44.113235 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:34:44.113200    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7652705-54c0-4597-90a4-c3b0762276f6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f7652705-54c0-4597-90a4-c3b0762276f6" (UID: "f7652705-54c0-4597-90a4-c3b0762276f6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:34:44.113300 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:34:44.113286    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7652705-54c0-4597-90a4-c3b0762276f6-cabundle-cert\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:34:44.113339 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:34:44.113303    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7652705-54c0-4597-90a4-c3b0762276f6-kserve-provision-location\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:34:44.113339 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:34:44.113313    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7652705-54c0-4597-90a4-c3b0762276f6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:34:44.114808 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:34:44.114791    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7652705-54c0-4597-90a4-c3b0762276f6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f7652705-54c0-4597-90a4-c3b0762276f6" (UID: "f7652705-54c0-4597-90a4-c3b0762276f6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:34:44.114869 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:34:44.114850    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7652705-54c0-4597-90a4-c3b0762276f6-kube-api-access-vtwfw" (OuterVolumeSpecName: "kube-api-access-vtwfw") pod "f7652705-54c0-4597-90a4-c3b0762276f6" (UID: "f7652705-54c0-4597-90a4-c3b0762276f6"). InnerVolumeSpecName "kube-api-access-vtwfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:34:44.214588 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:34:44.214493    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7652705-54c0-4597-90a4-c3b0762276f6-proxy-tls\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:34:44.214588 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:34:44.214537    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtwfw\" (UniqueName: \"kubernetes.io/projected/f7652705-54c0-4597-90a4-c3b0762276f6-kube-api-access-vtwfw\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:34:44.377946 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:34:44.377910    2563 generic.go:356] "Generic (PLEG): container finished" podID="f7652705-54c0-4597-90a4-c3b0762276f6" containerID="18ab2591c1f017003e1413cdb572f8e4072cf2fda8d7c92797a0a56ddf5bcb2e" exitCode=137
Sep 17 22:34:44.377946 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:34:44.377962    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-v7q8v" event={"ID":"f7652705-54c0-4597-90a4-c3b0762276f6","Type":"ContainerDied","Data":"18ab2591c1f017003e1413cdb572f8e4072cf2fda8d7c92797a0a56ddf5bcb2e"}
Sep 17 22:34:44.378241 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:34:44.377974    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-v7q8v"
Sep 17 22:34:44.378241 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:34:44.378034    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-v7q8v" event={"ID":"f7652705-54c0-4597-90a4-c3b0762276f6","Type":"ContainerDied","Data":"71e8166989d2a62a4508155809e4bcd52ce7dc4efccaa5c46a08aa7a013ba664"}
Sep 17 22:34:44.378241 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:34:44.378056    2563 scope.go:117] "RemoveContainer" containerID="49948b26f99e89335c0bcba4acc98c5402a9f83b4f87c280ae668c35adfb72db"
Sep 17 22:34:44.386834 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:34:44.386819    2563 scope.go:117] "RemoveContainer" containerID="18ab2591c1f017003e1413cdb572f8e4072cf2fda8d7c92797a0a56ddf5bcb2e"
Sep 17 22:34:44.393886 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:34:44.393865    2563 scope.go:117] "RemoveContainer" containerID="ac41a9c4209a73272145488825cce0c05eb4e85292c47c77f55346e5d5720afc"
Sep 17 22:34:44.398815 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:34:44.398796    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-v7q8v"]
Sep 17 22:34:44.401130 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:34:44.401116    2563 scope.go:117] "RemoveContainer" containerID="49948b26f99e89335c0bcba4acc98c5402a9f83b4f87c280ae668c35adfb72db"
Sep 17 22:34:44.401392 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:34:44.401374    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49948b26f99e89335c0bcba4acc98c5402a9f83b4f87c280ae668c35adfb72db\": container with ID starting with 49948b26f99e89335c0bcba4acc98c5402a9f83b4f87c280ae668c35adfb72db not found: ID does not exist" containerID="49948b26f99e89335c0bcba4acc98c5402a9f83b4f87c280ae668c35adfb72db"
Sep 17 22:34:44.401464 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:34:44.401401    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49948b26f99e89335c0bcba4acc98c5402a9f83b4f87c280ae668c35adfb72db"} err="failed to get container status \"49948b26f99e89335c0bcba4acc98c5402a9f83b4f87c280ae668c35adfb72db\": rpc error: code = NotFound desc = could not find container \"49948b26f99e89335c0bcba4acc98c5402a9f83b4f87c280ae668c35adfb72db\": container with ID starting with 49948b26f99e89335c0bcba4acc98c5402a9f83b4f87c280ae668c35adfb72db not found: ID does not exist"
Sep 17 22:34:44.401464 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:34:44.401424    2563 scope.go:117] "RemoveContainer" containerID="18ab2591c1f017003e1413cdb572f8e4072cf2fda8d7c92797a0a56ddf5bcb2e"
Sep 17 22:34:44.401679 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:34:44.401658    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18ab2591c1f017003e1413cdb572f8e4072cf2fda8d7c92797a0a56ddf5bcb2e\": container with ID starting with 18ab2591c1f017003e1413cdb572f8e4072cf2fda8d7c92797a0a56ddf5bcb2e not found: ID does not exist" containerID="18ab2591c1f017003e1413cdb572f8e4072cf2fda8d7c92797a0a56ddf5bcb2e"
Sep 17 22:34:44.401718 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:34:44.401685    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18ab2591c1f017003e1413cdb572f8e4072cf2fda8d7c92797a0a56ddf5bcb2e"} err="failed to get container status \"18ab2591c1f017003e1413cdb572f8e4072cf2fda8d7c92797a0a56ddf5bcb2e\": rpc error: code = NotFound desc = could not find container \"18ab2591c1f017003e1413cdb572f8e4072cf2fda8d7c92797a0a56ddf5bcb2e\": container with ID starting with 18ab2591c1f017003e1413cdb572f8e4072cf2fda8d7c92797a0a56ddf5bcb2e not found: ID does not exist"
Sep 17 22:34:44.401718 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:34:44.401702    2563 scope.go:117] "RemoveContainer" containerID="ac41a9c4209a73272145488825cce0c05eb4e85292c47c77f55346e5d5720afc"
Sep 17 22:34:44.401915 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:34:44.401899    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac41a9c4209a73272145488825cce0c05eb4e85292c47c77f55346e5d5720afc\": container with ID starting with ac41a9c4209a73272145488825cce0c05eb4e85292c47c77f55346e5d5720afc not found: ID does not exist" containerID="ac41a9c4209a73272145488825cce0c05eb4e85292c47c77f55346e5d5720afc"
Sep 17 22:34:44.401970 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:34:44.401921    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac41a9c4209a73272145488825cce0c05eb4e85292c47c77f55346e5d5720afc"} err="failed to get container status \"ac41a9c4209a73272145488825cce0c05eb4e85292c47c77f55346e5d5720afc\": rpc error: code = NotFound desc = could not find container \"ac41a9c4209a73272145488825cce0c05eb4e85292c47c77f55346e5d5720afc\": container with ID starting with ac41a9c4209a73272145488825cce0c05eb4e85292c47c77f55346e5d5720afc not found: ID does not exist"
Sep 17 22:34:44.404965 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:34:44.404945    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-v7q8v"]
Sep 17 22:34:44.913635 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:34:44.913607    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7652705-54c0-4597-90a4-c3b0762276f6" path="/var/lib/kubelet/pods/f7652705-54c0-4597-90a4-c3b0762276f6/volumes"
Sep 17 22:35:26.858887 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:35:26.858797    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wvgbx"]
Sep 17 22:35:26.859498 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:35:26.859243    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="689c1955-688f-4f55-a8e6-822ac7b779b8" containerName="kube-rbac-proxy"
Sep 17 22:35:26.859498 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:35:26.859263    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="689c1955-688f-4f55-a8e6-822ac7b779b8" containerName="kube-rbac-proxy"
Sep 17 22:35:26.859498 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:35:26.859273    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e92ff6c4-7ef8-4500-a219-6d0e61773381" containerName="kube-rbac-proxy"
Sep 17 22:35:26.859498 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:35:26.859281    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e92ff6c4-7ef8-4500-a219-6d0e61773381" containerName="kube-rbac-proxy"
Sep 17 22:35:26.859498 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:35:26.859292    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e92ff6c4-7ef8-4500-a219-6d0e61773381" containerName="kserve-container"
Sep 17 22:35:26.859498 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:35:26.859301    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e92ff6c4-7ef8-4500-a219-6d0e61773381" containerName="kserve-container"
Sep 17 22:35:26.859498 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:35:26.859319    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7652705-54c0-4597-90a4-c3b0762276f6" containerName="kube-rbac-proxy"
Sep 17 22:35:26.859498 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:35:26.859327    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7652705-54c0-4597-90a4-c3b0762276f6" containerName="kube-rbac-proxy"
Sep 17 22:35:26.859498 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:35:26.859337    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7652705-54c0-4597-90a4-c3b0762276f6" containerName="storage-initializer"
Sep 17 22:35:26.859498 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:35:26.859344    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7652705-54c0-4597-90a4-c3b0762276f6" containerName="storage-initializer"
Sep 17 22:35:26.859498 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:35:26.859354    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7652705-54c0-4597-90a4-c3b0762276f6" containerName="kserve-container"
Sep 17 22:35:26.859498 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:35:26.859362    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7652705-54c0-4597-90a4-c3b0762276f6" containerName="kserve-container"
Sep 17 22:35:26.859498 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:35:26.859373    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="689c1955-688f-4f55-a8e6-822ac7b779b8" containerName="storage-initializer"
Sep 17 22:35:26.859498 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:35:26.859381    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="689c1955-688f-4f55-a8e6-822ac7b779b8" containerName="storage-initializer"
Sep 17 22:35:26.859498 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:35:26.859400    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="689c1955-688f-4f55-a8e6-822ac7b779b8" containerName="kserve-container"
Sep 17 22:35:26.859498 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:35:26.859408    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="689c1955-688f-4f55-a8e6-822ac7b779b8" containerName="kserve-container"
Sep 17 22:35:26.859498 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:35:26.859417    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e92ff6c4-7ef8-4500-a219-6d0e61773381" containerName="storage-initializer"
Sep 17 22:35:26.859498 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:35:26.859425    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e92ff6c4-7ef8-4500-a219-6d0e61773381" containerName="storage-initializer"
Sep 17 22:35:26.859498 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:35:26.859496    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="689c1955-688f-4f55-a8e6-822ac7b779b8" containerName="kserve-container"
Sep 17 22:35:26.859498 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:35:26.859509    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="689c1955-688f-4f55-a8e6-822ac7b779b8" containerName="kube-rbac-proxy"
Sep 17 22:35:26.860571 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:35:26.859517    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7652705-54c0-4597-90a4-c3b0762276f6" containerName="kube-rbac-proxy"
Sep 17 22:35:26.860571 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:35:26.859527    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e92ff6c4-7ef8-4500-a219-6d0e61773381" containerName="kserve-container"
Sep 17 22:35:26.860571 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:35:26.859537    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e92ff6c4-7ef8-4500-a219-6d0e61773381" containerName="kube-rbac-proxy"
Sep 17 22:35:26.860571 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:35:26.859547    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7652705-54c0-4597-90a4-c3b0762276f6" containerName="kserve-container"
Sep 17 22:35:26.863036 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:35:26.863014    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wvgbx"
Sep 17 22:35:26.865455 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:35:26.865427    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-predictor-serving-cert\""
Sep 17 22:35:26.865575 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:35:26.865493    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 17 22:35:26.865575 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:35:26.865546    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 22:35:26.865699 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:35:26.865628    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 22:35:26.865699 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:35:26.865643    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 22:35:26.865699 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:35:26.865664    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:35:26.866615 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:35:26.866600    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v4p4m\""
Sep 17 22:35:26.873083 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:35:26.873062    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wvgbx"]
Sep 17 22:35:26.948885 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:35:26.948851    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ccd5e00-b688-42fb-a2de-294457d99e4b-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wvgbx\" (UID: \"1ccd5e00-b688-42fb-a2de-294457d99e4b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wvgbx"
Sep 17 22:35:26.949065 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:35:26.948897    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ccd5e00-b688-42fb-a2de-294457d99e4b-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wvgbx\" (UID: \"1ccd5e00-b688-42fb-a2de-294457d99e4b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wvgbx"
Sep 17 22:35:26.949065 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:35:26.948944    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ccd5e00-b688-42fb-a2de-294457d99e4b-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wvgbx\" (UID: \"1ccd5e00-b688-42fb-a2de-294457d99e4b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wvgbx"
Sep 17 22:35:26.949065 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:35:26.948975    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c5v6\" (UniqueName: \"kubernetes.io/projected/1ccd5e00-b688-42fb-a2de-294457d99e4b-kube-api-access-4c5v6\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wvgbx\" (UID: \"1ccd5e00-b688-42fb-a2de-294457d99e4b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wvgbx"
Sep 17 22:35:26.949065 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:35:26.949019    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ccd5e00-b688-42fb-a2de-294457d99e4b-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wvgbx\" (UID: \"1ccd5e00-b688-42fb-a2de-294457d99e4b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wvgbx"
Sep 17 22:35:27.049601 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:35:27.049565    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ccd5e00-b688-42fb-a2de-294457d99e4b-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wvgbx\" (UID: \"1ccd5e00-b688-42fb-a2de-294457d99e4b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wvgbx"
Sep 17 22:35:27.049601 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:35:27.049604    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ccd5e00-b688-42fb-a2de-294457d99e4b-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wvgbx\" (UID: \"1ccd5e00-b688-42fb-a2de-294457d99e4b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wvgbx"
Sep 17 22:35:27.049829 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:35:27.049634    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ccd5e00-b688-42fb-a2de-294457d99e4b-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wvgbx\" (UID: \"1ccd5e00-b688-42fb-a2de-294457d99e4b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wvgbx"
Sep 17 22:35:27.049829 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:35:27.049680    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c5v6\" (UniqueName: \"kubernetes.io/projected/1ccd5e00-b688-42fb-a2de-294457d99e4b-kube-api-access-4c5v6\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wvgbx\" (UID: \"1ccd5e00-b688-42fb-a2de-294457d99e4b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wvgbx"
Sep 17 22:35:27.049829 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:35:27.049702    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ccd5e00-b688-42fb-a2de-294457d99e4b-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wvgbx\" (UID: \"1ccd5e00-b688-42fb-a2de-294457d99e4b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wvgbx"
Sep 17 22:35:27.049829 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:35:27.049765    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-serving-cert: secret "isvc-xgboost-v2-mlserver-predictor-serving-cert" not found
Sep 17 22:35:27.049990 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:35:27.049873    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ccd5e00-b688-42fb-a2de-294457d99e4b-proxy-tls podName:1ccd5e00-b688-42fb-a2de-294457d99e4b nodeName:}" failed. No retries permitted until 2026-09-17 22:35:27.549850431 +0000 UTC m=+2238.717803395 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1ccd5e00-b688-42fb-a2de-294457d99e4b-proxy-tls") pod "isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wvgbx" (UID: "1ccd5e00-b688-42fb-a2de-294457d99e4b") : secret "isvc-xgboost-v2-mlserver-predictor-serving-cert" not found
Sep 17 22:35:27.050163 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:35:27.050123    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ccd5e00-b688-42fb-a2de-294457d99e4b-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wvgbx\" (UID: \"1ccd5e00-b688-42fb-a2de-294457d99e4b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wvgbx"
Sep 17 22:35:27.050482 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:35:27.050460    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ccd5e00-b688-42fb-a2de-294457d99e4b-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wvgbx\" (UID: \"1ccd5e00-b688-42fb-a2de-294457d99e4b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wvgbx"
Sep 17 22:35:27.050521 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:35:27.050460    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ccd5e00-b688-42fb-a2de-294457d99e4b-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wvgbx\" (UID: \"1ccd5e00-b688-42fb-a2de-294457d99e4b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wvgbx"
Sep 17 22:35:27.057886 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:35:27.057856    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c5v6\" (UniqueName: \"kubernetes.io/projected/1ccd5e00-b688-42fb-a2de-294457d99e4b-kube-api-access-4c5v6\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wvgbx\" (UID: \"1ccd5e00-b688-42fb-a2de-294457d99e4b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wvgbx"
Sep 17 22:35:27.553931 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:35:27.553892    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ccd5e00-b688-42fb-a2de-294457d99e4b-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wvgbx\" (UID: \"1ccd5e00-b688-42fb-a2de-294457d99e4b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wvgbx"
Sep 17 22:35:27.556353 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:35:27.556318    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ccd5e00-b688-42fb-a2de-294457d99e4b-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wvgbx\" (UID: \"1ccd5e00-b688-42fb-a2de-294457d99e4b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wvgbx"
Sep 17 22:35:27.773302 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:35:27.773267    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wvgbx"
Sep 17 22:35:28.043527 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:35:28.043451    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wvgbx"]
Sep 17 22:35:28.045766 ip-10-0-138-45 kubenswrapper[2563]: W0917 22:35:28.045736    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ccd5e00_b688_42fb_a2de_294457d99e4b.slice/crio-e216b5e2d6c719f6501c4cee5855e44dc397f512f1b94ca54a333a3518e8f0b4 WatchSource:0}: Error finding container e216b5e2d6c719f6501c4cee5855e44dc397f512f1b94ca54a333a3518e8f0b4: Status 404 returned error can't find the container with id e216b5e2d6c719f6501c4cee5855e44dc397f512f1b94ca54a333a3518e8f0b4
Sep 17 22:35:28.543062 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:35:28.543027    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wvgbx" event={"ID":"1ccd5e00-b688-42fb-a2de-294457d99e4b","Type":"ContainerStarted","Data":"2ff0980715908166a6ee9bb26fae80018dcc99485b3861ab635221dfe2afde70"}
Sep 17 22:35:28.543229 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:35:28.543109    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wvgbx" event={"ID":"1ccd5e00-b688-42fb-a2de-294457d99e4b","Type":"ContainerStarted","Data":"e216b5e2d6c719f6501c4cee5855e44dc397f512f1b94ca54a333a3518e8f0b4"}
Sep 17 22:35:32.560884 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:35:32.560852    2563 generic.go:356] "Generic (PLEG): container finished" podID="1ccd5e00-b688-42fb-a2de-294457d99e4b" containerID="2ff0980715908166a6ee9bb26fae80018dcc99485b3861ab635221dfe2afde70" exitCode=0
Sep 17 22:35:32.561293 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:35:32.560895    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wvgbx" event={"ID":"1ccd5e00-b688-42fb-a2de-294457d99e4b","Type":"ContainerDied","Data":"2ff0980715908166a6ee9bb26fae80018dcc99485b3861ab635221dfe2afde70"}
Sep 17 22:35:33.567326 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:35:33.567295    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wvgbx" event={"ID":"1ccd5e00-b688-42fb-a2de-294457d99e4b","Type":"ContainerStarted","Data":"5b935d884df6705201cffe984ba006833523b77b1ce0fa47d8862ffeaa4e2b09"}
Sep 17 22:35:33.567742 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:35:33.567355    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wvgbx" event={"ID":"1ccd5e00-b688-42fb-a2de-294457d99e4b","Type":"ContainerStarted","Data":"81c76ad8e88715225d50878533fb447167a242bcdff9c08546ce364ee6748762"}
Sep 17 22:35:33.567742 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:35:33.567566    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wvgbx"
Sep 17 22:35:33.567742 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:35:33.567625    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wvgbx"
Sep 17 22:35:33.587584 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:35:33.587537    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wvgbx" podStartSLOduration=3.751624861 podStartE2EDuration="7.58752155s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.835896689s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:35:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:35:33.585439078 +0000 UTC m=+2244.753392062" watchObservedRunningTime="2026-09-17 22:35:33.58752155 +0000 UTC m=+2244.755474533"
Sep 17 22:35:39.576116 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:35:39.576086    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wvgbx"
Sep 17 22:36:09.580663 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:09.580629    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wvgbx"
Sep 17 22:36:16.915402 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:16.915366    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wvgbx"]
Sep 17 22:36:16.915788 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:16.915612    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wvgbx" podUID="1ccd5e00-b688-42fb-a2de-294457d99e4b" containerName="kserve-container" containerID="cri-o://81c76ad8e88715225d50878533fb447167a242bcdff9c08546ce364ee6748762" gracePeriod=30
Sep 17 22:36:16.915788 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:16.915684    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wvgbx" podUID="1ccd5e00-b688-42fb-a2de-294457d99e4b" containerName="kube-rbac-proxy" containerID="cri-o://5b935d884df6705201cffe984ba006833523b77b1ce0fa47d8862ffeaa4e2b09" gracePeriod=30
Sep 17 22:36:17.740040 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:17.740009    2563 generic.go:356] "Generic (PLEG): container finished" podID="1ccd5e00-b688-42fb-a2de-294457d99e4b" containerID="5b935d884df6705201cffe984ba006833523b77b1ce0fa47d8862ffeaa4e2b09" exitCode=2
Sep 17 22:36:17.740253 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:17.740087    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wvgbx" event={"ID":"1ccd5e00-b688-42fb-a2de-294457d99e4b","Type":"ContainerDied","Data":"5b935d884df6705201cffe984ba006833523b77b1ce0fa47d8862ffeaa4e2b09"}
Sep 17 22:36:19.572674 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:19.572630    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wvgbx" podUID="1ccd5e00-b688-42fb-a2de-294457d99e4b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.45:8643/healthz\": dial tcp 10.133.0.45:8643: connect: connection refused"
Sep 17 22:36:19.577105 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:19.577072    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wvgbx" podUID="1ccd5e00-b688-42fb-a2de-294457d99e4b" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.45:8080/v2/models/isvc-xgboost-v2-mlserver/ready\": dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 17 22:36:22.113718 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:22.113670    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vg7rj"]
Sep 17 22:36:22.117318 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:22.117297    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vg7rj"
Sep 17 22:36:22.119796 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:22.119777    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-predictor-serving-cert\""
Sep 17 22:36:22.119899 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:22.119781    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 17 22:36:22.127036 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:22.127016    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vg7rj"]
Sep 17 22:36:22.212026 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:22.211995    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81abb38e-21cb-49cc-81bc-5404c2f24368-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-vg7rj\" (UID: \"81abb38e-21cb-49cc-81bc-5404c2f24368\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vg7rj"
Sep 17 22:36:22.212026 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:22.212029    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81abb38e-21cb-49cc-81bc-5404c2f24368-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-vg7rj\" (UID: \"81abb38e-21cb-49cc-81bc-5404c2f24368\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vg7rj"
Sep 17 22:36:22.212256 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:22.212105    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwv6h\" (UniqueName: \"kubernetes.io/projected/81abb38e-21cb-49cc-81bc-5404c2f24368-kube-api-access-rwv6h\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-vg7rj\" (UID: \"81abb38e-21cb-49cc-81bc-5404c2f24368\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vg7rj"
Sep 17 22:36:22.212256 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:22.212162    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81abb38e-21cb-49cc-81bc-5404c2f24368-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-vg7rj\" (UID: \"81abb38e-21cb-49cc-81bc-5404c2f24368\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vg7rj"
Sep 17 22:36:22.212256 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:22.212188    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81abb38e-21cb-49cc-81bc-5404c2f24368-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-vg7rj\" (UID: \"81abb38e-21cb-49cc-81bc-5404c2f24368\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vg7rj"
Sep 17 22:36:22.313439 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:22.313409    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81abb38e-21cb-49cc-81bc-5404c2f24368-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-vg7rj\" (UID: \"81abb38e-21cb-49cc-81bc-5404c2f24368\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vg7rj"
Sep 17 22:36:22.313613 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:22.313446    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81abb38e-21cb-49cc-81bc-5404c2f24368-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-vg7rj\" (UID: \"81abb38e-21cb-49cc-81bc-5404c2f24368\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vg7rj"
Sep 17 22:36:22.313613 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:22.313497    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwv6h\" (UniqueName: \"kubernetes.io/projected/81abb38e-21cb-49cc-81bc-5404c2f24368-kube-api-access-rwv6h\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-vg7rj\" (UID: \"81abb38e-21cb-49cc-81bc-5404c2f24368\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vg7rj"
Sep 17 22:36:22.313613 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:22.313534    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81abb38e-21cb-49cc-81bc-5404c2f24368-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-vg7rj\" (UID: \"81abb38e-21cb-49cc-81bc-5404c2f24368\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vg7rj"
Sep 17 22:36:22.313613 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:22.313555    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81abb38e-21cb-49cc-81bc-5404c2f24368-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-vg7rj\" (UID: \"81abb38e-21cb-49cc-81bc-5404c2f24368\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vg7rj"
Sep 17 22:36:22.313989 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:22.313970    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81abb38e-21cb-49cc-81bc-5404c2f24368-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-vg7rj\" (UID: \"81abb38e-21cb-49cc-81bc-5404c2f24368\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vg7rj"
Sep 17 22:36:22.314208 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:22.314189    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81abb38e-21cb-49cc-81bc-5404c2f24368-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-vg7rj\" (UID: \"81abb38e-21cb-49cc-81bc-5404c2f24368\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vg7rj"
Sep 17 22:36:22.314344 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:22.314324    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81abb38e-21cb-49cc-81bc-5404c2f24368-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-vg7rj\" (UID: \"81abb38e-21cb-49cc-81bc-5404c2f24368\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vg7rj"
Sep 17 22:36:22.315791 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:22.315773    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81abb38e-21cb-49cc-81bc-5404c2f24368-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-vg7rj\" (UID: \"81abb38e-21cb-49cc-81bc-5404c2f24368\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vg7rj"
Sep 17 22:36:22.321690 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:22.321671    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwv6h\" (UniqueName: \"kubernetes.io/projected/81abb38e-21cb-49cc-81bc-5404c2f24368-kube-api-access-rwv6h\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-vg7rj\" (UID: \"81abb38e-21cb-49cc-81bc-5404c2f24368\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vg7rj"
Sep 17 22:36:22.426784 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:22.426752    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vg7rj"
Sep 17 22:36:22.697558 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:22.697530    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vg7rj"]
Sep 17 22:36:22.700105 ip-10-0-138-45 kubenswrapper[2563]: W0917 22:36:22.700078    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81abb38e_21cb_49cc_81bc_5404c2f24368.slice/crio-41e3f2ccba6d5cff016a2e183c104be635d65d3051cc55e5041b2adc5355687d WatchSource:0}: Error finding container 41e3f2ccba6d5cff016a2e183c104be635d65d3051cc55e5041b2adc5355687d: Status 404 returned error can't find the container with id 41e3f2ccba6d5cff016a2e183c104be635d65d3051cc55e5041b2adc5355687d
Sep 17 22:36:22.761897 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:22.761869    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vg7rj" event={"ID":"81abb38e-21cb-49cc-81bc-5404c2f24368","Type":"ContainerStarted","Data":"4a08e7b670de997b468d8f252f5eb42d64797a26d0bf0ca3ecda70a46f0de3f5"}
Sep 17 22:36:22.762036 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:22.761927    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vg7rj" event={"ID":"81abb38e-21cb-49cc-81bc-5404c2f24368","Type":"ContainerStarted","Data":"41e3f2ccba6d5cff016a2e183c104be635d65d3051cc55e5041b2adc5355687d"}
Sep 17 22:36:23.246608 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:23.246578    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wvgbx"
Sep 17 22:36:23.321535 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:23.321504    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4c5v6\" (UniqueName: \"kubernetes.io/projected/1ccd5e00-b688-42fb-a2de-294457d99e4b-kube-api-access-4c5v6\") pod \"1ccd5e00-b688-42fb-a2de-294457d99e4b\" (UID: \"1ccd5e00-b688-42fb-a2de-294457d99e4b\") "
Sep 17 22:36:23.321689 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:23.321555    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ccd5e00-b688-42fb-a2de-294457d99e4b-cabundle-cert\") pod \"1ccd5e00-b688-42fb-a2de-294457d99e4b\" (UID: \"1ccd5e00-b688-42fb-a2de-294457d99e4b\") "
Sep 17 22:36:23.321689 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:23.321610    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ccd5e00-b688-42fb-a2de-294457d99e4b-kube-rbac-proxy-sar-config\") pod \"1ccd5e00-b688-42fb-a2de-294457d99e4b\" (UID: \"1ccd5e00-b688-42fb-a2de-294457d99e4b\") "
Sep 17 22:36:23.321689 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:23.321663    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ccd5e00-b688-42fb-a2de-294457d99e4b-proxy-tls\") pod \"1ccd5e00-b688-42fb-a2de-294457d99e4b\" (UID: \"1ccd5e00-b688-42fb-a2de-294457d99e4b\") "
Sep 17 22:36:23.321875 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:23.321760    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ccd5e00-b688-42fb-a2de-294457d99e4b-kserve-provision-location\") pod \"1ccd5e00-b688-42fb-a2de-294457d99e4b\" (UID: \"1ccd5e00-b688-42fb-a2de-294457d99e4b\") "
Sep 17 22:36:23.322312 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:23.322002    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ccd5e00-b688-42fb-a2de-294457d99e4b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1ccd5e00-b688-42fb-a2de-294457d99e4b" (UID: "1ccd5e00-b688-42fb-a2de-294457d99e4b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:36:23.322312 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:23.322001    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ccd5e00-b688-42fb-a2de-294457d99e4b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1ccd5e00-b688-42fb-a2de-294457d99e4b" (UID: "1ccd5e00-b688-42fb-a2de-294457d99e4b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:36:23.322312 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:23.322225    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ccd5e00-b688-42fb-a2de-294457d99e4b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1ccd5e00-b688-42fb-a2de-294457d99e4b" (UID: "1ccd5e00-b688-42fb-a2de-294457d99e4b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:36:23.323707 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:23.323675    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ccd5e00-b688-42fb-a2de-294457d99e4b-kube-api-access-4c5v6" (OuterVolumeSpecName: "kube-api-access-4c5v6") pod "1ccd5e00-b688-42fb-a2de-294457d99e4b" (UID: "1ccd5e00-b688-42fb-a2de-294457d99e4b"). InnerVolumeSpecName "kube-api-access-4c5v6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:36:23.323919 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:23.323897    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ccd5e00-b688-42fb-a2de-294457d99e4b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1ccd5e00-b688-42fb-a2de-294457d99e4b" (UID: "1ccd5e00-b688-42fb-a2de-294457d99e4b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:36:23.422470 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:23.422443    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ccd5e00-b688-42fb-a2de-294457d99e4b-kserve-provision-location\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:36:23.422470 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:23.422468    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4c5v6\" (UniqueName: \"kubernetes.io/projected/1ccd5e00-b688-42fb-a2de-294457d99e4b-kube-api-access-4c5v6\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:36:23.422470 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:23.422477    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ccd5e00-b688-42fb-a2de-294457d99e4b-cabundle-cert\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:36:23.422689 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:23.422486    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ccd5e00-b688-42fb-a2de-294457d99e4b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:36:23.422689 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:23.422495    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ccd5e00-b688-42fb-a2de-294457d99e4b-proxy-tls\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:36:23.768137 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:23.768102    2563 generic.go:356] "Generic (PLEG): container finished" podID="1ccd5e00-b688-42fb-a2de-294457d99e4b" containerID="81c76ad8e88715225d50878533fb447167a242bcdff9c08546ce364ee6748762" exitCode=0
Sep 17 22:36:23.768349 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:23.768175    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wvgbx"
Sep 17 22:36:23.768349 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:23.768178    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wvgbx" event={"ID":"1ccd5e00-b688-42fb-a2de-294457d99e4b","Type":"ContainerDied","Data":"81c76ad8e88715225d50878533fb447167a242bcdff9c08546ce364ee6748762"}
Sep 17 22:36:23.768349 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:23.768246    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wvgbx" event={"ID":"1ccd5e00-b688-42fb-a2de-294457d99e4b","Type":"ContainerDied","Data":"e216b5e2d6c719f6501c4cee5855e44dc397f512f1b94ca54a333a3518e8f0b4"}
Sep 17 22:36:23.768349 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:23.768283    2563 scope.go:117] "RemoveContainer" containerID="5b935d884df6705201cffe984ba006833523b77b1ce0fa47d8862ffeaa4e2b09"
Sep 17 22:36:23.776955 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:23.776936    2563 scope.go:117] "RemoveContainer" containerID="81c76ad8e88715225d50878533fb447167a242bcdff9c08546ce364ee6748762"
Sep 17 22:36:23.784100 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:23.784083    2563 scope.go:117] "RemoveContainer" containerID="2ff0980715908166a6ee9bb26fae80018dcc99485b3861ab635221dfe2afde70"
Sep 17 22:36:23.790739 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:23.790713    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wvgbx"]
Sep 17 22:36:23.791523 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:23.791490    2563 scope.go:117] "RemoveContainer" containerID="5b935d884df6705201cffe984ba006833523b77b1ce0fa47d8862ffeaa4e2b09"
Sep 17 22:36:23.791779 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:36:23.791759    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b935d884df6705201cffe984ba006833523b77b1ce0fa47d8862ffeaa4e2b09\": container with ID starting with 5b935d884df6705201cffe984ba006833523b77b1ce0fa47d8862ffeaa4e2b09 not found: ID does not exist" containerID="5b935d884df6705201cffe984ba006833523b77b1ce0fa47d8862ffeaa4e2b09"
Sep 17 22:36:23.791922 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:23.791784    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b935d884df6705201cffe984ba006833523b77b1ce0fa47d8862ffeaa4e2b09"} err="failed to get container status \"5b935d884df6705201cffe984ba006833523b77b1ce0fa47d8862ffeaa4e2b09\": rpc error: code = NotFound desc = could not find container \"5b935d884df6705201cffe984ba006833523b77b1ce0fa47d8862ffeaa4e2b09\": container with ID starting with 5b935d884df6705201cffe984ba006833523b77b1ce0fa47d8862ffeaa4e2b09 not found: ID does not exist"
Sep 17 22:36:23.791922 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:23.791800    2563 scope.go:117] "RemoveContainer" containerID="81c76ad8e88715225d50878533fb447167a242bcdff9c08546ce364ee6748762"
Sep 17 22:36:23.792099 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:36:23.792079    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81c76ad8e88715225d50878533fb447167a242bcdff9c08546ce364ee6748762\": container with ID starting with 81c76ad8e88715225d50878533fb447167a242bcdff9c08546ce364ee6748762 not found: ID does not exist" containerID="81c76ad8e88715225d50878533fb447167a242bcdff9c08546ce364ee6748762"
Sep 17 22:36:23.792429 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:23.792119    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81c76ad8e88715225d50878533fb447167a242bcdff9c08546ce364ee6748762"} err="failed to get container status \"81c76ad8e88715225d50878533fb447167a242bcdff9c08546ce364ee6748762\": rpc error: code = NotFound desc = could not find container \"81c76ad8e88715225d50878533fb447167a242bcdff9c08546ce364ee6748762\": container with ID starting with 81c76ad8e88715225d50878533fb447167a242bcdff9c08546ce364ee6748762 not found: ID does not exist"
Sep 17 22:36:23.792429 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:23.792187    2563 scope.go:117] "RemoveContainer" containerID="2ff0980715908166a6ee9bb26fae80018dcc99485b3861ab635221dfe2afde70"
Sep 17 22:36:23.792581 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:36:23.792497    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ff0980715908166a6ee9bb26fae80018dcc99485b3861ab635221dfe2afde70\": container with ID starting with 2ff0980715908166a6ee9bb26fae80018dcc99485b3861ab635221dfe2afde70 not found: ID does not exist" containerID="2ff0980715908166a6ee9bb26fae80018dcc99485b3861ab635221dfe2afde70"
Sep 17 22:36:23.792581 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:23.792525    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ff0980715908166a6ee9bb26fae80018dcc99485b3861ab635221dfe2afde70"} err="failed to get container status \"2ff0980715908166a6ee9bb26fae80018dcc99485b3861ab635221dfe2afde70\": rpc error: code = NotFound desc = could not find container \"2ff0980715908166a6ee9bb26fae80018dcc99485b3861ab635221dfe2afde70\": container with ID starting with 2ff0980715908166a6ee9bb26fae80018dcc99485b3861ab635221dfe2afde70 not found: ID does not exist"
Sep 17 22:36:23.793997 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:23.793975    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wvgbx"]
Sep 17 22:36:24.914274 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:24.914245    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ccd5e00-b688-42fb-a2de-294457d99e4b" path="/var/lib/kubelet/pods/1ccd5e00-b688-42fb-a2de-294457d99e4b/volumes"
Sep 17 22:36:26.782981 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:26.782950    2563 generic.go:356] "Generic (PLEG): container finished" podID="81abb38e-21cb-49cc-81bc-5404c2f24368" containerID="4a08e7b670de997b468d8f252f5eb42d64797a26d0bf0ca3ecda70a46f0de3f5" exitCode=0
Sep 17 22:36:26.783339 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:26.783013    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vg7rj" event={"ID":"81abb38e-21cb-49cc-81bc-5404c2f24368","Type":"ContainerDied","Data":"4a08e7b670de997b468d8f252f5eb42d64797a26d0bf0ca3ecda70a46f0de3f5"}
Sep 17 22:36:27.788902 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:27.788872    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vg7rj" event={"ID":"81abb38e-21cb-49cc-81bc-5404c2f24368","Type":"ContainerStarted","Data":"756623d0975385588a58c1bdcd77ab80b6f31d670462d08cc41deb9f53beedf3"}
Sep 17 22:36:27.789278 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:27.788926    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vg7rj" event={"ID":"81abb38e-21cb-49cc-81bc-5404c2f24368","Type":"ContainerStarted","Data":"24d634b4261d513f547da360e95fff4b43471e04339a7e711ea64e1c18e9d492"}
Sep 17 22:36:27.789278 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:27.789114    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vg7rj"
Sep 17 22:36:27.789278 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:27.789184    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vg7rj"
Sep 17 22:36:27.808915 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:27.808872    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vg7rj" podStartSLOduration=1.909021176 podStartE2EDuration="5.808858897s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.899837721s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:36:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:36:27.806562812 +0000 UTC m=+2298.974515804" watchObservedRunningTime="2026-09-17 22:36:27.808858897 +0000 UTC m=+2298.976811879"
Sep 17 22:36:33.798278 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:36:33.798252    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vg7rj"
Sep 17 22:37:03.802404 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:37:03.802331    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vg7rj"
Sep 17 22:37:12.183136 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:37:12.183099    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vg7rj"]
Sep 17 22:37:12.183646 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:37:12.183417    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vg7rj" podUID="81abb38e-21cb-49cc-81bc-5404c2f24368" containerName="kserve-container" containerID="cri-o://24d634b4261d513f547da360e95fff4b43471e04339a7e711ea64e1c18e9d492" gracePeriod=30
Sep 17 22:37:12.183646 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:37:12.183493    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vg7rj" podUID="81abb38e-21cb-49cc-81bc-5404c2f24368" containerName="kube-rbac-proxy" containerID="cri-o://756623d0975385588a58c1bdcd77ab80b6f31d670462d08cc41deb9f53beedf3" gracePeriod=30
Sep 17 22:37:12.969362 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:37:12.969326    2563 generic.go:356] "Generic (PLEG): container finished" podID="81abb38e-21cb-49cc-81bc-5404c2f24368" containerID="756623d0975385588a58c1bdcd77ab80b6f31d670462d08cc41deb9f53beedf3" exitCode=2
Sep 17 22:37:12.969533 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:37:12.969397    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vg7rj" event={"ID":"81abb38e-21cb-49cc-81bc-5404c2f24368","Type":"ContainerDied","Data":"756623d0975385588a58c1bdcd77ab80b6f31d670462d08cc41deb9f53beedf3"}
Sep 17 22:37:13.793455 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:37:13.793417    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vg7rj" podUID="81abb38e-21cb-49cc-81bc-5404c2f24368" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.46:8643/healthz\": dial tcp 10.133.0.46:8643: connect: connection refused"
Sep 17 22:37:18.620608 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:37:18.620586    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vg7rj"
Sep 17 22:37:18.670090 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:37:18.670065    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81abb38e-21cb-49cc-81bc-5404c2f24368-kserve-provision-location\") pod \"81abb38e-21cb-49cc-81bc-5404c2f24368\" (UID: \"81abb38e-21cb-49cc-81bc-5404c2f24368\") "
Sep 17 22:37:18.670230 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:37:18.670111    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81abb38e-21cb-49cc-81bc-5404c2f24368-cabundle-cert\") pod \"81abb38e-21cb-49cc-81bc-5404c2f24368\" (UID: \"81abb38e-21cb-49cc-81bc-5404c2f24368\") "
Sep 17 22:37:18.670230 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:37:18.670164    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81abb38e-21cb-49cc-81bc-5404c2f24368-proxy-tls\") pod \"81abb38e-21cb-49cc-81bc-5404c2f24368\" (UID: \"81abb38e-21cb-49cc-81bc-5404c2f24368\") "
Sep 17 22:37:18.670230 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:37:18.670190    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rwv6h\" (UniqueName: \"kubernetes.io/projected/81abb38e-21cb-49cc-81bc-5404c2f24368-kube-api-access-rwv6h\") pod \"81abb38e-21cb-49cc-81bc-5404c2f24368\" (UID: \"81abb38e-21cb-49cc-81bc-5404c2f24368\") "
Sep 17 22:37:18.670230 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:37:18.670205    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81abb38e-21cb-49cc-81bc-5404c2f24368-kube-rbac-proxy-sar-config\") pod \"81abb38e-21cb-49cc-81bc-5404c2f24368\" (UID: \"81abb38e-21cb-49cc-81bc-5404c2f24368\") "
Sep 17 22:37:18.670519 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:37:18.670379    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81abb38e-21cb-49cc-81bc-5404c2f24368-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "81abb38e-21cb-49cc-81bc-5404c2f24368" (UID: "81abb38e-21cb-49cc-81bc-5404c2f24368"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:37:18.670583 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:37:18.670521    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81abb38e-21cb-49cc-81bc-5404c2f24368-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "81abb38e-21cb-49cc-81bc-5404c2f24368" (UID: "81abb38e-21cb-49cc-81bc-5404c2f24368"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:37:18.670583 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:37:18.670572    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81abb38e-21cb-49cc-81bc-5404c2f24368-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "81abb38e-21cb-49cc-81bc-5404c2f24368" (UID: "81abb38e-21cb-49cc-81bc-5404c2f24368"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:37:18.672202 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:37:18.672180    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81abb38e-21cb-49cc-81bc-5404c2f24368-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "81abb38e-21cb-49cc-81bc-5404c2f24368" (UID: "81abb38e-21cb-49cc-81bc-5404c2f24368"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:37:18.672314 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:37:18.672293    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81abb38e-21cb-49cc-81bc-5404c2f24368-kube-api-access-rwv6h" (OuterVolumeSpecName: "kube-api-access-rwv6h") pod "81abb38e-21cb-49cc-81bc-5404c2f24368" (UID: "81abb38e-21cb-49cc-81bc-5404c2f24368"). InnerVolumeSpecName "kube-api-access-rwv6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:37:18.770842 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:37:18.770766    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81abb38e-21cb-49cc-81bc-5404c2f24368-kserve-provision-location\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:37:18.770842 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:37:18.770794    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81abb38e-21cb-49cc-81bc-5404c2f24368-cabundle-cert\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:37:18.770842 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:37:18.770806    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81abb38e-21cb-49cc-81bc-5404c2f24368-proxy-tls\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:37:18.770842 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:37:18.770817    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwv6h\" (UniqueName: \"kubernetes.io/projected/81abb38e-21cb-49cc-81bc-5404c2f24368-kube-api-access-rwv6h\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:37:18.770842 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:37:18.770829    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81abb38e-21cb-49cc-81bc-5404c2f24368-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:37:18.994888 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:37:18.994851    2563 generic.go:356] "Generic (PLEG): container finished" podID="81abb38e-21cb-49cc-81bc-5404c2f24368" containerID="24d634b4261d513f547da360e95fff4b43471e04339a7e711ea64e1c18e9d492" exitCode=0
Sep 17 22:37:18.995032 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:37:18.994897    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vg7rj" event={"ID":"81abb38e-21cb-49cc-81bc-5404c2f24368","Type":"ContainerDied","Data":"24d634b4261d513f547da360e95fff4b43471e04339a7e711ea64e1c18e9d492"}
Sep 17 22:37:18.995032 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:37:18.994909    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vg7rj"
Sep 17 22:37:18.995032 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:37:18.994940    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vg7rj" event={"ID":"81abb38e-21cb-49cc-81bc-5404c2f24368","Type":"ContainerDied","Data":"41e3f2ccba6d5cff016a2e183c104be635d65d3051cc55e5041b2adc5355687d"}
Sep 17 22:37:18.995032 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:37:18.994965    2563 scope.go:117] "RemoveContainer" containerID="756623d0975385588a58c1bdcd77ab80b6f31d670462d08cc41deb9f53beedf3"
Sep 17 22:37:19.002949 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:37:19.002932    2563 scope.go:117] "RemoveContainer" containerID="24d634b4261d513f547da360e95fff4b43471e04339a7e711ea64e1c18e9d492"
Sep 17 22:37:19.010014 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:37:19.009998    2563 scope.go:117] "RemoveContainer" containerID="4a08e7b670de997b468d8f252f5eb42d64797a26d0bf0ca3ecda70a46f0de3f5"
Sep 17 22:37:19.011053 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:37:19.011029    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vg7rj"]
Sep 17 22:37:19.014630 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:37:19.014610    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vg7rj"]
Sep 17 22:37:19.017452 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:37:19.017436    2563 scope.go:117] "RemoveContainer" containerID="756623d0975385588a58c1bdcd77ab80b6f31d670462d08cc41deb9f53beedf3"
Sep 17 22:37:19.017716 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:37:19.017700    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"756623d0975385588a58c1bdcd77ab80b6f31d670462d08cc41deb9f53beedf3\": container with ID starting with 756623d0975385588a58c1bdcd77ab80b6f31d670462d08cc41deb9f53beedf3 not found: ID does not exist" containerID="756623d0975385588a58c1bdcd77ab80b6f31d670462d08cc41deb9f53beedf3"
Sep 17 22:37:19.017776 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:37:19.017722    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"756623d0975385588a58c1bdcd77ab80b6f31d670462d08cc41deb9f53beedf3"} err="failed to get container status \"756623d0975385588a58c1bdcd77ab80b6f31d670462d08cc41deb9f53beedf3\": rpc error: code = NotFound desc = could not find container \"756623d0975385588a58c1bdcd77ab80b6f31d670462d08cc41deb9f53beedf3\": container with ID starting with 756623d0975385588a58c1bdcd77ab80b6f31d670462d08cc41deb9f53beedf3 not found: ID does not exist"
Sep 17 22:37:19.017776 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:37:19.017738    2563 scope.go:117] "RemoveContainer" containerID="24d634b4261d513f547da360e95fff4b43471e04339a7e711ea64e1c18e9d492"
Sep 17 22:37:19.018003 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:37:19.017985    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24d634b4261d513f547da360e95fff4b43471e04339a7e711ea64e1c18e9d492\": container with ID starting with 24d634b4261d513f547da360e95fff4b43471e04339a7e711ea64e1c18e9d492 not found: ID does not exist" containerID="24d634b4261d513f547da360e95fff4b43471e04339a7e711ea64e1c18e9d492"
Sep 17 22:37:19.018050 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:37:19.018008    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24d634b4261d513f547da360e95fff4b43471e04339a7e711ea64e1c18e9d492"} err="failed to get container status \"24d634b4261d513f547da360e95fff4b43471e04339a7e711ea64e1c18e9d492\": rpc error: code = NotFound desc = could not find container \"24d634b4261d513f547da360e95fff4b43471e04339a7e711ea64e1c18e9d492\": container with ID starting with 24d634b4261d513f547da360e95fff4b43471e04339a7e711ea64e1c18e9d492 not found: ID does not exist"
Sep 17 22:37:19.018050 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:37:19.018024    2563 scope.go:117] "RemoveContainer" containerID="4a08e7b670de997b468d8f252f5eb42d64797a26d0bf0ca3ecda70a46f0de3f5"
Sep 17 22:37:19.018326 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:37:19.018304    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a08e7b670de997b468d8f252f5eb42d64797a26d0bf0ca3ecda70a46f0de3f5\": container with ID starting with 4a08e7b670de997b468d8f252f5eb42d64797a26d0bf0ca3ecda70a46f0de3f5 not found: ID does not exist" containerID="4a08e7b670de997b468d8f252f5eb42d64797a26d0bf0ca3ecda70a46f0de3f5"
Sep 17 22:37:19.018387 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:37:19.018334    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a08e7b670de997b468d8f252f5eb42d64797a26d0bf0ca3ecda70a46f0de3f5"} err="failed to get container status \"4a08e7b670de997b468d8f252f5eb42d64797a26d0bf0ca3ecda70a46f0de3f5\": rpc error: code = NotFound desc = could not find container \"4a08e7b670de997b468d8f252f5eb42d64797a26d0bf0ca3ecda70a46f0de3f5\": container with ID starting with 4a08e7b670de997b468d8f252f5eb42d64797a26d0bf0ca3ecda70a46f0de3f5 not found: ID does not exist"
Sep 17 22:37:20.913280 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:37:20.913250    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81abb38e-21cb-49cc-81bc-5404c2f24368" path="/var/lib/kubelet/pods/81abb38e-21cb-49cc-81bc-5404c2f24368/volumes"
Sep 17 22:38:34.846795 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:34.846720    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-9b4tl"]
Sep 17 22:38:34.847255 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:34.847054    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ccd5e00-b688-42fb-a2de-294457d99e4b" containerName="kube-rbac-proxy"
Sep 17 22:38:34.847255 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:34.847069    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ccd5e00-b688-42fb-a2de-294457d99e4b" containerName="kube-rbac-proxy"
Sep 17 22:38:34.847255 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:34.847079    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ccd5e00-b688-42fb-a2de-294457d99e4b" containerName="storage-initializer"
Sep 17 22:38:34.847255 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:34.847085    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ccd5e00-b688-42fb-a2de-294457d99e4b" containerName="storage-initializer"
Sep 17 22:38:34.847255 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:34.847092    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ccd5e00-b688-42fb-a2de-294457d99e4b" containerName="kserve-container"
Sep 17 22:38:34.847255 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:34.847098    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ccd5e00-b688-42fb-a2de-294457d99e4b" containerName="kserve-container"
Sep 17 22:38:34.847255 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:34.847106    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81abb38e-21cb-49cc-81bc-5404c2f24368" containerName="kube-rbac-proxy"
Sep 17 22:38:34.847255 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:34.847111    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="81abb38e-21cb-49cc-81bc-5404c2f24368" containerName="kube-rbac-proxy"
Sep 17 22:38:34.847255 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:34.847120    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81abb38e-21cb-49cc-81bc-5404c2f24368" containerName="kserve-container"
Sep 17 22:38:34.847255 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:34.847125    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="81abb38e-21cb-49cc-81bc-5404c2f24368" containerName="kserve-container"
Sep 17 22:38:34.847255 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:34.847153    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81abb38e-21cb-49cc-81bc-5404c2f24368" containerName="storage-initializer"
Sep 17 22:38:34.847255 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:34.847161    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="81abb38e-21cb-49cc-81bc-5404c2f24368" containerName="storage-initializer"
Sep 17 22:38:34.847255 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:34.847205    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="81abb38e-21cb-49cc-81bc-5404c2f24368" containerName="kube-rbac-proxy"
Sep 17 22:38:34.847255 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:34.847213    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="81abb38e-21cb-49cc-81bc-5404c2f24368" containerName="kserve-container"
Sep 17 22:38:34.847255 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:34.847220    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ccd5e00-b688-42fb-a2de-294457d99e4b" containerName="kserve-container"
Sep 17 22:38:34.847255 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:34.847227    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ccd5e00-b688-42fb-a2de-294457d99e4b" containerName="kube-rbac-proxy"
Sep 17 22:38:34.850402 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:34.850387    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-9b4tl"
Sep 17 22:38:34.852828 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:34.852806    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:38:34.852963 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:34.852840    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 22:38:34.852963 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:34.852876    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 22:38:34.852963 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:34.852907    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 22:38:34.852963 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:34.852923    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v4p4m\""
Sep 17 22:38:34.852963 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:34.852877    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 17 22:38:34.853236 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:34.853018    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-predictor-serving-cert\""
Sep 17 22:38:34.860882 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:34.860862    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-9b4tl"]
Sep 17 22:38:34.976901 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:34.976871    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e295041-f6e6-4c4e-8138-8d82e7e4914a-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-9b4tl\" (UID: \"1e295041-f6e6-4c4e-8138-8d82e7e4914a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-9b4tl"
Sep 17 22:38:34.977047 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:34.976909    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e295041-f6e6-4c4e-8138-8d82e7e4914a-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-9b4tl\" (UID: \"1e295041-f6e6-4c4e-8138-8d82e7e4914a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-9b4tl"
Sep 17 22:38:34.977047 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:34.976932    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vxfc\" (UniqueName: \"kubernetes.io/projected/1e295041-f6e6-4c4e-8138-8d82e7e4914a-kube-api-access-8vxfc\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-9b4tl\" (UID: \"1e295041-f6e6-4c4e-8138-8d82e7e4914a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-9b4tl"
Sep 17 22:38:34.977047 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:34.977020    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e295041-f6e6-4c4e-8138-8d82e7e4914a-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-9b4tl\" (UID: \"1e295041-f6e6-4c4e-8138-8d82e7e4914a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-9b4tl"
Sep 17 22:38:34.977167 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:34.977059    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e295041-f6e6-4c4e-8138-8d82e7e4914a-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-9b4tl\" (UID: \"1e295041-f6e6-4c4e-8138-8d82e7e4914a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-9b4tl"
Sep 17 22:38:35.078215 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:35.078185    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e295041-f6e6-4c4e-8138-8d82e7e4914a-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-9b4tl\" (UID: \"1e295041-f6e6-4c4e-8138-8d82e7e4914a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-9b4tl"
Sep 17 22:38:35.078380 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:35.078225    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e295041-f6e6-4c4e-8138-8d82e7e4914a-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-9b4tl\" (UID: \"1e295041-f6e6-4c4e-8138-8d82e7e4914a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-9b4tl"
Sep 17 22:38:35.078380 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:35.078248    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vxfc\" (UniqueName: \"kubernetes.io/projected/1e295041-f6e6-4c4e-8138-8d82e7e4914a-kube-api-access-8vxfc\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-9b4tl\" (UID: \"1e295041-f6e6-4c4e-8138-8d82e7e4914a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-9b4tl"
Sep 17 22:38:35.078380 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:35.078272    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e295041-f6e6-4c4e-8138-8d82e7e4914a-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-9b4tl\" (UID: \"1e295041-f6e6-4c4e-8138-8d82e7e4914a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-9b4tl"
Sep 17 22:38:35.078380 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:35.078295    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e295041-f6e6-4c4e-8138-8d82e7e4914a-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-9b4tl\" (UID: \"1e295041-f6e6-4c4e-8138-8d82e7e4914a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-9b4tl"
Sep 17 22:38:35.078734 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:35.078701    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e295041-f6e6-4c4e-8138-8d82e7e4914a-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-9b4tl\" (UID: \"1e295041-f6e6-4c4e-8138-8d82e7e4914a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-9b4tl"
Sep 17 22:38:35.078947 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:35.078926    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e295041-f6e6-4c4e-8138-8d82e7e4914a-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-9b4tl\" (UID: \"1e295041-f6e6-4c4e-8138-8d82e7e4914a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-9b4tl"
Sep 17 22:38:35.078947 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:35.078939    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e295041-f6e6-4c4e-8138-8d82e7e4914a-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-9b4tl\" (UID: \"1e295041-f6e6-4c4e-8138-8d82e7e4914a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-9b4tl"
Sep 17 22:38:35.080580 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:35.080565    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e295041-f6e6-4c4e-8138-8d82e7e4914a-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-9b4tl\" (UID: \"1e295041-f6e6-4c4e-8138-8d82e7e4914a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-9b4tl"
Sep 17 22:38:35.085906 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:35.085885    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vxfc\" (UniqueName: \"kubernetes.io/projected/1e295041-f6e6-4c4e-8138-8d82e7e4914a-kube-api-access-8vxfc\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-9b4tl\" (UID: \"1e295041-f6e6-4c4e-8138-8d82e7e4914a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-9b4tl"
Sep 17 22:38:35.160523 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:35.160488    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-9b4tl"
Sep 17 22:38:35.434903 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:35.434874    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-9b4tl"]
Sep 17 22:38:36.290057 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:36.290004    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-9b4tl" event={"ID":"1e295041-f6e6-4c4e-8138-8d82e7e4914a","Type":"ContainerStarted","Data":"1ad06f0d678cff6b0e2fde749854fde07c47123cca6255aec570096312e995c1"}
Sep 17 22:38:36.290470 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:36.290119    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-9b4tl" event={"ID":"1e295041-f6e6-4c4e-8138-8d82e7e4914a","Type":"ContainerStarted","Data":"cd29455e324fd0870e511b3e71436c2aaa8fc5996e28b38c74848fc823d6edd9"}
Sep 17 22:38:39.303825 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:39.303751    2563 generic.go:356] "Generic (PLEG): container finished" podID="1e295041-f6e6-4c4e-8138-8d82e7e4914a" containerID="1ad06f0d678cff6b0e2fde749854fde07c47123cca6255aec570096312e995c1" exitCode=0
Sep 17 22:38:39.303825 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:39.303786    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-9b4tl" event={"ID":"1e295041-f6e6-4c4e-8138-8d82e7e4914a","Type":"ContainerDied","Data":"1ad06f0d678cff6b0e2fde749854fde07c47123cca6255aec570096312e995c1"}
Sep 17 22:38:40.309803 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:40.309763    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-9b4tl" event={"ID":"1e295041-f6e6-4c4e-8138-8d82e7e4914a","Type":"ContainerStarted","Data":"a735e08f82928f71213697825d78e77f5140e62d916bed60935a893118df6b8b"}
Sep 17 22:38:40.310292 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:40.309831    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-9b4tl" event={"ID":"1e295041-f6e6-4c4e-8138-8d82e7e4914a","Type":"ContainerStarted","Data":"5e841a3e785f85f5e9d7613fe10c4156a26d67e4b3f0186dcaecfcb7d8d241a4"}
Sep 17 22:38:40.310292 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:40.310024    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-9b4tl"
Sep 17 22:38:40.329079 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:40.329033    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-9b4tl" podStartSLOduration=2.665923035 podStartE2EDuration="6.329016817s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.663093782s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:38:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:38:40.328099073 +0000 UTC m=+2431.496052060" watchObservedRunningTime="2026-09-17 22:38:40.329016817 +0000 UTC m=+2431.496969798"
Sep 17 22:38:41.313935 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:41.313900    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-9b4tl"
Sep 17 22:38:43.111343 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:43.111314    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fsdj4"]
Sep 17 22:38:43.114496 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:43.114476    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fsdj4"
Sep 17 22:38:43.116793 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:43.116773    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-predictor-serving-cert\""
Sep 17 22:38:43.116920 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:43.116829    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 17 22:38:43.124269 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:43.124246    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fsdj4"]
Sep 17 22:38:43.244664 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:43.244624    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/beffab2d-4a96-4234-be69-005f9c7f90ec-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-fsdj4\" (UID: \"beffab2d-4a96-4234-be69-005f9c7f90ec\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fsdj4"
Sep 17 22:38:43.244845 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:43.244676    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/beffab2d-4a96-4234-be69-005f9c7f90ec-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-fsdj4\" (UID: \"beffab2d-4a96-4234-be69-005f9c7f90ec\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fsdj4"
Sep 17 22:38:43.244845 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:43.244756    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/beffab2d-4a96-4234-be69-005f9c7f90ec-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-fsdj4\" (UID: \"beffab2d-4a96-4234-be69-005f9c7f90ec\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fsdj4"
Sep 17 22:38:43.244845 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:43.244820    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/beffab2d-4a96-4234-be69-005f9c7f90ec-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-fsdj4\" (UID: \"beffab2d-4a96-4234-be69-005f9c7f90ec\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fsdj4"
Sep 17 22:38:43.244845 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:43.244839    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9fwq\" (UniqueName: \"kubernetes.io/projected/beffab2d-4a96-4234-be69-005f9c7f90ec-kube-api-access-n9fwq\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-fsdj4\" (UID: \"beffab2d-4a96-4234-be69-005f9c7f90ec\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fsdj4"
Sep 17 22:38:43.345607 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:43.345562    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/beffab2d-4a96-4234-be69-005f9c7f90ec-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-fsdj4\" (UID: \"beffab2d-4a96-4234-be69-005f9c7f90ec\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fsdj4"
Sep 17 22:38:43.345607 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:43.345615    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/beffab2d-4a96-4234-be69-005f9c7f90ec-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-fsdj4\" (UID: \"beffab2d-4a96-4234-be69-005f9c7f90ec\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fsdj4"
Sep 17 22:38:43.345870 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:43.345656    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/beffab2d-4a96-4234-be69-005f9c7f90ec-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-fsdj4\" (UID: \"beffab2d-4a96-4234-be69-005f9c7f90ec\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fsdj4"
Sep 17 22:38:43.345870 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:43.345707    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/beffab2d-4a96-4234-be69-005f9c7f90ec-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-fsdj4\" (UID: \"beffab2d-4a96-4234-be69-005f9c7f90ec\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fsdj4"
Sep 17 22:38:43.345870 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:43.345729    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9fwq\" (UniqueName: \"kubernetes.io/projected/beffab2d-4a96-4234-be69-005f9c7f90ec-kube-api-access-n9fwq\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-fsdj4\" (UID: \"beffab2d-4a96-4234-be69-005f9c7f90ec\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fsdj4"
Sep 17 22:38:43.346086 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:43.346011    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/beffab2d-4a96-4234-be69-005f9c7f90ec-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-fsdj4\" (UID: \"beffab2d-4a96-4234-be69-005f9c7f90ec\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fsdj4"
Sep 17 22:38:43.346411 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:43.346386    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/beffab2d-4a96-4234-be69-005f9c7f90ec-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-fsdj4\" (UID: \"beffab2d-4a96-4234-be69-005f9c7f90ec\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fsdj4"
Sep 17 22:38:43.346411 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:43.346406    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/beffab2d-4a96-4234-be69-005f9c7f90ec-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-fsdj4\" (UID: \"beffab2d-4a96-4234-be69-005f9c7f90ec\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fsdj4"
Sep 17 22:38:43.348095 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:43.348067    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/beffab2d-4a96-4234-be69-005f9c7f90ec-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-fsdj4\" (UID: \"beffab2d-4a96-4234-be69-005f9c7f90ec\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fsdj4"
Sep 17 22:38:43.354286 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:43.354263    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9fwq\" (UniqueName: \"kubernetes.io/projected/beffab2d-4a96-4234-be69-005f9c7f90ec-kube-api-access-n9fwq\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-fsdj4\" (UID: \"beffab2d-4a96-4234-be69-005f9c7f90ec\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fsdj4"
Sep 17 22:38:43.425315 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:43.425283    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fsdj4"
Sep 17 22:38:43.698545 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:43.698329    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fsdj4"]
Sep 17 22:38:43.701098 ip-10-0-138-45 kubenswrapper[2563]: W0917 22:38:43.701069    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeffab2d_4a96_4234_be69_005f9c7f90ec.slice/crio-b2ea547724ced5b83472cd47294acf474dd8d8e27a4edbfb78eb19b35c8dcc20 WatchSource:0}: Error finding container b2ea547724ced5b83472cd47294acf474dd8d8e27a4edbfb78eb19b35c8dcc20: Status 404 returned error can't find the container with id b2ea547724ced5b83472cd47294acf474dd8d8e27a4edbfb78eb19b35c8dcc20
Sep 17 22:38:44.327907 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:44.327869    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fsdj4" event={"ID":"beffab2d-4a96-4234-be69-005f9c7f90ec","Type":"ContainerStarted","Data":"8a888b91247023fffdaa50ce009bda98c8e9702a64bda32874d3c38b70b4e124"}
Sep 17 22:38:44.328343 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:44.327935    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fsdj4" event={"ID":"beffab2d-4a96-4234-be69-005f9c7f90ec","Type":"ContainerStarted","Data":"b2ea547724ced5b83472cd47294acf474dd8d8e27a4edbfb78eb19b35c8dcc20"}
Sep 17 22:38:47.323452 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:47.323421    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-9b4tl"
Sep 17 22:38:48.345351 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:48.345316    2563 generic.go:356] "Generic (PLEG): container finished" podID="beffab2d-4a96-4234-be69-005f9c7f90ec" containerID="8a888b91247023fffdaa50ce009bda98c8e9702a64bda32874d3c38b70b4e124" exitCode=0
Sep 17 22:38:48.345751 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:48.345386    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fsdj4" event={"ID":"beffab2d-4a96-4234-be69-005f9c7f90ec","Type":"ContainerDied","Data":"8a888b91247023fffdaa50ce009bda98c8e9702a64bda32874d3c38b70b4e124"}
Sep 17 22:38:49.351260 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:49.351227    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fsdj4" event={"ID":"beffab2d-4a96-4234-be69-005f9c7f90ec","Type":"ContainerStarted","Data":"83146943ea4bfa50df9a16f17f6977ec7ad995906027dedc39de87d52eeabf77"}
Sep 17 22:38:49.351662 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:49.351285    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fsdj4" event={"ID":"beffab2d-4a96-4234-be69-005f9c7f90ec","Type":"ContainerStarted","Data":"86e689b3693164f17392a43396f2d1d1c652dd80e3592982b377c9fd9246e33d"}
Sep 17 22:38:49.351662 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:49.351504    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fsdj4"
Sep 17 22:38:49.351662 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:49.351642    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fsdj4"
Sep 17 22:38:49.369355 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:49.369303    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fsdj4" podStartSLOduration=2.433257162 podStartE2EDuration="6.369287229s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.936030067s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:38:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:38:49.368299546 +0000 UTC m=+2440.536252565" watchObservedRunningTime="2026-09-17 22:38:49.369287229 +0000 UTC m=+2440.537240221"
Sep 17 22:38:49.658099 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:49.658071    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nrkd5_34b8ef2b-97ba-4562-87ee-8da0cb4604c7/ovn-acl-logging/0.log"
Sep 17 22:38:49.661288 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:49.661269    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nrkd5_34b8ef2b-97ba-4562-87ee-8da0cb4604c7/ovn-acl-logging/0.log"
Sep 17 22:38:55.360921 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:38:55.360894    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fsdj4"
Sep 17 22:39:17.327092 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:17.327063    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-9b4tl"
Sep 17 22:39:24.900714 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:24.900679    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-9b4tl"]
Sep 17 22:39:24.901128 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:24.900931    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-9b4tl" podUID="1e295041-f6e6-4c4e-8138-8d82e7e4914a" containerName="kserve-container" containerID="cri-o://5e841a3e785f85f5e9d7613fe10c4156a26d67e4b3f0186dcaecfcb7d8d241a4" gracePeriod=30
Sep 17 22:39:24.901128 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:24.901009    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-9b4tl" podUID="1e295041-f6e6-4c4e-8138-8d82e7e4914a" containerName="kube-rbac-proxy" containerID="cri-o://a735e08f82928f71213697825d78e77f5140e62d916bed60935a893118df6b8b" gracePeriod=30
Sep 17 22:39:25.452721 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:25.452661    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fsdj4" podUID="beffab2d-4a96-4234-be69-005f9c7f90ec" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 17 22:39:25.495332 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:25.495299    2563 generic.go:356] "Generic (PLEG): container finished" podID="1e295041-f6e6-4c4e-8138-8d82e7e4914a" containerID="a735e08f82928f71213697825d78e77f5140e62d916bed60935a893118df6b8b" exitCode=2
Sep 17 22:39:25.495512 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:25.495364    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-9b4tl" event={"ID":"1e295041-f6e6-4c4e-8138-8d82e7e4914a","Type":"ContainerDied","Data":"a735e08f82928f71213697825d78e77f5140e62d916bed60935a893118df6b8b"}
Sep 17 22:39:25.941973 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:25.941939    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-9b4tl"
Sep 17 22:39:26.012317 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:26.012227    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e295041-f6e6-4c4e-8138-8d82e7e4914a-kube-rbac-proxy-sar-config\") pod \"1e295041-f6e6-4c4e-8138-8d82e7e4914a\" (UID: \"1e295041-f6e6-4c4e-8138-8d82e7e4914a\") "
Sep 17 22:39:26.012317 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:26.012270    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8vxfc\" (UniqueName: \"kubernetes.io/projected/1e295041-f6e6-4c4e-8138-8d82e7e4914a-kube-api-access-8vxfc\") pod \"1e295041-f6e6-4c4e-8138-8d82e7e4914a\" (UID: \"1e295041-f6e6-4c4e-8138-8d82e7e4914a\") "
Sep 17 22:39:26.012317 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:26.012301    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e295041-f6e6-4c4e-8138-8d82e7e4914a-cabundle-cert\") pod \"1e295041-f6e6-4c4e-8138-8d82e7e4914a\" (UID: \"1e295041-f6e6-4c4e-8138-8d82e7e4914a\") "
Sep 17 22:39:26.012573 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:26.012326    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e295041-f6e6-4c4e-8138-8d82e7e4914a-proxy-tls\") pod \"1e295041-f6e6-4c4e-8138-8d82e7e4914a\" (UID: \"1e295041-f6e6-4c4e-8138-8d82e7e4914a\") "
Sep 17 22:39:26.012573 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:26.012384    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e295041-f6e6-4c4e-8138-8d82e7e4914a-kserve-provision-location\") pod \"1e295041-f6e6-4c4e-8138-8d82e7e4914a\" (UID: \"1e295041-f6e6-4c4e-8138-8d82e7e4914a\") "
Sep 17 22:39:26.012720 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:26.012690    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e295041-f6e6-4c4e-8138-8d82e7e4914a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1e295041-f6e6-4c4e-8138-8d82e7e4914a" (UID: "1e295041-f6e6-4c4e-8138-8d82e7e4914a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:39:26.012720 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:26.012710    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e295041-f6e6-4c4e-8138-8d82e7e4914a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1e295041-f6e6-4c4e-8138-8d82e7e4914a" (UID: "1e295041-f6e6-4c4e-8138-8d82e7e4914a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:39:26.012720 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:26.012727    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e295041-f6e6-4c4e-8138-8d82e7e4914a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1e295041-f6e6-4c4e-8138-8d82e7e4914a" (UID: "1e295041-f6e6-4c4e-8138-8d82e7e4914a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:39:26.014594 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:26.014570    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e295041-f6e6-4c4e-8138-8d82e7e4914a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1e295041-f6e6-4c4e-8138-8d82e7e4914a" (UID: "1e295041-f6e6-4c4e-8138-8d82e7e4914a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:39:26.014684 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:26.014592    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e295041-f6e6-4c4e-8138-8d82e7e4914a-kube-api-access-8vxfc" (OuterVolumeSpecName: "kube-api-access-8vxfc") pod "1e295041-f6e6-4c4e-8138-8d82e7e4914a" (UID: "1e295041-f6e6-4c4e-8138-8d82e7e4914a"). InnerVolumeSpecName "kube-api-access-8vxfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:39:26.113608 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:26.113563    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e295041-f6e6-4c4e-8138-8d82e7e4914a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:39:26.113608 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:26.113598    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vxfc\" (UniqueName: \"kubernetes.io/projected/1e295041-f6e6-4c4e-8138-8d82e7e4914a-kube-api-access-8vxfc\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:39:26.113608 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:26.113613    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e295041-f6e6-4c4e-8138-8d82e7e4914a-cabundle-cert\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:39:26.113958 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:26.113626    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e295041-f6e6-4c4e-8138-8d82e7e4914a-proxy-tls\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:39:26.113958 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:26.113637    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e295041-f6e6-4c4e-8138-8d82e7e4914a-kserve-provision-location\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:39:26.500786 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:26.500750    2563 generic.go:356] "Generic (PLEG): container finished" podID="1e295041-f6e6-4c4e-8138-8d82e7e4914a" containerID="5e841a3e785f85f5e9d7613fe10c4156a26d67e4b3f0186dcaecfcb7d8d241a4" exitCode=0
Sep 17 22:39:26.501000 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:26.500809    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-9b4tl"
Sep 17 22:39:26.501000 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:26.500836    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-9b4tl" event={"ID":"1e295041-f6e6-4c4e-8138-8d82e7e4914a","Type":"ContainerDied","Data":"5e841a3e785f85f5e9d7613fe10c4156a26d67e4b3f0186dcaecfcb7d8d241a4"}
Sep 17 22:39:26.501000 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:26.500899    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-9b4tl" event={"ID":"1e295041-f6e6-4c4e-8138-8d82e7e4914a","Type":"ContainerDied","Data":"cd29455e324fd0870e511b3e71436c2aaa8fc5996e28b38c74848fc823d6edd9"}
Sep 17 22:39:26.501000 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:26.500931    2563 scope.go:117] "RemoveContainer" containerID="a735e08f82928f71213697825d78e77f5140e62d916bed60935a893118df6b8b"
Sep 17 22:39:26.509682 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:26.509661    2563 scope.go:117] "RemoveContainer" containerID="5e841a3e785f85f5e9d7613fe10c4156a26d67e4b3f0186dcaecfcb7d8d241a4"
Sep 17 22:39:26.516805 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:26.516785    2563 scope.go:117] "RemoveContainer" containerID="1ad06f0d678cff6b0e2fde749854fde07c47123cca6255aec570096312e995c1"
Sep 17 22:39:26.522081 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:26.522054    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-9b4tl"]
Sep 17 22:39:26.524188 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:26.524134    2563 scope.go:117] "RemoveContainer" containerID="a735e08f82928f71213697825d78e77f5140e62d916bed60935a893118df6b8b"
Sep 17 22:39:26.524532 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:39:26.524499    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a735e08f82928f71213697825d78e77f5140e62d916bed60935a893118df6b8b\": container with ID starting with a735e08f82928f71213697825d78e77f5140e62d916bed60935a893118df6b8b not found: ID does not exist" containerID="a735e08f82928f71213697825d78e77f5140e62d916bed60935a893118df6b8b"
Sep 17 22:39:26.524662 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:26.524534    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a735e08f82928f71213697825d78e77f5140e62d916bed60935a893118df6b8b"} err="failed to get container status \"a735e08f82928f71213697825d78e77f5140e62d916bed60935a893118df6b8b\": rpc error: code = NotFound desc = could not find container \"a735e08f82928f71213697825d78e77f5140e62d916bed60935a893118df6b8b\": container with ID starting with a735e08f82928f71213697825d78e77f5140e62d916bed60935a893118df6b8b not found: ID does not exist"
Sep 17 22:39:26.524662 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:26.524560    2563 scope.go:117] "RemoveContainer" containerID="5e841a3e785f85f5e9d7613fe10c4156a26d67e4b3f0186dcaecfcb7d8d241a4"
Sep 17 22:39:26.524961 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:39:26.524936    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e841a3e785f85f5e9d7613fe10c4156a26d67e4b3f0186dcaecfcb7d8d241a4\": container with ID starting with 5e841a3e785f85f5e9d7613fe10c4156a26d67e4b3f0186dcaecfcb7d8d241a4 not found: ID does not exist" containerID="5e841a3e785f85f5e9d7613fe10c4156a26d67e4b3f0186dcaecfcb7d8d241a4"
Sep 17 22:39:26.525038 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:26.524967    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e841a3e785f85f5e9d7613fe10c4156a26d67e4b3f0186dcaecfcb7d8d241a4"} err="failed to get container status \"5e841a3e785f85f5e9d7613fe10c4156a26d67e4b3f0186dcaecfcb7d8d241a4\": rpc error: code = NotFound desc = could not find container \"5e841a3e785f85f5e9d7613fe10c4156a26d67e4b3f0186dcaecfcb7d8d241a4\": container with ID starting with 5e841a3e785f85f5e9d7613fe10c4156a26d67e4b3f0186dcaecfcb7d8d241a4 not found: ID does not exist"
Sep 17 22:39:26.525038 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:26.525002    2563 scope.go:117] "RemoveContainer" containerID="1ad06f0d678cff6b0e2fde749854fde07c47123cca6255aec570096312e995c1"
Sep 17 22:39:26.525284 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:39:26.525262    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ad06f0d678cff6b0e2fde749854fde07c47123cca6255aec570096312e995c1\": container with ID starting with 1ad06f0d678cff6b0e2fde749854fde07c47123cca6255aec570096312e995c1 not found: ID does not exist" containerID="1ad06f0d678cff6b0e2fde749854fde07c47123cca6255aec570096312e995c1"
Sep 17 22:39:26.525350 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:26.525289    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ad06f0d678cff6b0e2fde749854fde07c47123cca6255aec570096312e995c1"} err="failed to get container status \"1ad06f0d678cff6b0e2fde749854fde07c47123cca6255aec570096312e995c1\": rpc error: code = NotFound desc = could not find container \"1ad06f0d678cff6b0e2fde749854fde07c47123cca6255aec570096312e995c1\": container with ID starting with 1ad06f0d678cff6b0e2fde749854fde07c47123cca6255aec570096312e995c1 not found: ID does not exist"
Sep 17 22:39:26.526367 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:26.526349    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-9b4tl"]
Sep 17 22:39:26.913547 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:26.913518    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e295041-f6e6-4c4e-8138-8d82e7e4914a" path="/var/lib/kubelet/pods/1e295041-f6e6-4c4e-8138-8d82e7e4914a/volumes"
Sep 17 22:39:35.363386 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:35.363355    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fsdj4"
Sep 17 22:39:43.187813 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:43.187778    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fsdj4"]
Sep 17 22:39:43.188225 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:43.188057    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fsdj4" podUID="beffab2d-4a96-4234-be69-005f9c7f90ec" containerName="kserve-container" containerID="cri-o://86e689b3693164f17392a43396f2d1d1c652dd80e3592982b377c9fd9246e33d" gracePeriod=30
Sep 17 22:39:43.188225 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:43.188074    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fsdj4" podUID="beffab2d-4a96-4234-be69-005f9c7f90ec" containerName="kube-rbac-proxy" containerID="cri-o://83146943ea4bfa50df9a16f17f6977ec7ad995906027dedc39de87d52eeabf77" gracePeriod=30
Sep 17 22:39:43.572593 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:43.572503    2563 generic.go:356] "Generic (PLEG): container finished" podID="beffab2d-4a96-4234-be69-005f9c7f90ec" containerID="83146943ea4bfa50df9a16f17f6977ec7ad995906027dedc39de87d52eeabf77" exitCode=2
Sep 17 22:39:43.572593 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:43.572554    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fsdj4" event={"ID":"beffab2d-4a96-4234-be69-005f9c7f90ec","Type":"ContainerDied","Data":"83146943ea4bfa50df9a16f17f6977ec7ad995906027dedc39de87d52eeabf77"}
Sep 17 22:39:45.355829 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:45.355792    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fsdj4" podUID="beffab2d-4a96-4234-be69-005f9c7f90ec" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.48:8643/healthz\": dial tcp 10.133.0.48:8643: connect: connection refused"
Sep 17 22:39:46.402321 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:46.402281    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fsdj4" podUID="beffab2d-4a96-4234-be69-005f9c7f90ec" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.48:8080/v2/models/isvc-xgboost-v2-runtime/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 17 22:39:50.356030 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:50.355988    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fsdj4" podUID="beffab2d-4a96-4234-be69-005f9c7f90ec" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.48:8643/healthz\": dial tcp 10.133.0.48:8643: connect: connection refused"
Sep 17 22:39:50.930414 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:50.930390    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fsdj4"
Sep 17 22:39:51.022921 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:51.022835    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n9fwq\" (UniqueName: \"kubernetes.io/projected/beffab2d-4a96-4234-be69-005f9c7f90ec-kube-api-access-n9fwq\") pod \"beffab2d-4a96-4234-be69-005f9c7f90ec\" (UID: \"beffab2d-4a96-4234-be69-005f9c7f90ec\") "
Sep 17 22:39:51.022921 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:51.022873    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/beffab2d-4a96-4234-be69-005f9c7f90ec-cabundle-cert\") pod \"beffab2d-4a96-4234-be69-005f9c7f90ec\" (UID: \"beffab2d-4a96-4234-be69-005f9c7f90ec\") "
Sep 17 22:39:51.022921 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:51.022904    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/beffab2d-4a96-4234-be69-005f9c7f90ec-proxy-tls\") pod \"beffab2d-4a96-4234-be69-005f9c7f90ec\" (UID: \"beffab2d-4a96-4234-be69-005f9c7f90ec\") "
Sep 17 22:39:51.023231 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:51.022933    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/beffab2d-4a96-4234-be69-005f9c7f90ec-kube-rbac-proxy-sar-config\") pod \"beffab2d-4a96-4234-be69-005f9c7f90ec\" (UID: \"beffab2d-4a96-4234-be69-005f9c7f90ec\") "
Sep 17 22:39:51.023231 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:51.022964    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/beffab2d-4a96-4234-be69-005f9c7f90ec-kserve-provision-location\") pod \"beffab2d-4a96-4234-be69-005f9c7f90ec\" (UID: \"beffab2d-4a96-4234-be69-005f9c7f90ec\") "
Sep 17 22:39:51.023356 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:51.023332    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/beffab2d-4a96-4234-be69-005f9c7f90ec-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "beffab2d-4a96-4234-be69-005f9c7f90ec" (UID: "beffab2d-4a96-4234-be69-005f9c7f90ec"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:39:51.023399 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:51.023329    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/beffab2d-4a96-4234-be69-005f9c7f90ec-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "beffab2d-4a96-4234-be69-005f9c7f90ec" (UID: "beffab2d-4a96-4234-be69-005f9c7f90ec"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:39:51.023399 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:51.023328    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beffab2d-4a96-4234-be69-005f9c7f90ec-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "beffab2d-4a96-4234-be69-005f9c7f90ec" (UID: "beffab2d-4a96-4234-be69-005f9c7f90ec"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:39:51.025026 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:51.025005    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/beffab2d-4a96-4234-be69-005f9c7f90ec-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "beffab2d-4a96-4234-be69-005f9c7f90ec" (UID: "beffab2d-4a96-4234-be69-005f9c7f90ec"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:39:51.025088 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:51.025057    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beffab2d-4a96-4234-be69-005f9c7f90ec-kube-api-access-n9fwq" (OuterVolumeSpecName: "kube-api-access-n9fwq") pod "beffab2d-4a96-4234-be69-005f9c7f90ec" (UID: "beffab2d-4a96-4234-be69-005f9c7f90ec"). InnerVolumeSpecName "kube-api-access-n9fwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:39:51.124025 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:51.123992    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/beffab2d-4a96-4234-be69-005f9c7f90ec-kserve-provision-location\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:39:51.124025 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:51.124017    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9fwq\" (UniqueName: \"kubernetes.io/projected/beffab2d-4a96-4234-be69-005f9c7f90ec-kube-api-access-n9fwq\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:39:51.124025 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:51.124026    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/beffab2d-4a96-4234-be69-005f9c7f90ec-cabundle-cert\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:39:51.124025 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:51.124035    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/beffab2d-4a96-4234-be69-005f9c7f90ec-proxy-tls\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:39:51.124301 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:51.124043    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/beffab2d-4a96-4234-be69-005f9c7f90ec-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:39:51.605488 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:51.605452    2563 generic.go:356] "Generic (PLEG): container finished" podID="beffab2d-4a96-4234-be69-005f9c7f90ec" containerID="86e689b3693164f17392a43396f2d1d1c652dd80e3592982b377c9fd9246e33d" exitCode=0
Sep 17 22:39:51.605867 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:51.605508    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fsdj4"
Sep 17 22:39:51.605867 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:51.605528    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fsdj4" event={"ID":"beffab2d-4a96-4234-be69-005f9c7f90ec","Type":"ContainerDied","Data":"86e689b3693164f17392a43396f2d1d1c652dd80e3592982b377c9fd9246e33d"}
Sep 17 22:39:51.605867 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:51.605576    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fsdj4" event={"ID":"beffab2d-4a96-4234-be69-005f9c7f90ec","Type":"ContainerDied","Data":"b2ea547724ced5b83472cd47294acf474dd8d8e27a4edbfb78eb19b35c8dcc20"}
Sep 17 22:39:51.605867 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:51.605605    2563 scope.go:117] "RemoveContainer" containerID="83146943ea4bfa50df9a16f17f6977ec7ad995906027dedc39de87d52eeabf77"
Sep 17 22:39:51.613988 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:51.613820    2563 scope.go:117] "RemoveContainer" containerID="86e689b3693164f17392a43396f2d1d1c652dd80e3592982b377c9fd9246e33d"
Sep 17 22:39:51.620793 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:51.620775    2563 scope.go:117] "RemoveContainer" containerID="8a888b91247023fffdaa50ce009bda98c8e9702a64bda32874d3c38b70b4e124"
Sep 17 22:39:51.628718 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:51.628696    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fsdj4"]
Sep 17 22:39:51.628793 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:51.628718    2563 scope.go:117] "RemoveContainer" containerID="83146943ea4bfa50df9a16f17f6977ec7ad995906027dedc39de87d52eeabf77"
Sep 17 22:39:51.628989 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:39:51.628973    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83146943ea4bfa50df9a16f17f6977ec7ad995906027dedc39de87d52eeabf77\": container with ID starting with 83146943ea4bfa50df9a16f17f6977ec7ad995906027dedc39de87d52eeabf77 not found: ID does not exist" containerID="83146943ea4bfa50df9a16f17f6977ec7ad995906027dedc39de87d52eeabf77"
Sep 17 22:39:51.629058 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:51.628999    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83146943ea4bfa50df9a16f17f6977ec7ad995906027dedc39de87d52eeabf77"} err="failed to get container status \"83146943ea4bfa50df9a16f17f6977ec7ad995906027dedc39de87d52eeabf77\": rpc error: code = NotFound desc = could not find container \"83146943ea4bfa50df9a16f17f6977ec7ad995906027dedc39de87d52eeabf77\": container with ID starting with 83146943ea4bfa50df9a16f17f6977ec7ad995906027dedc39de87d52eeabf77 not found: ID does not exist"
Sep 17 22:39:51.629058 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:51.629021    2563 scope.go:117] "RemoveContainer" containerID="86e689b3693164f17392a43396f2d1d1c652dd80e3592982b377c9fd9246e33d"
Sep 17 22:39:51.629295 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:39:51.629278    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86e689b3693164f17392a43396f2d1d1c652dd80e3592982b377c9fd9246e33d\": container with ID starting with 86e689b3693164f17392a43396f2d1d1c652dd80e3592982b377c9fd9246e33d not found: ID does not exist" containerID="86e689b3693164f17392a43396f2d1d1c652dd80e3592982b377c9fd9246e33d"
Sep 17 22:39:51.629343 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:51.629301    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86e689b3693164f17392a43396f2d1d1c652dd80e3592982b377c9fd9246e33d"} err="failed to get container status \"86e689b3693164f17392a43396f2d1d1c652dd80e3592982b377c9fd9246e33d\": rpc error: code = NotFound desc = could not find container \"86e689b3693164f17392a43396f2d1d1c652dd80e3592982b377c9fd9246e33d\": container with ID starting with 86e689b3693164f17392a43396f2d1d1c652dd80e3592982b377c9fd9246e33d not found: ID does not exist"
Sep 17 22:39:51.629343 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:51.629316    2563 scope.go:117] "RemoveContainer" containerID="8a888b91247023fffdaa50ce009bda98c8e9702a64bda32874d3c38b70b4e124"
Sep 17 22:39:51.629567 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:39:51.629548    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a888b91247023fffdaa50ce009bda98c8e9702a64bda32874d3c38b70b4e124\": container with ID starting with 8a888b91247023fffdaa50ce009bda98c8e9702a64bda32874d3c38b70b4e124 not found: ID does not exist" containerID="8a888b91247023fffdaa50ce009bda98c8e9702a64bda32874d3c38b70b4e124"
Sep 17 22:39:51.629606 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:51.629572    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a888b91247023fffdaa50ce009bda98c8e9702a64bda32874d3c38b70b4e124"} err="failed to get container status \"8a888b91247023fffdaa50ce009bda98c8e9702a64bda32874d3c38b70b4e124\": rpc error: code = NotFound desc = could not find container \"8a888b91247023fffdaa50ce009bda98c8e9702a64bda32874d3c38b70b4e124\": container with ID starting with 8a888b91247023fffdaa50ce009bda98c8e9702a64bda32874d3c38b70b4e124 not found: ID does not exist"
Sep 17 22:39:51.631837 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:51.631819    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fsdj4"]
Sep 17 22:39:52.913742 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:39:52.913649    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="beffab2d-4a96-4234-be69-005f9c7f90ec" path="/var/lib/kubelet/pods/beffab2d-4a96-4234-be69-005f9c7f90ec/volumes"
Sep 17 22:40:05.423238 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:05.423202    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-52d6w"]
Sep 17 22:40:05.423692 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:05.423478    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="beffab2d-4a96-4234-be69-005f9c7f90ec" containerName="kube-rbac-proxy"
Sep 17 22:40:05.423692 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:05.423488    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="beffab2d-4a96-4234-be69-005f9c7f90ec" containerName="kube-rbac-proxy"
Sep 17 22:40:05.423692 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:05.423499    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e295041-f6e6-4c4e-8138-8d82e7e4914a" containerName="storage-initializer"
Sep 17 22:40:05.423692 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:05.423506    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e295041-f6e6-4c4e-8138-8d82e7e4914a" containerName="storage-initializer"
Sep 17 22:40:05.423692 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:05.423515    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="beffab2d-4a96-4234-be69-005f9c7f90ec" containerName="kserve-container"
Sep 17 22:40:05.423692 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:05.423522    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="beffab2d-4a96-4234-be69-005f9c7f90ec" containerName="kserve-container"
Sep 17 22:40:05.423692 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:05.423531    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e295041-f6e6-4c4e-8138-8d82e7e4914a" containerName="kserve-container"
Sep 17 22:40:05.423692 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:05.423536    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e295041-f6e6-4c4e-8138-8d82e7e4914a" containerName="kserve-container"
Sep 17 22:40:05.423692 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:05.423549    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="beffab2d-4a96-4234-be69-005f9c7f90ec" containerName="storage-initializer"
Sep 17 22:40:05.423692 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:05.423555    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="beffab2d-4a96-4234-be69-005f9c7f90ec" containerName="storage-initializer"
Sep 17 22:40:05.423692 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:05.423564    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e295041-f6e6-4c4e-8138-8d82e7e4914a" containerName="kube-rbac-proxy"
Sep 17 22:40:05.423692 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:05.423569    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e295041-f6e6-4c4e-8138-8d82e7e4914a" containerName="kube-rbac-proxy"
Sep 17 22:40:05.423692 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:05.423616    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="beffab2d-4a96-4234-be69-005f9c7f90ec" containerName="kube-rbac-proxy"
Sep 17 22:40:05.423692 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:05.423625    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e295041-f6e6-4c4e-8138-8d82e7e4914a" containerName="kserve-container"
Sep 17 22:40:05.423692 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:05.423630    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="beffab2d-4a96-4234-be69-005f9c7f90ec" containerName="kserve-container"
Sep 17 22:40:05.423692 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:05.423638    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e295041-f6e6-4c4e-8138-8d82e7e4914a" containerName="kube-rbac-proxy"
Sep 17 22:40:05.428389 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:05.428368    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-52d6w"
Sep 17 22:40:05.430882 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:05.430850    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 17 22:40:05.430882 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:05.430861    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 22:40:05.431085 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:05.430966    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-predictor-serving-cert\""
Sep 17 22:40:05.431085 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:05.430985    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v4p4m\""
Sep 17 22:40:05.431085 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:05.430983    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 22:40:05.431085 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:05.431050    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 22:40:05.431611 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:05.431597    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:40:05.437242 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:05.437217    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-52d6w"]
Sep 17 22:40:05.538073 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:05.538034    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55fb39c8-47fa-466d-b3d5-723d02eddf7e-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-52d6w\" (UID: \"55fb39c8-47fa-466d-b3d5-723d02eddf7e\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-52d6w"
Sep 17 22:40:05.538300 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:05.538103    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55fb39c8-47fa-466d-b3d5-723d02eddf7e-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-52d6w\" (UID: \"55fb39c8-47fa-466d-b3d5-723d02eddf7e\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-52d6w"
Sep 17 22:40:05.538300 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:05.538175    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr5pp\" (UniqueName: \"kubernetes.io/projected/55fb39c8-47fa-466d-b3d5-723d02eddf7e-kube-api-access-wr5pp\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-52d6w\" (UID: \"55fb39c8-47fa-466d-b3d5-723d02eddf7e\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-52d6w"
Sep 17 22:40:05.538300 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:05.538209    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55fb39c8-47fa-466d-b3d5-723d02eddf7e-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-52d6w\" (UID: \"55fb39c8-47fa-466d-b3d5-723d02eddf7e\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-52d6w"
Sep 17 22:40:05.538300 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:05.538235    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55fb39c8-47fa-466d-b3d5-723d02eddf7e-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-52d6w\" (UID: \"55fb39c8-47fa-466d-b3d5-723d02eddf7e\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-52d6w"
Sep 17 22:40:05.638962 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:05.638926    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55fb39c8-47fa-466d-b3d5-723d02eddf7e-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-52d6w\" (UID: \"55fb39c8-47fa-466d-b3d5-723d02eddf7e\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-52d6w"
Sep 17 22:40:05.638962 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:05.638964    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr5pp\" (UniqueName: \"kubernetes.io/projected/55fb39c8-47fa-466d-b3d5-723d02eddf7e-kube-api-access-wr5pp\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-52d6w\" (UID: \"55fb39c8-47fa-466d-b3d5-723d02eddf7e\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-52d6w"
Sep 17 22:40:05.639204 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:05.638992    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55fb39c8-47fa-466d-b3d5-723d02eddf7e-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-52d6w\" (UID: \"55fb39c8-47fa-466d-b3d5-723d02eddf7e\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-52d6w"
Sep 17 22:40:05.639204 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:05.639048    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55fb39c8-47fa-466d-b3d5-723d02eddf7e-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-52d6w\" (UID: \"55fb39c8-47fa-466d-b3d5-723d02eddf7e\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-52d6w"
Sep 17 22:40:05.639204 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:05.639115    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55fb39c8-47fa-466d-b3d5-723d02eddf7e-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-52d6w\" (UID: \"55fb39c8-47fa-466d-b3d5-723d02eddf7e\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-52d6w"
Sep 17 22:40:05.639529 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:05.639507    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55fb39c8-47fa-466d-b3d5-723d02eddf7e-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-52d6w\" (UID: \"55fb39c8-47fa-466d-b3d5-723d02eddf7e\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-52d6w"
Sep 17 22:40:05.639831 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:05.639810    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55fb39c8-47fa-466d-b3d5-723d02eddf7e-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-52d6w\" (UID: \"55fb39c8-47fa-466d-b3d5-723d02eddf7e\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-52d6w"
Sep 17 22:40:05.639878 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:05.639811    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55fb39c8-47fa-466d-b3d5-723d02eddf7e-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-52d6w\" (UID: \"55fb39c8-47fa-466d-b3d5-723d02eddf7e\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-52d6w"
Sep 17 22:40:05.641590 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:05.641564    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55fb39c8-47fa-466d-b3d5-723d02eddf7e-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-52d6w\" (UID: \"55fb39c8-47fa-466d-b3d5-723d02eddf7e\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-52d6w"
Sep 17 22:40:05.647095 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:05.647063    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr5pp\" (UniqueName: \"kubernetes.io/projected/55fb39c8-47fa-466d-b3d5-723d02eddf7e-kube-api-access-wr5pp\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-52d6w\" (UID: \"55fb39c8-47fa-466d-b3d5-723d02eddf7e\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-52d6w"
Sep 17 22:40:05.739676 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:05.739571    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-52d6w"
Sep 17 22:40:06.008544 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:06.008518    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-52d6w"]
Sep 17 22:40:06.010472 ip-10-0-138-45 kubenswrapper[2563]: W0917 22:40:06.010439    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55fb39c8_47fa_466d_b3d5_723d02eddf7e.slice/crio-302d8da826bd30a8026a0b18434e61395ff04f76c9a02f3b25d0eeaed8139c92 WatchSource:0}: Error finding container 302d8da826bd30a8026a0b18434e61395ff04f76c9a02f3b25d0eeaed8139c92: Status 404 returned error can't find the container with id 302d8da826bd30a8026a0b18434e61395ff04f76c9a02f3b25d0eeaed8139c92
Sep 17 22:40:06.666992 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:06.666959    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-52d6w" event={"ID":"55fb39c8-47fa-466d-b3d5-723d02eddf7e","Type":"ContainerStarted","Data":"04abd0c8bc51cee2823b30b373d03c06fa188c085b271acd2cd4253a9c890e8a"}
Sep 17 22:40:06.667390 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:06.667019    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-52d6w" event={"ID":"55fb39c8-47fa-466d-b3d5-723d02eddf7e","Type":"ContainerStarted","Data":"302d8da826bd30a8026a0b18434e61395ff04f76c9a02f3b25d0eeaed8139c92"}
Sep 17 22:40:10.688711 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:10.688677    2563 generic.go:356] "Generic (PLEG): container finished" podID="55fb39c8-47fa-466d-b3d5-723d02eddf7e" containerID="04abd0c8bc51cee2823b30b373d03c06fa188c085b271acd2cd4253a9c890e8a" exitCode=0
Sep 17 22:40:10.689099 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:10.688748    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-52d6w" event={"ID":"55fb39c8-47fa-466d-b3d5-723d02eddf7e","Type":"ContainerDied","Data":"04abd0c8bc51cee2823b30b373d03c06fa188c085b271acd2cd4253a9c890e8a"}
Sep 17 22:40:11.694897 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:11.694865    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-52d6w" event={"ID":"55fb39c8-47fa-466d-b3d5-723d02eddf7e","Type":"ContainerStarted","Data":"2d58b35479cc760a2d549ba965b81569d1649eb4493f3406883097be58759620"}
Sep 17 22:40:11.695302 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:11.694935    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-52d6w" event={"ID":"55fb39c8-47fa-466d-b3d5-723d02eddf7e","Type":"ContainerStarted","Data":"543d69828c8f5209f05e14ee62ae51ac62a7ae236c5649fef1b8638089b3811b"}
Sep 17 22:40:11.695302 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:11.695174    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-52d6w"
Sep 17 22:40:11.695408 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:11.695343    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-52d6w"
Sep 17 22:40:11.714637 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:11.714584    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-52d6w" podStartSLOduration=2.643282439 podStartE2EDuration="6.714569345s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.071286906s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:40:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:40:11.713203791 +0000 UTC m=+2522.881156772" watchObservedRunningTime="2026-09-17 22:40:11.714569345 +0000 UTC m=+2522.882522326"
Sep 17 22:40:17.705053 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:17.705017    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-52d6w"
Sep 17 22:40:47.708893 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:47.708858    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-52d6w"
Sep 17 22:40:55.483302 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:55.483259    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-52d6w"]
Sep 17 22:40:55.483895 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:55.483505    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-52d6w" podUID="55fb39c8-47fa-466d-b3d5-723d02eddf7e" containerName="kserve-container" containerID="cri-o://543d69828c8f5209f05e14ee62ae51ac62a7ae236c5649fef1b8638089b3811b" gracePeriod=30
Sep 17 22:40:55.483895 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:55.483558    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-52d6w" podUID="55fb39c8-47fa-466d-b3d5-723d02eddf7e" containerName="kube-rbac-proxy" containerID="cri-o://2d58b35479cc760a2d549ba965b81569d1649eb4493f3406883097be58759620" gracePeriod=30
Sep 17 22:40:55.876472 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:55.876382    2563 generic.go:356] "Generic (PLEG): container finished" podID="55fb39c8-47fa-466d-b3d5-723d02eddf7e" containerID="2d58b35479cc760a2d549ba965b81569d1649eb4493f3406883097be58759620" exitCode=2
Sep 17 22:40:55.876613 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:55.876463    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-52d6w" event={"ID":"55fb39c8-47fa-466d-b3d5-723d02eddf7e","Type":"ContainerDied","Data":"2d58b35479cc760a2d549ba965b81569d1649eb4493f3406883097be58759620"}
Sep 17 22:40:56.712526 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:56.712501    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-52d6w"
Sep 17 22:40:56.766920 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:56.766893    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55fb39c8-47fa-466d-b3d5-723d02eddf7e-kserve-provision-location\") pod \"55fb39c8-47fa-466d-b3d5-723d02eddf7e\" (UID: \"55fb39c8-47fa-466d-b3d5-723d02eddf7e\") "
Sep 17 22:40:56.767069 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:56.766932    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55fb39c8-47fa-466d-b3d5-723d02eddf7e-cabundle-cert\") pod \"55fb39c8-47fa-466d-b3d5-723d02eddf7e\" (UID: \"55fb39c8-47fa-466d-b3d5-723d02eddf7e\") "
Sep 17 22:40:56.767069 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:56.766974    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wr5pp\" (UniqueName: \"kubernetes.io/projected/55fb39c8-47fa-466d-b3d5-723d02eddf7e-kube-api-access-wr5pp\") pod \"55fb39c8-47fa-466d-b3d5-723d02eddf7e\" (UID: \"55fb39c8-47fa-466d-b3d5-723d02eddf7e\") "
Sep 17 22:40:56.767069 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:56.767001    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55fb39c8-47fa-466d-b3d5-723d02eddf7e-proxy-tls\") pod \"55fb39c8-47fa-466d-b3d5-723d02eddf7e\" (UID: \"55fb39c8-47fa-466d-b3d5-723d02eddf7e\") "
Sep 17 22:40:56.767069 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:56.767041    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55fb39c8-47fa-466d-b3d5-723d02eddf7e-kube-rbac-proxy-sar-config\") pod \"55fb39c8-47fa-466d-b3d5-723d02eddf7e\" (UID: \"55fb39c8-47fa-466d-b3d5-723d02eddf7e\") "
Sep 17 22:40:56.767310 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:56.767285    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55fb39c8-47fa-466d-b3d5-723d02eddf7e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "55fb39c8-47fa-466d-b3d5-723d02eddf7e" (UID: "55fb39c8-47fa-466d-b3d5-723d02eddf7e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:40:56.767376 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:56.767334    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55fb39c8-47fa-466d-b3d5-723d02eddf7e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "55fb39c8-47fa-466d-b3d5-723d02eddf7e" (UID: "55fb39c8-47fa-466d-b3d5-723d02eddf7e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:40:56.767508 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:56.767486    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55fb39c8-47fa-466d-b3d5-723d02eddf7e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "55fb39c8-47fa-466d-b3d5-723d02eddf7e" (UID: "55fb39c8-47fa-466d-b3d5-723d02eddf7e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:40:56.769075 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:56.769055    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55fb39c8-47fa-466d-b3d5-723d02eddf7e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "55fb39c8-47fa-466d-b3d5-723d02eddf7e" (UID: "55fb39c8-47fa-466d-b3d5-723d02eddf7e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:40:56.769162 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:56.769086    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55fb39c8-47fa-466d-b3d5-723d02eddf7e-kube-api-access-wr5pp" (OuterVolumeSpecName: "kube-api-access-wr5pp") pod "55fb39c8-47fa-466d-b3d5-723d02eddf7e" (UID: "55fb39c8-47fa-466d-b3d5-723d02eddf7e"). InnerVolumeSpecName "kube-api-access-wr5pp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:40:56.867651 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:56.867547    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55fb39c8-47fa-466d-b3d5-723d02eddf7e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:40:56.867651 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:56.867574    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55fb39c8-47fa-466d-b3d5-723d02eddf7e-kserve-provision-location\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:40:56.867651 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:56.867583    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55fb39c8-47fa-466d-b3d5-723d02eddf7e-cabundle-cert\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:40:56.867651 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:56.867594    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr5pp\" (UniqueName: \"kubernetes.io/projected/55fb39c8-47fa-466d-b3d5-723d02eddf7e-kube-api-access-wr5pp\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:40:56.867651 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:56.867604    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55fb39c8-47fa-466d-b3d5-723d02eddf7e-proxy-tls\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:40:56.882121 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:56.882094    2563 generic.go:356] "Generic (PLEG): container finished" podID="55fb39c8-47fa-466d-b3d5-723d02eddf7e" containerID="543d69828c8f5209f05e14ee62ae51ac62a7ae236c5649fef1b8638089b3811b" exitCode=0
Sep 17 22:40:56.882256 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:56.882163    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-52d6w"
Sep 17 22:40:56.882256 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:56.882180    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-52d6w" event={"ID":"55fb39c8-47fa-466d-b3d5-723d02eddf7e","Type":"ContainerDied","Data":"543d69828c8f5209f05e14ee62ae51ac62a7ae236c5649fef1b8638089b3811b"}
Sep 17 22:40:56.882256 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:56.882250    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-52d6w" event={"ID":"55fb39c8-47fa-466d-b3d5-723d02eddf7e","Type":"ContainerDied","Data":"302d8da826bd30a8026a0b18434e61395ff04f76c9a02f3b25d0eeaed8139c92"}
Sep 17 22:40:56.882366 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:56.882290    2563 scope.go:117] "RemoveContainer" containerID="2d58b35479cc760a2d549ba965b81569d1649eb4493f3406883097be58759620"
Sep 17 22:40:56.891170 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:56.891133    2563 scope.go:117] "RemoveContainer" containerID="543d69828c8f5209f05e14ee62ae51ac62a7ae236c5649fef1b8638089b3811b"
Sep 17 22:40:56.898426 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:56.898411    2563 scope.go:117] "RemoveContainer" containerID="04abd0c8bc51cee2823b30b373d03c06fa188c085b271acd2cd4253a9c890e8a"
Sep 17 22:40:56.903804 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:56.903779    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-52d6w"]
Sep 17 22:40:56.905507 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:56.905488    2563 scope.go:117] "RemoveContainer" containerID="2d58b35479cc760a2d549ba965b81569d1649eb4493f3406883097be58759620"
Sep 17 22:40:56.905844 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:40:56.905820    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d58b35479cc760a2d549ba965b81569d1649eb4493f3406883097be58759620\": container with ID starting with 2d58b35479cc760a2d549ba965b81569d1649eb4493f3406883097be58759620 not found: ID does not exist" containerID="2d58b35479cc760a2d549ba965b81569d1649eb4493f3406883097be58759620"
Sep 17 22:40:56.905934 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:56.905854    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d58b35479cc760a2d549ba965b81569d1649eb4493f3406883097be58759620"} err="failed to get container status \"2d58b35479cc760a2d549ba965b81569d1649eb4493f3406883097be58759620\": rpc error: code = NotFound desc = could not find container \"2d58b35479cc760a2d549ba965b81569d1649eb4493f3406883097be58759620\": container with ID starting with 2d58b35479cc760a2d549ba965b81569d1649eb4493f3406883097be58759620 not found: ID does not exist"
Sep 17 22:40:56.905934 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:56.905871    2563 scope.go:117] "RemoveContainer" containerID="543d69828c8f5209f05e14ee62ae51ac62a7ae236c5649fef1b8638089b3811b"
Sep 17 22:40:56.906232 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:40:56.906211    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"543d69828c8f5209f05e14ee62ae51ac62a7ae236c5649fef1b8638089b3811b\": container with ID starting with 543d69828c8f5209f05e14ee62ae51ac62a7ae236c5649fef1b8638089b3811b not found: ID does not exist" containerID="543d69828c8f5209f05e14ee62ae51ac62a7ae236c5649fef1b8638089b3811b"
Sep 17 22:40:56.906296 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:56.906238    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"543d69828c8f5209f05e14ee62ae51ac62a7ae236c5649fef1b8638089b3811b"} err="failed to get container status \"543d69828c8f5209f05e14ee62ae51ac62a7ae236c5649fef1b8638089b3811b\": rpc error: code = NotFound desc = could not find container \"543d69828c8f5209f05e14ee62ae51ac62a7ae236c5649fef1b8638089b3811b\": container with ID starting with 543d69828c8f5209f05e14ee62ae51ac62a7ae236c5649fef1b8638089b3811b not found: ID does not exist"
Sep 17 22:40:56.906296 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:56.906253    2563 scope.go:117] "RemoveContainer" containerID="04abd0c8bc51cee2823b30b373d03c06fa188c085b271acd2cd4253a9c890e8a"
Sep 17 22:40:56.906487 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:40:56.906469    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04abd0c8bc51cee2823b30b373d03c06fa188c085b271acd2cd4253a9c890e8a\": container with ID starting with 04abd0c8bc51cee2823b30b373d03c06fa188c085b271acd2cd4253a9c890e8a not found: ID does not exist" containerID="04abd0c8bc51cee2823b30b373d03c06fa188c085b271acd2cd4253a9c890e8a"
Sep 17 22:40:56.906543 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:56.906497    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04abd0c8bc51cee2823b30b373d03c06fa188c085b271acd2cd4253a9c890e8a"} err="failed to get container status \"04abd0c8bc51cee2823b30b373d03c06fa188c085b271acd2cd4253a9c890e8a\": rpc error: code = NotFound desc = could not find container \"04abd0c8bc51cee2823b30b373d03c06fa188c085b271acd2cd4253a9c890e8a\": container with ID starting with 04abd0c8bc51cee2823b30b373d03c06fa188c085b271acd2cd4253a9c890e8a not found: ID does not exist"
Sep 17 22:40:56.907166 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:56.907121    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-52d6w"]
Sep 17 22:40:56.912976 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:40:56.912958    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55fb39c8-47fa-466d-b3d5-723d02eddf7e" path="/var/lib/kubelet/pods/55fb39c8-47fa-466d-b3d5-723d02eddf7e/volumes"
Sep 17 22:41:24.367296 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:41:24.367220    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-ldqbj"]
Sep 17 22:41:24.367766 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:41:24.367524    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55fb39c8-47fa-466d-b3d5-723d02eddf7e" containerName="kube-rbac-proxy"
Sep 17 22:41:24.367766 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:41:24.367535    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="55fb39c8-47fa-466d-b3d5-723d02eddf7e" containerName="kube-rbac-proxy"
Sep 17 22:41:24.367766 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:41:24.367546    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55fb39c8-47fa-466d-b3d5-723d02eddf7e" containerName="storage-initializer"
Sep 17 22:41:24.367766 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:41:24.367552    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="55fb39c8-47fa-466d-b3d5-723d02eddf7e" containerName="storage-initializer"
Sep 17 22:41:24.367766 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:41:24.367565    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55fb39c8-47fa-466d-b3d5-723d02eddf7e" containerName="kserve-container"
Sep 17 22:41:24.367766 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:41:24.367570    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="55fb39c8-47fa-466d-b3d5-723d02eddf7e" containerName="kserve-container"
Sep 17 22:41:24.367766 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:41:24.367615    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="55fb39c8-47fa-466d-b3d5-723d02eddf7e" containerName="kserve-container"
Sep 17 22:41:24.367766 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:41:24.367623    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="55fb39c8-47fa-466d-b3d5-723d02eddf7e" containerName="kube-rbac-proxy"
Sep 17 22:41:24.370594 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:41:24.370580    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-ldqbj"
Sep 17 22:41:24.373026 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:41:24.373006    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 17 22:41:24.373174 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:41:24.373131    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 22:41:24.373247 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:41:24.373172    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-kube-rbac-proxy-sar-config\""
Sep 17 22:41:24.374095 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:41:24.374076    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-predictor-serving-cert\""
Sep 17 22:41:24.374165 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:41:24.374082    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v4p4m\""
Sep 17 22:41:24.374207 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:41:24.374193    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 22:41:24.382545 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:41:24.382522    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-ldqbj"]
Sep 17 22:41:24.497346 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:41:24.497311    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0c6e1cc0-ba76-4b4a-a335-bed7161b74e7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-64bc8d7b4d-ldqbj\" (UID: \"0c6e1cc0-ba76-4b4a-a335-bed7161b74e7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-ldqbj"
Sep 17 22:41:24.497523 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:41:24.497365    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c6e1cc0-ba76-4b4a-a335-bed7161b74e7-proxy-tls\") pod \"isvc-sklearn-s3-predictor-64bc8d7b4d-ldqbj\" (UID: \"0c6e1cc0-ba76-4b4a-a335-bed7161b74e7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-ldqbj"
Sep 17 22:41:24.497523 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:41:24.497406    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blvzk\" (UniqueName: \"kubernetes.io/projected/0c6e1cc0-ba76-4b4a-a335-bed7161b74e7-kube-api-access-blvzk\") pod \"isvc-sklearn-s3-predictor-64bc8d7b4d-ldqbj\" (UID: \"0c6e1cc0-ba76-4b4a-a335-bed7161b74e7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-ldqbj"
Sep 17 22:41:24.497523 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:41:24.497452    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0c6e1cc0-ba76-4b4a-a335-bed7161b74e7-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-64bc8d7b4d-ldqbj\" (UID: \"0c6e1cc0-ba76-4b4a-a335-bed7161b74e7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-ldqbj"
Sep 17 22:41:24.597826 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:41:24.597790    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0c6e1cc0-ba76-4b4a-a335-bed7161b74e7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-64bc8d7b4d-ldqbj\" (UID: \"0c6e1cc0-ba76-4b4a-a335-bed7161b74e7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-ldqbj"
Sep 17 22:41:24.598014 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:41:24.597857    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c6e1cc0-ba76-4b4a-a335-bed7161b74e7-proxy-tls\") pod \"isvc-sklearn-s3-predictor-64bc8d7b4d-ldqbj\" (UID: \"0c6e1cc0-ba76-4b4a-a335-bed7161b74e7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-ldqbj"
Sep 17 22:41:24.598014 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:41:24.597882    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blvzk\" (UniqueName: \"kubernetes.io/projected/0c6e1cc0-ba76-4b4a-a335-bed7161b74e7-kube-api-access-blvzk\") pod \"isvc-sklearn-s3-predictor-64bc8d7b4d-ldqbj\" (UID: \"0c6e1cc0-ba76-4b4a-a335-bed7161b74e7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-ldqbj"
Sep 17 22:41:24.598014 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:41:24.597906    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0c6e1cc0-ba76-4b4a-a335-bed7161b74e7-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-64bc8d7b4d-ldqbj\" (UID: \"0c6e1cc0-ba76-4b4a-a335-bed7161b74e7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-ldqbj"
Sep 17 22:41:24.598347 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:41:24.598328    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0c6e1cc0-ba76-4b4a-a335-bed7161b74e7-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-64bc8d7b4d-ldqbj\" (UID: \"0c6e1cc0-ba76-4b4a-a335-bed7161b74e7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-ldqbj"
Sep 17 22:41:24.598521 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:41:24.598499    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0c6e1cc0-ba76-4b4a-a335-bed7161b74e7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-64bc8d7b4d-ldqbj\" (UID: \"0c6e1cc0-ba76-4b4a-a335-bed7161b74e7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-ldqbj"
Sep 17 22:41:24.600339 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:41:24.600324    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c6e1cc0-ba76-4b4a-a335-bed7161b74e7-proxy-tls\") pod \"isvc-sklearn-s3-predictor-64bc8d7b4d-ldqbj\" (UID: \"0c6e1cc0-ba76-4b4a-a335-bed7161b74e7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-ldqbj"
Sep 17 22:41:24.606080 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:41:24.606061    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blvzk\" (UniqueName: \"kubernetes.io/projected/0c6e1cc0-ba76-4b4a-a335-bed7161b74e7-kube-api-access-blvzk\") pod \"isvc-sklearn-s3-predictor-64bc8d7b4d-ldqbj\" (UID: \"0c6e1cc0-ba76-4b4a-a335-bed7161b74e7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-ldqbj"
Sep 17 22:41:24.680236 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:41:24.680205    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-ldqbj"
Sep 17 22:41:24.948815 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:41:24.948793    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-ldqbj"]
Sep 17 22:41:24.950680 ip-10-0-138-45 kubenswrapper[2563]: W0917 22:41:24.950656    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c6e1cc0_ba76_4b4a_a335_bed7161b74e7.slice/crio-098b4655e3174b3aa4e0cf367e9eb218ebc0c7239284c5f5f2d61e0c84d616a3 WatchSource:0}: Error finding container 098b4655e3174b3aa4e0cf367e9eb218ebc0c7239284c5f5f2d61e0c84d616a3: Status 404 returned error can't find the container with id 098b4655e3174b3aa4e0cf367e9eb218ebc0c7239284c5f5f2d61e0c84d616a3
Sep 17 22:41:24.988405 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:41:24.988374    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-ldqbj" event={"ID":"0c6e1cc0-ba76-4b4a-a335-bed7161b74e7","Type":"ContainerStarted","Data":"098b4655e3174b3aa4e0cf367e9eb218ebc0c7239284c5f5f2d61e0c84d616a3"}
Sep 17 22:41:25.994287 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:41:25.994243    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-ldqbj" event={"ID":"0c6e1cc0-ba76-4b4a-a335-bed7161b74e7","Type":"ContainerStarted","Data":"55bf1902fb9257bec73d0b9ae319feb3dae876be9ac1ad22b30b21cd91b087f7"}
Sep 17 22:41:26.999298 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:41:26.999264    2563 generic.go:356] "Generic (PLEG): container finished" podID="0c6e1cc0-ba76-4b4a-a335-bed7161b74e7" containerID="55bf1902fb9257bec73d0b9ae319feb3dae876be9ac1ad22b30b21cd91b087f7" exitCode=0
Sep 17 22:41:26.999696 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:41:26.999309    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-ldqbj" event={"ID":"0c6e1cc0-ba76-4b4a-a335-bed7161b74e7","Type":"ContainerDied","Data":"55bf1902fb9257bec73d0b9ae319feb3dae876be9ac1ad22b30b21cd91b087f7"}
Sep 17 22:41:28.004926 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:41:28.004895    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-ldqbj" event={"ID":"0c6e1cc0-ba76-4b4a-a335-bed7161b74e7","Type":"ContainerStarted","Data":"7663a8b9dc1c4f0019b00c24f664f9c226e2ec06f1d915d0ae1cb28ea45b437e"}
Sep 17 22:41:28.005339 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:41:28.004959    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-ldqbj" event={"ID":"0c6e1cc0-ba76-4b4a-a335-bed7161b74e7","Type":"ContainerStarted","Data":"0fc2ce88eff0cfa609c2b1a6adef18cd4cc38bbd7b5491112d88df4dd0b246f9"}
Sep 17 22:41:28.005339 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:41:28.005038    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-ldqbj"
Sep 17 22:41:28.038087 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:41:28.038023    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-ldqbj" podStartSLOduration=2.980486242 podStartE2EDuration="4.038001598s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.057515356s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:41:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:41:28.035715695 +0000 UTC m=+2599.203668679" watchObservedRunningTime="2026-09-17 22:41:28.038001598 +0000 UTC m=+2599.205954580"
Sep 17 22:41:29.008484 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:41:29.008446    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-ldqbj"
Sep 17 22:41:29.009756 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:41:29.009731    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-ldqbj" podUID="0c6e1cc0-ba76-4b4a-a335-bed7161b74e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 17 22:41:30.011722 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:41:30.011677    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-ldqbj" podUID="0c6e1cc0-ba76-4b4a-a335-bed7161b74e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 17 22:41:35.016801 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:41:35.016773    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-ldqbj"
Sep 17 22:41:35.017349 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:41:35.017321    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-ldqbj" podUID="0c6e1cc0-ba76-4b4a-a335-bed7161b74e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 17 22:41:45.018098 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:41:45.018057    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-ldqbj" podUID="0c6e1cc0-ba76-4b4a-a335-bed7161b74e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 17 22:41:55.017318 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:41:55.017278    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-ldqbj" podUID="0c6e1cc0-ba76-4b4a-a335-bed7161b74e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 17 22:42:05.017790 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:42:05.017752    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-ldqbj" podUID="0c6e1cc0-ba76-4b4a-a335-bed7161b74e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 17 22:42:15.018177 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:42:15.018120    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-ldqbj" podUID="0c6e1cc0-ba76-4b4a-a335-bed7161b74e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 17 22:42:25.018074 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:42:25.018030    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-ldqbj" podUID="0c6e1cc0-ba76-4b4a-a335-bed7161b74e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 17 22:42:35.018160 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:42:35.018105    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-ldqbj" podUID="0c6e1cc0-ba76-4b4a-a335-bed7161b74e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 17 22:42:45.018293 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:42:45.018260    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-ldqbj"
Sep 17 22:42:54.472317 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:42:54.472231    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-ldqbj"]
Sep 17 22:42:54.472762 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:42:54.472557    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-ldqbj" podUID="0c6e1cc0-ba76-4b4a-a335-bed7161b74e7" containerName="kserve-container" containerID="cri-o://0fc2ce88eff0cfa609c2b1a6adef18cd4cc38bbd7b5491112d88df4dd0b246f9" gracePeriod=30
Sep 17 22:42:54.472762 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:42:54.472609    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-ldqbj" podUID="0c6e1cc0-ba76-4b4a-a335-bed7161b74e7" containerName="kube-rbac-proxy" containerID="cri-o://7663a8b9dc1c4f0019b00c24f664f9c226e2ec06f1d915d0ae1cb28ea45b437e" gracePeriod=30
Sep 17 22:42:55.012901 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:42:55.012864    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-ldqbj" podUID="0c6e1cc0-ba76-4b4a-a335-bed7161b74e7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.50:8643/healthz\": dial tcp 10.133.0.50:8643: connect: connection refused"
Sep 17 22:42:55.018303 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:42:55.018281    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-ldqbj" podUID="0c6e1cc0-ba76-4b4a-a335-bed7161b74e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 17 22:42:55.341637 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:42:55.341552    2563 generic.go:356] "Generic (PLEG): container finished" podID="0c6e1cc0-ba76-4b4a-a335-bed7161b74e7" containerID="7663a8b9dc1c4f0019b00c24f664f9c226e2ec06f1d915d0ae1cb28ea45b437e" exitCode=2
Sep 17 22:42:55.341805 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:42:55.341633    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-ldqbj" event={"ID":"0c6e1cc0-ba76-4b4a-a335-bed7161b74e7","Type":"ContainerDied","Data":"7663a8b9dc1c4f0019b00c24f664f9c226e2ec06f1d915d0ae1cb28ea45b437e"}
Sep 17 22:42:58.997272 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:42:58.997249    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-ldqbj"
Sep 17 22:42:59.074498 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:42:59.074470    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c6e1cc0-ba76-4b4a-a335-bed7161b74e7-proxy-tls\") pod \"0c6e1cc0-ba76-4b4a-a335-bed7161b74e7\" (UID: \"0c6e1cc0-ba76-4b4a-a335-bed7161b74e7\") "
Sep 17 22:42:59.074675 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:42:59.074526    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0c6e1cc0-ba76-4b4a-a335-bed7161b74e7-kserve-provision-location\") pod \"0c6e1cc0-ba76-4b4a-a335-bed7161b74e7\" (UID: \"0c6e1cc0-ba76-4b4a-a335-bed7161b74e7\") "
Sep 17 22:42:59.074675 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:42:59.074569    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0c6e1cc0-ba76-4b4a-a335-bed7161b74e7-kube-rbac-proxy-sar-config\") pod \"0c6e1cc0-ba76-4b4a-a335-bed7161b74e7\" (UID: \"0c6e1cc0-ba76-4b4a-a335-bed7161b74e7\") "
Sep 17 22:42:59.074675 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:42:59.074598    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-blvzk\" (UniqueName: \"kubernetes.io/projected/0c6e1cc0-ba76-4b4a-a335-bed7161b74e7-kube-api-access-blvzk\") pod \"0c6e1cc0-ba76-4b4a-a335-bed7161b74e7\" (UID: \"0c6e1cc0-ba76-4b4a-a335-bed7161b74e7\") "
Sep 17 22:42:59.074893 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:42:59.074865    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c6e1cc0-ba76-4b4a-a335-bed7161b74e7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0c6e1cc0-ba76-4b4a-a335-bed7161b74e7" (UID: "0c6e1cc0-ba76-4b4a-a335-bed7161b74e7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:42:59.074999 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:42:59.074888    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c6e1cc0-ba76-4b4a-a335-bed7161b74e7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0c6e1cc0-ba76-4b4a-a335-bed7161b74e7" (UID: "0c6e1cc0-ba76-4b4a-a335-bed7161b74e7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:42:59.076520 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:42:59.076501    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c6e1cc0-ba76-4b4a-a335-bed7161b74e7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0c6e1cc0-ba76-4b4a-a335-bed7161b74e7" (UID: "0c6e1cc0-ba76-4b4a-a335-bed7161b74e7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:42:59.076664 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:42:59.076643    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c6e1cc0-ba76-4b4a-a335-bed7161b74e7-kube-api-access-blvzk" (OuterVolumeSpecName: "kube-api-access-blvzk") pod "0c6e1cc0-ba76-4b4a-a335-bed7161b74e7" (UID: "0c6e1cc0-ba76-4b4a-a335-bed7161b74e7"). InnerVolumeSpecName "kube-api-access-blvzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:42:59.176131 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:42:59.176105    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0c6e1cc0-ba76-4b4a-a335-bed7161b74e7-kserve-provision-location\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:42:59.176131 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:42:59.176127    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0c6e1cc0-ba76-4b4a-a335-bed7161b74e7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:42:59.176319 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:42:59.176137    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blvzk\" (UniqueName: \"kubernetes.io/projected/0c6e1cc0-ba76-4b4a-a335-bed7161b74e7-kube-api-access-blvzk\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:42:59.176319 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:42:59.176177    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c6e1cc0-ba76-4b4a-a335-bed7161b74e7-proxy-tls\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:42:59.359059 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:42:59.359031    2563 generic.go:356] "Generic (PLEG): container finished" podID="0c6e1cc0-ba76-4b4a-a335-bed7161b74e7" containerID="0fc2ce88eff0cfa609c2b1a6adef18cd4cc38bbd7b5491112d88df4dd0b246f9" exitCode=0
Sep 17 22:42:59.359214 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:42:59.359095    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-ldqbj"
Sep 17 22:42:59.359214 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:42:59.359115    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-ldqbj" event={"ID":"0c6e1cc0-ba76-4b4a-a335-bed7161b74e7","Type":"ContainerDied","Data":"0fc2ce88eff0cfa609c2b1a6adef18cd4cc38bbd7b5491112d88df4dd0b246f9"}
Sep 17 22:42:59.359214 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:42:59.359199    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-ldqbj" event={"ID":"0c6e1cc0-ba76-4b4a-a335-bed7161b74e7","Type":"ContainerDied","Data":"098b4655e3174b3aa4e0cf367e9eb218ebc0c7239284c5f5f2d61e0c84d616a3"}
Sep 17 22:42:59.359341 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:42:59.359239    2563 scope.go:117] "RemoveContainer" containerID="7663a8b9dc1c4f0019b00c24f664f9c226e2ec06f1d915d0ae1cb28ea45b437e"
Sep 17 22:42:59.371747 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:42:59.371726    2563 scope.go:117] "RemoveContainer" containerID="0fc2ce88eff0cfa609c2b1a6adef18cd4cc38bbd7b5491112d88df4dd0b246f9"
Sep 17 22:42:59.378564 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:42:59.378546    2563 scope.go:117] "RemoveContainer" containerID="55bf1902fb9257bec73d0b9ae319feb3dae876be9ac1ad22b30b21cd91b087f7"
Sep 17 22:42:59.385032 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:42:59.385007    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-ldqbj"]
Sep 17 22:42:59.386161 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:42:59.386133    2563 scope.go:117] "RemoveContainer" containerID="7663a8b9dc1c4f0019b00c24f664f9c226e2ec06f1d915d0ae1cb28ea45b437e"
Sep 17 22:42:59.386457 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:42:59.386431    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7663a8b9dc1c4f0019b00c24f664f9c226e2ec06f1d915d0ae1cb28ea45b437e\": container with ID starting with 7663a8b9dc1c4f0019b00c24f664f9c226e2ec06f1d915d0ae1cb28ea45b437e not found: ID does not exist" containerID="7663a8b9dc1c4f0019b00c24f664f9c226e2ec06f1d915d0ae1cb28ea45b437e"
Sep 17 22:42:59.386559 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:42:59.386466    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7663a8b9dc1c4f0019b00c24f664f9c226e2ec06f1d915d0ae1cb28ea45b437e"} err="failed to get container status \"7663a8b9dc1c4f0019b00c24f664f9c226e2ec06f1d915d0ae1cb28ea45b437e\": rpc error: code = NotFound desc = could not find container \"7663a8b9dc1c4f0019b00c24f664f9c226e2ec06f1d915d0ae1cb28ea45b437e\": container with ID starting with 7663a8b9dc1c4f0019b00c24f664f9c226e2ec06f1d915d0ae1cb28ea45b437e not found: ID does not exist"
Sep 17 22:42:59.386559 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:42:59.386483    2563 scope.go:117] "RemoveContainer" containerID="0fc2ce88eff0cfa609c2b1a6adef18cd4cc38bbd7b5491112d88df4dd0b246f9"
Sep 17 22:42:59.386812 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:42:59.386787    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fc2ce88eff0cfa609c2b1a6adef18cd4cc38bbd7b5491112d88df4dd0b246f9\": container with ID starting with 0fc2ce88eff0cfa609c2b1a6adef18cd4cc38bbd7b5491112d88df4dd0b246f9 not found: ID does not exist" containerID="0fc2ce88eff0cfa609c2b1a6adef18cd4cc38bbd7b5491112d88df4dd0b246f9"
Sep 17 22:42:59.386933 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:42:59.386845    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fc2ce88eff0cfa609c2b1a6adef18cd4cc38bbd7b5491112d88df4dd0b246f9"} err="failed to get container status \"0fc2ce88eff0cfa609c2b1a6adef18cd4cc38bbd7b5491112d88df4dd0b246f9\": rpc error: code = NotFound desc = could not find container \"0fc2ce88eff0cfa609c2b1a6adef18cd4cc38bbd7b5491112d88df4dd0b246f9\": container with ID starting with 0fc2ce88eff0cfa609c2b1a6adef18cd4cc38bbd7b5491112d88df4dd0b246f9 not found: ID does not exist"
Sep 17 22:42:59.386933 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:42:59.386865    2563 scope.go:117] "RemoveContainer" containerID="55bf1902fb9257bec73d0b9ae319feb3dae876be9ac1ad22b30b21cd91b087f7"
Sep 17 22:42:59.387330 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:42:59.387302    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55bf1902fb9257bec73d0b9ae319feb3dae876be9ac1ad22b30b21cd91b087f7\": container with ID starting with 55bf1902fb9257bec73d0b9ae319feb3dae876be9ac1ad22b30b21cd91b087f7 not found: ID does not exist" containerID="55bf1902fb9257bec73d0b9ae319feb3dae876be9ac1ad22b30b21cd91b087f7"
Sep 17 22:42:59.387413 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:42:59.387338    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55bf1902fb9257bec73d0b9ae319feb3dae876be9ac1ad22b30b21cd91b087f7"} err="failed to get container status \"55bf1902fb9257bec73d0b9ae319feb3dae876be9ac1ad22b30b21cd91b087f7\": rpc error: code = NotFound desc = could not find container \"55bf1902fb9257bec73d0b9ae319feb3dae876be9ac1ad22b30b21cd91b087f7\": container with ID starting with 55bf1902fb9257bec73d0b9ae319feb3dae876be9ac1ad22b30b21cd91b087f7 not found: ID does not exist"
Sep 17 22:42:59.388363 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:42:59.388347    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-ldqbj"]
Sep 17 22:43:00.914097 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:43:00.914064    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c6e1cc0-ba76-4b4a-a335-bed7161b74e7" path="/var/lib/kubelet/pods/0c6e1cc0-ba76-4b4a-a335-bed7161b74e7/volumes"
Sep 17 22:43:49.682557 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:43:49.682526    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nrkd5_34b8ef2b-97ba-4562-87ee-8da0cb4604c7/ovn-acl-logging/0.log"
Sep 17 22:43:49.685807 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:43:49.685784    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nrkd5_34b8ef2b-97ba-4562-87ee-8da0cb4604c7/ovn-acl-logging/0.log"
Sep 17 22:43:52.547763 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:43:52.547731    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bszqq"]
Sep 17 22:43:52.548151 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:43:52.548004    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c6e1cc0-ba76-4b4a-a335-bed7161b74e7" containerName="kserve-container"
Sep 17 22:43:52.548151 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:43:52.548015    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c6e1cc0-ba76-4b4a-a335-bed7161b74e7" containerName="kserve-container"
Sep 17 22:43:52.548151 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:43:52.548024    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c6e1cc0-ba76-4b4a-a335-bed7161b74e7" containerName="storage-initializer"
Sep 17 22:43:52.548151 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:43:52.548029    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c6e1cc0-ba76-4b4a-a335-bed7161b74e7" containerName="storage-initializer"
Sep 17 22:43:52.548151 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:43:52.548038    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c6e1cc0-ba76-4b4a-a335-bed7161b74e7" containerName="kube-rbac-proxy"
Sep 17 22:43:52.548151 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:43:52.548044    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c6e1cc0-ba76-4b4a-a335-bed7161b74e7" containerName="kube-rbac-proxy"
Sep 17 22:43:52.548151 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:43:52.548097    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c6e1cc0-ba76-4b4a-a335-bed7161b74e7" containerName="kube-rbac-proxy"
Sep 17 22:43:52.548151 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:43:52.548104    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c6e1cc0-ba76-4b4a-a335-bed7161b74e7" containerName="kserve-container"
Sep 17 22:43:52.551096 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:43:52.551081    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bszqq"
Sep 17 22:43:52.553591 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:43:52.553567    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 22:43:52.553591 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:43:52.553567    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:43:52.553759 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:43:52.553605    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-predictor-serving-cert\""
Sep 17 22:43:52.553759 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:43:52.553592    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v4p4m\""
Sep 17 22:43:52.554371 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:43:52.554354    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-kube-rbac-proxy-sar-config\""
Sep 17 22:43:52.554371 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:43:52.554369    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 22:43:52.554536 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:43:52.554410    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 22:43:52.561010 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:43:52.560990    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bszqq"]
Sep 17 22:43:52.597120 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:43:52.597096    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ca0f9ab-d9e6-4030-8266-b7eb29138264-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-bszqq\" (UID: \"4ca0f9ab-d9e6-4030-8266-b7eb29138264\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bszqq"
Sep 17 22:43:52.597262 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:43:52.597126    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ca0f9ab-d9e6-4030-8266-b7eb29138264-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-bszqq\" (UID: \"4ca0f9ab-d9e6-4030-8266-b7eb29138264\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bszqq"
Sep 17 22:43:52.597262 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:43:52.597205    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swx4x\" (UniqueName: \"kubernetes.io/projected/4ca0f9ab-d9e6-4030-8266-b7eb29138264-kube-api-access-swx4x\") pod \"isvc-paddle-runtime-predictor-58b99fff76-bszqq\" (UID: \"4ca0f9ab-d9e6-4030-8266-b7eb29138264\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bszqq"
Sep 17 22:43:52.597356 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:43:52.597290    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ca0f9ab-d9e6-4030-8266-b7eb29138264-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-bszqq\" (UID: \"4ca0f9ab-d9e6-4030-8266-b7eb29138264\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bszqq"
Sep 17 22:43:52.597356 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:43:52.597337    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ca0f9ab-d9e6-4030-8266-b7eb29138264-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-bszqq\" (UID: \"4ca0f9ab-d9e6-4030-8266-b7eb29138264\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bszqq"
Sep 17 22:43:52.697844 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:43:52.697814    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ca0f9ab-d9e6-4030-8266-b7eb29138264-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-bszqq\" (UID: \"4ca0f9ab-d9e6-4030-8266-b7eb29138264\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bszqq"
Sep 17 22:43:52.698002 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:43:52.697858    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ca0f9ab-d9e6-4030-8266-b7eb29138264-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-bszqq\" (UID: \"4ca0f9ab-d9e6-4030-8266-b7eb29138264\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bszqq"
Sep 17 22:43:52.698002 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:43:52.697884    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ca0f9ab-d9e6-4030-8266-b7eb29138264-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-bszqq\" (UID: \"4ca0f9ab-d9e6-4030-8266-b7eb29138264\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bszqq"
Sep 17 22:43:52.698002 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:43:52.697967    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-runtime-predictor-serving-cert: secret "isvc-paddle-runtime-predictor-serving-cert" not found
Sep 17 22:43:52.698184 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:43:52.697998    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ca0f9ab-d9e6-4030-8266-b7eb29138264-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-bszqq\" (UID: \"4ca0f9ab-d9e6-4030-8266-b7eb29138264\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bszqq"
Sep 17 22:43:52.698184 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:43:52.698048    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ca0f9ab-d9e6-4030-8266-b7eb29138264-proxy-tls podName:4ca0f9ab-d9e6-4030-8266-b7eb29138264 nodeName:}" failed. No retries permitted until 2026-09-17 22:43:53.198026376 +0000 UTC m=+2744.365979337 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4ca0f9ab-d9e6-4030-8266-b7eb29138264-proxy-tls") pod "isvc-paddle-runtime-predictor-58b99fff76-bszqq" (UID: "4ca0f9ab-d9e6-4030-8266-b7eb29138264") : secret "isvc-paddle-runtime-predictor-serving-cert" not found
Sep 17 22:43:52.698184 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:43:52.698088    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swx4x\" (UniqueName: \"kubernetes.io/projected/4ca0f9ab-d9e6-4030-8266-b7eb29138264-kube-api-access-swx4x\") pod \"isvc-paddle-runtime-predictor-58b99fff76-bszqq\" (UID: \"4ca0f9ab-d9e6-4030-8266-b7eb29138264\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bszqq"
Sep 17 22:43:52.698365 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:43:52.698347    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ca0f9ab-d9e6-4030-8266-b7eb29138264-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-bszqq\" (UID: \"4ca0f9ab-d9e6-4030-8266-b7eb29138264\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bszqq"
Sep 17 22:43:52.698601 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:43:52.698581    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ca0f9ab-d9e6-4030-8266-b7eb29138264-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-bszqq\" (UID: \"4ca0f9ab-d9e6-4030-8266-b7eb29138264\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bszqq"
Sep 17 22:43:52.698669 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:43:52.698613    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ca0f9ab-d9e6-4030-8266-b7eb29138264-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-bszqq\" (UID: \"4ca0f9ab-d9e6-4030-8266-b7eb29138264\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bszqq"
Sep 17 22:43:52.705944 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:43:52.705924    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swx4x\" (UniqueName: \"kubernetes.io/projected/4ca0f9ab-d9e6-4030-8266-b7eb29138264-kube-api-access-swx4x\") pod \"isvc-paddle-runtime-predictor-58b99fff76-bszqq\" (UID: \"4ca0f9ab-d9e6-4030-8266-b7eb29138264\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bszqq"
Sep 17 22:43:53.202698 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:43:53.202661    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ca0f9ab-d9e6-4030-8266-b7eb29138264-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-bszqq\" (UID: \"4ca0f9ab-d9e6-4030-8266-b7eb29138264\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bszqq"
Sep 17 22:43:53.204966 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:43:53.204948    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ca0f9ab-d9e6-4030-8266-b7eb29138264-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-bszqq\" (UID: \"4ca0f9ab-d9e6-4030-8266-b7eb29138264\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bszqq"
Sep 17 22:43:53.461387 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:43:53.461304    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bszqq"
Sep 17 22:43:53.730242 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:43:53.730217    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bszqq"]
Sep 17 22:43:53.732176 ip-10-0-138-45 kubenswrapper[2563]: W0917 22:43:53.732126    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ca0f9ab_d9e6_4030_8266_b7eb29138264.slice/crio-5625a55503ff55d77c6ccd4f2f9813bc8d098167d7dc99d53753298859a96483 WatchSource:0}: Error finding container 5625a55503ff55d77c6ccd4f2f9813bc8d098167d7dc99d53753298859a96483: Status 404 returned error can't find the container with id 5625a55503ff55d77c6ccd4f2f9813bc8d098167d7dc99d53753298859a96483
Sep 17 22:43:54.575336 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:43:54.575299    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bszqq" event={"ID":"4ca0f9ab-d9e6-4030-8266-b7eb29138264","Type":"ContainerStarted","Data":"25af9aa328c4b5c662bf298a2e958ad1c39bfa419a2a2aaab2f4e3331c1a833a"}
Sep 17 22:43:54.575512 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:43:54.575357    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bszqq" event={"ID":"4ca0f9ab-d9e6-4030-8266-b7eb29138264","Type":"ContainerStarted","Data":"5625a55503ff55d77c6ccd4f2f9813bc8d098167d7dc99d53753298859a96483"}
Sep 17 22:43:58.592546 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:43:58.592513    2563 generic.go:356] "Generic (PLEG): container finished" podID="4ca0f9ab-d9e6-4030-8266-b7eb29138264" containerID="25af9aa328c4b5c662bf298a2e958ad1c39bfa419a2a2aaab2f4e3331c1a833a" exitCode=0
Sep 17 22:43:58.592953 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:43:58.592584    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bszqq" event={"ID":"4ca0f9ab-d9e6-4030-8266-b7eb29138264","Type":"ContainerDied","Data":"25af9aa328c4b5c662bf298a2e958ad1c39bfa419a2a2aaab2f4e3331c1a833a"}
Sep 17 22:43:58.593782 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:43:58.593767    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 22:44:10.650756 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:44:10.650717    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bszqq" event={"ID":"4ca0f9ab-d9e6-4030-8266-b7eb29138264","Type":"ContainerStarted","Data":"89018437d36d4ddbe672ff9fe595141319d51017b7aa6cd1479fc2a380365de5"}
Sep 17 22:44:10.651212 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:44:10.650792    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bszqq" event={"ID":"4ca0f9ab-d9e6-4030-8266-b7eb29138264","Type":"ContainerStarted","Data":"0ffd851d01e4807d3cec98445ac3e6edbb1f1428bebf7b76845b6d1f0611d6ed"}
Sep 17 22:44:10.651212 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:44:10.651008    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bszqq"
Sep 17 22:44:10.669432 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:44:10.669385    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bszqq" podStartSLOduration=3.289067023 podStartE2EDuration="18.669369803s" totalImagesPullingTime="11.382065199s" totalInitContainerRuntime="3.998237581s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:43:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:44:10.66772427 +0000 UTC m=+2761.835677252" watchObservedRunningTime="2026-09-17 22:44:10.669369803 +0000 UTC m=+2761.837322784"
Sep 17 22:44:11.654684 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:44:11.654655    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bszqq"
Sep 17 22:44:11.656162 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:44:11.656115    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bszqq" podUID="4ca0f9ab-d9e6-4030-8266-b7eb29138264" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 17 22:44:12.659057 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:44:12.659012    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bszqq" podUID="4ca0f9ab-d9e6-4030-8266-b7eb29138264" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 17 22:44:17.663085 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:44:17.663058    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bszqq"
Sep 17 22:44:17.663579 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:44:17.663550    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bszqq" podUID="4ca0f9ab-d9e6-4030-8266-b7eb29138264" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 17 22:44:27.663662 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:44:27.663569    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bszqq" podUID="4ca0f9ab-d9e6-4030-8266-b7eb29138264" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 17 22:44:37.663570 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:44:37.663531    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bszqq" podUID="4ca0f9ab-d9e6-4030-8266-b7eb29138264" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 17 22:44:47.664223 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:44:47.664180    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bszqq" podUID="4ca0f9ab-d9e6-4030-8266-b7eb29138264" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 17 22:44:57.664317 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:44:57.664283    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bszqq"
Sep 17 22:45:04.021527 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:04.021489    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bszqq"]
Sep 17 22:45:04.021998 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:04.021724    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bszqq" podUID="4ca0f9ab-d9e6-4030-8266-b7eb29138264" containerName="kserve-container" containerID="cri-o://0ffd851d01e4807d3cec98445ac3e6edbb1f1428bebf7b76845b6d1f0611d6ed" gracePeriod=30
Sep 17 22:45:04.021998 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:04.021801    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bszqq" podUID="4ca0f9ab-d9e6-4030-8266-b7eb29138264" containerName="kube-rbac-proxy" containerID="cri-o://89018437d36d4ddbe672ff9fe595141319d51017b7aa6cd1479fc2a380365de5" gracePeriod=30
Sep 17 22:45:04.866723 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:04.866688    2563 generic.go:356] "Generic (PLEG): container finished" podID="4ca0f9ab-d9e6-4030-8266-b7eb29138264" containerID="89018437d36d4ddbe672ff9fe595141319d51017b7aa6cd1479fc2a380365de5" exitCode=2
Sep 17 22:45:04.866942 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:04.866759    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bszqq" event={"ID":"4ca0f9ab-d9e6-4030-8266-b7eb29138264","Type":"ContainerDied","Data":"89018437d36d4ddbe672ff9fe595141319d51017b7aa6cd1479fc2a380365de5"}
Sep 17 22:45:07.057447 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:07.057421    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bszqq"
Sep 17 22:45:07.209390 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:07.209362    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ca0f9ab-d9e6-4030-8266-b7eb29138264-proxy-tls\") pod \"4ca0f9ab-d9e6-4030-8266-b7eb29138264\" (UID: \"4ca0f9ab-d9e6-4030-8266-b7eb29138264\") "
Sep 17 22:45:07.209576 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:07.209407    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ca0f9ab-d9e6-4030-8266-b7eb29138264-kube-rbac-proxy-sar-config\") pod \"4ca0f9ab-d9e6-4030-8266-b7eb29138264\" (UID: \"4ca0f9ab-d9e6-4030-8266-b7eb29138264\") "
Sep 17 22:45:07.209576 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:07.209447    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ca0f9ab-d9e6-4030-8266-b7eb29138264-kserve-provision-location\") pod \"4ca0f9ab-d9e6-4030-8266-b7eb29138264\" (UID: \"4ca0f9ab-d9e6-4030-8266-b7eb29138264\") "
Sep 17 22:45:07.209698 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:07.209650    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-swx4x\" (UniqueName: \"kubernetes.io/projected/4ca0f9ab-d9e6-4030-8266-b7eb29138264-kube-api-access-swx4x\") pod \"4ca0f9ab-d9e6-4030-8266-b7eb29138264\" (UID: \"4ca0f9ab-d9e6-4030-8266-b7eb29138264\") "
Sep 17 22:45:07.209698 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:07.209681    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ca0f9ab-d9e6-4030-8266-b7eb29138264-cabundle-cert\") pod \"4ca0f9ab-d9e6-4030-8266-b7eb29138264\" (UID: \"4ca0f9ab-d9e6-4030-8266-b7eb29138264\") "
Sep 17 22:45:07.209819 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:07.209795    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ca0f9ab-d9e6-4030-8266-b7eb29138264-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4ca0f9ab-d9e6-4030-8266-b7eb29138264" (UID: "4ca0f9ab-d9e6-4030-8266-b7eb29138264"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:45:07.210054 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:07.210035    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ca0f9ab-d9e6-4030-8266-b7eb29138264-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:45:07.210116 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:07.210098    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ca0f9ab-d9e6-4030-8266-b7eb29138264-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4ca0f9ab-d9e6-4030-8266-b7eb29138264" (UID: "4ca0f9ab-d9e6-4030-8266-b7eb29138264"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:45:07.211533 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:07.211501    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ca0f9ab-d9e6-4030-8266-b7eb29138264-kube-api-access-swx4x" (OuterVolumeSpecName: "kube-api-access-swx4x") pod "4ca0f9ab-d9e6-4030-8266-b7eb29138264" (UID: "4ca0f9ab-d9e6-4030-8266-b7eb29138264"). InnerVolumeSpecName "kube-api-access-swx4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:45:07.211533 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:07.211517    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ca0f9ab-d9e6-4030-8266-b7eb29138264-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4ca0f9ab-d9e6-4030-8266-b7eb29138264" (UID: "4ca0f9ab-d9e6-4030-8266-b7eb29138264"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:45:07.223834 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:07.223809    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ca0f9ab-d9e6-4030-8266-b7eb29138264-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4ca0f9ab-d9e6-4030-8266-b7eb29138264" (UID: "4ca0f9ab-d9e6-4030-8266-b7eb29138264"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:45:07.310395 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:07.310371    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ca0f9ab-d9e6-4030-8266-b7eb29138264-kserve-provision-location\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:45:07.310395 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:07.310392    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swx4x\" (UniqueName: \"kubernetes.io/projected/4ca0f9ab-d9e6-4030-8266-b7eb29138264-kube-api-access-swx4x\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:45:07.310545 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:07.310404    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ca0f9ab-d9e6-4030-8266-b7eb29138264-cabundle-cert\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:45:07.310545 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:07.310413    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ca0f9ab-d9e6-4030-8266-b7eb29138264-proxy-tls\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:45:07.879643 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:07.879610    2563 generic.go:356] "Generic (PLEG): container finished" podID="4ca0f9ab-d9e6-4030-8266-b7eb29138264" containerID="0ffd851d01e4807d3cec98445ac3e6edbb1f1428bebf7b76845b6d1f0611d6ed" exitCode=0
Sep 17 22:45:07.879817 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:07.879660    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bszqq"
Sep 17 22:45:07.879817 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:07.879696    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bszqq" event={"ID":"4ca0f9ab-d9e6-4030-8266-b7eb29138264","Type":"ContainerDied","Data":"0ffd851d01e4807d3cec98445ac3e6edbb1f1428bebf7b76845b6d1f0611d6ed"}
Sep 17 22:45:07.879817 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:07.879747    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bszqq" event={"ID":"4ca0f9ab-d9e6-4030-8266-b7eb29138264","Type":"ContainerDied","Data":"5625a55503ff55d77c6ccd4f2f9813bc8d098167d7dc99d53753298859a96483"}
Sep 17 22:45:07.879817 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:07.879772    2563 scope.go:117] "RemoveContainer" containerID="89018437d36d4ddbe672ff9fe595141319d51017b7aa6cd1479fc2a380365de5"
Sep 17 22:45:07.888315 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:07.888299    2563 scope.go:117] "RemoveContainer" containerID="0ffd851d01e4807d3cec98445ac3e6edbb1f1428bebf7b76845b6d1f0611d6ed"
Sep 17 22:45:07.895193 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:07.895174    2563 scope.go:117] "RemoveContainer" containerID="25af9aa328c4b5c662bf298a2e958ad1c39bfa419a2a2aaab2f4e3331c1a833a"
Sep 17 22:45:07.901680 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:07.901620    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bszqq"]
Sep 17 22:45:07.904109 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:07.904039    2563 scope.go:117] "RemoveContainer" containerID="89018437d36d4ddbe672ff9fe595141319d51017b7aa6cd1479fc2a380365de5"
Sep 17 22:45:07.904707 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:45:07.904683    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89018437d36d4ddbe672ff9fe595141319d51017b7aa6cd1479fc2a380365de5\": container with ID starting with 89018437d36d4ddbe672ff9fe595141319d51017b7aa6cd1479fc2a380365de5 not found: ID does not exist" containerID="89018437d36d4ddbe672ff9fe595141319d51017b7aa6cd1479fc2a380365de5"
Sep 17 22:45:07.904778 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:07.904714    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89018437d36d4ddbe672ff9fe595141319d51017b7aa6cd1479fc2a380365de5"} err="failed to get container status \"89018437d36d4ddbe672ff9fe595141319d51017b7aa6cd1479fc2a380365de5\": rpc error: code = NotFound desc = could not find container \"89018437d36d4ddbe672ff9fe595141319d51017b7aa6cd1479fc2a380365de5\": container with ID starting with 89018437d36d4ddbe672ff9fe595141319d51017b7aa6cd1479fc2a380365de5 not found: ID does not exist"
Sep 17 22:45:07.904778 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:07.904731    2563 scope.go:117] "RemoveContainer" containerID="0ffd851d01e4807d3cec98445ac3e6edbb1f1428bebf7b76845b6d1f0611d6ed"
Sep 17 22:45:07.905057 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:45:07.905039    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ffd851d01e4807d3cec98445ac3e6edbb1f1428bebf7b76845b6d1f0611d6ed\": container with ID starting with 0ffd851d01e4807d3cec98445ac3e6edbb1f1428bebf7b76845b6d1f0611d6ed not found: ID does not exist" containerID="0ffd851d01e4807d3cec98445ac3e6edbb1f1428bebf7b76845b6d1f0611d6ed"
Sep 17 22:45:07.905117 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:07.905061    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ffd851d01e4807d3cec98445ac3e6edbb1f1428bebf7b76845b6d1f0611d6ed"} err="failed to get container status \"0ffd851d01e4807d3cec98445ac3e6edbb1f1428bebf7b76845b6d1f0611d6ed\": rpc error: code = NotFound desc = could not find container \"0ffd851d01e4807d3cec98445ac3e6edbb1f1428bebf7b76845b6d1f0611d6ed\": container with ID starting with 0ffd851d01e4807d3cec98445ac3e6edbb1f1428bebf7b76845b6d1f0611d6ed not found: ID does not exist"
Sep 17 22:45:07.905117 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:07.905077    2563 scope.go:117] "RemoveContainer" containerID="25af9aa328c4b5c662bf298a2e958ad1c39bfa419a2a2aaab2f4e3331c1a833a"
Sep 17 22:45:07.905338 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:45:07.905323    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25af9aa328c4b5c662bf298a2e958ad1c39bfa419a2a2aaab2f4e3331c1a833a\": container with ID starting with 25af9aa328c4b5c662bf298a2e958ad1c39bfa419a2a2aaab2f4e3331c1a833a not found: ID does not exist" containerID="25af9aa328c4b5c662bf298a2e958ad1c39bfa419a2a2aaab2f4e3331c1a833a"
Sep 17 22:45:07.905383 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:07.905345    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25af9aa328c4b5c662bf298a2e958ad1c39bfa419a2a2aaab2f4e3331c1a833a"} err="failed to get container status \"25af9aa328c4b5c662bf298a2e958ad1c39bfa419a2a2aaab2f4e3331c1a833a\": rpc error: code = NotFound desc = could not find container \"25af9aa328c4b5c662bf298a2e958ad1c39bfa419a2a2aaab2f4e3331c1a833a\": container with ID starting with 25af9aa328c4b5c662bf298a2e958ad1c39bfa419a2a2aaab2f4e3331c1a833a not found: ID does not exist"
Sep 17 22:45:07.905429 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:07.905414    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bszqq"]
Sep 17 22:45:08.913443 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:08.913406    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ca0f9ab-d9e6-4030-8266-b7eb29138264" path="/var/lib/kubelet/pods/4ca0f9ab-d9e6-4030-8266-b7eb29138264/volumes"
Sep 17 22:45:09.175332 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:09.175243    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-j5sk7"]
Sep 17 22:45:09.175621 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:09.175604    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ca0f9ab-d9e6-4030-8266-b7eb29138264" containerName="kube-rbac-proxy"
Sep 17 22:45:09.175702 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:09.175624    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ca0f9ab-d9e6-4030-8266-b7eb29138264" containerName="kube-rbac-proxy"
Sep 17 22:45:09.175702 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:09.175637    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ca0f9ab-d9e6-4030-8266-b7eb29138264" containerName="kserve-container"
Sep 17 22:45:09.175702 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:09.175646    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ca0f9ab-d9e6-4030-8266-b7eb29138264" containerName="kserve-container"
Sep 17 22:45:09.175702 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:09.175658    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ca0f9ab-d9e6-4030-8266-b7eb29138264" containerName="storage-initializer"
Sep 17 22:45:09.175702 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:09.175668    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ca0f9ab-d9e6-4030-8266-b7eb29138264" containerName="storage-initializer"
Sep 17 22:45:09.175968 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:09.175735    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ca0f9ab-d9e6-4030-8266-b7eb29138264" containerName="kserve-container"
Sep 17 22:45:09.175968 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:09.175746    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ca0f9ab-d9e6-4030-8266-b7eb29138264" containerName="kube-rbac-proxy"
Sep 17 22:45:09.180268 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:09.180248    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-j5sk7"
Sep 17 22:45:09.182678 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:09.182649    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-predictor-serving-cert\""
Sep 17 22:45:09.182678 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:09.182663    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:45:09.182846 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:09.182662    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 22:45:09.182846 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:09.182818    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 22:45:09.183073 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:09.183050    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v4p4m\""
Sep 17 22:45:09.183451 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:09.183432    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 17 22:45:09.183548 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:09.183457    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 22:45:09.188861 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:09.188841    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-j5sk7"]
Sep 17 22:45:09.325794 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:09.325751    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afa1a8ef-edc0-42cd-8eab-1f893e362a1f-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-j5sk7\" (UID: \"afa1a8ef-edc0-42cd-8eab-1f893e362a1f\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-j5sk7"
Sep 17 22:45:09.325956 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:09.325799    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzn8x\" (UniqueName: \"kubernetes.io/projected/afa1a8ef-edc0-42cd-8eab-1f893e362a1f-kube-api-access-wzn8x\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-j5sk7\" (UID: \"afa1a8ef-edc0-42cd-8eab-1f893e362a1f\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-j5sk7"
Sep 17 22:45:09.325956 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:09.325828    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/afa1a8ef-edc0-42cd-8eab-1f893e362a1f-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-j5sk7\" (UID: \"afa1a8ef-edc0-42cd-8eab-1f893e362a1f\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-j5sk7"
Sep 17 22:45:09.325956 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:09.325847    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/afa1a8ef-edc0-42cd-8eab-1f893e362a1f-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-j5sk7\" (UID: \"afa1a8ef-edc0-42cd-8eab-1f893e362a1f\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-j5sk7"
Sep 17 22:45:09.325956 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:09.325864    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/afa1a8ef-edc0-42cd-8eab-1f893e362a1f-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-j5sk7\" (UID: \"afa1a8ef-edc0-42cd-8eab-1f893e362a1f\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-j5sk7"
Sep 17 22:45:09.426428 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:09.426342    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/afa1a8ef-edc0-42cd-8eab-1f893e362a1f-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-j5sk7\" (UID: \"afa1a8ef-edc0-42cd-8eab-1f893e362a1f\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-j5sk7"
Sep 17 22:45:09.426428 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:09.426382    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/afa1a8ef-edc0-42cd-8eab-1f893e362a1f-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-j5sk7\" (UID: \"afa1a8ef-edc0-42cd-8eab-1f893e362a1f\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-j5sk7"
Sep 17 22:45:09.426655 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:09.426441    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afa1a8ef-edc0-42cd-8eab-1f893e362a1f-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-j5sk7\" (UID: \"afa1a8ef-edc0-42cd-8eab-1f893e362a1f\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-j5sk7"
Sep 17 22:45:09.426655 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:09.426481    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzn8x\" (UniqueName: \"kubernetes.io/projected/afa1a8ef-edc0-42cd-8eab-1f893e362a1f-kube-api-access-wzn8x\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-j5sk7\" (UID: \"afa1a8ef-edc0-42cd-8eab-1f893e362a1f\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-j5sk7"
Sep 17 22:45:09.426655 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:09.426526    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/afa1a8ef-edc0-42cd-8eab-1f893e362a1f-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-j5sk7\" (UID: \"afa1a8ef-edc0-42cd-8eab-1f893e362a1f\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-j5sk7"
Sep 17 22:45:09.426820 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:09.426752    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/afa1a8ef-edc0-42cd-8eab-1f893e362a1f-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-j5sk7\" (UID: \"afa1a8ef-edc0-42cd-8eab-1f893e362a1f\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-j5sk7"
Sep 17 22:45:09.427125 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:09.427105    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/afa1a8ef-edc0-42cd-8eab-1f893e362a1f-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-j5sk7\" (UID: \"afa1a8ef-edc0-42cd-8eab-1f893e362a1f\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-j5sk7"
Sep 17 22:45:09.427233 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:09.427172    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/afa1a8ef-edc0-42cd-8eab-1f893e362a1f-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-j5sk7\" (UID: \"afa1a8ef-edc0-42cd-8eab-1f893e362a1f\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-j5sk7"
Sep 17 22:45:09.428857 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:09.428841    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afa1a8ef-edc0-42cd-8eab-1f893e362a1f-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-j5sk7\" (UID: \"afa1a8ef-edc0-42cd-8eab-1f893e362a1f\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-j5sk7"
Sep 17 22:45:09.434235 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:09.434213    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzn8x\" (UniqueName: \"kubernetes.io/projected/afa1a8ef-edc0-42cd-8eab-1f893e362a1f-kube-api-access-wzn8x\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-j5sk7\" (UID: \"afa1a8ef-edc0-42cd-8eab-1f893e362a1f\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-j5sk7"
Sep 17 22:45:09.490414 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:09.490386    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-j5sk7"
Sep 17 22:45:09.760239 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:09.760216    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-j5sk7"]
Sep 17 22:45:09.762556 ip-10-0-138-45 kubenswrapper[2563]: W0917 22:45:09.762531    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafa1a8ef_edc0_42cd_8eab_1f893e362a1f.slice/crio-10a3058ac2430698cb59b625641c87184ab878d0e83b0506d3fbe1bd87a82cc6 WatchSource:0}: Error finding container 10a3058ac2430698cb59b625641c87184ab878d0e83b0506d3fbe1bd87a82cc6: Status 404 returned error can't find the container with id 10a3058ac2430698cb59b625641c87184ab878d0e83b0506d3fbe1bd87a82cc6
Sep 17 22:45:09.889284 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:09.889243    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-j5sk7" event={"ID":"afa1a8ef-edc0-42cd-8eab-1f893e362a1f","Type":"ContainerStarted","Data":"25ec95a83ff2c6fb908b0851a52b26d15b804fe116c020939556be4f8f5b6c2f"}
Sep 17 22:45:09.889477 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:09.889315    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-j5sk7" event={"ID":"afa1a8ef-edc0-42cd-8eab-1f893e362a1f","Type":"ContainerStarted","Data":"10a3058ac2430698cb59b625641c87184ab878d0e83b0506d3fbe1bd87a82cc6"}
Sep 17 22:45:14.908640 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:14.908605    2563 generic.go:356] "Generic (PLEG): container finished" podID="afa1a8ef-edc0-42cd-8eab-1f893e362a1f" containerID="25ec95a83ff2c6fb908b0851a52b26d15b804fe116c020939556be4f8f5b6c2f" exitCode=0
Sep 17 22:45:14.912340 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:14.912316    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-j5sk7" event={"ID":"afa1a8ef-edc0-42cd-8eab-1f893e362a1f","Type":"ContainerDied","Data":"25ec95a83ff2c6fb908b0851a52b26d15b804fe116c020939556be4f8f5b6c2f"}
Sep 17 22:45:15.914085 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:15.914050    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-j5sk7" event={"ID":"afa1a8ef-edc0-42cd-8eab-1f893e362a1f","Type":"ContainerStarted","Data":"e483a8ec7bbb3ed69d840de18c50b9dd413849908ca2bcda62641fb40a5509c8"}
Sep 17 22:45:15.914554 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:15.914118    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-j5sk7" event={"ID":"afa1a8ef-edc0-42cd-8eab-1f893e362a1f","Type":"ContainerStarted","Data":"7e946bcffbaa87ea4cbdc397c028110665859637373283ef0477ae26742e7a2c"}
Sep 17 22:45:15.914554 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:15.914344    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-j5sk7"
Sep 17 22:45:15.932123 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:15.932078    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-j5sk7" podStartSLOduration=2.815032029 podStartE2EDuration="6.932064848s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.117032819s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:45:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:45:15.931558069 +0000 UTC m=+2827.099511053" watchObservedRunningTime="2026-09-17 22:45:15.932064848 +0000 UTC m=+2827.100017829"
Sep 17 22:45:16.918155 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:16.918114    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-j5sk7"
Sep 17 22:45:16.919427 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:16.919408    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-j5sk7" podUID="afa1a8ef-edc0-42cd-8eab-1f893e362a1f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 17 22:45:17.922034 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:17.921998    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-j5sk7" podUID="afa1a8ef-edc0-42cd-8eab-1f893e362a1f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 17 22:45:22.927213 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:22.927186    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-j5sk7"
Sep 17 22:45:22.927788 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:22.927674    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-j5sk7" podUID="afa1a8ef-edc0-42cd-8eab-1f893e362a1f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 17 22:45:32.927770 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:32.927733    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-j5sk7" podUID="afa1a8ef-edc0-42cd-8eab-1f893e362a1f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 17 22:45:42.928319 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:42.928277    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-j5sk7" podUID="afa1a8ef-edc0-42cd-8eab-1f893e362a1f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 17 22:45:52.928416 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:45:52.928332    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-j5sk7" podUID="afa1a8ef-edc0-42cd-8eab-1f893e362a1f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 17 22:46:02.928295 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:02.928265    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-j5sk7"
Sep 17 22:46:10.778497 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:10.778466    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-j5sk7"]
Sep 17 22:46:10.778970 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:10.778775    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-j5sk7" podUID="afa1a8ef-edc0-42cd-8eab-1f893e362a1f" containerName="kserve-container" containerID="cri-o://7e946bcffbaa87ea4cbdc397c028110665859637373283ef0477ae26742e7a2c" gracePeriod=30
Sep 17 22:46:10.778970 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:10.778864    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-j5sk7" podUID="afa1a8ef-edc0-42cd-8eab-1f893e362a1f" containerName="kube-rbac-proxy" containerID="cri-o://e483a8ec7bbb3ed69d840de18c50b9dd413849908ca2bcda62641fb40a5509c8" gracePeriod=30
Sep 17 22:46:11.130123 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:11.130039    2563 generic.go:356] "Generic (PLEG): container finished" podID="afa1a8ef-edc0-42cd-8eab-1f893e362a1f" containerID="e483a8ec7bbb3ed69d840de18c50b9dd413849908ca2bcda62641fb40a5509c8" exitCode=2
Sep 17 22:46:11.130123 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:11.130110    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-j5sk7" event={"ID":"afa1a8ef-edc0-42cd-8eab-1f893e362a1f","Type":"ContainerDied","Data":"e483a8ec7bbb3ed69d840de18c50b9dd413849908ca2bcda62641fb40a5509c8"}
Sep 17 22:46:12.922698 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:12.922660    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-j5sk7" podUID="afa1a8ef-edc0-42cd-8eab-1f893e362a1f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.52:8643/healthz\": dial tcp 10.133.0.52:8643: connect: connection refused"
Sep 17 22:46:12.928120 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:12.928094    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-j5sk7" podUID="afa1a8ef-edc0-42cd-8eab-1f893e362a1f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 17 22:46:13.918235 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:13.918212    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-j5sk7"
Sep 17 22:46:14.049200 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:14.049116    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/afa1a8ef-edc0-42cd-8eab-1f893e362a1f-cabundle-cert\") pod \"afa1a8ef-edc0-42cd-8eab-1f893e362a1f\" (UID: \"afa1a8ef-edc0-42cd-8eab-1f893e362a1f\") "
Sep 17 22:46:14.049200 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:14.049172    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/afa1a8ef-edc0-42cd-8eab-1f893e362a1f-kube-rbac-proxy-sar-config\") pod \"afa1a8ef-edc0-42cd-8eab-1f893e362a1f\" (UID: \"afa1a8ef-edc0-42cd-8eab-1f893e362a1f\") "
Sep 17 22:46:14.049633 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:14.049236    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/afa1a8ef-edc0-42cd-8eab-1f893e362a1f-kserve-provision-location\") pod \"afa1a8ef-edc0-42cd-8eab-1f893e362a1f\" (UID: \"afa1a8ef-edc0-42cd-8eab-1f893e362a1f\") "
Sep 17 22:46:14.049633 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:14.049256    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afa1a8ef-edc0-42cd-8eab-1f893e362a1f-proxy-tls\") pod \"afa1a8ef-edc0-42cd-8eab-1f893e362a1f\" (UID: \"afa1a8ef-edc0-42cd-8eab-1f893e362a1f\") "
Sep 17 22:46:14.049633 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:14.049295    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wzn8x\" (UniqueName: \"kubernetes.io/projected/afa1a8ef-edc0-42cd-8eab-1f893e362a1f-kube-api-access-wzn8x\") pod \"afa1a8ef-edc0-42cd-8eab-1f893e362a1f\" (UID: \"afa1a8ef-edc0-42cd-8eab-1f893e362a1f\") "
Sep 17 22:46:14.049633 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:14.049528    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afa1a8ef-edc0-42cd-8eab-1f893e362a1f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "afa1a8ef-edc0-42cd-8eab-1f893e362a1f" (UID: "afa1a8ef-edc0-42cd-8eab-1f893e362a1f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:46:14.049633 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:14.049587    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afa1a8ef-edc0-42cd-8eab-1f893e362a1f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "afa1a8ef-edc0-42cd-8eab-1f893e362a1f" (UID: "afa1a8ef-edc0-42cd-8eab-1f893e362a1f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:46:14.051355 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:14.051332    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afa1a8ef-edc0-42cd-8eab-1f893e362a1f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "afa1a8ef-edc0-42cd-8eab-1f893e362a1f" (UID: "afa1a8ef-edc0-42cd-8eab-1f893e362a1f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:46:14.051516 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:14.051500    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afa1a8ef-edc0-42cd-8eab-1f893e362a1f-kube-api-access-wzn8x" (OuterVolumeSpecName: "kube-api-access-wzn8x") pod "afa1a8ef-edc0-42cd-8eab-1f893e362a1f" (UID: "afa1a8ef-edc0-42cd-8eab-1f893e362a1f"). InnerVolumeSpecName "kube-api-access-wzn8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:46:14.058908 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:14.058885    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afa1a8ef-edc0-42cd-8eab-1f893e362a1f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "afa1a8ef-edc0-42cd-8eab-1f893e362a1f" (UID: "afa1a8ef-edc0-42cd-8eab-1f893e362a1f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:46:14.143085 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:14.143054    2563 generic.go:356] "Generic (PLEG): container finished" podID="afa1a8ef-edc0-42cd-8eab-1f893e362a1f" containerID="7e946bcffbaa87ea4cbdc397c028110665859637373283ef0477ae26742e7a2c" exitCode=0
Sep 17 22:46:14.143259 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:14.143117    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-j5sk7"
Sep 17 22:46:14.143259 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:14.143133    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-j5sk7" event={"ID":"afa1a8ef-edc0-42cd-8eab-1f893e362a1f","Type":"ContainerDied","Data":"7e946bcffbaa87ea4cbdc397c028110665859637373283ef0477ae26742e7a2c"}
Sep 17 22:46:14.143259 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:14.143220    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-j5sk7" event={"ID":"afa1a8ef-edc0-42cd-8eab-1f893e362a1f","Type":"ContainerDied","Data":"10a3058ac2430698cb59b625641c87184ab878d0e83b0506d3fbe1bd87a82cc6"}
Sep 17 22:46:14.143368 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:14.143259    2563 scope.go:117] "RemoveContainer" containerID="e483a8ec7bbb3ed69d840de18c50b9dd413849908ca2bcda62641fb40a5509c8"
Sep 17 22:46:14.149815 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:14.149794    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/afa1a8ef-edc0-42cd-8eab-1f893e362a1f-kserve-provision-location\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:46:14.149925 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:14.149816    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afa1a8ef-edc0-42cd-8eab-1f893e362a1f-proxy-tls\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:46:14.149925 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:14.149836    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzn8x\" (UniqueName: \"kubernetes.io/projected/afa1a8ef-edc0-42cd-8eab-1f893e362a1f-kube-api-access-wzn8x\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:46:14.149925 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:14.149849    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/afa1a8ef-edc0-42cd-8eab-1f893e362a1f-cabundle-cert\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:46:14.149925 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:14.149862    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/afa1a8ef-edc0-42cd-8eab-1f893e362a1f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:46:14.151170 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:14.151150    2563 scope.go:117] "RemoveContainer" containerID="7e946bcffbaa87ea4cbdc397c028110665859637373283ef0477ae26742e7a2c"
Sep 17 22:46:14.158210 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:14.158193    2563 scope.go:117] "RemoveContainer" containerID="25ec95a83ff2c6fb908b0851a52b26d15b804fe116c020939556be4f8f5b6c2f"
Sep 17 22:46:14.163398 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:14.163377    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-j5sk7"]
Sep 17 22:46:14.165804 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:14.165788    2563 scope.go:117] "RemoveContainer" containerID="e483a8ec7bbb3ed69d840de18c50b9dd413849908ca2bcda62641fb40a5509c8"
Sep 17 22:46:14.166201 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:46:14.166171    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e483a8ec7bbb3ed69d840de18c50b9dd413849908ca2bcda62641fb40a5509c8\": container with ID starting with e483a8ec7bbb3ed69d840de18c50b9dd413849908ca2bcda62641fb40a5509c8 not found: ID does not exist" containerID="e483a8ec7bbb3ed69d840de18c50b9dd413849908ca2bcda62641fb40a5509c8"
Sep 17 22:46:14.166276 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:14.166215    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e483a8ec7bbb3ed69d840de18c50b9dd413849908ca2bcda62641fb40a5509c8"} err="failed to get container status \"e483a8ec7bbb3ed69d840de18c50b9dd413849908ca2bcda62641fb40a5509c8\": rpc error: code = NotFound desc = could not find container \"e483a8ec7bbb3ed69d840de18c50b9dd413849908ca2bcda62641fb40a5509c8\": container with ID starting with e483a8ec7bbb3ed69d840de18c50b9dd413849908ca2bcda62641fb40a5509c8 not found: ID does not exist"
Sep 17 22:46:14.166276 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:14.166239    2563 scope.go:117] "RemoveContainer" containerID="7e946bcffbaa87ea4cbdc397c028110665859637373283ef0477ae26742e7a2c"
Sep 17 22:46:14.166547 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:46:14.166527    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e946bcffbaa87ea4cbdc397c028110665859637373283ef0477ae26742e7a2c\": container with ID starting with 7e946bcffbaa87ea4cbdc397c028110665859637373283ef0477ae26742e7a2c not found: ID does not exist" containerID="7e946bcffbaa87ea4cbdc397c028110665859637373283ef0477ae26742e7a2c"
Sep 17 22:46:14.166709 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:14.166556    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e946bcffbaa87ea4cbdc397c028110665859637373283ef0477ae26742e7a2c"} err="failed to get container status \"7e946bcffbaa87ea4cbdc397c028110665859637373283ef0477ae26742e7a2c\": rpc error: code = NotFound desc = could not find container \"7e946bcffbaa87ea4cbdc397c028110665859637373283ef0477ae26742e7a2c\": container with ID starting with 7e946bcffbaa87ea4cbdc397c028110665859637373283ef0477ae26742e7a2c not found: ID does not exist"
Sep 17 22:46:14.166709 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:14.166575    2563 scope.go:117] "RemoveContainer" containerID="25ec95a83ff2c6fb908b0851a52b26d15b804fe116c020939556be4f8f5b6c2f"
Sep 17 22:46:14.166887 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:46:14.166871    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25ec95a83ff2c6fb908b0851a52b26d15b804fe116c020939556be4f8f5b6c2f\": container with ID starting with 25ec95a83ff2c6fb908b0851a52b26d15b804fe116c020939556be4f8f5b6c2f not found: ID does not exist" containerID="25ec95a83ff2c6fb908b0851a52b26d15b804fe116c020939556be4f8f5b6c2f"
Sep 17 22:46:14.166925 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:14.166892    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25ec95a83ff2c6fb908b0851a52b26d15b804fe116c020939556be4f8f5b6c2f"} err="failed to get container status \"25ec95a83ff2c6fb908b0851a52b26d15b804fe116c020939556be4f8f5b6c2f\": rpc error: code = NotFound desc = could not find container \"25ec95a83ff2c6fb908b0851a52b26d15b804fe116c020939556be4f8f5b6c2f\": container with ID starting with 25ec95a83ff2c6fb908b0851a52b26d15b804fe116c020939556be4f8f5b6c2f not found: ID does not exist"
Sep 17 22:46:14.167619 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:14.167601    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-j5sk7"]
Sep 17 22:46:14.913928 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:14.913898    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="afa1a8ef-edc0-42cd-8eab-1f893e362a1f" path="/var/lib/kubelet/pods/afa1a8ef-edc0-42cd-8eab-1f893e362a1f/volumes"
Sep 17 22:46:15.990036 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:15.989993    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5ppvb"]
Sep 17 22:46:15.990450 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:15.990283    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afa1a8ef-edc0-42cd-8eab-1f893e362a1f" containerName="kube-rbac-proxy"
Sep 17 22:46:15.990450 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:15.990295    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="afa1a8ef-edc0-42cd-8eab-1f893e362a1f" containerName="kube-rbac-proxy"
Sep 17 22:46:15.990450 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:15.990308    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afa1a8ef-edc0-42cd-8eab-1f893e362a1f" containerName="storage-initializer"
Sep 17 22:46:15.990450 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:15.990314    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="afa1a8ef-edc0-42cd-8eab-1f893e362a1f" containerName="storage-initializer"
Sep 17 22:46:15.990450 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:15.990331    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afa1a8ef-edc0-42cd-8eab-1f893e362a1f" containerName="kserve-container"
Sep 17 22:46:15.990450 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:15.990337    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="afa1a8ef-edc0-42cd-8eab-1f893e362a1f" containerName="kserve-container"
Sep 17 22:46:15.990450 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:15.990380    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="afa1a8ef-edc0-42cd-8eab-1f893e362a1f" containerName="kserve-container"
Sep 17 22:46:15.990450 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:15.990388    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="afa1a8ef-edc0-42cd-8eab-1f893e362a1f" containerName="kube-rbac-proxy"
Sep 17 22:46:15.994852 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:15.994834    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5ppvb"
Sep 17 22:46:15.997533 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:15.997507    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v4p4m\""
Sep 17 22:46:15.997856 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:15.997842    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 22:46:15.998060 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:15.998043    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:46:15.998155 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:15.998095    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 22:46:15.998155 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:15.998100    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-predictor-serving-cert\""
Sep 17 22:46:15.998155 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:15.998104    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-kube-rbac-proxy-sar-config\""
Sep 17 22:46:15.998329 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:15.998306    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 22:46:16.005704 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:16.005683    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5ppvb"]
Sep 17 22:46:16.167432 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:16.167398    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cda22250-770b-47a1-b257-eca3e9cbf4e6-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-5ppvb\" (UID: \"cda22250-770b-47a1-b257-eca3e9cbf4e6\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5ppvb"
Sep 17 22:46:16.167432 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:16.167431    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cda22250-770b-47a1-b257-eca3e9cbf4e6-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-5ppvb\" (UID: \"cda22250-770b-47a1-b257-eca3e9cbf4e6\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5ppvb"
Sep 17 22:46:16.167676 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:16.167462    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87vzf\" (UniqueName: \"kubernetes.io/projected/cda22250-770b-47a1-b257-eca3e9cbf4e6-kube-api-access-87vzf\") pod \"isvc-pmml-predictor-8674795f59-5ppvb\" (UID: \"cda22250-770b-47a1-b257-eca3e9cbf4e6\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5ppvb"
Sep 17 22:46:16.167676 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:16.167537    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cda22250-770b-47a1-b257-eca3e9cbf4e6-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-5ppvb\" (UID: \"cda22250-770b-47a1-b257-eca3e9cbf4e6\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5ppvb"
Sep 17 22:46:16.167676 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:16.167585    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cda22250-770b-47a1-b257-eca3e9cbf4e6-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-5ppvb\" (UID: \"cda22250-770b-47a1-b257-eca3e9cbf4e6\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5ppvb"
Sep 17 22:46:16.268340 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:16.268240    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cda22250-770b-47a1-b257-eca3e9cbf4e6-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-5ppvb\" (UID: \"cda22250-770b-47a1-b257-eca3e9cbf4e6\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5ppvb"
Sep 17 22:46:16.268340 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:16.268289    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cda22250-770b-47a1-b257-eca3e9cbf4e6-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-5ppvb\" (UID: \"cda22250-770b-47a1-b257-eca3e9cbf4e6\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5ppvb"
Sep 17 22:46:16.268340 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:16.268318    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87vzf\" (UniqueName: \"kubernetes.io/projected/cda22250-770b-47a1-b257-eca3e9cbf4e6-kube-api-access-87vzf\") pod \"isvc-pmml-predictor-8674795f59-5ppvb\" (UID: \"cda22250-770b-47a1-b257-eca3e9cbf4e6\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5ppvb"
Sep 17 22:46:16.268340 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:16.268348    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cda22250-770b-47a1-b257-eca3e9cbf4e6-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-5ppvb\" (UID: \"cda22250-770b-47a1-b257-eca3e9cbf4e6\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5ppvb"
Sep 17 22:46:16.268689 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:16.268372    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cda22250-770b-47a1-b257-eca3e9cbf4e6-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-5ppvb\" (UID: \"cda22250-770b-47a1-b257-eca3e9cbf4e6\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5ppvb"
Sep 17 22:46:16.268816 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:16.268793    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cda22250-770b-47a1-b257-eca3e9cbf4e6-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-5ppvb\" (UID: \"cda22250-770b-47a1-b257-eca3e9cbf4e6\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5ppvb"
Sep 17 22:46:16.268968 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:16.268951    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cda22250-770b-47a1-b257-eca3e9cbf4e6-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-5ppvb\" (UID: \"cda22250-770b-47a1-b257-eca3e9cbf4e6\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5ppvb"
Sep 17 22:46:16.269014 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:16.268984    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cda22250-770b-47a1-b257-eca3e9cbf4e6-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-5ppvb\" (UID: \"cda22250-770b-47a1-b257-eca3e9cbf4e6\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5ppvb"
Sep 17 22:46:16.270815 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:16.270799    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cda22250-770b-47a1-b257-eca3e9cbf4e6-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-5ppvb\" (UID: \"cda22250-770b-47a1-b257-eca3e9cbf4e6\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5ppvb"
Sep 17 22:46:16.277533 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:16.277511    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87vzf\" (UniqueName: \"kubernetes.io/projected/cda22250-770b-47a1-b257-eca3e9cbf4e6-kube-api-access-87vzf\") pod \"isvc-pmml-predictor-8674795f59-5ppvb\" (UID: \"cda22250-770b-47a1-b257-eca3e9cbf4e6\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5ppvb"
Sep 17 22:46:16.304534 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:16.304507    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5ppvb"
Sep 17 22:46:16.575736 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:16.575504    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5ppvb"]
Sep 17 22:46:16.578318 ip-10-0-138-45 kubenswrapper[2563]: W0917 22:46:16.578286    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcda22250_770b_47a1_b257_eca3e9cbf4e6.slice/crio-c4d450b8f30aa7ce73f3a237c7f30e8d7eab6793165a2e6929e007bd4cd7f398 WatchSource:0}: Error finding container c4d450b8f30aa7ce73f3a237c7f30e8d7eab6793165a2e6929e007bd4cd7f398: Status 404 returned error can't find the container with id c4d450b8f30aa7ce73f3a237c7f30e8d7eab6793165a2e6929e007bd4cd7f398
Sep 17 22:46:17.155973 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:17.155937    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5ppvb" event={"ID":"cda22250-770b-47a1-b257-eca3e9cbf4e6","Type":"ContainerStarted","Data":"b8f52676dd31529438cb24dc32607cc1354cea6f763c7ac86fccefa274666d87"}
Sep 17 22:46:17.156381 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:17.155999    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5ppvb" event={"ID":"cda22250-770b-47a1-b257-eca3e9cbf4e6","Type":"ContainerStarted","Data":"c4d450b8f30aa7ce73f3a237c7f30e8d7eab6793165a2e6929e007bd4cd7f398"}
Sep 17 22:46:21.172545 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:21.172511    2563 generic.go:356] "Generic (PLEG): container finished" podID="cda22250-770b-47a1-b257-eca3e9cbf4e6" containerID="b8f52676dd31529438cb24dc32607cc1354cea6f763c7ac86fccefa274666d87" exitCode=0
Sep 17 22:46:21.172917 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:21.172583    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5ppvb" event={"ID":"cda22250-770b-47a1-b257-eca3e9cbf4e6","Type":"ContainerDied","Data":"b8f52676dd31529438cb24dc32607cc1354cea6f763c7ac86fccefa274666d87"}
Sep 17 22:46:29.213659 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:29.213624    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5ppvb" event={"ID":"cda22250-770b-47a1-b257-eca3e9cbf4e6","Type":"ContainerStarted","Data":"2bedc19009ddf5ab8a29ac995b22a8f8f6893676dfa4a00d8cd4e4e5540a90bd"}
Sep 17 22:46:29.214071 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:29.213684    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5ppvb" event={"ID":"cda22250-770b-47a1-b257-eca3e9cbf4e6","Type":"ContainerStarted","Data":"77076ea90e2055373526577ab87a8fe7ceb77bbecd08a9950b320451a513c629"}
Sep 17 22:46:29.214071 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:29.213884    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5ppvb"
Sep 17 22:46:29.237734 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:29.237683    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5ppvb" podStartSLOduration=3.261662095 podStartE2EDuration="14.23766927s" totalImagesPullingTime="7.145157205s" totalInitContainerRuntime="3.83084997s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:46:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:46:29.235589441 +0000 UTC m=+2900.403542422" watchObservedRunningTime="2026-09-17 22:46:29.23766927 +0000 UTC m=+2900.405622250"
Sep 17 22:46:30.217853 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:30.217818    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5ppvb"
Sep 17 22:46:30.218503 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:30.218477    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5ppvb" podUID="cda22250-770b-47a1-b257-eca3e9cbf4e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 17 22:46:31.221424 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:31.221378    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5ppvb" podUID="cda22250-770b-47a1-b257-eca3e9cbf4e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 17 22:46:36.225433 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:36.225403    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5ppvb"
Sep 17 22:46:36.225992 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:36.225865    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5ppvb" podUID="cda22250-770b-47a1-b257-eca3e9cbf4e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 17 22:46:46.225971 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:46.225932    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5ppvb" podUID="cda22250-770b-47a1-b257-eca3e9cbf4e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 17 22:46:56.226634 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:46:56.226589    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5ppvb" podUID="cda22250-770b-47a1-b257-eca3e9cbf4e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 17 22:47:06.226192 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:47:06.226132    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5ppvb" podUID="cda22250-770b-47a1-b257-eca3e9cbf4e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 17 22:47:16.226136 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:47:16.226091    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5ppvb" podUID="cda22250-770b-47a1-b257-eca3e9cbf4e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 17 22:47:26.226652 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:47:26.226572    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5ppvb" podUID="cda22250-770b-47a1-b257-eca3e9cbf4e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 17 22:47:36.226673 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:47:36.226627    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5ppvb" podUID="cda22250-770b-47a1-b257-eca3e9cbf4e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 17 22:47:39.909483 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:47:39.909438    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5ppvb" podUID="cda22250-770b-47a1-b257-eca3e9cbf4e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 17 22:47:49.909993 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:47:49.909954    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5ppvb" podUID="cda22250-770b-47a1-b257-eca3e9cbf4e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 17 22:47:59.910369 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:47:59.910338    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5ppvb"
Sep 17 22:48:07.566694 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:48:07.566658    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5ppvb"]
Sep 17 22:48:07.567125 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:48:07.566979    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5ppvb" podUID="cda22250-770b-47a1-b257-eca3e9cbf4e6" containerName="kserve-container" containerID="cri-o://77076ea90e2055373526577ab87a8fe7ceb77bbecd08a9950b320451a513c629" gracePeriod=30
Sep 17 22:48:07.567125 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:48:07.567007    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5ppvb" podUID="cda22250-770b-47a1-b257-eca3e9cbf4e6" containerName="kube-rbac-proxy" containerID="cri-o://2bedc19009ddf5ab8a29ac995b22a8f8f6893676dfa4a00d8cd4e4e5540a90bd" gracePeriod=30
Sep 17 22:48:08.598199 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:48:08.598166    2563 generic.go:356] "Generic (PLEG): container finished" podID="cda22250-770b-47a1-b257-eca3e9cbf4e6" containerID="2bedc19009ddf5ab8a29ac995b22a8f8f6893676dfa4a00d8cd4e4e5540a90bd" exitCode=2
Sep 17 22:48:08.598570 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:48:08.598239    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5ppvb" event={"ID":"cda22250-770b-47a1-b257-eca3e9cbf4e6","Type":"ContainerDied","Data":"2bedc19009ddf5ab8a29ac995b22a8f8f6893676dfa4a00d8cd4e4e5540a90bd"}
Sep 17 22:48:09.910103 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:48:09.910059    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5ppvb" podUID="cda22250-770b-47a1-b257-eca3e9cbf4e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 17 22:48:11.221515 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:48:11.221470    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5ppvb" podUID="cda22250-770b-47a1-b257-eca3e9cbf4e6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.53:8643/healthz\": dial tcp 10.133.0.53:8643: connect: connection refused"
Sep 17 22:48:11.612777 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:48:11.612745    2563 generic.go:356] "Generic (PLEG): container finished" podID="cda22250-770b-47a1-b257-eca3e9cbf4e6" containerID="77076ea90e2055373526577ab87a8fe7ceb77bbecd08a9950b320451a513c629" exitCode=0
Sep 17 22:48:11.612917 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:48:11.612818    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5ppvb" event={"ID":"cda22250-770b-47a1-b257-eca3e9cbf4e6","Type":"ContainerDied","Data":"77076ea90e2055373526577ab87a8fe7ceb77bbecd08a9950b320451a513c629"}
Sep 17 22:48:11.703171 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:48:11.703132    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5ppvb"
Sep 17 22:48:11.723654 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:48:11.723626    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cda22250-770b-47a1-b257-eca3e9cbf4e6-proxy-tls\") pod \"cda22250-770b-47a1-b257-eca3e9cbf4e6\" (UID: \"cda22250-770b-47a1-b257-eca3e9cbf4e6\") "
Sep 17 22:48:11.723782 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:48:11.723680    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cda22250-770b-47a1-b257-eca3e9cbf4e6-kserve-provision-location\") pod \"cda22250-770b-47a1-b257-eca3e9cbf4e6\" (UID: \"cda22250-770b-47a1-b257-eca3e9cbf4e6\") "
Sep 17 22:48:11.723782 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:48:11.723743    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-87vzf\" (UniqueName: \"kubernetes.io/projected/cda22250-770b-47a1-b257-eca3e9cbf4e6-kube-api-access-87vzf\") pod \"cda22250-770b-47a1-b257-eca3e9cbf4e6\" (UID: \"cda22250-770b-47a1-b257-eca3e9cbf4e6\") "
Sep 17 22:48:11.723782 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:48:11.723779    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cda22250-770b-47a1-b257-eca3e9cbf4e6-kube-rbac-proxy-sar-config\") pod \"cda22250-770b-47a1-b257-eca3e9cbf4e6\" (UID: \"cda22250-770b-47a1-b257-eca3e9cbf4e6\") "
Sep 17 22:48:11.723950 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:48:11.723803    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cda22250-770b-47a1-b257-eca3e9cbf4e6-cabundle-cert\") pod \"cda22250-770b-47a1-b257-eca3e9cbf4e6\" (UID: \"cda22250-770b-47a1-b257-eca3e9cbf4e6\") "
Sep 17 22:48:11.723950 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:48:11.723928    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cda22250-770b-47a1-b257-eca3e9cbf4e6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cda22250-770b-47a1-b257-eca3e9cbf4e6" (UID: "cda22250-770b-47a1-b257-eca3e9cbf4e6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:48:11.724164 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:48:11.724127    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cda22250-770b-47a1-b257-eca3e9cbf4e6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cda22250-770b-47a1-b257-eca3e9cbf4e6" (UID: "cda22250-770b-47a1-b257-eca3e9cbf4e6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:48:11.724248 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:48:11.724212    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cda22250-770b-47a1-b257-eca3e9cbf4e6-kserve-provision-location\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:48:11.724301 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:48:11.724288    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cda22250-770b-47a1-b257-eca3e9cbf4e6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cda22250-770b-47a1-b257-eca3e9cbf4e6" (UID: "cda22250-770b-47a1-b257-eca3e9cbf4e6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:48:11.725953 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:48:11.725927    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cda22250-770b-47a1-b257-eca3e9cbf4e6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cda22250-770b-47a1-b257-eca3e9cbf4e6" (UID: "cda22250-770b-47a1-b257-eca3e9cbf4e6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:48:11.726051 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:48:11.726012    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cda22250-770b-47a1-b257-eca3e9cbf4e6-kube-api-access-87vzf" (OuterVolumeSpecName: "kube-api-access-87vzf") pod "cda22250-770b-47a1-b257-eca3e9cbf4e6" (UID: "cda22250-770b-47a1-b257-eca3e9cbf4e6"). InnerVolumeSpecName "kube-api-access-87vzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:48:11.825333 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:48:11.825244    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87vzf\" (UniqueName: \"kubernetes.io/projected/cda22250-770b-47a1-b257-eca3e9cbf4e6-kube-api-access-87vzf\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:48:11.825333 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:48:11.825281    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cda22250-770b-47a1-b257-eca3e9cbf4e6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:48:11.825333 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:48:11.825291    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cda22250-770b-47a1-b257-eca3e9cbf4e6-cabundle-cert\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:48:11.825333 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:48:11.825299    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cda22250-770b-47a1-b257-eca3e9cbf4e6-proxy-tls\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:48:12.619157 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:48:12.619122    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5ppvb"
Sep 17 22:48:12.619571 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:48:12.619123    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5ppvb" event={"ID":"cda22250-770b-47a1-b257-eca3e9cbf4e6","Type":"ContainerDied","Data":"c4d450b8f30aa7ce73f3a237c7f30e8d7eab6793165a2e6929e007bd4cd7f398"}
Sep 17 22:48:12.619571 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:48:12.619293    2563 scope.go:117] "RemoveContainer" containerID="2bedc19009ddf5ab8a29ac995b22a8f8f6893676dfa4a00d8cd4e4e5540a90bd"
Sep 17 22:48:12.629756 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:48:12.629735    2563 scope.go:117] "RemoveContainer" containerID="77076ea90e2055373526577ab87a8fe7ceb77bbecd08a9950b320451a513c629"
Sep 17 22:48:12.636855 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:48:12.636841    2563 scope.go:117] "RemoveContainer" containerID="b8f52676dd31529438cb24dc32607cc1354cea6f763c7ac86fccefa274666d87"
Sep 17 22:48:12.644603 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:48:12.644582    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5ppvb"]
Sep 17 22:48:12.650761 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:48:12.650741    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-5ppvb"]
Sep 17 22:48:12.913772 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:48:12.913736    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cda22250-770b-47a1-b257-eca3e9cbf4e6" path="/var/lib/kubelet/pods/cda22250-770b-47a1-b257-eca3e9cbf4e6/volumes"
Sep 17 22:48:49.707359 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:48:49.707333    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nrkd5_34b8ef2b-97ba-4562-87ee-8da0cb4604c7/ovn-acl-logging/0.log"
Sep 17 22:48:49.710125 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:48:49.710103    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nrkd5_34b8ef2b-97ba-4562-87ee-8da0cb4604c7/ovn-acl-logging/0.log"
Sep 17 22:51:55.124616 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:51:55.124533    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-347c27-predictor-8fb656fb7-tm9dc"]
Sep 17 22:51:55.125108 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:51:55.124936    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cda22250-770b-47a1-b257-eca3e9cbf4e6" containerName="kserve-container"
Sep 17 22:51:55.125108 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:51:55.124952    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cda22250-770b-47a1-b257-eca3e9cbf4e6" containerName="kserve-container"
Sep 17 22:51:55.125108 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:51:55.124975    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cda22250-770b-47a1-b257-eca3e9cbf4e6" containerName="storage-initializer"
Sep 17 22:51:55.125108 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:51:55.124983    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cda22250-770b-47a1-b257-eca3e9cbf4e6" containerName="storage-initializer"
Sep 17 22:51:55.125108 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:51:55.124992    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cda22250-770b-47a1-b257-eca3e9cbf4e6" containerName="kube-rbac-proxy"
Sep 17 22:51:55.125108 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:51:55.125001    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cda22250-770b-47a1-b257-eca3e9cbf4e6" containerName="kube-rbac-proxy"
Sep 17 22:51:55.125108 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:51:55.125091    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cda22250-770b-47a1-b257-eca3e9cbf4e6" containerName="kube-rbac-proxy"
Sep 17 22:51:55.125108 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:51:55.125104    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cda22250-770b-47a1-b257-eca3e9cbf4e6" containerName="kserve-container"
Sep 17 22:51:55.128352 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:51:55.128332    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-347c27-predictor-8fb656fb7-tm9dc"
Sep 17 22:51:55.130772 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:51:55.130748    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:51:55.130880 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:51:55.130786    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-347c27-predictor-serving-cert\""
Sep 17 22:51:55.130880 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:51:55.130748    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-347c27-kube-rbac-proxy-sar-config\""
Sep 17 22:51:55.130880 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:51:55.130805    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 22:51:55.130880 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:51:55.130828    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 22:51:55.130880 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:51:55.130796    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-347c27\""
Sep 17 22:51:55.131967 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:51:55.131949    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-347c27-dockercfg-zjxrf\""
Sep 17 22:51:55.138744 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:51:55.138721    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-347c27-predictor-8fb656fb7-tm9dc"]
Sep 17 22:51:55.187858 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:51:55.187828    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50d94d53-7581-47e9-b608-7834c9341981-kserve-provision-location\") pod \"isvc-secondary-347c27-predictor-8fb656fb7-tm9dc\" (UID: \"50d94d53-7581-47e9-b608-7834c9341981\") " pod="kserve-ci-e2e-test/isvc-secondary-347c27-predictor-8fb656fb7-tm9dc"
Sep 17 22:51:55.187997 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:51:55.187863    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh46c\" (UniqueName: \"kubernetes.io/projected/50d94d53-7581-47e9-b608-7834c9341981-kube-api-access-xh46c\") pod \"isvc-secondary-347c27-predictor-8fb656fb7-tm9dc\" (UID: \"50d94d53-7581-47e9-b608-7834c9341981\") " pod="kserve-ci-e2e-test/isvc-secondary-347c27-predictor-8fb656fb7-tm9dc"
Sep 17 22:51:55.187997 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:51:55.187888    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50d94d53-7581-47e9-b608-7834c9341981-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-347c27-predictor-8fb656fb7-tm9dc\" (UID: \"50d94d53-7581-47e9-b608-7834c9341981\") " pod="kserve-ci-e2e-test/isvc-secondary-347c27-predictor-8fb656fb7-tm9dc"
Sep 17 22:51:55.187997 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:51:55.187912    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50d94d53-7581-47e9-b608-7834c9341981-proxy-tls\") pod \"isvc-secondary-347c27-predictor-8fb656fb7-tm9dc\" (UID: \"50d94d53-7581-47e9-b608-7834c9341981\") " pod="kserve-ci-e2e-test/isvc-secondary-347c27-predictor-8fb656fb7-tm9dc"
Sep 17 22:51:55.187997 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:51:55.187957    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50d94d53-7581-47e9-b608-7834c9341981-cabundle-cert\") pod \"isvc-secondary-347c27-predictor-8fb656fb7-tm9dc\" (UID: \"50d94d53-7581-47e9-b608-7834c9341981\") " pod="kserve-ci-e2e-test/isvc-secondary-347c27-predictor-8fb656fb7-tm9dc"
Sep 17 22:51:55.289127 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:51:55.289096    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50d94d53-7581-47e9-b608-7834c9341981-kserve-provision-location\") pod \"isvc-secondary-347c27-predictor-8fb656fb7-tm9dc\" (UID: \"50d94d53-7581-47e9-b608-7834c9341981\") " pod="kserve-ci-e2e-test/isvc-secondary-347c27-predictor-8fb656fb7-tm9dc"
Sep 17 22:51:55.289127 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:51:55.289130    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh46c\" (UniqueName: \"kubernetes.io/projected/50d94d53-7581-47e9-b608-7834c9341981-kube-api-access-xh46c\") pod \"isvc-secondary-347c27-predictor-8fb656fb7-tm9dc\" (UID: \"50d94d53-7581-47e9-b608-7834c9341981\") " pod="kserve-ci-e2e-test/isvc-secondary-347c27-predictor-8fb656fb7-tm9dc"
Sep 17 22:51:55.289314 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:51:55.289181    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50d94d53-7581-47e9-b608-7834c9341981-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-347c27-predictor-8fb656fb7-tm9dc\" (UID: \"50d94d53-7581-47e9-b608-7834c9341981\") " pod="kserve-ci-e2e-test/isvc-secondary-347c27-predictor-8fb656fb7-tm9dc"
Sep 17 22:51:55.289314 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:51:55.289211    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50d94d53-7581-47e9-b608-7834c9341981-proxy-tls\") pod \"isvc-secondary-347c27-predictor-8fb656fb7-tm9dc\" (UID: \"50d94d53-7581-47e9-b608-7834c9341981\") " pod="kserve-ci-e2e-test/isvc-secondary-347c27-predictor-8fb656fb7-tm9dc"
Sep 17 22:51:55.289314 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:51:55.289265    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50d94d53-7581-47e9-b608-7834c9341981-cabundle-cert\") pod \"isvc-secondary-347c27-predictor-8fb656fb7-tm9dc\" (UID: \"50d94d53-7581-47e9-b608-7834c9341981\") " pod="kserve-ci-e2e-test/isvc-secondary-347c27-predictor-8fb656fb7-tm9dc"
Sep 17 22:51:55.289580 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:51:55.289559    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50d94d53-7581-47e9-b608-7834c9341981-kserve-provision-location\") pod \"isvc-secondary-347c27-predictor-8fb656fb7-tm9dc\" (UID: \"50d94d53-7581-47e9-b608-7834c9341981\") " pod="kserve-ci-e2e-test/isvc-secondary-347c27-predictor-8fb656fb7-tm9dc"
Sep 17 22:51:55.289881 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:51:55.289863    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50d94d53-7581-47e9-b608-7834c9341981-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-347c27-predictor-8fb656fb7-tm9dc\" (UID: \"50d94d53-7581-47e9-b608-7834c9341981\") " pod="kserve-ci-e2e-test/isvc-secondary-347c27-predictor-8fb656fb7-tm9dc"
Sep 17 22:51:55.289966 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:51:55.289865    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50d94d53-7581-47e9-b608-7834c9341981-cabundle-cert\") pod \"isvc-secondary-347c27-predictor-8fb656fb7-tm9dc\" (UID: \"50d94d53-7581-47e9-b608-7834c9341981\") " pod="kserve-ci-e2e-test/isvc-secondary-347c27-predictor-8fb656fb7-tm9dc"
Sep 17 22:51:55.291577 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:51:55.291560    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50d94d53-7581-47e9-b608-7834c9341981-proxy-tls\") pod \"isvc-secondary-347c27-predictor-8fb656fb7-tm9dc\" (UID: \"50d94d53-7581-47e9-b608-7834c9341981\") " pod="kserve-ci-e2e-test/isvc-secondary-347c27-predictor-8fb656fb7-tm9dc"
Sep 17 22:51:55.297349 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:51:55.297332    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh46c\" (UniqueName: \"kubernetes.io/projected/50d94d53-7581-47e9-b608-7834c9341981-kube-api-access-xh46c\") pod \"isvc-secondary-347c27-predictor-8fb656fb7-tm9dc\" (UID: \"50d94d53-7581-47e9-b608-7834c9341981\") " pod="kserve-ci-e2e-test/isvc-secondary-347c27-predictor-8fb656fb7-tm9dc"
Sep 17 22:51:55.438003 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:51:55.437969    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-347c27-predictor-8fb656fb7-tm9dc"
Sep 17 22:51:55.707796 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:51:55.707733    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-347c27-predictor-8fb656fb7-tm9dc"]
Sep 17 22:51:55.709882 ip-10-0-138-45 kubenswrapper[2563]: W0917 22:51:55.709854    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50d94d53_7581_47e9_b608_7834c9341981.slice/crio-22526144b0a1b762faf87925d379b028565dc65527a7878b7dcd138e85ce98a4 WatchSource:0}: Error finding container 22526144b0a1b762faf87925d379b028565dc65527a7878b7dcd138e85ce98a4: Status 404 returned error can't find the container with id 22526144b0a1b762faf87925d379b028565dc65527a7878b7dcd138e85ce98a4
Sep 17 22:51:56.464544 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:51:56.464503    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-347c27-predictor-8fb656fb7-tm9dc" event={"ID":"50d94d53-7581-47e9-b608-7834c9341981","Type":"ContainerStarted","Data":"5588fa77d1207a7da621a7eca5a126423560b489467669ce1bd0f1ca17db7696"}
Sep 17 22:51:56.464950 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:51:56.464583    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-347c27-predictor-8fb656fb7-tm9dc" event={"ID":"50d94d53-7581-47e9-b608-7834c9341981","Type":"ContainerStarted","Data":"22526144b0a1b762faf87925d379b028565dc65527a7878b7dcd138e85ce98a4"}
Sep 17 22:52:00.485861 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:00.485835    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-347c27-predictor-8fb656fb7-tm9dc_50d94d53-7581-47e9-b608-7834c9341981/storage-initializer/0.log"
Sep 17 22:52:00.486255 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:00.485872    2563 generic.go:356] "Generic (PLEG): container finished" podID="50d94d53-7581-47e9-b608-7834c9341981" containerID="5588fa77d1207a7da621a7eca5a126423560b489467669ce1bd0f1ca17db7696" exitCode=1
Sep 17 22:52:00.486255 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:00.485916    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-347c27-predictor-8fb656fb7-tm9dc" event={"ID":"50d94d53-7581-47e9-b608-7834c9341981","Type":"ContainerDied","Data":"5588fa77d1207a7da621a7eca5a126423560b489467669ce1bd0f1ca17db7696"}
Sep 17 22:52:01.491037 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:01.491009    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-347c27-predictor-8fb656fb7-tm9dc_50d94d53-7581-47e9-b608-7834c9341981/storage-initializer/0.log"
Sep 17 22:52:01.491520 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:01.491114    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-347c27-predictor-8fb656fb7-tm9dc" event={"ID":"50d94d53-7581-47e9-b608-7834c9341981","Type":"ContainerStarted","Data":"cd10df4053f3fb2265e01a1dff247850e2fcf5503ebc55374574fe9ecb94e694"}
Sep 17 22:52:07.515578 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:07.515548    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-347c27-predictor-8fb656fb7-tm9dc_50d94d53-7581-47e9-b608-7834c9341981/storage-initializer/1.log"
Sep 17 22:52:07.515980 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:07.515945    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-347c27-predictor-8fb656fb7-tm9dc_50d94d53-7581-47e9-b608-7834c9341981/storage-initializer/0.log"
Sep 17 22:52:07.516026 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:07.515978    2563 generic.go:356] "Generic (PLEG): container finished" podID="50d94d53-7581-47e9-b608-7834c9341981" containerID="cd10df4053f3fb2265e01a1dff247850e2fcf5503ebc55374574fe9ecb94e694" exitCode=1
Sep 17 22:52:07.516026 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:07.516011    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-347c27-predictor-8fb656fb7-tm9dc" event={"ID":"50d94d53-7581-47e9-b608-7834c9341981","Type":"ContainerDied","Data":"cd10df4053f3fb2265e01a1dff247850e2fcf5503ebc55374574fe9ecb94e694"}
Sep 17 22:52:07.516092 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:07.516069    2563 scope.go:117] "RemoveContainer" containerID="5588fa77d1207a7da621a7eca5a126423560b489467669ce1bd0f1ca17db7696"
Sep 17 22:52:07.516418 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:07.516394    2563 scope.go:117] "RemoveContainer" containerID="5588fa77d1207a7da621a7eca5a126423560b489467669ce1bd0f1ca17db7696"
Sep 17 22:52:07.526735 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:52:07.526713    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-347c27-predictor-8fb656fb7-tm9dc_kserve-ci-e2e-test(50d94d53-7581-47e9-b608-7834c9341981)\"" pod="kserve-ci-e2e-test/isvc-secondary-347c27-predictor-8fb656fb7-tm9dc" podUID="50d94d53-7581-47e9-b608-7834c9341981"
Sep 17 22:52:07.526972 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:52:07.526949    2563 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-347c27-predictor-8fb656fb7-tm9dc_kserve-ci-e2e-test_50d94d53-7581-47e9-b608-7834c9341981_0 in pod sandbox 22526144b0a1b762faf87925d379b028565dc65527a7878b7dcd138e85ce98a4 from index: no such id: '5588fa77d1207a7da621a7eca5a126423560b489467669ce1bd0f1ca17db7696'" containerID="5588fa77d1207a7da621a7eca5a126423560b489467669ce1bd0f1ca17db7696"
Sep 17 22:52:07.527023 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:07.526982    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5588fa77d1207a7da621a7eca5a126423560b489467669ce1bd0f1ca17db7696"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-347c27-predictor-8fb656fb7-tm9dc_kserve-ci-e2e-test_50d94d53-7581-47e9-b608-7834c9341981_0 in pod sandbox 22526144b0a1b762faf87925d379b028565dc65527a7878b7dcd138e85ce98a4 from index: no such id: '5588fa77d1207a7da621a7eca5a126423560b489467669ce1bd0f1ca17db7696'"
Sep 17 22:52:08.521812 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:08.521783    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-347c27-predictor-8fb656fb7-tm9dc_50d94d53-7581-47e9-b608-7834c9341981/storage-initializer/1.log"
Sep 17 22:52:11.123321 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:11.123292    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-347c27-predictor-8fb656fb7-tm9dc"]
Sep 17 22:52:11.251945 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:11.251924    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-347c27-predictor-8fb656fb7-tm9dc_50d94d53-7581-47e9-b608-7834c9341981/storage-initializer/1.log"
Sep 17 22:52:11.255696 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:11.255680    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-347c27-predictor-8fb656fb7-tm9dc"
Sep 17 22:52:11.430693 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:11.430664    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50d94d53-7581-47e9-b608-7834c9341981-kserve-provision-location\") pod \"50d94d53-7581-47e9-b608-7834c9341981\" (UID: \"50d94d53-7581-47e9-b608-7834c9341981\") "
Sep 17 22:52:11.430884 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:11.430721    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50d94d53-7581-47e9-b608-7834c9341981-kube-rbac-proxy-sar-config\") pod \"50d94d53-7581-47e9-b608-7834c9341981\" (UID: \"50d94d53-7581-47e9-b608-7834c9341981\") "
Sep 17 22:52:11.430884 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:11.430756    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50d94d53-7581-47e9-b608-7834c9341981-cabundle-cert\") pod \"50d94d53-7581-47e9-b608-7834c9341981\" (UID: \"50d94d53-7581-47e9-b608-7834c9341981\") "
Sep 17 22:52:11.430884 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:11.430784    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50d94d53-7581-47e9-b608-7834c9341981-proxy-tls\") pod \"50d94d53-7581-47e9-b608-7834c9341981\" (UID: \"50d94d53-7581-47e9-b608-7834c9341981\") "
Sep 17 22:52:11.430884 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:11.430834    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xh46c\" (UniqueName: \"kubernetes.io/projected/50d94d53-7581-47e9-b608-7834c9341981-kube-api-access-xh46c\") pod \"50d94d53-7581-47e9-b608-7834c9341981\" (UID: \"50d94d53-7581-47e9-b608-7834c9341981\") "
Sep 17 22:52:11.431114 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:11.430950    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50d94d53-7581-47e9-b608-7834c9341981-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "50d94d53-7581-47e9-b608-7834c9341981" (UID: "50d94d53-7581-47e9-b608-7834c9341981"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:52:11.431207 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:11.431128    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50d94d53-7581-47e9-b608-7834c9341981-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "50d94d53-7581-47e9-b608-7834c9341981" (UID: "50d94d53-7581-47e9-b608-7834c9341981"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:52:11.431207 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:11.431175    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50d94d53-7581-47e9-b608-7834c9341981-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "50d94d53-7581-47e9-b608-7834c9341981" (UID: "50d94d53-7581-47e9-b608-7834c9341981"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:52:11.432819 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:11.432795    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50d94d53-7581-47e9-b608-7834c9341981-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "50d94d53-7581-47e9-b608-7834c9341981" (UID: "50d94d53-7581-47e9-b608-7834c9341981"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:52:11.432907 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:11.432889    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50d94d53-7581-47e9-b608-7834c9341981-kube-api-access-xh46c" (OuterVolumeSpecName: "kube-api-access-xh46c") pod "50d94d53-7581-47e9-b608-7834c9341981" (UID: "50d94d53-7581-47e9-b608-7834c9341981"). InnerVolumeSpecName "kube-api-access-xh46c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:52:11.531598 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:11.531561    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50d94d53-7581-47e9-b608-7834c9341981-kserve-provision-location\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:52:11.531598 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:11.531599    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50d94d53-7581-47e9-b608-7834c9341981-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:52:11.531817 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:11.531614    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50d94d53-7581-47e9-b608-7834c9341981-cabundle-cert\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:52:11.531817 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:11.531628    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50d94d53-7581-47e9-b608-7834c9341981-proxy-tls\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:52:11.531817 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:11.531641    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xh46c\" (UniqueName: \"kubernetes.io/projected/50d94d53-7581-47e9-b608-7834c9341981-kube-api-access-xh46c\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:52:11.536403 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:11.536382    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-347c27-predictor-8fb656fb7-tm9dc_50d94d53-7581-47e9-b608-7834c9341981/storage-initializer/1.log"
Sep 17 22:52:11.536554 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:11.536448    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-347c27-predictor-8fb656fb7-tm9dc" event={"ID":"50d94d53-7581-47e9-b608-7834c9341981","Type":"ContainerDied","Data":"22526144b0a1b762faf87925d379b028565dc65527a7878b7dcd138e85ce98a4"}
Sep 17 22:52:11.536554 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:11.536496    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-347c27-predictor-8fb656fb7-tm9dc"
Sep 17 22:52:11.536554 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:11.536514    2563 scope.go:117] "RemoveContainer" containerID="cd10df4053f3fb2265e01a1dff247850e2fcf5503ebc55374574fe9ecb94e694"
Sep 17 22:52:11.573000 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:11.572970    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-347c27-predictor-8fb656fb7-tm9dc"]
Sep 17 22:52:11.574416 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:11.574388    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-347c27-predictor-8fb656fb7-tm9dc"]
Sep 17 22:52:12.913559 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:12.913530    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50d94d53-7581-47e9-b608-7834c9341981" path="/var/lib/kubelet/pods/50d94d53-7581-47e9-b608-7834c9341981/volumes"
Sep 17 22:52:17.171459 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:17.171426    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-q8tq9/must-gather-gtfp8"]
Sep 17 22:52:17.171806 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:17.171723    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50d94d53-7581-47e9-b608-7834c9341981" containerName="storage-initializer"
Sep 17 22:52:17.171806 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:17.171734    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="50d94d53-7581-47e9-b608-7834c9341981" containerName="storage-initializer"
Sep 17 22:52:17.171806 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:17.171745    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50d94d53-7581-47e9-b608-7834c9341981" containerName="storage-initializer"
Sep 17 22:52:17.171806 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:17.171751    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="50d94d53-7581-47e9-b608-7834c9341981" containerName="storage-initializer"
Sep 17 22:52:17.171806 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:17.171804    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="50d94d53-7581-47e9-b608-7834c9341981" containerName="storage-initializer"
Sep 17 22:52:17.171806 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:17.171810    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="50d94d53-7581-47e9-b608-7834c9341981" containerName="storage-initializer"
Sep 17 22:52:17.179394 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:17.179369    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q8tq9/must-gather-gtfp8"
Sep 17 22:52:17.181527 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:17.181506    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-q8tq9\"/\"kube-root-ca.crt\""
Sep 17 22:52:17.181628 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:17.181531    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-q8tq9\"/\"openshift-service-ca.crt\""
Sep 17 22:52:17.187156 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:17.187122    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q8tq9/must-gather-gtfp8"]
Sep 17 22:52:17.278947 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:17.278914    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swrwh\" (UniqueName: \"kubernetes.io/projected/56148841-b0ae-4c3e-adc6-0edcab6eaecc-kube-api-access-swrwh\") pod \"must-gather-gtfp8\" (UID: \"56148841-b0ae-4c3e-adc6-0edcab6eaecc\") " pod="openshift-must-gather-q8tq9/must-gather-gtfp8"
Sep 17 22:52:17.279113 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:17.278966    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/56148841-b0ae-4c3e-adc6-0edcab6eaecc-must-gather-output\") pod \"must-gather-gtfp8\" (UID: \"56148841-b0ae-4c3e-adc6-0edcab6eaecc\") " pod="openshift-must-gather-q8tq9/must-gather-gtfp8"
Sep 17 22:52:17.379573 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:17.379537    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/56148841-b0ae-4c3e-adc6-0edcab6eaecc-must-gather-output\") pod \"must-gather-gtfp8\" (UID: \"56148841-b0ae-4c3e-adc6-0edcab6eaecc\") " pod="openshift-must-gather-q8tq9/must-gather-gtfp8"
Sep 17 22:52:17.379748 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:17.379609    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swrwh\" (UniqueName: \"kubernetes.io/projected/56148841-b0ae-4c3e-adc6-0edcab6eaecc-kube-api-access-swrwh\") pod \"must-gather-gtfp8\" (UID: \"56148841-b0ae-4c3e-adc6-0edcab6eaecc\") " pod="openshift-must-gather-q8tq9/must-gather-gtfp8"
Sep 17 22:52:17.379974 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:17.379952    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/56148841-b0ae-4c3e-adc6-0edcab6eaecc-must-gather-output\") pod \"must-gather-gtfp8\" (UID: \"56148841-b0ae-4c3e-adc6-0edcab6eaecc\") " pod="openshift-must-gather-q8tq9/must-gather-gtfp8"
Sep 17 22:52:17.387780 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:17.387752    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swrwh\" (UniqueName: \"kubernetes.io/projected/56148841-b0ae-4c3e-adc6-0edcab6eaecc-kube-api-access-swrwh\") pod \"must-gather-gtfp8\" (UID: \"56148841-b0ae-4c3e-adc6-0edcab6eaecc\") " pod="openshift-must-gather-q8tq9/must-gather-gtfp8"
Sep 17 22:52:17.499978 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:17.499887    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q8tq9/must-gather-gtfp8"
Sep 17 22:52:17.767573 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:17.767488    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q8tq9/must-gather-gtfp8"]
Sep 17 22:52:17.770296 ip-10-0-138-45 kubenswrapper[2563]: W0917 22:52:17.770265    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod56148841_b0ae_4c3e_adc6_0edcab6eaecc.slice/crio-80259e10169519e750ef60b38fdd3f5a52a01e1a45fd6f7c5873570e6e99f78b WatchSource:0}: Error finding container 80259e10169519e750ef60b38fdd3f5a52a01e1a45fd6f7c5873570e6e99f78b: Status 404 returned error can't find the container with id 80259e10169519e750ef60b38fdd3f5a52a01e1a45fd6f7c5873570e6e99f78b
Sep 17 22:52:17.772014 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:17.771994    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 22:52:18.572367 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:18.572317    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q8tq9/must-gather-gtfp8" event={"ID":"56148841-b0ae-4c3e-adc6-0edcab6eaecc","Type":"ContainerStarted","Data":"80259e10169519e750ef60b38fdd3f5a52a01e1a45fd6f7c5873570e6e99f78b"}
Sep 17 22:52:24.604013 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:24.603979    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q8tq9/must-gather-gtfp8" event={"ID":"56148841-b0ae-4c3e-adc6-0edcab6eaecc","Type":"ContainerStarted","Data":"0558ccb2e7917e42aceb82dc97b96258c9913bf28f19bea604a62d2289e41b91"}
Sep 17 22:52:24.604013 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:24.604024    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q8tq9/must-gather-gtfp8" event={"ID":"56148841-b0ae-4c3e-adc6-0edcab6eaecc","Type":"ContainerStarted","Data":"2b517b4e0497d741666c456af9f0edc3f1ae77e17c43e9dbac3ee9a55da77db9"}
Sep 17 22:52:24.619191 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:24.619124    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-q8tq9/must-gather-gtfp8" podStartSLOduration=1.697734211 podStartE2EDuration="7.619107499s" totalImagesPullingTime="5.921373288s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:52:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:52:24.618927717 +0000 UTC m=+3255.786880700" watchObservedRunningTime="2026-09-17 22:52:24.619107499 +0000 UTC m=+3255.787060502"
Sep 17 22:52:51.722744 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:51.722711    2563 generic.go:356] "Generic (PLEG): container finished" podID="56148841-b0ae-4c3e-adc6-0edcab6eaecc" containerID="2b517b4e0497d741666c456af9f0edc3f1ae77e17c43e9dbac3ee9a55da77db9" exitCode=0
Sep 17 22:52:51.723211 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:51.722782    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q8tq9/must-gather-gtfp8" event={"ID":"56148841-b0ae-4c3e-adc6-0edcab6eaecc","Type":"ContainerDied","Data":"2b517b4e0497d741666c456af9f0edc3f1ae77e17c43e9dbac3ee9a55da77db9"}
Sep 17 22:52:51.723211 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:51.723090    2563 scope.go:117] "RemoveContainer" containerID="2b517b4e0497d741666c456af9f0edc3f1ae77e17c43e9dbac3ee9a55da77db9"
Sep 17 22:52:52.071151 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:52.071043    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-q8tq9_must-gather-gtfp8_56148841-b0ae-4c3e-adc6-0edcab6eaecc/gather/0.log"
Sep 17 22:52:55.723212 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:55.723177    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-wxx9x_79a6aec6-e29a-45cf-881f-447b87f04557/global-pull-secret-syncer/0.log"
Sep 17 22:52:55.848300 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:55.848271    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-j6ntd_b93158dd-d718-446e-8e3b-38a2235ed7a7/konnectivity-agent/0.log"
Sep 17 22:52:55.901922 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:55.901885    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-45.ec2.internal_2e7e38f229d83af38da836b565cddb01/haproxy/0.log"
Sep 17 22:52:57.617163 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:57.617118    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-q8tq9/must-gather-gtfp8"]
Sep 17 22:52:57.617553 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:57.617332    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-q8tq9/must-gather-gtfp8" podUID="56148841-b0ae-4c3e-adc6-0edcab6eaecc" containerName="copy" containerID="cri-o://0558ccb2e7917e42aceb82dc97b96258c9913bf28f19bea604a62d2289e41b91" gracePeriod=2
Sep 17 22:52:57.621432 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:57.620866    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-q8tq9/must-gather-gtfp8"]
Sep 17 22:52:57.748994 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:57.748971    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-q8tq9_must-gather-gtfp8_56148841-b0ae-4c3e-adc6-0edcab6eaecc/copy/0.log"
Sep 17 22:52:57.749396 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:57.749371    2563 generic.go:356] "Generic (PLEG): container finished" podID="56148841-b0ae-4c3e-adc6-0edcab6eaecc" containerID="0558ccb2e7917e42aceb82dc97b96258c9913bf28f19bea604a62d2289e41b91" exitCode=143
Sep 17 22:52:57.845812 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:57.845790    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-q8tq9_must-gather-gtfp8_56148841-b0ae-4c3e-adc6-0edcab6eaecc/copy/0.log"
Sep 17 22:52:57.848433 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:57.848415    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q8tq9/must-gather-gtfp8"
Sep 17 22:52:57.850523 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:52:57.850502    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-gtfp8\" is forbidden: User \"system:node:ip-10-0-138-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-q8tq9\": no relationship found between node 'ip-10-0-138-45.ec2.internal' and this object" podUID="56148841-b0ae-4c3e-adc6-0edcab6eaecc" pod="openshift-must-gather-q8tq9/must-gather-gtfp8"
Sep 17 22:52:57.937316 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:57.937285    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/56148841-b0ae-4c3e-adc6-0edcab6eaecc-must-gather-output\") pod \"56148841-b0ae-4c3e-adc6-0edcab6eaecc\" (UID: \"56148841-b0ae-4c3e-adc6-0edcab6eaecc\") "
Sep 17 22:52:57.937484 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:57.937365    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-swrwh\" (UniqueName: \"kubernetes.io/projected/56148841-b0ae-4c3e-adc6-0edcab6eaecc-kube-api-access-swrwh\") pod \"56148841-b0ae-4c3e-adc6-0edcab6eaecc\" (UID: \"56148841-b0ae-4c3e-adc6-0edcab6eaecc\") "
Sep 17 22:52:57.938966 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:57.938936    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56148841-b0ae-4c3e-adc6-0edcab6eaecc-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "56148841-b0ae-4c3e-adc6-0edcab6eaecc" (UID: "56148841-b0ae-4c3e-adc6-0edcab6eaecc"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:52:57.939505 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:57.939483    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56148841-b0ae-4c3e-adc6-0edcab6eaecc-kube-api-access-swrwh" (OuterVolumeSpecName: "kube-api-access-swrwh") pod "56148841-b0ae-4c3e-adc6-0edcab6eaecc" (UID: "56148841-b0ae-4c3e-adc6-0edcab6eaecc"). InnerVolumeSpecName "kube-api-access-swrwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:52:58.038401 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:58.038360    2563 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/56148841-b0ae-4c3e-adc6-0edcab6eaecc-must-gather-output\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:52:58.038401 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:58.038392    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swrwh\" (UniqueName: \"kubernetes.io/projected/56148841-b0ae-4c3e-adc6-0edcab6eaecc-kube-api-access-swrwh\") on node \"ip-10-0-138-45.ec2.internal\" DevicePath \"\""
Sep 17 22:52:58.754916 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:58.754885    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-q8tq9_must-gather-gtfp8_56148841-b0ae-4c3e-adc6-0edcab6eaecc/copy/0.log"
Sep 17 22:52:58.755377 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:58.755279    2563 scope.go:117] "RemoveContainer" containerID="0558ccb2e7917e42aceb82dc97b96258c9913bf28f19bea604a62d2289e41b91"
Sep 17 22:52:58.755377 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:58.755322    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q8tq9/must-gather-gtfp8"
Sep 17 22:52:58.757353 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:52:58.757328    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-gtfp8\" is forbidden: User \"system:node:ip-10-0-138-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-q8tq9\": no relationship found between node 'ip-10-0-138-45.ec2.internal' and this object" podUID="56148841-b0ae-4c3e-adc6-0edcab6eaecc" pod="openshift-must-gather-q8tq9/must-gather-gtfp8"
Sep 17 22:52:58.762906 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:58.762885    2563 scope.go:117] "RemoveContainer" containerID="2b517b4e0497d741666c456af9f0edc3f1ae77e17c43e9dbac3ee9a55da77db9"
Sep 17 22:52:58.764764 ip-10-0-138-45 kubenswrapper[2563]: E0917 22:52:58.764740    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-gtfp8\" is forbidden: User \"system:node:ip-10-0-138-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-q8tq9\": no relationship found between node 'ip-10-0-138-45.ec2.internal' and this object" podUID="56148841-b0ae-4c3e-adc6-0edcab6eaecc" pod="openshift-must-gather-q8tq9/must-gather-gtfp8"
Sep 17 22:52:58.913646 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:52:58.913616    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56148841-b0ae-4c3e-adc6-0edcab6eaecc" path="/var/lib/kubelet/pods/56148841-b0ae-4c3e-adc6-0edcab6eaecc/volumes"
Sep 17 22:53:00.408590 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:00.408560    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-58ccf7d5fc-sg9zb_49ac9268-c95a-4df8-a119-9843598e243d/metrics-server/0.log"
Sep 17 22:53:00.557471 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:00.557447    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-klbh9_3eebb3bc-62b7-464c-8199-d97145f2c8cb/node-exporter/0.log"
Sep 17 22:53:00.578222 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:00.578202    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-klbh9_3eebb3bc-62b7-464c-8199-d97145f2c8cb/kube-rbac-proxy/0.log"
Sep 17 22:53:00.600447 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:00.600418    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-klbh9_3eebb3bc-62b7-464c-8199-d97145f2c8cb/init-textfile/0.log"
Sep 17 22:53:00.705709 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:00.705634    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-jqq5f_2123e624-b230-4c2b-a8c4-eaad63f9fbcb/kube-rbac-proxy-main/0.log"
Sep 17 22:53:00.728858 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:00.728830    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-jqq5f_2123e624-b230-4c2b-a8c4-eaad63f9fbcb/kube-rbac-proxy-self/0.log"
Sep 17 22:53:00.749931 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:00.749905    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-jqq5f_2123e624-b230-4c2b-a8c4-eaad63f9fbcb/openshift-state-metrics/0.log"
Sep 17 22:53:00.800624 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:00.800599    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cd0340a2-0b4d-42e1-8661-f5f15afc9b56/prometheus/0.log"
Sep 17 22:53:00.818469 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:00.818436    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cd0340a2-0b4d-42e1-8661-f5f15afc9b56/config-reloader/0.log"
Sep 17 22:53:00.844830 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:00.844807    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cd0340a2-0b4d-42e1-8661-f5f15afc9b56/thanos-sidecar/0.log"
Sep 17 22:53:00.869080 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:00.869056    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cd0340a2-0b4d-42e1-8661-f5f15afc9b56/kube-rbac-proxy-web/0.log"
Sep 17 22:53:00.891746 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:00.891725    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cd0340a2-0b4d-42e1-8661-f5f15afc9b56/kube-rbac-proxy/0.log"
Sep 17 22:53:00.916317 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:00.916301    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cd0340a2-0b4d-42e1-8661-f5f15afc9b56/kube-rbac-proxy-thanos/0.log"
Sep 17 22:53:00.943227 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:00.943210    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cd0340a2-0b4d-42e1-8661-f5f15afc9b56/init-config-reloader/0.log"
Sep 17 22:53:00.977593 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:00.977523    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-sjdcx_12b8424b-4eee-4a25-a2a2-407d9fbcf929/prometheus-operator/0.log"
Sep 17 22:53:00.997491 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:00.997473    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-sjdcx_12b8424b-4eee-4a25-a2a2-407d9fbcf929/kube-rbac-proxy/0.log"
Sep 17 22:53:01.023168 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:01.023132    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5c6747dc7b-zwlxc_7cc7320d-2524-475c-abc8-1b97ca677476/prometheus-operator-admission-webhook/0.log"
Sep 17 22:53:03.601772 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:03.601734    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6594f4c75-drqjx_3befbbf9-41ff-4f57-9cfd-aee679772893/download-server/0.log"
Sep 17 22:53:04.656732 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:04.656698    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kfvtg/perf-node-gather-daemonset-kwh87"]
Sep 17 22:53:04.657134 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:04.656973    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56148841-b0ae-4c3e-adc6-0edcab6eaecc" containerName="gather"
Sep 17 22:53:04.657134 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:04.656984    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="56148841-b0ae-4c3e-adc6-0edcab6eaecc" containerName="gather"
Sep 17 22:53:04.657134 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:04.656998    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56148841-b0ae-4c3e-adc6-0edcab6eaecc" containerName="copy"
Sep 17 22:53:04.657134 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:04.657003    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="56148841-b0ae-4c3e-adc6-0edcab6eaecc" containerName="copy"
Sep 17 22:53:04.657134 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:04.657057    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="56148841-b0ae-4c3e-adc6-0edcab6eaecc" containerName="copy"
Sep 17 22:53:04.657134 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:04.657064    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="56148841-b0ae-4c3e-adc6-0edcab6eaecc" containerName="gather"
Sep 17 22:53:04.662212 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:04.662193    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kfvtg/perf-node-gather-daemonset-kwh87"
Sep 17 22:53:04.664537 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:04.664514    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kfvtg\"/\"kube-root-ca.crt\""
Sep 17 22:53:04.665561 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:04.665544    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kfvtg\"/\"default-dockercfg-8n8bw\""
Sep 17 22:53:04.665643 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:04.665605    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kfvtg\"/\"openshift-service-ca.crt\""
Sep 17 22:53:04.670421 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:04.670398    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kfvtg/perf-node-gather-daemonset-kwh87"]
Sep 17 22:53:04.692291 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:04.692264    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bkmw\" (UniqueName: \"kubernetes.io/projected/608afc4a-cc7f-485c-9994-7fcc23c02c23-kube-api-access-9bkmw\") pod \"perf-node-gather-daemonset-kwh87\" (UID: \"608afc4a-cc7f-485c-9994-7fcc23c02c23\") " pod="openshift-must-gather-kfvtg/perf-node-gather-daemonset-kwh87"
Sep 17 22:53:04.692425 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:04.692296    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/608afc4a-cc7f-485c-9994-7fcc23c02c23-podres\") pod \"perf-node-gather-daemonset-kwh87\" (UID: \"608afc4a-cc7f-485c-9994-7fcc23c02c23\") " pod="openshift-must-gather-kfvtg/perf-node-gather-daemonset-kwh87"
Sep 17 22:53:04.692425 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:04.692329    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/608afc4a-cc7f-485c-9994-7fcc23c02c23-lib-modules\") pod \"perf-node-gather-daemonset-kwh87\" (UID: \"608afc4a-cc7f-485c-9994-7fcc23c02c23\") " pod="openshift-must-gather-kfvtg/perf-node-gather-daemonset-kwh87"
Sep 17 22:53:04.692425 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:04.692400    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/608afc4a-cc7f-485c-9994-7fcc23c02c23-proc\") pod \"perf-node-gather-daemonset-kwh87\" (UID: \"608afc4a-cc7f-485c-9994-7fcc23c02c23\") " pod="openshift-must-gather-kfvtg/perf-node-gather-daemonset-kwh87"
Sep 17 22:53:04.692541 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:04.692435    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/608afc4a-cc7f-485c-9994-7fcc23c02c23-sys\") pod \"perf-node-gather-daemonset-kwh87\" (UID: \"608afc4a-cc7f-485c-9994-7fcc23c02c23\") " pod="openshift-must-gather-kfvtg/perf-node-gather-daemonset-kwh87"
Sep 17 22:53:04.793210 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:04.793174    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bkmw\" (UniqueName: \"kubernetes.io/projected/608afc4a-cc7f-485c-9994-7fcc23c02c23-kube-api-access-9bkmw\") pod \"perf-node-gather-daemonset-kwh87\" (UID: \"608afc4a-cc7f-485c-9994-7fcc23c02c23\") " pod="openshift-must-gather-kfvtg/perf-node-gather-daemonset-kwh87"
Sep 17 22:53:04.793210 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:04.793215    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/608afc4a-cc7f-485c-9994-7fcc23c02c23-podres\") pod \"perf-node-gather-daemonset-kwh87\" (UID: \"608afc4a-cc7f-485c-9994-7fcc23c02c23\") " pod="openshift-must-gather-kfvtg/perf-node-gather-daemonset-kwh87"
Sep 17 22:53:04.793471 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:04.793254    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/608afc4a-cc7f-485c-9994-7fcc23c02c23-lib-modules\") pod \"perf-node-gather-daemonset-kwh87\" (UID: \"608afc4a-cc7f-485c-9994-7fcc23c02c23\") " pod="openshift-must-gather-kfvtg/perf-node-gather-daemonset-kwh87"
Sep 17 22:53:04.793471 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:04.793380    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/608afc4a-cc7f-485c-9994-7fcc23c02c23-proc\") pod \"perf-node-gather-daemonset-kwh87\" (UID: \"608afc4a-cc7f-485c-9994-7fcc23c02c23\") " pod="openshift-must-gather-kfvtg/perf-node-gather-daemonset-kwh87"
Sep 17 22:53:04.793471 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:04.793415    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/608afc4a-cc7f-485c-9994-7fcc23c02c23-sys\") pod \"perf-node-gather-daemonset-kwh87\" (UID: \"608afc4a-cc7f-485c-9994-7fcc23c02c23\") " pod="openshift-must-gather-kfvtg/perf-node-gather-daemonset-kwh87"
Sep 17 22:53:04.793471 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:04.793384    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/608afc4a-cc7f-485c-9994-7fcc23c02c23-podres\") pod \"perf-node-gather-daemonset-kwh87\" (UID: \"608afc4a-cc7f-485c-9994-7fcc23c02c23\") " pod="openshift-must-gather-kfvtg/perf-node-gather-daemonset-kwh87"
Sep 17 22:53:04.793471 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:04.793416    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/608afc4a-cc7f-485c-9994-7fcc23c02c23-proc\") pod \"perf-node-gather-daemonset-kwh87\" (UID: \"608afc4a-cc7f-485c-9994-7fcc23c02c23\") " pod="openshift-must-gather-kfvtg/perf-node-gather-daemonset-kwh87"
Sep 17 22:53:04.793471 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:04.793386    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/608afc4a-cc7f-485c-9994-7fcc23c02c23-lib-modules\") pod \"perf-node-gather-daemonset-kwh87\" (UID: \"608afc4a-cc7f-485c-9994-7fcc23c02c23\") " pod="openshift-must-gather-kfvtg/perf-node-gather-daemonset-kwh87"
Sep 17 22:53:04.793719 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:04.793493    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/608afc4a-cc7f-485c-9994-7fcc23c02c23-sys\") pod \"perf-node-gather-daemonset-kwh87\" (UID: \"608afc4a-cc7f-485c-9994-7fcc23c02c23\") " pod="openshift-must-gather-kfvtg/perf-node-gather-daemonset-kwh87"
Sep 17 22:53:04.800997 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:04.800968    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bkmw\" (UniqueName: \"kubernetes.io/projected/608afc4a-cc7f-485c-9994-7fcc23c02c23-kube-api-access-9bkmw\") pod \"perf-node-gather-daemonset-kwh87\" (UID: \"608afc4a-cc7f-485c-9994-7fcc23c02c23\") " pod="openshift-must-gather-kfvtg/perf-node-gather-daemonset-kwh87"
Sep 17 22:53:04.972275 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:04.972191    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kfvtg/perf-node-gather-daemonset-kwh87"
Sep 17 22:53:05.086712 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:05.086680    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kcnsx_0e64d86f-9913-4a28-a1dc-1810710456bc/dns/0.log"
Sep 17 22:53:05.108096 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:05.108068    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kcnsx_0e64d86f-9913-4a28-a1dc-1810710456bc/kube-rbac-proxy/0.log"
Sep 17 22:53:05.235073 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:05.234992    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-n2fp6_9d2ac593-98b6-4b38-adef-a1bfbf1a168f/dns-node-resolver/0.log"
Sep 17 22:53:05.239050 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:05.239026    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kfvtg/perf-node-gather-daemonset-kwh87"]
Sep 17 22:53:05.241599 ip-10-0-138-45 kubenswrapper[2563]: W0917 22:53:05.241574    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod608afc4a_cc7f_485c_9994_7fcc23c02c23.slice/crio-3f77876a9f2656fc0b806d3708dc426e9a89d2bf85b085f0712792f80600ea88 WatchSource:0}: Error finding container 3f77876a9f2656fc0b806d3708dc426e9a89d2bf85b085f0712792f80600ea88: Status 404 returned error can't find the container with id 3f77876a9f2656fc0b806d3708dc426e9a89d2bf85b085f0712792f80600ea88
Sep 17 22:53:05.713311 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:05.713273    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-988484c4f-ggmxg_84f54040-ce25-49f7-a00d-d0231b434a33/registry/0.log"
Sep 17 22:53:05.762500 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:05.762471    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hzx5m_c20c2a19-2b30-48fb-bb14-b5321cf2f4d0/node-ca/0.log"
Sep 17 22:53:05.783102 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:05.783071    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kfvtg/perf-node-gather-daemonset-kwh87" event={"ID":"608afc4a-cc7f-485c-9994-7fcc23c02c23","Type":"ContainerStarted","Data":"50b7bbce0255ee2a719d86e4bf14b9cd8904e346811c46405fd3df13b398d0a2"}
Sep 17 22:53:05.783291 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:05.783117    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kfvtg/perf-node-gather-daemonset-kwh87" event={"ID":"608afc4a-cc7f-485c-9994-7fcc23c02c23","Type":"ContainerStarted","Data":"3f77876a9f2656fc0b806d3708dc426e9a89d2bf85b085f0712792f80600ea88"}
Sep 17 22:53:05.783291 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:05.783158    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-kfvtg/perf-node-gather-daemonset-kwh87"
Sep 17 22:53:05.800215 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:05.800123    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kfvtg/perf-node-gather-daemonset-kwh87" podStartSLOduration=1.800108858 podStartE2EDuration="1.800108858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 22:53:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:53:05.799207615 +0000 UTC m=+3296.967160600" watchObservedRunningTime="2026-09-17 22:53:05.800108858 +0000 UTC m=+3296.968061951"
Sep 17 22:53:06.857810 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:06.857769    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-p6kpt_01db16ed-7748-4c99-a5ce-7f98df2f5e7a/serve-healthcheck-canary/0.log"
Sep 17 22:53:07.324115 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:07.324083    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l9x89_cd8b7977-5b29-4e94-b08c-b4de9c50deaf/kube-rbac-proxy/0.log"
Sep 17 22:53:07.343987 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:07.343956    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l9x89_cd8b7977-5b29-4e94-b08c-b4de9c50deaf/exporter/0.log"
Sep 17 22:53:07.364449 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:07.364426    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l9x89_cd8b7977-5b29-4e94-b08c-b4de9c50deaf/extractor/0.log"
Sep 17 22:53:10.112473 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:10.112437    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-qddlq_30d8c766-4389-4e58-ba47-812160a0d19d/s3-init/0.log"
Sep 17 22:53:10.137860 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:10.137830    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-qddlq_30d8c766-4389-4e58-ba47-812160a0d19d/download-hf-model/0.log"
Sep 17 22:53:10.170945 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:10.170914    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-6z54j_a9dba7ec-6fad-445d-ae48-429e1d1cd9d0/s3-tls-init-custom/0.log"
Sep 17 22:53:10.188425 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:10.188400    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-6z54j_a9dba7ec-6fad-445d-ae48-429e1d1cd9d0/download-hf-model/0.log"
Sep 17 22:53:10.228895 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:10.228864    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-r46kx_340c5f4c-7f8b-4ac3-8821-ac96bb524c0a/s3-tls-init-serving/0.log"
Sep 17 22:53:10.247434 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:10.247406    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-r46kx_340c5f4c-7f8b-4ac3-8821-ac96bb524c0a/download-hf-model/0.log"
Sep 17 22:53:11.796709 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:11.796685    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-kfvtg/perf-node-gather-daemonset-kwh87"
Sep 17 22:53:14.509189 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:14.509135    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-r4fmm_ac585f9d-01e3-403f-a392-6081f15a9d9b/migrator/0.log"
Sep 17 22:53:14.531013 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:14.530984    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-r4fmm_ac585f9d-01e3-403f-a392-6081f15a9d9b/graceful-termination/0.log"
Sep 17 22:53:15.944585 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:15.944561    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wjt9h_f004b93f-9438-4008-ad17-92150090d28c/kube-multus-additional-cni-plugins/0.log"
Sep 17 22:53:15.965443 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:15.965411    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wjt9h_f004b93f-9438-4008-ad17-92150090d28c/egress-router-binary-copy/0.log"
Sep 17 22:53:15.989446 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:15.989423    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wjt9h_f004b93f-9438-4008-ad17-92150090d28c/cni-plugins/0.log"
Sep 17 22:53:16.010252 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:16.010212    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wjt9h_f004b93f-9438-4008-ad17-92150090d28c/bond-cni-plugin/0.log"
Sep 17 22:53:16.031209 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:16.031161    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wjt9h_f004b93f-9438-4008-ad17-92150090d28c/routeoverride-cni/0.log"
Sep 17 22:53:16.052905 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:16.052879    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wjt9h_f004b93f-9438-4008-ad17-92150090d28c/whereabouts-cni-bincopy/0.log"
Sep 17 22:53:16.115899 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:16.115867    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wjt9h_f004b93f-9438-4008-ad17-92150090d28c/whereabouts-cni/0.log"
Sep 17 22:53:16.323410 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:16.323381    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-f6t75_e5348473-b853-4265-8876-9e0b2c8943d8/kube-multus/0.log"
Sep 17 22:53:16.463037 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:16.463009    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xkk4g_45d5f94d-990c-4f9e-b1ab-980021c00716/network-metrics-daemon/0.log"
Sep 17 22:53:16.480015 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:16.479990    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xkk4g_45d5f94d-990c-4f9e-b1ab-980021c00716/kube-rbac-proxy/0.log"
Sep 17 22:53:17.676289 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:17.676258    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nrkd5_34b8ef2b-97ba-4562-87ee-8da0cb4604c7/ovn-controller/0.log"
Sep 17 22:53:17.692333 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:17.692310    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nrkd5_34b8ef2b-97ba-4562-87ee-8da0cb4604c7/ovn-acl-logging/0.log"
Sep 17 22:53:17.692557 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:17.692540    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nrkd5_34b8ef2b-97ba-4562-87ee-8da0cb4604c7/ovn-acl-logging/1.log"
Sep 17 22:53:17.709727 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:17.709701    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nrkd5_34b8ef2b-97ba-4562-87ee-8da0cb4604c7/kube-rbac-proxy-node/0.log"
Sep 17 22:53:17.730026 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:17.730003    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nrkd5_34b8ef2b-97ba-4562-87ee-8da0cb4604c7/kube-rbac-proxy-ovn-metrics/0.log"
Sep 17 22:53:17.748814 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:17.748793    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nrkd5_34b8ef2b-97ba-4562-87ee-8da0cb4604c7/northd/0.log"
Sep 17 22:53:17.770067 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:17.770042    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nrkd5_34b8ef2b-97ba-4562-87ee-8da0cb4604c7/nbdb/0.log"
Sep 17 22:53:17.790228 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:17.790202    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nrkd5_34b8ef2b-97ba-4562-87ee-8da0cb4604c7/sbdb/0.log"
Sep 17 22:53:17.847287 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:17.847264    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nrkd5_34b8ef2b-97ba-4562-87ee-8da0cb4604c7/ovnkube-controller/0.log"
Sep 17 22:53:18.880007 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:18.879974    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-bnj94_3b714c1c-9f7e-430e-bc07-94cf3a75921c/network-check-target-container/0.log"
Sep 17 22:53:19.818307 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:19.818277    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-tkphq_d6f6a71b-842d-48f0-93fc-28006b5c122f/iptables-alerter/0.log"
Sep 17 22:53:20.478780 ip-10-0-138-45 kubenswrapper[2563]: I0917 22:53:20.478746    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-26gb4_992a8c01-f2ce-4523-9a33-56e12b134be2/tuned/0.log"