Sep 17 17:40:46.050493 ip-10-0-131-14 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 17:40:49.350263 ip-10-0-131-14 kubenswrapper[2569]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:40:49.350263 ip-10-0-131-14 kubenswrapper[2569]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 17:40:49.350263 ip-10-0-131-14 kubenswrapper[2569]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:40:49.350263 ip-10-0-131-14 kubenswrapper[2569]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 17:40:49.350263 ip-10-0-131-14 kubenswrapper[2569]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:40:49.351063 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.350335    2569 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 17:40:49.354519 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354498    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:40:49.354519 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354518    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:40:49.354519 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354522    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:40:49.354615 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354525    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:40:49.354615 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354529    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:40:49.354615 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354534    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:40:49.354615 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354538    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:40:49.354615 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354542    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:40:49.354615 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354544    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:40:49.354615 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354554    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:40:49.354615 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354557    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:40:49.354615 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354560    2569 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:40:49.354615 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354562    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:40:49.354615 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354566    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:40:49.354615 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354568    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:40:49.354615 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354572    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:40:49.354615 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354574    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:40:49.354615 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354577    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:40:49.354615 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354579    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:40:49.354615 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354582    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:40:49.354615 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354586    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:40:49.354615 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354590    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:40:49.355065 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354593    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:40:49.355065 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354597    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:40:49.355065 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354600    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:40:49.355065 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354603    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:40:49.355065 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354607    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:40:49.355065 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354610    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:40:49.355065 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354613    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:40:49.355065 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354615    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:40:49.355065 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354618    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:40:49.355065 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354621    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:40:49.355065 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354624    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:40:49.355065 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354626    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:40:49.355065 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354629    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:40:49.355065 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354631    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:40:49.355065 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354635    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:40:49.355065 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354638    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:40:49.355065 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354640    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:40:49.355065 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354643    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:40:49.355065 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354646    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:40:49.355591 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354648    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:40:49.355591 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354651    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:40:49.355591 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354654    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:40:49.355591 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354656    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:40:49.355591 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354659    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:40:49.355591 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354661    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:40:49.355591 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354664    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:40:49.355591 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354666    2569 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:40:49.355591 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354668    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:40:49.355591 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354671    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:40:49.355591 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354673    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:40:49.355591 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354676    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:40:49.355591 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354678    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:40:49.355591 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354680    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:40:49.355591 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354684    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:40:49.355591 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354686    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:40:49.355591 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354689    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:40:49.355591 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354692    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:40:49.355591 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354694    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:40:49.355591 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354697    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:40:49.356300 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354700    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:40:49.356300 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354702    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:40:49.356300 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354705    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:40:49.356300 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354709    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:40:49.356300 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354712    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:40:49.356300 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354715    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:40:49.356300 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354718    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:40:49.356300 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354720    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:40:49.356300 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354723    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:40:49.356300 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354726    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:40:49.356300 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354728    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:40:49.356300 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354731    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:40:49.356300 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354733    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:40:49.356300 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354736    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:40:49.356300 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354740    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:40:49.356300 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354743    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:40:49.356300 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354746    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:40:49.356300 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354748    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:40:49.356300 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354751    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:40:49.356300 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354753    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:40:49.357129 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354756    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:40:49.357129 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354758    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:40:49.357129 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354761    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:40:49.357129 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354764    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:40:49.357129 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354766    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:40:49.357129 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354769    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:40:49.357129 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354772    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:40:49.357129 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354775    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:40:49.357129 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354778    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:40:49.357129 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354781    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:40:49.357129 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354783    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:40:49.357129 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354786    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:40:49.357129 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354788    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:40:49.357129 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354791    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:40:49.357129 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354793    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:40:49.357129 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354796    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:40:49.357129 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354798    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:40:49.357129 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.354801    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:40:49.357129 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355534    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:40:49.357129 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355541    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:40:49.357949 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355544    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:40:49.357949 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355547    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:40:49.357949 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355550    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:40:49.357949 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355553    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:40:49.357949 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355555    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:40:49.357949 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355558    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:40:49.357949 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355562    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:40:49.357949 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355565    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:40:49.357949 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355568    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:40:49.357949 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355571    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:40:49.357949 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355573    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:40:49.357949 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355576    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:40:49.357949 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355579    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:40:49.357949 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355581    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:40:49.357949 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355584    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:40:49.357949 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355586    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:40:49.357949 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355589    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:40:49.357949 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355592    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:40:49.357949 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355595    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:40:49.357949 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355598    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:40:49.358837 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355600    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:40:49.358837 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355603    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:40:49.358837 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355606    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:40:49.358837 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355609    2569 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:40:49.358837 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355611    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:40:49.358837 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355614    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:40:49.358837 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355617    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:40:49.358837 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355619    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:40:49.358837 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355622    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:40:49.358837 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355624    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:40:49.358837 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355627    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:40:49.358837 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355629    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:40:49.358837 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355632    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:40:49.358837 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355634    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:40:49.358837 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355638    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:40:49.358837 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355641    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:40:49.358837 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355643    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:40:49.358837 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355646    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:40:49.358837 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355648    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:40:49.358837 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355651    2569 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:40:49.359720 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355656    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:40:49.359720 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355659    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:40:49.359720 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355661    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:40:49.359720 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355664    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:40:49.359720 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355666    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:40:49.359720 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355669    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:40:49.359720 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355671    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:40:49.359720 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355674    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:40:49.359720 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355677    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:40:49.359720 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355680    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:40:49.359720 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355683    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:40:49.359720 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355686    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:40:49.359720 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355688    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:40:49.359720 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355691    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:40:49.359720 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355694    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:40:49.359720 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355697    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:40:49.359720 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355701    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:40:49.359720 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355704    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:40:49.359720 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355708    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:40:49.359720 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355711    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:40:49.360320 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355714    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:40:49.360320 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355717    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:40:49.360320 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355720    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:40:49.360320 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355723    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:40:49.360320 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355726    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:40:49.360320 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355728    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:40:49.360320 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355731    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:40:49.360320 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355733    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:40:49.360320 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355737    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:40:49.360320 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355739    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:40:49.360320 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355742    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:40:49.360320 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355745    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:40:49.360320 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355747    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:40:49.360320 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355751    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:40:49.360320 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355753    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:40:49.360320 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355756    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:40:49.360320 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355758    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:40:49.360320 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355761    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:40:49.360320 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355763    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:40:49.360860 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355766    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:40:49.360860 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355768    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:40:49.360860 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355771    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:40:49.360860 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355774    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:40:49.360860 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355779    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:40:49.360860 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355782    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:40:49.360860 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355784    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:40:49.360860 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355787    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:40:49.360860 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355789    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:40:49.360860 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355792    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:40:49.360860 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355794    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:40:49.360860 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355797    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:40:49.360860 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355800    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:40:49.360860 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355802    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:40:49.360860 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355805    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:40:49.360860 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355809    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:40:49.360860 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355813    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:40:49.360860 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.355816    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:40:49.360860 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357574    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:40:49.361455 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357592    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:40:49.361455 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357598    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:40:49.361455 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357603    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:40:49.361455 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357608    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:40:49.361455 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357613    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:40:49.361455 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357618    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:40:49.361455 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357623    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:40:49.361455 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357627    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:40:49.361455 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357632    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:40:49.361455 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357637    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:40:49.361455 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357641    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:40:49.361455 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357646    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:40:49.361455 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357650    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:40:49.361455 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357656    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:40:49.361455 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357662    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:40:49.361455 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357667    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:40:49.361455 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357672    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:40:49.361455 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357676    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:40:49.361455 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357680    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:40:49.361455 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357691    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:40:49.362132 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357696    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:40:49.362132 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357701    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:40:49.362132 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357705    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:40:49.362132 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357709    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:40:49.362132 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357713    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:40:49.362132 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357718    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:40:49.362132 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357722    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:40:49.362132 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357726    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:40:49.362132 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357730    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:40:49.362132 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357735    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:40:49.362132 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357739    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:40:49.362132 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357743    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:40:49.362132 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357750    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:40:49.362132 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357754    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:40:49.362132 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357758    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:40:49.362132 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357763    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:40:49.362132 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357767    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:40:49.362132 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357771    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:40:49.362132 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357775    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:40:49.362645 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357779    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:40:49.362645 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357784    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:40:49.362645 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357788    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:40:49.362645 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357794    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:40:49.362645 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357798    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:40:49.362645 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357802    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:40:49.362645 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357809    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:40:49.362645 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357815    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:40:49.362645 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357819    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:40:49.362645 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357823    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:40:49.362645 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357829    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:40:49.362645 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357833    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:40:49.362645 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357838    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:40:49.362645 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357842    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:40:49.362645 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357846    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:40:49.362645 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357850    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:40:49.362645 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357855    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:40:49.362645 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357859    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:40:49.362645 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357863    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:40:49.362645 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357867    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:40:49.363249 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357872    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:40:49.363249 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357876    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:40:49.363249 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357880    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:40:49.363249 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357885    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:40:49.363249 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357889    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:40:49.363249 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357894    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:40:49.363249 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357906    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:40:49.363249 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357911    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:40:49.363249 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357916    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:40:49.363249 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357921    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:40:49.363249 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357925    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:40:49.363249 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357929    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:40:49.363249 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357934    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:40:49.363249 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357938    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:40:49.363249 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357942    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:40:49.363249 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357946    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:40:49.363249 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357950    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:40:49.363249 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357954    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:40:49.363249 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357958    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:40:49.363249 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357961    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:40:49.363738 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357965    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:40:49.363738 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357969    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:40:49.363738 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357976    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:40:49.363738 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357981    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:40:49.363738 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357986    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:40:49.363738 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357990    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:40:49.363738 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357994    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:40:49.363738 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.357999    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:40:49.363738 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.358003    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:40:49.363738 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.358007    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:40:49.363738 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.358011    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:40:49.363738 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.358016    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:40:49.363738 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.358020    2569 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:40:49.363738 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.358024    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:40:49.363738 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.358029    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:40:49.363738 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.358033    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:40:49.363738 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.358037    2569 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:40:49.363738 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.358042    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:40:49.363738 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.358047    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:40:49.363738 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358168    2569 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 17:40:49.364283 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358225    2569 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 17:40:49.364283 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358236    2569 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 17:40:49.364283 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358243    2569 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 17:40:49.364283 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358251    2569 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 17:40:49.364283 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358257    2569 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 17:40:49.364283 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358263    2569 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 17:40:49.364283 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358270    2569 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 17:40:49.364283 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358275    2569 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 17:40:49.364283 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358281    2569 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 17:40:49.364283 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358286    2569 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 17:40:49.364283 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358292    2569 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 17:40:49.364283 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358297    2569 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 17:40:49.364283 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358302    2569 flags.go:64] FLAG: --cgroup-root=""
Sep 17 17:40:49.364283 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358307    2569 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 17:40:49.364283 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358312    2569 flags.go:64] FLAG: --client-ca-file=""
Sep 17 17:40:49.364283 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358316    2569 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 17:40:49.364283 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358323    2569 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 17:40:49.364283 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358331    2569 flags.go:64] FLAG: --cluster-domain=""
Sep 17 17:40:49.364283 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358336    2569 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 17:40:49.364283 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358341    2569 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 17:40:49.364283 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358346    2569 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 17:40:49.364283 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358352    2569 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 17:40:49.364283 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358357    2569 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 17:40:49.364283 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358362    2569 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 17:40:49.364869 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358367    2569 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 17:40:49.364869 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358373    2569 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 17:40:49.364869 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358378    2569 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 17:40:49.364869 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358383    2569 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 17:40:49.364869 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358388    2569 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 17:40:49.364869 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358394    2569 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 17:40:49.364869 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358399    2569 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 17:40:49.364869 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358406    2569 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 17:40:49.364869 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358412    2569 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 17:40:49.364869 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358417    2569 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 17:40:49.364869 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358421    2569 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 17:40:49.364869 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358427    2569 flags.go:64] FLAG: --enable-server="true"
Sep 17 17:40:49.364869 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358431    2569 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 17:40:49.364869 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358439    2569 flags.go:64] FLAG: --event-burst="100"
Sep 17 17:40:49.364869 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358444    2569 flags.go:64] FLAG: --event-qps="50"
Sep 17 17:40:49.364869 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358449    2569 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 17:40:49.364869 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358454    2569 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 17:40:49.364869 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358459    2569 flags.go:64] FLAG: --eviction-hard=""
Sep 17 17:40:49.364869 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358465    2569 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 17:40:49.364869 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358470    2569 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 17:40:49.364869 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358475    2569 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 17:40:49.364869 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358480    2569 flags.go:64] FLAG: --eviction-soft=""
Sep 17 17:40:49.364869 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358485    2569 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 17:40:49.364869 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358490    2569 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 17:40:49.364869 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358495    2569 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 17:40:49.365511 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358500    2569 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 17:40:49.365511 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358506    2569 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 17:40:49.365511 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358510    2569 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 17:40:49.365511 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358515    2569 flags.go:64] FLAG: --feature-gates=""
Sep 17 17:40:49.365511 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358525    2569 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 17:40:49.365511 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358530    2569 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 17:40:49.365511 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358535    2569 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 17:40:49.365511 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358540    2569 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 17:40:49.365511 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358545    2569 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 17:40:49.365511 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358550    2569 flags.go:64] FLAG: --help="false"
Sep 17 17:40:49.365511 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358555    2569 flags.go:64] FLAG: --hostname-override="ip-10-0-131-14.ec2.internal"
Sep 17 17:40:49.365511 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358561    2569 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 17:40:49.365511 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358566    2569 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 17:40:49.365511 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358571    2569 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 17:40:49.365511 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358577    2569 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 17:40:49.365511 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358583    2569 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 17:40:49.365511 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358588    2569 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 17:40:49.365511 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358593    2569 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 17:40:49.365511 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358598    2569 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 17:40:49.365511 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358603    2569 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 17:40:49.365511 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358607    2569 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 17:40:49.365511 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358613    2569 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 17:40:49.365511 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358618    2569 flags.go:64] FLAG: --kube-reserved=""
Sep 17 17:40:49.365511 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358622    2569 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 17:40:49.366159 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358627    2569 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 17:40:49.366159 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358632    2569 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 17:40:49.366159 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358637    2569 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 17:40:49.366159 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358642    2569 flags.go:64] FLAG: --lock-file=""
Sep 17 17:40:49.366159 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358646    2569 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 17:40:49.366159 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358651    2569 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 17:40:49.366159 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358656    2569 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 17:40:49.366159 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358664    2569 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 17:40:49.366159 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358669    2569 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 17:40:49.366159 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358675    2569 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 17:40:49.366159 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358680    2569 flags.go:64] FLAG: --logging-format="text"
Sep 17 17:40:49.366159 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358685    2569 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 17:40:49.366159 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358690    2569 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 17:40:49.366159 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358695    2569 flags.go:64] FLAG: --manifest-url=""
Sep 17 17:40:49.366159 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358699    2569 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 17:40:49.366159 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358707    2569 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 17:40:49.366159 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358711    2569 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 17:40:49.366159 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358718    2569 flags.go:64] FLAG: --max-pods="110"
Sep 17 17:40:49.366159 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358724    2569 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 17:40:49.366159 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358732    2569 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 17:40:49.366159 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358737    2569 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 17:40:49.366159 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358742    2569 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 17:40:49.366159 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358747    2569 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 17:40:49.366159 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358751    2569 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 17:40:49.366159 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358757    2569 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 17:40:49.366785 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358770    2569 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 17:40:49.366785 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358775    2569 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 17:40:49.366785 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358780    2569 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 17:40:49.366785 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358784    2569 flags.go:64] FLAG: --pod-cidr=""
Sep 17 17:40:49.366785 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358790    2569 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 17:40:49.366785 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358800    2569 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 17:40:49.366785 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358805    2569 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 17:40:49.366785 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358810    2569 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 17:40:49.366785 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358815    2569 flags.go:64] FLAG: --port="10250"
Sep 17 17:40:49.366785 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358820    2569 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 17:40:49.366785 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358825    2569 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0219a7aeedb4cb0d0"
Sep 17 17:40:49.366785 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358830    2569 flags.go:64] FLAG: --qos-reserved=""
Sep 17 17:40:49.366785 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358834    2569 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 17:40:49.366785 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358839    2569 flags.go:64] FLAG: --register-node="true"
Sep 17 17:40:49.366785 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358843    2569 flags.go:64] FLAG: --register-with-taints=""
Sep 17 17:40:49.366785 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358849    2569 flags.go:64] FLAG: --registry-burst="10"
Sep 17 17:40:49.366785 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358854    2569 flags.go:64] FLAG: --registry-qps="5"
Sep 17 17:40:49.366785 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358859    2569 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 17:40:49.366785 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358864    2569 flags.go:64] FLAG: --reserved-memory=""
Sep 17 17:40:49.366785 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358870    2569 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 17:40:49.366785 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358875    2569 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 17:40:49.366785 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358880    2569 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 17:40:49.366785 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358885    2569 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 17:40:49.366785 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358890    2569 flags.go:64] FLAG: --runonce="false"
Sep 17 17:40:49.366785 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358895    2569 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 17:40:49.367420 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358900    2569 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 17:40:49.367420 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358905    2569 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 17:40:49.367420 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358912    2569 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 17:40:49.367420 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358916    2569 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 17:40:49.367420 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358921    2569 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 17:40:49.367420 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358927    2569 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 17:40:49.367420 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358932    2569 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 17:40:49.367420 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358936    2569 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 17:40:49.367420 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358941    2569 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 17:40:49.367420 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358946    2569 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 17:40:49.367420 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358950    2569 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 17:40:49.367420 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358956    2569 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 17:40:49.367420 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358961    2569 flags.go:64] FLAG: --system-cgroups=""
Sep 17 17:40:49.367420 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358965    2569 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 17:40:49.367420 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358975    2569 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 17:40:49.367420 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358980    2569 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 17:40:49.367420 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358984    2569 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 17:40:49.367420 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358991    2569 flags.go:64] FLAG: --tls-min-version=""
Sep 17 17:40:49.367420 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.358996    2569 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 17:40:49.367420 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.359001    2569 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 17:40:49.367420 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.359006    2569 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 17:40:49.367420 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.359011    2569 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 17:40:49.367420 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.359017    2569 flags.go:64] FLAG: --v="2"
Sep 17 17:40:49.367420 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.359024    2569 flags.go:64] FLAG: --version="false"
Sep 17 17:40:49.367420 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.359030    2569 flags.go:64] FLAG: --vmodule=""
Sep 17 17:40:49.368056 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.359037    2569 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 17:40:49.368056 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.359042    2569 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 17:40:49.368056 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359239    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:40:49.368056 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359247    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:40:49.368056 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359252    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:40:49.368056 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359257    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:40:49.368056 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359262    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:40:49.368056 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359266    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:40:49.368056 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359270    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:40:49.368056 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359277    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:40:49.368056 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359281    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:40:49.368056 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359286    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:40:49.368056 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359290    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:40:49.368056 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359295    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:40:49.368056 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359299    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:40:49.368056 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359304    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:40:49.368056 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359308    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:40:49.368056 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359313    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:40:49.368056 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359317    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:40:49.368056 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359321    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:40:49.368573 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359325    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:40:49.368573 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359332    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:40:49.368573 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359336    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:40:49.368573 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359340    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:40:49.368573 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359344    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:40:49.368573 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359348    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:40:49.368573 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359352    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:40:49.368573 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359356    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:40:49.368573 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359360    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:40:49.368573 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359365    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:40:49.368573 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359369    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:40:49.368573 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359373    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:40:49.368573 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359377    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:40:49.368573 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359383    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:40:49.368573 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359387    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:40:49.368573 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359391    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:40:49.368573 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359395    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:40:49.368573 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359400    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:40:49.368573 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359404    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:40:49.368573 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359409    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:40:49.369059 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359416    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:40:49.369059 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359423    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:40:49.369059 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359428    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:40:49.369059 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359433    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:40:49.369059 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359439    2569 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:40:49.369059 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359444    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:40:49.369059 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359449    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:40:49.369059 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359453    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:40:49.369059 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359458    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:40:49.369059 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359462    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:40:49.369059 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359467    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:40:49.369059 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359471    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:40:49.369059 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359476    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:40:49.369059 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359482    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:40:49.369059 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359486    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:40:49.369059 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359491    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:40:49.369059 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359495    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:40:49.369059 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359500    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:40:49.369059 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359504    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:40:49.369059 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359508    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:40:49.369620 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359512    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:40:49.369620 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359516    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:40:49.369620 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359520    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:40:49.369620 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359525    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:40:49.369620 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359555    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:40:49.369620 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359561    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:40:49.369620 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359567    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:40:49.369620 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359572    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:40:49.369620 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359576    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:40:49.369620 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359581    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:40:49.369620 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359586    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:40:49.369620 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359590    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:40:49.369620 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359594    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:40:49.369620 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359603    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:40:49.369620 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359608    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:40:49.369620 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359613    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:40:49.369620 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359618    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:40:49.369620 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359622    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:40:49.369620 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359627    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:40:49.370126 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359633    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:40:49.370126 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359639    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:40:49.370126 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359643    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:40:49.370126 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359648    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:40:49.370126 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359652    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:40:49.370126 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359657    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:40:49.370126 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359663    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:40:49.370126 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359667    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:40:49.370126 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359671    2569 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:40:49.370126 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359676    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:40:49.370126 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359680    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:40:49.370126 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359684    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:40:49.370126 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359688    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:40:49.370126 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359693    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:40:49.370126 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359698    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:40:49.370126 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359702    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:40:49.370126 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359706    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:40:49.370126 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359711    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:40:49.370126 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359715    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:40:49.370585 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359719    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:40:49.370585 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359724    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:40:49.370585 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.359728    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:40:49.370585 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.360169    2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:40:49.370585 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.367990    2569 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 17:40:49.370585 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.368009    2569 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 17:40:49.370585 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.368031    2569 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 17:40:49.370585 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.368037    2569 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 17:40:49.370585 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368115    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:40:49.370585 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368120    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:40:49.370585 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368125    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:40:49.370585 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368129    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:40:49.370585 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368133    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:40:49.370585 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368136    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:40:49.370938 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368139    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:40:49.370938 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368142    2569 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:40:49.370938 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368146    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:40:49.370938 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368149    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:40:49.370938 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368152    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:40:49.370938 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368155    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:40:49.370938 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368159    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:40:49.370938 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368162    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:40:49.370938 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368165    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:40:49.370938 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368169    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:40:49.370938 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368173    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:40:49.370938 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368176    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:40:49.370938 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368180    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:40:49.370938 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368183    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:40:49.370938 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368186    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:40:49.370938 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368189    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:40:49.370938 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368192    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:40:49.370938 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368195    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:40:49.370938 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368198    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:40:49.370938 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368202    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:40:49.371494 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368205    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:40:49.371494 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368207    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:40:49.371494 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368211    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:40:49.371494 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368213    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:40:49.371494 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368216    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:40:49.371494 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368219    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:40:49.371494 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368221    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:40:49.371494 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368224    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:40:49.371494 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368227    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:40:49.371494 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368230    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:40:49.371494 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368233    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:40:49.371494 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368235    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:40:49.371494 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368238    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:40:49.371494 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368241    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:40:49.371494 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368244    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:40:49.371494 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368247    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:40:49.371494 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368249    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:40:49.371494 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368252    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:40:49.371494 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368255    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:40:49.371494 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368258    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:40:49.371979 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368260    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:40:49.371979 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368263    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:40:49.371979 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368265    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:40:49.371979 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368268    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:40:49.371979 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368270    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:40:49.371979 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368273    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:40:49.371979 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368276    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:40:49.371979 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368278    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:40:49.371979 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368281    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:40:49.371979 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368284    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:40:49.371979 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368287    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:40:49.371979 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368290    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:40:49.371979 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368293    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:40:49.371979 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368296    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:40:49.371979 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368299    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:40:49.371979 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368302    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:40:49.371979 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368304    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:40:49.371979 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368307    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:40:49.371979 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368310    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:40:49.372471 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368312    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:40:49.372471 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368315    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:40:49.372471 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368318    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:40:49.372471 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368320    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:40:49.372471 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368323    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:40:49.372471 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368325    2569 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:40:49.372471 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368328    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:40:49.372471 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368331    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:40:49.372471 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368334    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:40:49.372471 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368337    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:40:49.372471 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368339    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:40:49.372471 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368342    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:40:49.372471 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368345    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:40:49.372471 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368347    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:40:49.372471 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368350    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:40:49.372471 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368352    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:40:49.372471 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368355    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:40:49.372471 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368357    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:40:49.372471 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368361    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:40:49.372471 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368365    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:40:49.372965 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368368    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:40:49.372965 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368370    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:40:49.372965 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368374    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:40:49.372965 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368377    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:40:49.372965 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368379    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:40:49.372965 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368382    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:40:49.372965 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368386    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:40:49.372965 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368389    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:40:49.372965 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368392    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:40:49.372965 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368395    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:40:49.372965 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368398    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:40:49.372965 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368400    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:40:49.372965 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368403    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:40:49.372965 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368405    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:40:49.373337 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.368411    2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:40:49.373337 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368527    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:40:49.373337 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368532    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:40:49.373337 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368536    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:40:49.373337 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368539    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:40:49.373337 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368542    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:40:49.373337 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368545    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:40:49.373337 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368549    2569 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:40:49.373337 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368552    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:40:49.373337 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368554    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:40:49.373337 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368557    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:40:49.373337 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368560    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:40:49.373337 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368563    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:40:49.373337 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368567    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:40:49.373337 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368570    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:40:49.373722 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368573    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:40:49.373722 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368575    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:40:49.373722 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368578    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:40:49.373722 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368581    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:40:49.373722 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368584    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:40:49.373722 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368586    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:40:49.373722 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368589    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:40:49.373722 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368592    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:40:49.373722 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368595    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:40:49.373722 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368598    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:40:49.373722 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368601    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:40:49.373722 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368604    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:40:49.373722 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368607    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:40:49.373722 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368609    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:40:49.373722 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368612    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:40:49.373722 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368614    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:40:49.373722 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368617    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:40:49.373722 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368620    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:40:49.373722 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368623    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:40:49.373722 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368625    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:40:49.374234 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368628    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:40:49.374234 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368630    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:40:49.374234 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368633    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:40:49.374234 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368635    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:40:49.374234 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368638    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:40:49.374234 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368640    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:40:49.374234 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368643    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:40:49.374234 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368645    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:40:49.374234 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368648    2569 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:40:49.374234 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368650    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:40:49.374234 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368653    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:40:49.374234 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368656    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:40:49.374234 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368658    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:40:49.374234 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368660    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:40:49.374234 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368663    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:40:49.374234 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368666    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:40:49.374234 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368668    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:40:49.374234 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368671    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:40:49.374234 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368673    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:40:49.374234 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368676    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:40:49.374727 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368679    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:40:49.374727 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368682    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:40:49.374727 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368686    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:40:49.374727 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368690    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:40:49.374727 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368693    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:40:49.374727 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368697    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:40:49.374727 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368699    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:40:49.374727 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368702    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:40:49.374727 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368705    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:40:49.374727 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368707    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:40:49.374727 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368710    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:40:49.374727 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368713    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:40:49.374727 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368715    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:40:49.374727 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368718    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:40:49.374727 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368720    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:40:49.374727 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368723    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:40:49.374727 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368725    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:40:49.374727 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368727    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:40:49.374727 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368730    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:40:49.375213 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368732    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:40:49.375213 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368736    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:40:49.375213 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368739    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:40:49.375213 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368742    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:40:49.375213 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368745    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:40:49.375213 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368748    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:40:49.375213 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368750    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:40:49.375213 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368753    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:40:49.375213 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368755    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:40:49.375213 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368758    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:40:49.375213 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368760    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:40:49.375213 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368763    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:40:49.375213 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368766    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:40:49.375213 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368769    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:40:49.375213 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368771    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:40:49.375213 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368774    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:40:49.375213 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368777    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:40:49.375213 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368781    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:40:49.375213 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368784    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:40:49.375678 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368786    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:40:49.375678 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368789    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:40:49.375678 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368791    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:40:49.375678 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368794    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:40:49.375678 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368796    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:40:49.375678 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368799    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:40:49.375678 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:49.368801    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:40:49.375678 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.368806    2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:40:49.375678 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.368934    2569 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 17:40:49.375678 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.370944    2569 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 17:40:49.375678 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.371692    2569 server.go:1019] "Starting client certificate rotation"
Sep 17 17:40:49.375678 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.371794    2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 17:40:49.375678 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.371836    2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 17:40:49.376038 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.373649    2569 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 17:40:49.376038 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.374792    2569 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 17:40:49.379936 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.379915    2569 log.go:25] "Validated CRI v1 runtime API"
Sep 17 17:40:49.381882 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.381864    2569 log.go:25] "Validated CRI v1 image API"
Sep 17 17:40:49.382703 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.382688    2569 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 17:40:49.386789 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.386759    2569 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 d288584a-3aa3-4d65-94b1-ea7738cdfb6d:/dev/nvme0n1p4 e7211f4e-abe7-4985-aada-65ee0bd65cc7:/dev/nvme0n1p3]
Sep 17 17:40:49.386843 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.386789    2569 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 17:40:49.392358 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.392235    2569 manager.go:216] Machine: {Timestamp:2026-09-17 17:40:49.391694011 +0000 UTC m=+1.895696627 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3109135 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b002685fff8a6cb19d27e4ab5f306 SystemUUID:ec2b0026-85ff-f8a6-cb19-d27e4ab5f306 BootID:9017f14c-f219-4881-9e8e-125dc14ea522 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f5:68:1f:2d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f5:68:1f:2d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:d2:f1:02:b2:cc:e1 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 17:40:49.392358 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.392343    2569 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 17:40:49.392504 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.392430    2569 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 17:40:49.393783 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.393756    2569 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 17:40:49.393934 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.393786    2569 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-14.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 17:40:49.394780 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.394770    2569 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 17:40:49.394813 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.394784    2569 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 17:40:49.394813 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.394797    2569 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 17:40:49.394813 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.394808    2569 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 17:40:49.394897 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.394884    2569 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 17:40:49.395988 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.395970    2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 17:40:49.398354 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.398342    2569 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 17:40:49.398487 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.398477    2569 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 17:40:49.398527 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.398507    2569 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 17:40:49.398527 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.398517    2569 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 17:40:49.398598 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.398531    2569 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 17:40:49.398598 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.398540    2569 kubelet.go:398] "Adding apiserver pod source"
Sep 17 17:40:49.398598 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.398549    2569 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 17:40:49.398947 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.398934    2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 17:40:49.398990 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.398957    2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 17:40:49.403239 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.403220    2569 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 17:40:49.405237 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.405224    2569 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 17:40:49.405282 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.405254    2569 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 17:40:49.406196 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.406178    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 17:40:49.406196 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.406196    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 17:40:49.406196 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.406202    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 17:40:49.406352 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.406207    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 17:40:49.406352 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.406216    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 17:40:49.406352 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.406222    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 17:40:49.406352 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.406238    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 17:40:49.406352 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.406246    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 17:40:49.406352 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.406253    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 17:40:49.406352 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.406259    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 17:40:49.406352 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.406268    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 17:40:49.406352 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.406273    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 17:40:49.406352 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.406300    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 17:40:49.406352 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.406308    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 17:40:49.408867 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.408847    2569 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-14.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 17:40:49.408966 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:49.408924    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 17:40:49.408966 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:49.408928    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-14.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 17:40:49.410274 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.410254    2569 server.go:1268] "Started kubelet"
Sep 17 17:40:49.410372 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.410325    2569 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 17:40:49.410372 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.410332    2569 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 17:40:49.410443 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.410421    2569 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 17:40:49.411349 ip-10-0-131-14 systemd[1]: Started Kubernetes Kubelet.
Sep 17 17:40:49.411626 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.411398    2569 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 17:40:49.411626 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.411430    2569 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 17:40:49.416311 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:49.416276    2569 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 17:40:49.416960 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.416946    2569 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 17:40:49.417024 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.416962    2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 17:40:49.417067 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:49.417035    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-14.ec2.internal\" not found"
Sep 17 17:40:49.417427 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.417412    2569 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 17:40:49.417572 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.417548    2569 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 17:40:49.417665 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.417645    2569 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 17:40:49.417790 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.417779    2569 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 17:40:49.417790 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.417787    2569 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 17:40:49.418183 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.418162    2569 factory.go:153] Registering CRI-O factory
Sep 17 17:40:49.418183 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.418180    2569 factory.go:223] Registration of the crio container factory successfully
Sep 17 17:40:49.418304 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.418232    2569 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 17:40:49.418304 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.418241    2569 factory.go:55] Registering systemd factory
Sep 17 17:40:49.418304 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.418249    2569 factory.go:223] Registration of the systemd container factory successfully
Sep 17 17:40:49.418304 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.418265    2569 factory.go:103] Registering Raw factory
Sep 17 17:40:49.418304 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.418279    2569 manager.go:1196] Started watching for new ooms in manager
Sep 17 17:40:49.418641 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.418628    2569 manager.go:349] Starting recovery of all containers
Sep 17 17:40:49.439155 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.438960    2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hkjll"
Sep 17 17:40:49.439287 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.439177    2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hkjll"
Sep 17 17:40:49.443717 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:49.443686    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 17:40:49.443937 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:49.443910    2569 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-14.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 17:40:49.444992 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:49.443856    2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-14.ec2.internal.18d62c84796388a4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-14.ec2.internal,UID:ip-10-0-131-14.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-14.ec2.internal,},FirstTimestamp:2026-09-17 17:40:49.410222244 +0000 UTC m=+1.914224927,LastTimestamp:2026-09-17 17:40:49.410222244 +0000 UTC m=+1.914224927,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-14.ec2.internal,}"
Sep 17 17:40:49.449026 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.448998    2569 manager.go:354] Recovery completed
Sep 17 17:40:49.453160 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.453145    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:40:49.455153 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.455139    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-14.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:40:49.455218 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.455167    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-14.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:40:49.455218 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.455178    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-14.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:40:49.455734 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.455720    2569 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 17:40:49.455794 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.455734    2569 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 17:40:49.455794 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.455740    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:40:49.455794 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.455751    2569 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 17:40:49.456435 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.456420    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-14.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:40:49.456486 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.456446    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-14.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:40:49.456486 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.456456    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-14.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:40:49.457185 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:49.457072    2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-14.ec2.internal.18d62c847c112161  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-14.ec2.internal,UID:ip-10-0-131-14.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-14.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-14.ec2.internal,},FirstTimestamp:2026-09-17 17:40:49.455153505 +0000 UTC m=+1.959156121,LastTimestamp:2026-09-17 17:40:49.455153505 +0000 UTC m=+1.959156121,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-14.ec2.internal,}"
Sep 17 17:40:49.457756 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.457743    2569 policy_none.go:49] "None policy: Start"
Sep 17 17:40:49.457807 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.457759    2569 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 17:40:49.457807 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.457769    2569 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 17:40:49.459749 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.459736    2569 policy_none.go:47] "Start"
Sep 17 17:40:49.512814 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.501221    2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 17:40:49.512814 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.502400    2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 17:40:49.512814 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.502448    2569 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 17:40:49.512814 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.502468    2569 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 17:40:49.512814 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:49.502509    2569 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Sep 17 17:40:49.512814 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.505035    2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:40:49.512814 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.505702    2569 manager.go:337] "Starting Device Plugin manager"
Sep 17 17:40:49.512814 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:49.505736    2569 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 17:40:49.512814 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.505746    2569 server.go:87] "Starting device plugin registration server"
Sep 17 17:40:49.512814 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.506053    2569 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 17:40:49.512814 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.506092    2569 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 17:40:49.512814 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.506181    2569 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 17:40:49.512814 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.506261    2569 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 17:40:49.512814 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.506270    2569 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 17:40:49.512814 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:49.506735    2569 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 17:40:49.512814 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:49.506782    2569 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-14.ec2.internal\" not found"
Sep 17 17:40:49.603417 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.603315    2569 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-14.ec2.internal"]
Sep 17 17:40:49.603417 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.603420    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:40:49.604984 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.604968    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-14.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:40:49.605060 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.604999    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-14.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:40:49.605060 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.605015    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-14.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:40:49.606198 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.606183    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:40:49.607108 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.607069    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-14.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:40:49.607208 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.607117    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-14.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:40:49.607208 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.607135    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-14.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:40:49.607208 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.607155    2569 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-14.ec2.internal"
Sep 17 17:40:49.607331 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.607317    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:40:49.607477 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.607463    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.ec2.internal"
Sep 17 17:40:49.607515 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.607494    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:40:49.608016 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.607999    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-14.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:40:49.608071 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.608028    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-14.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:40:49.608071 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.608038    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-14.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:40:49.608155 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.607999    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-14.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:40:49.608155 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.608118    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-14.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:40:49.608155 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.608132    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-14.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:40:49.610364 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.610348    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-14.ec2.internal"
Sep 17 17:40:49.610450 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.610372    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:40:49.611002 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.610982    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-14.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:40:49.611113 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.611006    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-14.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:40:49.611113 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.611016    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-14.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:40:49.615601 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.615588    2569 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-14.ec2.internal"
Sep 17 17:40:49.615658 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:49.615608    2569 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-14.ec2.internal\": node \"ip-10-0-131-14.ec2.internal\" not found"
Sep 17 17:40:49.618393 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.618376    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/273a3c53d0c1db5ff2ab130636ac4e73-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-14.ec2.internal\" (UID: \"273a3c53d0c1db5ff2ab130636ac4e73\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.ec2.internal"
Sep 17 17:40:49.618468 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.618399    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/273a3c53d0c1db5ff2ab130636ac4e73-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-14.ec2.internal\" (UID: \"273a3c53d0c1db5ff2ab130636ac4e73\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.ec2.internal"
Sep 17 17:40:49.618468 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.618414    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/73796eafa3737317490091fb30921865-config\") pod \"kube-apiserver-proxy-ip-10-0-131-14.ec2.internal\" (UID: \"73796eafa3737317490091fb30921865\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-14.ec2.internal"
Sep 17 17:40:49.625655 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:49.625639    2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-14.ec2.internal\" not found" node="ip-10-0-131-14.ec2.internal"
Sep 17 17:40:49.629114 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:49.629098    2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-14.ec2.internal\" not found" node="ip-10-0-131-14.ec2.internal"
Sep 17 17:40:49.636292 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:49.636275    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-14.ec2.internal\" not found"
Sep 17 17:40:49.719392 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.719356    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/73796eafa3737317490091fb30921865-config\") pod \"kube-apiserver-proxy-ip-10-0-131-14.ec2.internal\" (UID: \"73796eafa3737317490091fb30921865\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-14.ec2.internal"
Sep 17 17:40:49.719392 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.719389    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/273a3c53d0c1db5ff2ab130636ac4e73-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-14.ec2.internal\" (UID: \"273a3c53d0c1db5ff2ab130636ac4e73\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.ec2.internal"
Sep 17 17:40:49.719609 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.719406    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/273a3c53d0c1db5ff2ab130636ac4e73-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-14.ec2.internal\" (UID: \"273a3c53d0c1db5ff2ab130636ac4e73\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.ec2.internal"
Sep 17 17:40:49.719609 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.719449    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/273a3c53d0c1db5ff2ab130636ac4e73-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-14.ec2.internal\" (UID: \"273a3c53d0c1db5ff2ab130636ac4e73\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.ec2.internal"
Sep 17 17:40:49.719609 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.719460    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/73796eafa3737317490091fb30921865-config\") pod \"kube-apiserver-proxy-ip-10-0-131-14.ec2.internal\" (UID: \"73796eafa3737317490091fb30921865\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-14.ec2.internal"
Sep 17 17:40:49.719609 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.719481    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/273a3c53d0c1db5ff2ab130636ac4e73-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-14.ec2.internal\" (UID: \"273a3c53d0c1db5ff2ab130636ac4e73\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.ec2.internal"
Sep 17 17:40:49.736902 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:49.736873    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-14.ec2.internal\" not found"
Sep 17 17:40:49.837642 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:49.837613    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-14.ec2.internal\" not found"
Sep 17 17:40:49.929489 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.929426    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.ec2.internal"
Sep 17 17:40:49.932587 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:49.932568    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-14.ec2.internal"
Sep 17 17:40:49.937729 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:49.937711    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-14.ec2.internal\" not found"
Sep 17 17:40:50.038251 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:50.038208    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-14.ec2.internal\" not found"
Sep 17 17:40:50.138816 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:50.138785    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-14.ec2.internal\" not found"
Sep 17 17:40:50.239351 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:50.239268    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-14.ec2.internal\" not found"
Sep 17 17:40:50.339889 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:50.339855    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-14.ec2.internal\" not found"
Sep 17 17:40:50.372007 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:50.371985    2569 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 17:40:50.372660 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:50.372154    2569 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 17:40:50.440876 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:50.440849    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-14.ec2.internal\" not found"
Sep 17 17:40:50.446009 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:50.445979    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 17:35:49 +0000 UTC" deadline="2028-03-05 23:39:42.357483371 +0000 UTC"
Sep 17 17:40:50.446009 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:50.446008    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12845h58m51.911478746s"
Sep 17 17:40:50.541958 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:50.541878    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-14.ec2.internal\" not found"
Sep 17 17:40:50.600526 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:50.600494    2569 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:40:50.603808 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:50.603791    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:40:50.642043 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:50.642005    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-14.ec2.internal\" not found"
Sep 17 17:40:50.742516 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:50.742478    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-14.ec2.internal\" not found"
Sep 17 17:40:50.787302 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:50.787267    2569 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:40:50.829167 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:50.829070    2569 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:40:50.917706 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:50.917676    2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.ec2.internal"
Sep 17 17:40:50.928770 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:50.928749    2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 17:40:50.929552 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:50.929534    2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-14.ec2.internal"
Sep 17 17:40:50.936825 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:50.936810    2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 17:40:51.312541 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.312288    2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:40:51.399870 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.399832    2569 apiserver.go:52] "Watching apiserver"
Sep 17 17:40:51.406006 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.405979    2569 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 17:40:51.406435 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.406410    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9psk","openshift-dns/node-resolver-qn9k7","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.ec2.internal","openshift-multus/multus-fjfrk","openshift-multus/network-metrics-daemon-rzvhs","openshift-network-operator/iptables-alerter-hfw2c","kube-system/global-pull-secret-syncer-cfb52","kube-system/kube-apiserver-proxy-ip-10-0-131-14.ec2.internal","openshift-cluster-node-tuning-operator/tuned-csv2f","openshift-image-registry/node-ca-nbmpx","openshift-multus/multus-additional-cni-plugins-mwh5v","openshift-network-diagnostics/network-check-target-ngfbr","openshift-ovn-kubernetes/ovnkube-node-k4mqn","kube-system/konnectivity-agent-qn6lp"]
Sep 17 17:40:51.408921 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.408896    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cfb52"
Sep 17 17:40:51.409037 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:51.408984    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cfb52" podUID="3f18fbf0-4d7d-4621-9260-199ef7cc0e0d"
Sep 17 17:40:51.411261 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.411234    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn"
Sep 17 17:40:51.413540 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.413515    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qn6lp"
Sep 17 17:40:51.415025 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.414565    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 17:40:51.415025 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.414700    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 17:40:51.415025 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.414763    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 17:40:51.415025 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.414867    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-45qgh\""
Sep 17 17:40:51.415025 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.414924    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 17:40:51.415340 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.415129    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 17:40:51.415340 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.415135    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 17:40:51.415914 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.415892    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fjfrk"
Sep 17 17:40:51.416280 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.416258    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 17:40:51.416363 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.416284    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-cd75v\""
Sep 17 17:40:51.416363 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.416326    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 17:40:51.417979 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.417959    2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 17:40:51.418636 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.418535    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rzvhs"
Sep 17 17:40:51.418636 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:51.418600    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rzvhs" podUID="caf785f7-c156-4b6f-8c82-4c1efe5cd710"
Sep 17 17:40:51.421139 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.421121    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hfw2c"
Sep 17 17:40:51.423358 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.423014    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-csv2f"
Sep 17 17:40:51.424738 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.424561    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 17:40:51.426168 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.425411    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9psk"
Sep 17 17:40:51.427400 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.426889    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 17:40:51.427400 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.426997    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 17:40:51.427400 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.427328    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-9t6xf\""
Sep 17 17:40:51.427581 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.427511    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 17:40:51.428689 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.427868    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f847c3c6-f2ab-43a0-b467-27db788f5b9f-host-var-lib-cni-multus\") pod \"multus-fjfrk\" (UID: \"f847c3c6-f2ab-43a0-b467-27db788f5b9f\") " pod="openshift-multus/multus-fjfrk"
Sep 17 17:40:51.428689 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.427917    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f847c3c6-f2ab-43a0-b467-27db788f5b9f-etc-kubernetes\") pod \"multus-fjfrk\" (UID: \"f847c3c6-f2ab-43a0-b467-27db788f5b9f\") " pod="openshift-multus/multus-fjfrk"
Sep 17 17:40:51.428689 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.427946    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff-host-slash\") pod \"ovnkube-node-k4mqn\" (UID: \"6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn"
Sep 17 17:40:51.428689 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.427974    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff-log-socket\") pod \"ovnkube-node-k4mqn\" (UID: \"6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn"
Sep 17 17:40:51.428689 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.428003    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff-env-overrides\") pod \"ovnkube-node-k4mqn\" (UID: \"6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn"
Sep 17 17:40:51.428689 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.428030    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff-ovnkube-script-lib\") pod \"ovnkube-node-k4mqn\" (UID: \"6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn"
Sep 17 17:40:51.428689 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.428057    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b86gx\" (UniqueName: \"kubernetes.io/projected/6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff-kube-api-access-b86gx\") pod \"ovnkube-node-k4mqn\" (UID: \"6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn"
Sep 17 17:40:51.428689 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.428098    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmskf\" (UniqueName: \"kubernetes.io/projected/bc71d7ac-eae9-4ef5-a2f8-f0292bdc51b6-kube-api-access-lmskf\") pod \"iptables-alerter-hfw2c\" (UID: \"bc71d7ac-eae9-4ef5-a2f8-f0292bdc51b6\") " pod="openshift-network-operator/iptables-alerter-hfw2c"
Sep 17 17:40:51.428689 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.428130    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3f18fbf0-4d7d-4621-9260-199ef7cc0e0d-kubelet-config\") pod \"global-pull-secret-syncer-cfb52\" (UID: \"3f18fbf0-4d7d-4621-9260-199ef7cc0e0d\") " pod="kube-system/global-pull-secret-syncer-cfb52"
Sep 17 17:40:51.428689 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.428157    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff-host-kubelet\") pod \"ovnkube-node-k4mqn\" (UID: \"6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn"
Sep 17 17:40:51.428689 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.428185    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff-host-cni-bin\") pod \"ovnkube-node-k4mqn\" (UID: \"6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn"
Sep 17 17:40:51.428689 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.428292    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f847c3c6-f2ab-43a0-b467-27db788f5b9f-cni-binary-copy\") pod \"multus-fjfrk\" (UID: \"f847c3c6-f2ab-43a0-b467-27db788f5b9f\") " pod="openshift-multus/multus-fjfrk"
Sep 17 17:40:51.428689 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.428321    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3f18fbf0-4d7d-4621-9260-199ef7cc0e0d-dbus\") pod \"global-pull-secret-syncer-cfb52\" (UID: \"3f18fbf0-4d7d-4621-9260-199ef7cc0e0d\") " pod="kube-system/global-pull-secret-syncer-cfb52"
Sep 17 17:40:51.428689 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.428349    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f847c3c6-f2ab-43a0-b467-27db788f5b9f-system-cni-dir\") pod \"multus-fjfrk\" (UID: \"f847c3c6-f2ab-43a0-b467-27db788f5b9f\") " pod="openshift-multus/multus-fjfrk"
Sep 17 17:40:51.428689 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.428381    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f847c3c6-f2ab-43a0-b467-27db788f5b9f-multus-socket-dir-parent\") pod \"multus-fjfrk\" (UID: \"f847c3c6-f2ab-43a0-b467-27db788f5b9f\") " pod="openshift-multus/multus-fjfrk"
Sep 17 17:40:51.428689 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.428410    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f847c3c6-f2ab-43a0-b467-27db788f5b9f-host-var-lib-kubelet\") pod \"multus-fjfrk\" (UID: \"f847c3c6-f2ab-43a0-b467-27db788f5b9f\") " pod="openshift-multus/multus-fjfrk"
Sep 17 17:40:51.428689 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.428477    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f847c3c6-f2ab-43a0-b467-27db788f5b9f-multus-conf-dir\") pod \"multus-fjfrk\" (UID: \"f847c3c6-f2ab-43a0-b467-27db788f5b9f\") " pod="openshift-multus/multus-fjfrk"
Sep 17 17:40:51.429505 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.428537    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f847c3c6-f2ab-43a0-b467-27db788f5b9f-multus-daemon-config\") pod \"multus-fjfrk\" (UID: \"f847c3c6-f2ab-43a0-b467-27db788f5b9f\") " pod="openshift-multus/multus-fjfrk"
Sep 17 17:40:51.429505 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.428694    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caf785f7-c156-4b6f-8c82-4c1efe5cd710-metrics-certs\") pod \"network-metrics-daemon-rzvhs\" (UID: \"caf785f7-c156-4b6f-8c82-4c1efe5cd710\") " pod="openshift-multus/network-metrics-daemon-rzvhs"
Sep 17 17:40:51.429505 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.428756    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff-var-lib-openvswitch\") pod \"ovnkube-node-k4mqn\" (UID: \"6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn"
Sep 17 17:40:51.429505 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.428831    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f847c3c6-f2ab-43a0-b467-27db788f5b9f-host-run-multus-certs\") pod \"multus-fjfrk\" (UID: \"f847c3c6-f2ab-43a0-b467-27db788f5b9f\") " pod="openshift-multus/multus-fjfrk"
Sep 17 17:40:51.429505 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.428862    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb5mg\" (UniqueName: \"kubernetes.io/projected/f847c3c6-f2ab-43a0-b467-27db788f5b9f-kube-api-access-hb5mg\") pod \"multus-fjfrk\" (UID: \"f847c3c6-f2ab-43a0-b467-27db788f5b9f\") " pod="openshift-multus/multus-fjfrk"
Sep 17 17:40:51.429986 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.429959    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bc71d7ac-eae9-4ef5-a2f8-f0292bdc51b6-iptables-alerter-script\") pod \"iptables-alerter-hfw2c\" (UID: \"bc71d7ac-eae9-4ef5-a2f8-f0292bdc51b6\") " pod="openshift-network-operator/iptables-alerter-hfw2c"
Sep 17 17:40:51.431480 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.430473    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qn9k7"
Sep 17 17:40:51.431480 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.430992    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:40:51.431480 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.431008    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff-host-run-ovn-kubernetes\") pod \"ovnkube-node-k4mqn\" (UID: \"6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn"
Sep 17 17:40:51.431480 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.431047    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/372b5295-ddc7-4999-9b48-c6cfb84d8e83-konnectivity-ca\") pod \"konnectivity-agent-qn6lp\" (UID: \"372b5295-ddc7-4999-9b48-c6cfb84d8e83\") " pod="kube-system/konnectivity-agent-qn6lp"
Sep 17 17:40:51.431480 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.431112    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f847c3c6-f2ab-43a0-b467-27db788f5b9f-multus-cni-dir\") pod \"multus-fjfrk\" (UID: \"f847c3c6-f2ab-43a0-b467-27db788f5b9f\") " pod="openshift-multus/multus-fjfrk"
Sep 17 17:40:51.431480 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.431145    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f847c3c6-f2ab-43a0-b467-27db788f5b9f-hostroot\") pod \"multus-fjfrk\" (UID: \"f847c3c6-f2ab-43a0-b467-27db788f5b9f\") " pod="openshift-multus/multus-fjfrk"
Sep 17 17:40:51.431480 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.431173    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phzkj\" (UniqueName: \"kubernetes.io/projected/caf785f7-c156-4b6f-8c82-4c1efe5cd710-kube-api-access-phzkj\") pod \"network-metrics-daemon-rzvhs\" (UID: \"caf785f7-c156-4b6f-8c82-4c1efe5cd710\") " pod="openshift-multus/network-metrics-daemon-rzvhs"
Sep 17 17:40:51.431480 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.431202    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3f18fbf0-4d7d-4621-9260-199ef7cc0e0d-original-pull-secret\") pod \"global-pull-secret-syncer-cfb52\" (UID: \"3f18fbf0-4d7d-4621-9260-199ef7cc0e0d\") " pod="kube-system/global-pull-secret-syncer-cfb52"
Sep 17 17:40:51.431480 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.431262    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff-systemd-units\") pod \"ovnkube-node-k4mqn\" (UID: \"6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn"
Sep 17 17:40:51.431480 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.431290    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff-etc-openvswitch\") pod \"ovnkube-node-k4mqn\" (UID: \"6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn"
Sep 17 17:40:51.431480 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.431300    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:40:51.431480 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.431320    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff-node-log\") pod \"ovnkube-node-k4mqn\" (UID: \"6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn"
Sep 17 17:40:51.431480 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.431350    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k4mqn\" (UID: \"6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn"
Sep 17 17:40:51.431480 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.431374    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff-ovnkube-config\") pod \"ovnkube-node-k4mqn\" (UID: \"6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn"
Sep 17 17:40:51.431480 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.431401    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/372b5295-ddc7-4999-9b48-c6cfb84d8e83-agent-certs\") pod \"konnectivity-agent-qn6lp\" (UID: \"372b5295-ddc7-4999-9b48-c6cfb84d8e83\") " pod="kube-system/konnectivity-agent-qn6lp"
Sep 17 17:40:51.431480 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.431467    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 17:40:51.431480 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.431469    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f847c3c6-f2ab-43a0-b467-27db788f5b9f-cnibin\") pod \"multus-fjfrk\" (UID: \"f847c3c6-f2ab-43a0-b467-27db788f5b9f\") " pod="openshift-multus/multus-fjfrk"
Sep 17 17:40:51.432301 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.431578    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff-host-cni-netd\") pod \"ovnkube-node-k4mqn\" (UID: \"6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn"
Sep 17 17:40:51.432301 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.431629    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f847c3c6-f2ab-43a0-b467-27db788f5b9f-host-run-k8s-cni-cncf-io\") pod \"multus-fjfrk\" (UID: \"f847c3c6-f2ab-43a0-b467-27db788f5b9f\") " pod="openshift-multus/multus-fjfrk"
Sep 17 17:40:51.432301 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.431694    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f847c3c6-f2ab-43a0-b467-27db788f5b9f-host-var-lib-cni-bin\") pod \"multus-fjfrk\" (UID: \"f847c3c6-f2ab-43a0-b467-27db788f5b9f\") " pod="openshift-multus/multus-fjfrk"
Sep 17 17:40:51.432301 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.431728    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bc71d7ac-eae9-4ef5-a2f8-f0292bdc51b6-host-slash\") pod \"iptables-alerter-hfw2c\" (UID: \"bc71d7ac-eae9-4ef5-a2f8-f0292bdc51b6\") " pod="openshift-network-operator/iptables-alerter-hfw2c"
Sep 17 17:40:51.432301 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.431755    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tt2nm\""
Sep 17 17:40:51.432301 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.431760    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff-host-run-netns\") pod \"ovnkube-node-k4mqn\" (UID: \"6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn"
Sep 17 17:40:51.432301 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.431911    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff-run-systemd\") pod \"ovnkube-node-k4mqn\" (UID: \"6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn"
Sep 17 17:40:51.432301 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.431937    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 17:40:51.432301 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.431941    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff-run-openvswitch\") pod \"ovnkube-node-k4mqn\" (UID: \"6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn"
Sep 17 17:40:51.432301 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.431994    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff-run-ovn\") pod \"ovnkube-node-k4mqn\" (UID: \"6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn"
Sep 17 17:40:51.432301 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.432056    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff-ovn-node-metrics-cert\") pod \"ovnkube-node-k4mqn\" (UID: \"6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn"
Sep 17 17:40:51.432301 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.432103    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f847c3c6-f2ab-43a0-b467-27db788f5b9f-host-run-netns\") pod \"multus-fjfrk\" (UID: \"f847c3c6-f2ab-43a0-b467-27db788f5b9f\") " pod="openshift-multus/multus-fjfrk"
Sep 17 17:40:51.432301 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.432122    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 17:40:51.432301 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.432260    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vc5qt\""
Sep 17 17:40:51.432301 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.432305    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 17:40:51.433148 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.432261    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 17:40:51.433191 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.433181    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 17:40:51.433373 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.433353    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-b9l7v\""
Sep 17 17:40:51.433628 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.433615    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nbmpx"
Sep 17 17:40:51.434028 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.434010    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 17:40:51.434147 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.434060    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-f9rw7\""
Sep 17 17:40:51.434287 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.434272    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 17:40:51.436536 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.436161    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mwh5v"
Sep 17 17:40:51.436536 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.436337    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 17:40:51.436714 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.436625    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 17:40:51.436714 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.436641    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 17:40:51.436714 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.436644    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-sxwd6\""
Sep 17 17:40:51.437948 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.437916    2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 17:40:51.438773 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.438751    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ngfbr"
Sep 17 17:40:51.438871 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:51.438816    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ngfbr" podUID="bf919600-b3d9-49db-b561-1e02cb809474"
Sep 17 17:40:51.440098 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.439149    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bwc8b\""
Sep 17 17:40:51.440098 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.439154    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 17:40:51.440098 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.439435    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 17:40:51.457177 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.457151    2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-4wvl4"
Sep 17 17:40:51.468833 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.468805    2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-4wvl4"
Sep 17 17:40:51.507636 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.507577    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-14.ec2.internal" event={"ID":"73796eafa3737317490091fb30921865","Type":"ContainerStarted","Data":"f576559c7c6be30238ae94740045614972b76e90465ca6639c6a1a85d828572c"}
Sep 17 17:40:51.509252 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.509223    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.ec2.internal" event={"ID":"273a3c53d0c1db5ff2ab130636ac4e73","Type":"ContainerStarted","Data":"6b7d71d09f8de34611b07d46c135c8cdff16d868b51774ee345f46edd2956647"}
Sep 17 17:40:51.519320 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.519289    2569 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 17:40:51.533059 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.533013    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f847c3c6-f2ab-43a0-b467-27db788f5b9f-cni-binary-copy\") pod \"multus-fjfrk\" (UID: \"f847c3c6-f2ab-43a0-b467-27db788f5b9f\") " pod="openshift-multus/multus-fjfrk"
Sep 17 17:40:51.533288 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.533072    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsfzs\" (UniqueName: \"kubernetes.io/projected/911990d2-9984-4b04-979c-1380a8bdb7a2-kube-api-access-hsfzs\") pod \"multus-additional-cni-plugins-mwh5v\" (UID: \"911990d2-9984-4b04-979c-1380a8bdb7a2\") " pod="openshift-multus/multus-additional-cni-plugins-mwh5v"
Sep 17 17:40:51.533288 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.533119    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3809c4cd-e5d9-4647-95f8-1abbeefd438e-socket-dir\") pod \"aws-ebs-csi-driver-node-b9psk\" (UID: \"3809c4cd-e5d9-4647-95f8-1abbeefd438e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9psk"
Sep 17 17:40:51.533288 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.533148    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3809c4cd-e5d9-4647-95f8-1abbeefd438e-etc-selinux\") pod \"aws-ebs-csi-driver-node-b9psk\" (UID: \"3809c4cd-e5d9-4647-95f8-1abbeefd438e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9psk"
Sep 17 17:40:51.533288 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.533170    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d40ba619-04c7-482d-b46f-ec561e79987c-etc-kubernetes\") pod \"tuned-csv2f\" (UID: \"d40ba619-04c7-482d-b46f-ec561e79987c\") " pod="openshift-cluster-node-tuning-operator/tuned-csv2f"
Sep 17 17:40:51.533288 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.533196    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f847c3c6-f2ab-43a0-b467-27db788f5b9f-system-cni-dir\") pod \"multus-fjfrk\" (UID: \"f847c3c6-f2ab-43a0-b467-27db788f5b9f\") " pod="openshift-multus/multus-fjfrk"
Sep 17 17:40:51.533288 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.533220    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f847c3c6-f2ab-43a0-b467-27db788f5b9f-multus-socket-dir-parent\") pod \"multus-fjfrk\" (UID: \"f847c3c6-f2ab-43a0-b467-27db788f5b9f\") " pod="openshift-multus/multus-fjfrk"
Sep 17 17:40:51.533288 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.533244    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f847c3c6-f2ab-43a0-b467-27db788f5b9f-host-var-lib-kubelet\") pod \"multus-fjfrk\" (UID: \"f847c3c6-f2ab-43a0-b467-27db788f5b9f\") " pod="openshift-multus/multus-fjfrk"
Sep 17 17:40:51.533288 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.533268    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f847c3c6-f2ab-43a0-b467-27db788f5b9f-multus-daemon-config\") pod \"multus-fjfrk\" (UID: \"f847c3c6-f2ab-43a0-b467-27db788f5b9f\") " pod="openshift-multus/multus-fjfrk"
Sep 17 17:40:51.533288 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.533290    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d40ba619-04c7-482d-b46f-ec561e79987c-etc-sysconfig\") pod \"tuned-csv2f\" (UID: \"d40ba619-04c7-482d-b46f-ec561e79987c\") " pod="openshift-cluster-node-tuning-operator/tuned-csv2f"
Sep 17 17:40:51.533660 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.533315    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb5mg\" (UniqueName: \"kubernetes.io/projected/f847c3c6-f2ab-43a0-b467-27db788f5b9f-kube-api-access-hb5mg\") pod \"multus-fjfrk\" (UID: \"f847c3c6-f2ab-43a0-b467-27db788f5b9f\") " pod="openshift-multus/multus-fjfrk"
Sep 17 17:40:51.533660 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.533349    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d40ba619-04c7-482d-b46f-ec561e79987c-etc-sysctl-d\") pod \"tuned-csv2f\" (UID: \"d40ba619-04c7-482d-b46f-ec561e79987c\") " pod="openshift-cluster-node-tuning-operator/tuned-csv2f"
Sep 17 17:40:51.533660 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.533374    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff-host-run-ovn-kubernetes\") pod \"ovnkube-node-k4mqn\" (UID: \"6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn"
Sep 17 17:40:51.533660 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.533403    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f847c3c6-f2ab-43a0-b467-27db788f5b9f-multus-cni-dir\") pod \"multus-fjfrk\" (UID: \"f847c3c6-f2ab-43a0-b467-27db788f5b9f\") " pod="openshift-multus/multus-fjfrk"
Sep 17 17:40:51.533660 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.533427    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f847c3c6-f2ab-43a0-b467-27db788f5b9f-hostroot\") pod \"multus-fjfrk\" (UID: \"f847c3c6-f2ab-43a0-b467-27db788f5b9f\") " pod="openshift-multus/multus-fjfrk"
Sep 17 17:40:51.533660 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.533463    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d40ba619-04c7-482d-b46f-ec561e79987c-etc-modprobe-d\") pod \"tuned-csv2f\" (UID: \"d40ba619-04c7-482d-b46f-ec561e79987c\") " pod="openshift-cluster-node-tuning-operator/tuned-csv2f"
Sep 17 17:40:51.533660 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.533488    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d40ba619-04c7-482d-b46f-ec561e79987c-run\") pod \"tuned-csv2f\" (UID: \"d40ba619-04c7-482d-b46f-ec561e79987c\") " pod="openshift-cluster-node-tuning-operator/tuned-csv2f"
Sep 17 17:40:51.533660 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.533517    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff-systemd-units\") pod \"ovnkube-node-k4mqn\" (UID: \"6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn"
Sep 17 17:40:51.533660 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.533544    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k4mqn\" (UID: \"6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn"
Sep 17 17:40:51.533660 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.533569    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/372b5295-ddc7-4999-9b48-c6cfb84d8e83-agent-certs\") pod \"konnectivity-agent-qn6lp\" (UID: \"372b5295-ddc7-4999-9b48-c6cfb84d8e83\") " pod="kube-system/konnectivity-agent-qn6lp"
Sep 17 17:40:51.533660 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.533618    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvc8l\" (UniqueName: \"kubernetes.io/projected/bf919600-b3d9-49db-b561-1e02cb809474-kube-api-access-qvc8l\") pod \"network-check-target-ngfbr\" (UID: \"bf919600-b3d9-49db-b561-1e02cb809474\") " pod="openshift-network-diagnostics/network-check-target-ngfbr"
Sep 17 17:40:51.533660 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.533642    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3809c4cd-e5d9-4647-95f8-1abbeefd438e-sys-fs\") pod \"aws-ebs-csi-driver-node-b9psk\" (UID: \"3809c4cd-e5d9-4647-95f8-1abbeefd438e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9psk"
Sep 17 17:40:51.534263 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.533670    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42dd1d14-0348-40e6-824e-90b1763291f1-tmp-dir\") pod \"node-resolver-qn9k7\" (UID: \"42dd1d14-0348-40e6-824e-90b1763291f1\") " pod="openshift-dns/node-resolver-qn9k7"
Sep 17 17:40:51.534263 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.533694    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d40ba619-04c7-482d-b46f-ec561e79987c-etc-sysctl-conf\") pod \"tuned-csv2f\" (UID: \"d40ba619-04c7-482d-b46f-ec561e79987c\") " pod="openshift-cluster-node-tuning-operator/tuned-csv2f"
Sep 17 17:40:51.534263 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.533708    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f847c3c6-f2ab-43a0-b467-27db788f5b9f-cni-binary-copy\") pod \"multus-fjfrk\" (UID: \"f847c3c6-f2ab-43a0-b467-27db788f5b9f\") " pod="openshift-multus/multus-fjfrk"
Sep 17 17:40:51.534263 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.533723    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f847c3c6-f2ab-43a0-b467-27db788f5b9f-host-run-k8s-cni-cncf-io\") pod \"multus-fjfrk\" (UID: \"f847c3c6-f2ab-43a0-b467-27db788f5b9f\") " pod="openshift-multus/multus-fjfrk"
Sep 17 17:40:51.534263 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.533764    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bc71d7ac-eae9-4ef5-a2f8-f0292bdc51b6-host-slash\") pod \"iptables-alerter-hfw2c\" (UID: \"bc71d7ac-eae9-4ef5-a2f8-f0292bdc51b6\") " pod="openshift-network-operator/iptables-alerter-hfw2c"
Sep 17 17:40:51.534263 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.533797    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff-host-run-ovn-kubernetes\") pod \"ovnkube-node-k4mqn\" (UID: \"6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn"
Sep 17 17:40:51.534263 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.533803    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr2bs\" (UniqueName: \"kubernetes.io/projected/42dd1d14-0348-40e6-824e-90b1763291f1-kube-api-access-fr2bs\") pod \"node-resolver-qn9k7\" (UID: \"42dd1d14-0348-40e6-824e-90b1763291f1\") " pod="openshift-dns/node-resolver-qn9k7"
Sep 17 17:40:51.534263 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.533830    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff-host-run-netns\") pod \"ovnkube-node-k4mqn\" (UID: \"6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn"
Sep 17 17:40:51.534263 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.533956    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f847c3c6-f2ab-43a0-b467-27db788f5b9f-multus-socket-dir-parent\") pod \"multus-fjfrk\" (UID: \"f847c3c6-f2ab-43a0-b467-27db788f5b9f\") " pod="openshift-multus/multus-fjfrk"
Sep 17 17:40:51.534263 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.533990    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f847c3c6-f2ab-43a0-b467-27db788f5b9f-host-var-lib-kubelet\") pod \"multus-fjfrk\" (UID: \"f847c3c6-f2ab-43a0-b467-27db788f5b9f\") " pod="openshift-multus/multus-fjfrk"
Sep 17 17:40:51.534263 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.534020    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff-run-openvswitch\") pod \"ovnkube-node-k4mqn\" (UID: \"6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn"
Sep 17 17:40:51.534263 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.534042    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff-run-ovn\") pod \"ovnkube-node-k4mqn\" (UID: \"6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn"
Sep 17 17:40:51.534263 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.534057    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f847c3c6-f2ab-43a0-b467-27db788f5b9f-host-var-lib-cni-multus\") pod \"multus-fjfrk\" (UID: \"f847c3c6-f2ab-43a0-b467-27db788f5b9f\") " pod="openshift-multus/multus-fjfrk"
Sep 17 17:40:51.534263 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.534093    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1b398422-1bbe-4f75-a7e6-a641cce3f5e8-host\") pod \"node-ca-nbmpx\" (UID: \"1b398422-1bbe-4f75-a7e6-a641cce3f5e8\") " pod="openshift-image-registry/node-ca-nbmpx"
Sep 17 17:40:51.534263 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.534114    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f847c3c6-f2ab-43a0-b467-27db788f5b9f-hostroot\") pod \"multus-fjfrk\" (UID: \"f847c3c6-f2ab-43a0-b467-27db788f5b9f\") " pod="openshift-multus/multus-fjfrk"
Sep 17 17:40:51.534263 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.534131    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d40ba619-04c7-482d-b46f-ec561e79987c-host\") pod \"tuned-csv2f\" (UID: \"d40ba619-04c7-482d-b46f-ec561e79987c\") " pod="openshift-cluster-node-tuning-operator/tuned-csv2f"
Sep 17 17:40:51.534263 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.534153    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff-host-slash\") pod \"ovnkube-node-k4mqn\" (UID: \"6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn"
Sep 17 17:40:51.534263 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.534176    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff-log-socket\") pod \"ovnkube-node-k4mqn\" (UID: \"6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn"
Sep 17 17:40:51.535034 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.534194    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff-env-overrides\") pod \"ovnkube-node-k4mqn\" (UID: \"6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn"
Sep 17 17:40:51.535034 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.534203    2569 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 17:40:51.535034 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.534224    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b86gx\" (UniqueName: \"kubernetes.io/projected/6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff-kube-api-access-b86gx\") pod \"ovnkube-node-k4mqn\" (UID: \"6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn"
Sep 17 17:40:51.535034 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.534287    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmskf\" (UniqueName: \"kubernetes.io/projected/bc71d7ac-eae9-4ef5-a2f8-f0292bdc51b6-kube-api-access-lmskf\") pod \"iptables-alerter-hfw2c\" (UID: \"bc71d7ac-eae9-4ef5-a2f8-f0292bdc51b6\") " pod="openshift-network-operator/iptables-alerter-hfw2c"
Sep 17 17:40:51.535034 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.534317    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/911990d2-9984-4b04-979c-1380a8bdb7a2-cnibin\") pod \"multus-additional-cni-plugins-mwh5v\" (UID: \"911990d2-9984-4b04-979c-1380a8bdb7a2\") " pod="openshift-multus/multus-additional-cni-plugins-mwh5v"
Sep 17 17:40:51.535034 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.534347    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/911990d2-9984-4b04-979c-1380a8bdb7a2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mwh5v\" (UID: \"911990d2-9984-4b04-979c-1380a8bdb7a2\") " pod="openshift-multus/multus-additional-cni-plugins-mwh5v"
Sep 17 17:40:51.535034 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.534376    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3f18fbf0-4d7d-4621-9260-199ef7cc0e0d-kubelet-config\") pod \"global-pull-secret-syncer-cfb52\" (UID: \"3f18fbf0-4d7d-4621-9260-199ef7cc0e0d\") " pod="kube-system/global-pull-secret-syncer-cfb52"
Sep 17 17:40:51.535034 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.534402    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff-host-kubelet\") pod \"ovnkube-node-k4mqn\" (UID: \"6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn"
Sep 17 17:40:51.535034 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.534427    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f847c3c6-f2ab-43a0-b467-27db788f5b9f-system-cni-dir\") pod \"multus-fjfrk\" (UID: \"f847c3c6-f2ab-43a0-b467-27db788f5b9f\") " pod="openshift-multus/multus-fjfrk"
Sep 17 17:40:51.535034 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.534541    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f847c3c6-f2ab-43a0-b467-27db788f5b9f-multus-daemon-config\") pod \"multus-fjfrk\" (UID: \"f847c3c6-f2ab-43a0-b467-27db788f5b9f\") " pod="openshift-multus/multus-fjfrk"
Sep 17 17:40:51.535034 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.534627    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff-host-cni-bin\") pod \"ovnkube-node-k4mqn\" (UID: \"6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn"
Sep 17 17:40:51.535034 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.534654    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3f18fbf0-4d7d-4621-9260-199ef7cc0e0d-dbus\") pod \"global-pull-secret-syncer-cfb52\" (UID: \"3f18fbf0-4d7d-4621-9260-199ef7cc0e0d\") " pod="kube-system/global-pull-secret-syncer-cfb52"
Sep 17 17:40:51.535034 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.534677    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f847c3c6-f2ab-43a0-b467-27db788f5b9f-multus-conf-dir\") pod \"multus-fjfrk\" (UID: \"f847c3c6-f2ab-43a0-b467-27db788f5b9f\") " pod="openshift-multus/multus-fjfrk"
Sep 17 17:40:51.535034 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.534699    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caf785f7-c156-4b6f-8c82-4c1efe5cd710-metrics-certs\") pod \"network-metrics-daemon-rzvhs\" (UID: \"caf785f7-c156-4b6f-8c82-4c1efe5cd710\") " pod="openshift-multus/network-metrics-daemon-rzvhs"
Sep 17 17:40:51.535034 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.534723    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/911990d2-9984-4b04-979c-1380a8bdb7a2-system-cni-dir\") pod \"multus-additional-cni-plugins-mwh5v\" (UID: \"911990d2-9984-4b04-979c-1380a8bdb7a2\") " pod="openshift-multus/multus-additional-cni-plugins-mwh5v"
Sep 17 17:40:51.535034 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.534745    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3809c4cd-e5d9-4647-95f8-1abbeefd438e-device-dir\") pod \"aws-ebs-csi-driver-node-b9psk\" (UID: \"3809c4cd-e5d9-4647-95f8-1abbeefd438e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9psk"
Sep 17 17:40:51.535034 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.534766    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d40ba619-04c7-482d-b46f-ec561e79987c-etc-systemd\") pod \"tuned-csv2f\" (UID: \"d40ba619-04c7-482d-b46f-ec561e79987c\") " pod="openshift-cluster-node-tuning-operator/tuned-csv2f"
Sep 17 17:40:51.535859 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.534786    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d40ba619-04c7-482d-b46f-ec561e79987c-lib-modules\") pod \"tuned-csv2f\" (UID: \"d40ba619-04c7-482d-b46f-ec561e79987c\") " pod="openshift-cluster-node-tuning-operator/tuned-csv2f"
Sep 17 17:40:51.535859 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.534807    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff-var-lib-openvswitch\") pod \"ovnkube-node-k4mqn\" (UID: \"6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn"
Sep 17 17:40:51.535859 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.534828    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f847c3c6-f2ab-43a0-b467-27db788f5b9f-host-run-multus-certs\") pod \"multus-fjfrk\" (UID: \"f847c3c6-f2ab-43a0-b467-27db788f5b9f\") " pod="openshift-multus/multus-fjfrk"
Sep 17 17:40:51.535859 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.534849    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bc71d7ac-eae9-4ef5-a2f8-f0292bdc51b6-iptables-alerter-script\") pod \"iptables-alerter-hfw2c\" (UID: \"bc71d7ac-eae9-4ef5-a2f8-f0292bdc51b6\") " pod="openshift-network-operator/iptables-alerter-hfw2c"
Sep 17 17:40:51.535859 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.534869    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3809c4cd-e5d9-4647-95f8-1abbeefd438e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b9psk\" (UID: \"3809c4cd-e5d9-4647-95f8-1abbeefd438e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9psk"
Sep 17 17:40:51.535859 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.534889    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24744\" (UniqueName: \"kubernetes.io/projected/3809c4cd-e5d9-4647-95f8-1abbeefd438e-kube-api-access-24744\") pod \"aws-ebs-csi-driver-node-b9psk\" (UID: \"3809c4cd-e5d9-4647-95f8-1abbeefd438e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9psk"
Sep 17 17:40:51.535859 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.534911    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/42dd1d14-0348-40e6-824e-90b1763291f1-hosts-file\") pod \"node-resolver-qn9k7\" (UID: \"42dd1d14-0348-40e6-824e-90b1763291f1\") " pod="openshift-dns/node-resolver-qn9k7"
Sep 17 17:40:51.535859 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.534916    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f847c3c6-f2ab-43a0-b467-27db788f5b9f-host-run-k8s-cni-cncf-io\") pod \"multus-fjfrk\" (UID: \"f847c3c6-f2ab-43a0-b467-27db788f5b9f\") " pod="openshift-multus/multus-fjfrk"
Sep 17 17:40:51.535859 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.534930    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d40ba619-04c7-482d-b46f-ec561e79987c-tmp\") pod \"tuned-csv2f\" (UID: \"d40ba619-04c7-482d-b46f-ec561e79987c\") " pod="openshift-cluster-node-tuning-operator/tuned-csv2f"
Sep 17 17:40:51.535859 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.534953    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/372b5295-ddc7-4999-9b48-c6cfb84d8e83-konnectivity-ca\") pod \"konnectivity-agent-qn6lp\" (UID: \"372b5295-ddc7-4999-9b48-c6cfb84d8e83\") " pod="kube-system/konnectivity-agent-qn6lp"
Sep 17 17:40:51.535859 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.534956    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff-systemd-units\") pod \"ovnkube-node-k4mqn\" (UID: \"6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn"
Sep 17 17:40:51.535859 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.534987    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phzkj\" (UniqueName: \"kubernetes.io/projected/caf785f7-c156-4b6f-8c82-4c1efe5cd710-kube-api-access-phzkj\") pod \"network-metrics-daemon-rzvhs\" (UID: \"caf785f7-c156-4b6f-8c82-4c1efe5cd710\") " pod="openshift-multus/network-metrics-daemon-rzvhs"
Sep 17 17:40:51.535859 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.535276    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff-host-kubelet\") pod \"ovnkube-node-k4mqn\" (UID: \"6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn"
Sep 17 17:40:51.535859 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.535339    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3809c4cd-e5d9-4647-95f8-1abbeefd438e-registration-dir\") pod \"aws-ebs-csi-driver-node-b9psk\" (UID: \"3809c4cd-e5d9-4647-95f8-1abbeefd438e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9psk"
Sep 17 17:40:51.535859 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.535368    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d40ba619-04c7-482d-b46f-ec561e79987c-var-lib-kubelet\") pod \"tuned-csv2f\" (UID: \"d40ba619-04c7-482d-b46f-ec561e79987c\") " pod="openshift-cluster-node-tuning-operator/tuned-csv2f"
Sep 17 17:40:51.535859 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.535398    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3f18fbf0-4d7d-4621-9260-199ef7cc0e0d-original-pull-secret\") pod \"global-pull-secret-syncer-cfb52\" (UID: \"3f18fbf0-4d7d-4621-9260-199ef7cc0e0d\") " pod="kube-system/global-pull-secret-syncer-cfb52"
Sep 17 17:40:51.535859 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.535427    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff-etc-openvswitch\") pod \"ovnkube-node-k4mqn\" (UID: \"6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn"
Sep 17 17:40:51.536659 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.535447    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/372b5295-ddc7-4999-9b48-c6cfb84d8e83-konnectivity-ca\") pod \"konnectivity-agent-qn6lp\" (UID: \"372b5295-ddc7-4999-9b48-c6cfb84d8e83\") " pod="kube-system/konnectivity-agent-qn6lp"
Sep 17 17:40:51.536659 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.535453    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff-node-log\") pod \"ovnkube-node-k4mqn\" (UID: \"6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn"
Sep 17 17:40:51.536659 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.535501    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff-ovnkube-config\") pod \"ovnkube-node-k4mqn\" (UID: \"6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn"
Sep 17 17:40:51.536659 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.535514    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f847c3c6-f2ab-43a0-b467-27db788f5b9f-multus-cni-dir\") pod \"multus-fjfrk\" (UID: \"f847c3c6-f2ab-43a0-b467-27db788f5b9f\") " pod="openshift-multus/multus-fjfrk"
Sep 17 17:40:51.536659 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.535543    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bc71d7ac-eae9-4ef5-a2f8-f0292bdc51b6-host-slash\") pod \"iptables-alerter-hfw2c\" (UID: \"bc71d7ac-eae9-4ef5-a2f8-f0292bdc51b6\") " pod="openshift-network-operator/iptables-alerter-hfw2c"
Sep 17 17:40:51.536659 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.535552    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff-var-lib-openvswitch\") pod \"ovnkube-node-k4mqn\" (UID: \"6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn"
Sep 17 17:40:51.536659 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.535557    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f847c3c6-f2ab-43a0-b467-27db788f5b9f-multus-conf-dir\") pod \"multus-fjfrk\" (UID: \"f847c3c6-f2ab-43a0-b467-27db788f5b9f\") " pod="openshift-multus/multus-fjfrk"
Sep 17 17:40:51.536659 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.535561    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3f18fbf0-4d7d-4621-9260-199ef7cc0e0d-dbus\") pod \"global-pull-secret-syncer-cfb52\" (UID: \"3f18fbf0-4d7d-4621-9260-199ef7cc0e0d\") " pod="kube-system/global-pull-secret-syncer-cfb52"
Sep 17 17:40:51.536659 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.535575    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f847c3c6-f2ab-43a0-b467-27db788f5b9f-cnibin\") pod \"multus-fjfrk\" (UID: \"f847c3c6-f2ab-43a0-b467-27db788f5b9f\") " pod="openshift-multus/multus-fjfrk"
Sep 17 17:40:51.536659 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.535590    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff-host-cni-bin\") pod \"ovnkube-node-k4mqn\" (UID: \"6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn"
Sep 17 17:40:51.536659 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.535595    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f847c3c6-f2ab-43a0-b467-27db788f5b9f-host-var-lib-cni-bin\") pod \"multus-fjfrk\" (UID: \"f847c3c6-f2ab-43a0-b467-27db788f5b9f\") " pod="openshift-multus/multus-fjfrk"
Sep 17 17:40:51.536659 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.535601    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f847c3c6-f2ab-43a0-b467-27db788f5b9f-host-run-multus-certs\") pod \"multus-fjfrk\" (UID: \"f847c3c6-f2ab-43a0-b467-27db788f5b9f\") " pod="openshift-multus/multus-fjfrk"
Sep 17 17:40:51.536659 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.535613    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d40ba619-04c7-482d-b46f-ec561e79987c-sys\") pod \"tuned-csv2f\" (UID: \"d40ba619-04c7-482d-b46f-ec561e79987c\") " pod="openshift-cluster-node-tuning-operator/tuned-csv2f"
Sep 17 17:40:51.536659 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.535636    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d40ba619-04c7-482d-b46f-ec561e79987c-etc-tuned\") pod \"tuned-csv2f\" (UID: \"d40ba619-04c7-482d-b46f-ec561e79987c\") " pod="openshift-cluster-node-tuning-operator/tuned-csv2f"
Sep 17 17:40:51.536659 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.535665    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff-log-socket\") pod \"ovnkube-node-k4mqn\" (UID: \"6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn"
Sep 17 17:40:51.536659 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:51.535720    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:40:51.536659 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:51.535769    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/caf785f7-c156-4b6f-8c82-4c1efe5cd710-metrics-certs podName:caf785f7-c156-4b6f-8c82-4c1efe5cd710 nodeName:}" failed. No retries permitted until 2026-09-17 17:40:52.035753554 +0000 UTC m=+4.539756156 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/caf785f7-c156-4b6f-8c82-4c1efe5cd710-metrics-certs") pod "network-metrics-daemon-rzvhs" (UID: "caf785f7-c156-4b6f-8c82-4c1efe5cd710") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:40:51.536659 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:51.535773    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:40:51.537525 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.535800    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k4mqn\" (UID: \"6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn"
Sep 17 17:40:51.537525 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:51.535822    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f18fbf0-4d7d-4621-9260-199ef7cc0e0d-original-pull-secret podName:3f18fbf0-4d7d-4621-9260-199ef7cc0e0d nodeName:}" failed. No retries permitted until 2026-09-17 17:40:52.035805386 +0000 UTC m=+4.539807997 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3f18fbf0-4d7d-4621-9260-199ef7cc0e0d-original-pull-secret") pod "global-pull-secret-syncer-cfb52" (UID: "3f18fbf0-4d7d-4621-9260-199ef7cc0e0d") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:40:51.537525 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.535831    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff-run-openvswitch\") pod \"ovnkube-node-k4mqn\" (UID: \"6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn"
Sep 17 17:40:51.537525 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.535856    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f847c3c6-f2ab-43a0-b467-27db788f5b9f-host-var-lib-cni-multus\") pod \"multus-fjfrk\" (UID: \"f847c3c6-f2ab-43a0-b467-27db788f5b9f\") " pod="openshift-multus/multus-fjfrk"
Sep 17 17:40:51.537525 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.535868    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff-etc-openvswitch\") pod \"ovnkube-node-k4mqn\" (UID: \"6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn"
Sep 17 17:40:51.537525 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.535886    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff-host-slash\") pod \"ovnkube-node-k4mqn\" (UID: \"6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn"
Sep 17 17:40:51.537525 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.535902    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff-host-run-netns\") pod \"ovnkube-node-k4mqn\" (UID: \"6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn"
Sep 17 17:40:51.537525 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.535914    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3f18fbf0-4d7d-4621-9260-199ef7cc0e0d-kubelet-config\") pod \"global-pull-secret-syncer-cfb52\" (UID: \"3f18fbf0-4d7d-4621-9260-199ef7cc0e0d\") " pod="kube-system/global-pull-secret-syncer-cfb52"
Sep 17 17:40:51.537525 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.535916    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff-node-log\") pod \"ovnkube-node-k4mqn\" (UID: \"6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn"
Sep 17 17:40:51.537525 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.535926    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff-run-ovn\") pod \"ovnkube-node-k4mqn\" (UID: \"6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn"
Sep 17 17:40:51.537525 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.535471    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff-env-overrides\") pod \"ovnkube-node-k4mqn\" (UID: \"6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn"
Sep 17 17:40:51.537525 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.535962    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f847c3c6-f2ab-43a0-b467-27db788f5b9f-host-var-lib-cni-bin\") pod \"multus-fjfrk\" (UID: \"f847c3c6-f2ab-43a0-b467-27db788f5b9f\") " pod="openshift-multus/multus-fjfrk"
Sep 17 17:40:51.537525 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.535976    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f847c3c6-f2ab-43a0-b467-27db788f5b9f-cnibin\") pod \"multus-fjfrk\" (UID: \"f847c3c6-f2ab-43a0-b467-27db788f5b9f\") " pod="openshift-multus/multus-fjfrk"
Sep 17 17:40:51.537525 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.536049    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff-host-cni-netd\") pod \"ovnkube-node-k4mqn\" (UID: \"6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn"
Sep 17 17:40:51.537525 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.536107    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/911990d2-9984-4b04-979c-1380a8bdb7a2-cni-binary-copy\") pod \"multus-additional-cni-plugins-mwh5v\" (UID: \"911990d2-9984-4b04-979c-1380a8bdb7a2\") " pod="openshift-multus/multus-additional-cni-plugins-mwh5v"
Sep 17 17:40:51.537525 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.536128    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff-host-cni-netd\") pod \"ovnkube-node-k4mqn\" (UID: \"6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn"
Sep 17 17:40:51.537525 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.536131    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bc71d7ac-eae9-4ef5-a2f8-f0292bdc51b6-iptables-alerter-script\") pod \"iptables-alerter-hfw2c\" (UID: \"bc71d7ac-eae9-4ef5-a2f8-f0292bdc51b6\") " pod="openshift-network-operator/iptables-alerter-hfw2c"
Sep 17 17:40:51.538470 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.536200    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/911990d2-9984-4b04-979c-1380a8bdb7a2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mwh5v\" (UID: \"911990d2-9984-4b04-979c-1380a8bdb7a2\") " pod="openshift-multus/multus-additional-cni-plugins-mwh5v"
Sep 17 17:40:51.538470 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.536222    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p56rv\" (UniqueName: \"kubernetes.io/projected/d40ba619-04c7-482d-b46f-ec561e79987c-kube-api-access-p56rv\") pod \"tuned-csv2f\" (UID: \"d40ba619-04c7-482d-b46f-ec561e79987c\") " pod="openshift-cluster-node-tuning-operator/tuned-csv2f"
Sep 17 17:40:51.538470 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.536244    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff-run-systemd\") pod \"ovnkube-node-k4mqn\" (UID: \"6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn"
Sep 17 17:40:51.538470 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.536267    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff-ovn-node-metrics-cert\") pod \"ovnkube-node-k4mqn\" (UID: \"6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn"
Sep 17 17:40:51.538470 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.536289    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff-run-systemd\") pod \"ovnkube-node-k4mqn\" (UID: \"6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn"
Sep 17 17:40:51.538470 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.536313    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f847c3c6-f2ab-43a0-b467-27db788f5b9f-host-run-netns\") pod \"multus-fjfrk\" (UID: \"f847c3c6-f2ab-43a0-b467-27db788f5b9f\") " pod="openshift-multus/multus-fjfrk"
Sep 17 17:40:51.538470 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.536342    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f847c3c6-f2ab-43a0-b467-27db788f5b9f-etc-kubernetes\") pod \"multus-fjfrk\" (UID: \"f847c3c6-f2ab-43a0-b467-27db788f5b9f\") " pod="openshift-multus/multus-fjfrk"
Sep 17 17:40:51.538470 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.536390    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f847c3c6-f2ab-43a0-b467-27db788f5b9f-host-run-netns\") pod \"multus-fjfrk\" (UID: \"f847c3c6-f2ab-43a0-b467-27db788f5b9f\") " pod="openshift-multus/multus-fjfrk"
Sep 17 17:40:51.538470 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.536403    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff-ovnkube-config\") pod \"ovnkube-node-k4mqn\" (UID: \"6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn"
Sep 17 17:40:51.538470 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.536422    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f847c3c6-f2ab-43a0-b467-27db788f5b9f-etc-kubernetes\") pod \"multus-fjfrk\" (UID: \"f847c3c6-f2ab-43a0-b467-27db788f5b9f\") " pod="openshift-multus/multus-fjfrk"
Sep 17 17:40:51.538470 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.536420    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/911990d2-9984-4b04-979c-1380a8bdb7a2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mwh5v\" (UID: \"911990d2-9984-4b04-979c-1380a8bdb7a2\") " pod="openshift-multus/multus-additional-cni-plugins-mwh5v"
Sep 17 17:40:51.538470 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.536451    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1b398422-1bbe-4f75-a7e6-a641cce3f5e8-serviceca\") pod \"node-ca-nbmpx\" (UID: \"1b398422-1bbe-4f75-a7e6-a641cce3f5e8\") " pod="openshift-image-registry/node-ca-nbmpx"
Sep 17 17:40:51.538470 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.536482    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qprbc\" (UniqueName: \"kubernetes.io/projected/1b398422-1bbe-4f75-a7e6-a641cce3f5e8-kube-api-access-qprbc\") pod \"node-ca-nbmpx\" (UID: \"1b398422-1bbe-4f75-a7e6-a641cce3f5e8\") " pod="openshift-image-registry/node-ca-nbmpx"
Sep 17 17:40:51.538470 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.536504    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff-ovnkube-script-lib\") pod \"ovnkube-node-k4mqn\" (UID: \"6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn"
Sep 17 17:40:51.538470 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.538424    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff-ovnkube-script-lib\") pod \"ovnkube-node-k4mqn\" (UID: \"6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn"
Sep 17 17:40:51.539731 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.539706    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/372b5295-ddc7-4999-9b48-c6cfb84d8e83-agent-certs\") pod \"konnectivity-agent-qn6lp\" (UID: \"372b5295-ddc7-4999-9b48-c6cfb84d8e83\") " pod="kube-system/konnectivity-agent-qn6lp"
Sep 17 17:40:51.540004 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.539986    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff-ovn-node-metrics-cert\") pod \"ovnkube-node-k4mqn\" (UID: \"6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn"
Sep 17 17:40:51.548496 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.548469    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb5mg\" (UniqueName: \"kubernetes.io/projected/f847c3c6-f2ab-43a0-b467-27db788f5b9f-kube-api-access-hb5mg\") pod \"multus-fjfrk\" (UID: \"f847c3c6-f2ab-43a0-b467-27db788f5b9f\") " pod="openshift-multus/multus-fjfrk"
Sep 17 17:40:51.548638 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.548585    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phzkj\" (UniqueName: \"kubernetes.io/projected/caf785f7-c156-4b6f-8c82-4c1efe5cd710-kube-api-access-phzkj\") pod \"network-metrics-daemon-rzvhs\" (UID: \"caf785f7-c156-4b6f-8c82-4c1efe5cd710\") " pod="openshift-multus/network-metrics-daemon-rzvhs"
Sep 17 17:40:51.548906 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.548879    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmskf\" (UniqueName: \"kubernetes.io/projected/bc71d7ac-eae9-4ef5-a2f8-f0292bdc51b6-kube-api-access-lmskf\") pod \"iptables-alerter-hfw2c\" (UID: \"bc71d7ac-eae9-4ef5-a2f8-f0292bdc51b6\") " pod="openshift-network-operator/iptables-alerter-hfw2c"
Sep 17 17:40:51.550100 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.550038    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b86gx\" (UniqueName: \"kubernetes.io/projected/6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff-kube-api-access-b86gx\") pod \"ovnkube-node-k4mqn\" (UID: \"6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn"
Sep 17 17:40:51.637633 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.637548    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/911990d2-9984-4b04-979c-1380a8bdb7a2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mwh5v\" (UID: \"911990d2-9984-4b04-979c-1380a8bdb7a2\") " pod="openshift-multus/multus-additional-cni-plugins-mwh5v"
Sep 17 17:40:51.637633 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.637592    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1b398422-1bbe-4f75-a7e6-a641cce3f5e8-serviceca\") pod \"node-ca-nbmpx\" (UID: \"1b398422-1bbe-4f75-a7e6-a641cce3f5e8\") " pod="openshift-image-registry/node-ca-nbmpx"
Sep 17 17:40:51.637843 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.637791    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qprbc\" (UniqueName: \"kubernetes.io/projected/1b398422-1bbe-4f75-a7e6-a641cce3f5e8-kube-api-access-qprbc\") pod \"node-ca-nbmpx\" (UID: \"1b398422-1bbe-4f75-a7e6-a641cce3f5e8\") " pod="openshift-image-registry/node-ca-nbmpx"
Sep 17 17:40:51.637843 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.637838    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsfzs\" (UniqueName: \"kubernetes.io/projected/911990d2-9984-4b04-979c-1380a8bdb7a2-kube-api-access-hsfzs\") pod \"multus-additional-cni-plugins-mwh5v\" (UID: \"911990d2-9984-4b04-979c-1380a8bdb7a2\") " pod="openshift-multus/multus-additional-cni-plugins-mwh5v"
Sep 17 17:40:51.637956 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.637887    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3809c4cd-e5d9-4647-95f8-1abbeefd438e-socket-dir\") pod \"aws-ebs-csi-driver-node-b9psk\" (UID: \"3809c4cd-e5d9-4647-95f8-1abbeefd438e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9psk"
Sep 17 17:40:51.637956 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.637933    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3809c4cd-e5d9-4647-95f8-1abbeefd438e-etc-selinux\") pod \"aws-ebs-csi-driver-node-b9psk\" (UID: \"3809c4cd-e5d9-4647-95f8-1abbeefd438e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9psk"
Sep 17 17:40:51.638121 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.637959    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d40ba619-04c7-482d-b46f-ec561e79987c-etc-kubernetes\") pod \"tuned-csv2f\" (UID: \"d40ba619-04c7-482d-b46f-ec561e79987c\") " pod="openshift-cluster-node-tuning-operator/tuned-csv2f"
Sep 17 17:40:51.638121 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.637990    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d40ba619-04c7-482d-b46f-ec561e79987c-etc-sysconfig\") pod \"tuned-csv2f\" (UID: \"d40ba619-04c7-482d-b46f-ec561e79987c\") " pod="openshift-cluster-node-tuning-operator/tuned-csv2f"
Sep 17 17:40:51.638121 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.638013    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d40ba619-04c7-482d-b46f-ec561e79987c-etc-sysctl-d\") pod \"tuned-csv2f\" (UID: \"d40ba619-04c7-482d-b46f-ec561e79987c\") " pod="openshift-cluster-node-tuning-operator/tuned-csv2f"
Sep 17 17:40:51.638121 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.638038    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d40ba619-04c7-482d-b46f-ec561e79987c-etc-modprobe-d\") pod \"tuned-csv2f\" (UID: \"d40ba619-04c7-482d-b46f-ec561e79987c\") " pod="openshift-cluster-node-tuning-operator/tuned-csv2f"
Sep 17 17:40:51.638121 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.638062    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d40ba619-04c7-482d-b46f-ec561e79987c-run\") pod \"tuned-csv2f\" (UID: \"d40ba619-04c7-482d-b46f-ec561e79987c\") " pod="openshift-cluster-node-tuning-operator/tuned-csv2f"
Sep 17 17:40:51.638121 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.638068    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3809c4cd-e5d9-4647-95f8-1abbeefd438e-socket-dir\") pod \"aws-ebs-csi-driver-node-b9psk\" (UID: \"3809c4cd-e5d9-4647-95f8-1abbeefd438e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9psk"
Sep 17 17:40:51.638121 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.638090    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3809c4cd-e5d9-4647-95f8-1abbeefd438e-etc-selinux\") pod \"aws-ebs-csi-driver-node-b9psk\" (UID: \"3809c4cd-e5d9-4647-95f8-1abbeefd438e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9psk"
Sep 17 17:40:51.638121 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.638104    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvc8l\" (UniqueName: \"kubernetes.io/projected/bf919600-b3d9-49db-b561-1e02cb809474-kube-api-access-qvc8l\") pod \"network-check-target-ngfbr\" (UID: \"bf919600-b3d9-49db-b561-1e02cb809474\") " pod="openshift-network-diagnostics/network-check-target-ngfbr"
Sep 17 17:40:51.638460 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.638166    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1b398422-1bbe-4f75-a7e6-a641cce3f5e8-serviceca\") pod \"node-ca-nbmpx\" (UID: \"1b398422-1bbe-4f75-a7e6-a641cce3f5e8\") " pod="openshift-image-registry/node-ca-nbmpx"
Sep 17 17:40:51.638460 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.638173    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d40ba619-04c7-482d-b46f-ec561e79987c-run\") pod \"tuned-csv2f\" (UID: \"d40ba619-04c7-482d-b46f-ec561e79987c\") " pod="openshift-cluster-node-tuning-operator/tuned-csv2f"
Sep 17 17:40:51.638460 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.638211    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/911990d2-9984-4b04-979c-1380a8bdb7a2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mwh5v\" (UID: \"911990d2-9984-4b04-979c-1380a8bdb7a2\") " pod="openshift-multus/multus-additional-cni-plugins-mwh5v"
Sep 17 17:40:51.638460 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.638221    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d40ba619-04c7-482d-b46f-ec561e79987c-etc-modprobe-d\") pod \"tuned-csv2f\" (UID: \"d40ba619-04c7-482d-b46f-ec561e79987c\") " pod="openshift-cluster-node-tuning-operator/tuned-csv2f"
Sep 17 17:40:51.638460 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.638240    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d40ba619-04c7-482d-b46f-ec561e79987c-etc-kubernetes\") pod \"tuned-csv2f\" (UID: \"d40ba619-04c7-482d-b46f-ec561e79987c\") " pod="openshift-cluster-node-tuning-operator/tuned-csv2f"
Sep 17 17:40:51.638460 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.638247    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d40ba619-04c7-482d-b46f-ec561e79987c-etc-sysctl-d\") pod \"tuned-csv2f\" (UID: \"d40ba619-04c7-482d-b46f-ec561e79987c\") " pod="openshift-cluster-node-tuning-operator/tuned-csv2f"
Sep 17 17:40:51.638460 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.638268    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3809c4cd-e5d9-4647-95f8-1abbeefd438e-sys-fs\") pod \"aws-ebs-csi-driver-node-b9psk\" (UID: \"3809c4cd-e5d9-4647-95f8-1abbeefd438e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9psk"
Sep 17 17:40:51.638460 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.638274    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d40ba619-04c7-482d-b46f-ec561e79987c-etc-sysconfig\") pod \"tuned-csv2f\" (UID: \"d40ba619-04c7-482d-b46f-ec561e79987c\") " pod="openshift-cluster-node-tuning-operator/tuned-csv2f"
Sep 17 17:40:51.638460 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.638307    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42dd1d14-0348-40e6-824e-90b1763291f1-tmp-dir\") pod \"node-resolver-qn9k7\" (UID: \"42dd1d14-0348-40e6-824e-90b1763291f1\") " pod="openshift-dns/node-resolver-qn9k7"
Sep 17 17:40:51.638460 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.638328    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3809c4cd-e5d9-4647-95f8-1abbeefd438e-sys-fs\") pod \"aws-ebs-csi-driver-node-b9psk\" (UID: \"3809c4cd-e5d9-4647-95f8-1abbeefd438e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9psk"
Sep 17 17:40:51.638460 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.638333    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d40ba619-04c7-482d-b46f-ec561e79987c-etc-sysctl-conf\") pod \"tuned-csv2f\" (UID: \"d40ba619-04c7-482d-b46f-ec561e79987c\") " pod="openshift-cluster-node-tuning-operator/tuned-csv2f"
Sep 17 17:40:51.638460 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.638362    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr2bs\" (UniqueName: \"kubernetes.io/projected/42dd1d14-0348-40e6-824e-90b1763291f1-kube-api-access-fr2bs\") pod \"node-resolver-qn9k7\" (UID: \"42dd1d14-0348-40e6-824e-90b1763291f1\") " pod="openshift-dns/node-resolver-qn9k7"
Sep 17 17:40:51.638460 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.638389    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1b398422-1bbe-4f75-a7e6-a641cce3f5e8-host\") pod \"node-ca-nbmpx\" (UID: \"1b398422-1bbe-4f75-a7e6-a641cce3f5e8\") " pod="openshift-image-registry/node-ca-nbmpx"
Sep 17 17:40:51.638460 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.638425    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1b398422-1bbe-4f75-a7e6-a641cce3f5e8-host\") pod \"node-ca-nbmpx\" (UID: \"1b398422-1bbe-4f75-a7e6-a641cce3f5e8\") " pod="openshift-image-registry/node-ca-nbmpx"
Sep 17 17:40:51.639122 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.638489    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d40ba619-04c7-482d-b46f-ec561e79987c-etc-sysctl-conf\") pod \"tuned-csv2f\" (UID: \"d40ba619-04c7-482d-b46f-ec561e79987c\") " pod="openshift-cluster-node-tuning-operator/tuned-csv2f"
Sep 17 17:40:51.639122 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.638508    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d40ba619-04c7-482d-b46f-ec561e79987c-host\") pod \"tuned-csv2f\" (UID: \"d40ba619-04c7-482d-b46f-ec561e79987c\") " pod="openshift-cluster-node-tuning-operator/tuned-csv2f"
Sep 17 17:40:51.639122 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.638537    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/911990d2-9984-4b04-979c-1380a8bdb7a2-cnibin\") pod \"multus-additional-cni-plugins-mwh5v\" (UID: \"911990d2-9984-4b04-979c-1380a8bdb7a2\") " pod="openshift-multus/multus-additional-cni-plugins-mwh5v"
Sep 17 17:40:51.639122 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.638553    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/911990d2-9984-4b04-979c-1380a8bdb7a2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mwh5v\" (UID: \"911990d2-9984-4b04-979c-1380a8bdb7a2\") " pod="openshift-multus/multus-additional-cni-plugins-mwh5v"
Sep 17 17:40:51.639122 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.638589    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/911990d2-9984-4b04-979c-1380a8bdb7a2-system-cni-dir\") pod \"multus-additional-cni-plugins-mwh5v\" (UID: \"911990d2-9984-4b04-979c-1380a8bdb7a2\") " pod="openshift-multus/multus-additional-cni-plugins-mwh5v"
Sep 17 17:40:51.639122 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.638591    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42dd1d14-0348-40e6-824e-90b1763291f1-tmp-dir\") pod \"node-resolver-qn9k7\" (UID: \"42dd1d14-0348-40e6-824e-90b1763291f1\") " pod="openshift-dns/node-resolver-qn9k7"
Sep 17 17:40:51.639122 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.638603    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/911990d2-9984-4b04-979c-1380a8bdb7a2-cnibin\") pod \"multus-additional-cni-plugins-mwh5v\" (UID: \"911990d2-9984-4b04-979c-1380a8bdb7a2\") " pod="openshift-multus/multus-additional-cni-plugins-mwh5v"
Sep 17 17:40:51.639122 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.638600    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d40ba619-04c7-482d-b46f-ec561e79987c-host\") pod \"tuned-csv2f\" (UID: \"d40ba619-04c7-482d-b46f-ec561e79987c\") " pod="openshift-cluster-node-tuning-operator/tuned-csv2f"
Sep 17 17:40:51.639122 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.638605    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3809c4cd-e5d9-4647-95f8-1abbeefd438e-device-dir\") pod \"aws-ebs-csi-driver-node-b9psk\" (UID: \"3809c4cd-e5d9-4647-95f8-1abbeefd438e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9psk"
Sep 17 17:40:51.639122 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.638634    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/911990d2-9984-4b04-979c-1380a8bdb7a2-system-cni-dir\") pod \"multus-additional-cni-plugins-mwh5v\" (UID: \"911990d2-9984-4b04-979c-1380a8bdb7a2\") " pod="openshift-multus/multus-additional-cni-plugins-mwh5v"
Sep 17 17:40:51.639122 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.638662    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d40ba619-04c7-482d-b46f-ec561e79987c-etc-systemd\") pod \"tuned-csv2f\" (UID: \"d40ba619-04c7-482d-b46f-ec561e79987c\") " pod="openshift-cluster-node-tuning-operator/tuned-csv2f"
Sep 17 17:40:51.639122 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.638687    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3809c4cd-e5d9-4647-95f8-1abbeefd438e-device-dir\") pod \"aws-ebs-csi-driver-node-b9psk\" (UID: \"3809c4cd-e5d9-4647-95f8-1abbeefd438e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9psk"
Sep 17 17:40:51.639122 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.638688    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d40ba619-04c7-482d-b46f-ec561e79987c-lib-modules\") pod \"tuned-csv2f\" (UID: \"d40ba619-04c7-482d-b46f-ec561e79987c\") " pod="openshift-cluster-node-tuning-operator/tuned-csv2f"
Sep 17 17:40:51.639122 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.638708    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d40ba619-04c7-482d-b46f-ec561e79987c-etc-systemd\") pod \"tuned-csv2f\" (UID: \"d40ba619-04c7-482d-b46f-ec561e79987c\") " pod="openshift-cluster-node-tuning-operator/tuned-csv2f"
Sep 17 17:40:51.639122 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.638724    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3809c4cd-e5d9-4647-95f8-1abbeefd438e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b9psk\" (UID: \"3809c4cd-e5d9-4647-95f8-1abbeefd438e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9psk"
Sep 17 17:40:51.639122 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.638740    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24744\" (UniqueName: \"kubernetes.io/projected/3809c4cd-e5d9-4647-95f8-1abbeefd438e-kube-api-access-24744\") pod \"aws-ebs-csi-driver-node-b9psk\" (UID: \"3809c4cd-e5d9-4647-95f8-1abbeefd438e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9psk"
Sep 17 17:40:51.639122 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.638756    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/42dd1d14-0348-40e6-824e-90b1763291f1-hosts-file\") pod \"node-resolver-qn9k7\" (UID: \"42dd1d14-0348-40e6-824e-90b1763291f1\") " pod="openshift-dns/node-resolver-qn9k7"
Sep 17 17:40:51.639838 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.638770    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d40ba619-04c7-482d-b46f-ec561e79987c-lib-modules\") pod \"tuned-csv2f\" (UID: \"d40ba619-04c7-482d-b46f-ec561e79987c\") " pod="openshift-cluster-node-tuning-operator/tuned-csv2f"
Sep 17 17:40:51.639838 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.638780    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d40ba619-04c7-482d-b46f-ec561e79987c-tmp\") pod \"tuned-csv2f\" (UID: \"d40ba619-04c7-482d-b46f-ec561e79987c\") " pod="openshift-cluster-node-tuning-operator/tuned-csv2f"
Sep 17 17:40:51.639838 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.638798    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3809c4cd-e5d9-4647-95f8-1abbeefd438e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b9psk\" (UID: \"3809c4cd-e5d9-4647-95f8-1abbeefd438e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9psk"
Sep 17 17:40:51.639838 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.638806    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3809c4cd-e5d9-4647-95f8-1abbeefd438e-registration-dir\") pod \"aws-ebs-csi-driver-node-b9psk\" (UID: \"3809c4cd-e5d9-4647-95f8-1abbeefd438e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9psk"
Sep 17 17:40:51.639838 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.638820    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/42dd1d14-0348-40e6-824e-90b1763291f1-hosts-file\") pod \"node-resolver-qn9k7\" (UID: \"42dd1d14-0348-40e6-824e-90b1763291f1\") " pod="openshift-dns/node-resolver-qn9k7"
Sep 17 17:40:51.639838 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.638830    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d40ba619-04c7-482d-b46f-ec561e79987c-var-lib-kubelet\") pod \"tuned-csv2f\" (UID: \"d40ba619-04c7-482d-b46f-ec561e79987c\") " pod="openshift-cluster-node-tuning-operator/tuned-csv2f"
Sep 17 17:40:51.639838 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.638868    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d40ba619-04c7-482d-b46f-ec561e79987c-sys\") pod \"tuned-csv2f\" (UID: \"d40ba619-04c7-482d-b46f-ec561e79987c\") " pod="openshift-cluster-node-tuning-operator/tuned-csv2f"
Sep 17 17:40:51.639838 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.638891    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d40ba619-04c7-482d-b46f-ec561e79987c-etc-tuned\") pod \"tuned-csv2f\" (UID: \"d40ba619-04c7-482d-b46f-ec561e79987c\") " pod="openshift-cluster-node-tuning-operator/tuned-csv2f"
Sep 17 17:40:51.639838 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.638903    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d40ba619-04c7-482d-b46f-ec561e79987c-var-lib-kubelet\") pod \"tuned-csv2f\" (UID: \"d40ba619-04c7-482d-b46f-ec561e79987c\") " pod="openshift-cluster-node-tuning-operator/tuned-csv2f"
Sep 17 17:40:51.639838 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.638869    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3809c4cd-e5d9-4647-95f8-1abbeefd438e-registration-dir\") pod \"aws-ebs-csi-driver-node-b9psk\" (UID: \"3809c4cd-e5d9-4647-95f8-1abbeefd438e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9psk"
Sep 17 17:40:51.639838 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.638913    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/911990d2-9984-4b04-979c-1380a8bdb7a2-cni-binary-copy\") pod \"multus-additional-cni-plugins-mwh5v\" (UID: \"911990d2-9984-4b04-979c-1380a8bdb7a2\") " pod="openshift-multus/multus-additional-cni-plugins-mwh5v"
Sep 17 17:40:51.639838 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.638948    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/911990d2-9984-4b04-979c-1380a8bdb7a2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mwh5v\" (UID: \"911990d2-9984-4b04-979c-1380a8bdb7a2\") " pod="openshift-multus/multus-additional-cni-plugins-mwh5v"
Sep 17 17:40:51.639838 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.638969    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/911990d2-9984-4b04-979c-1380a8bdb7a2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mwh5v\" (UID: \"911990d2-9984-4b04-979c-1380a8bdb7a2\") " pod="openshift-multus/multus-additional-cni-plugins-mwh5v"
Sep 17 17:40:51.639838 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.638974    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d40ba619-04c7-482d-b46f-ec561e79987c-sys\") pod \"tuned-csv2f\" (UID: \"d40ba619-04c7-482d-b46f-ec561e79987c\") " pod="openshift-cluster-node-tuning-operator/tuned-csv2f"
Sep 17 17:40:51.639838 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.638975    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p56rv\" (UniqueName: \"kubernetes.io/projected/d40ba619-04c7-482d-b46f-ec561e79987c-kube-api-access-p56rv\") pod \"tuned-csv2f\" (UID: \"d40ba619-04c7-482d-b46f-ec561e79987c\") " pod="openshift-cluster-node-tuning-operator/tuned-csv2f"
Sep 17 17:40:51.639838 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.639128    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/911990d2-9984-4b04-979c-1380a8bdb7a2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mwh5v\" (UID: \"911990d2-9984-4b04-979c-1380a8bdb7a2\") " pod="openshift-multus/multus-additional-cni-plugins-mwh5v"
Sep 17 17:40:51.639838 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.639327    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/911990d2-9984-4b04-979c-1380a8bdb7a2-cni-binary-copy\") pod \"multus-additional-cni-plugins-mwh5v\" (UID: \"911990d2-9984-4b04-979c-1380a8bdb7a2\") " pod="openshift-multus/multus-additional-cni-plugins-mwh5v"
Sep 17 17:40:51.641332 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.641310    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d40ba619-04c7-482d-b46f-ec561e79987c-etc-tuned\") pod \"tuned-csv2f\" (UID: \"d40ba619-04c7-482d-b46f-ec561e79987c\") " pod="openshift-cluster-node-tuning-operator/tuned-csv2f"
Sep 17 17:40:51.641447 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.641342    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d40ba619-04c7-482d-b46f-ec561e79987c-tmp\") pod \"tuned-csv2f\" (UID: \"d40ba619-04c7-482d-b46f-ec561e79987c\") " pod="openshift-cluster-node-tuning-operator/tuned-csv2f"
Sep 17 17:40:51.646285 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:51.646263    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:40:51.646389 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:51.646292    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:40:51.646389 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:51.646306    2569 projected.go:196] Error preparing data for projected volume kube-api-access-qvc8l for pod openshift-network-diagnostics/network-check-target-ngfbr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:40:51.646502 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:51.646391    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf919600-b3d9-49db-b561-1e02cb809474-kube-api-access-qvc8l podName:bf919600-b3d9-49db-b561-1e02cb809474 nodeName:}" failed. No retries permitted until 2026-09-17 17:40:52.146369988 +0000 UTC m=+4.650372610 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-qvc8l" (UniqueName: "kubernetes.io/projected/bf919600-b3d9-49db-b561-1e02cb809474-kube-api-access-qvc8l") pod "network-check-target-ngfbr" (UID: "bf919600-b3d9-49db-b561-1e02cb809474") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:40:51.648261 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.648209    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr2bs\" (UniqueName: \"kubernetes.io/projected/42dd1d14-0348-40e6-824e-90b1763291f1-kube-api-access-fr2bs\") pod \"node-resolver-qn9k7\" (UID: \"42dd1d14-0348-40e6-824e-90b1763291f1\") " pod="openshift-dns/node-resolver-qn9k7"
Sep 17 17:40:51.648261 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.648255    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qprbc\" (UniqueName: \"kubernetes.io/projected/1b398422-1bbe-4f75-a7e6-a641cce3f5e8-kube-api-access-qprbc\") pod \"node-ca-nbmpx\" (UID: \"1b398422-1bbe-4f75-a7e6-a641cce3f5e8\") " pod="openshift-image-registry/node-ca-nbmpx"
Sep 17 17:40:51.649869 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.649845    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsfzs\" (UniqueName: \"kubernetes.io/projected/911990d2-9984-4b04-979c-1380a8bdb7a2-kube-api-access-hsfzs\") pod \"multus-additional-cni-plugins-mwh5v\" (UID: \"911990d2-9984-4b04-979c-1380a8bdb7a2\") " pod="openshift-multus/multus-additional-cni-plugins-mwh5v"
Sep 17 17:40:51.650142 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.650122    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p56rv\" (UniqueName: \"kubernetes.io/projected/d40ba619-04c7-482d-b46f-ec561e79987c-kube-api-access-p56rv\") pod \"tuned-csv2f\" (UID: \"d40ba619-04c7-482d-b46f-ec561e79987c\") " pod="openshift-cluster-node-tuning-operator/tuned-csv2f"
Sep 17 17:40:51.650305 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.650290    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24744\" (UniqueName: \"kubernetes.io/projected/3809c4cd-e5d9-4647-95f8-1abbeefd438e-kube-api-access-24744\") pod \"aws-ebs-csi-driver-node-b9psk\" (UID: \"3809c4cd-e5d9-4647-95f8-1abbeefd438e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9psk"
Sep 17 17:40:51.726355 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.726318    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn"
Sep 17 17:40:51.735790 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.735767    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qn6lp"
Sep 17 17:40:51.746649 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.746620    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fjfrk"
Sep 17 17:40:51.754570 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.754543    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mwh5v"
Sep 17 17:40:51.763362 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.763338    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hfw2c"
Sep 17 17:40:51.769044 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.769025    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-csv2f"
Sep 17 17:40:51.776793 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.776774    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9psk"
Sep 17 17:40:51.784451 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.784431    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qn9k7"
Sep 17 17:40:51.788577 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:51.788554    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nbmpx"
Sep 17 17:40:52.041812 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:52.041724    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caf785f7-c156-4b6f-8c82-4c1efe5cd710-metrics-certs\") pod \"network-metrics-daemon-rzvhs\" (UID: \"caf785f7-c156-4b6f-8c82-4c1efe5cd710\") " pod="openshift-multus/network-metrics-daemon-rzvhs"
Sep 17 17:40:52.041812 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:52.041777    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3f18fbf0-4d7d-4621-9260-199ef7cc0e0d-original-pull-secret\") pod \"global-pull-secret-syncer-cfb52\" (UID: \"3f18fbf0-4d7d-4621-9260-199ef7cc0e0d\") " pod="kube-system/global-pull-secret-syncer-cfb52"
Sep 17 17:40:52.042029 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:52.041883    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:40:52.042029 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:52.042005    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/caf785f7-c156-4b6f-8c82-4c1efe5cd710-metrics-certs podName:caf785f7-c156-4b6f-8c82-4c1efe5cd710 nodeName:}" failed. No retries permitted until 2026-09-17 17:40:53.041984854 +0000 UTC m=+5.545987456 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/caf785f7-c156-4b6f-8c82-4c1efe5cd710-metrics-certs") pod "network-metrics-daemon-rzvhs" (UID: "caf785f7-c156-4b6f-8c82-4c1efe5cd710") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:40:52.042029 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:52.041906    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:40:52.042185 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:52.042089    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f18fbf0-4d7d-4621-9260-199ef7cc0e0d-original-pull-secret podName:3f18fbf0-4d7d-4621-9260-199ef7cc0e0d nodeName:}" failed. No retries permitted until 2026-09-17 17:40:53.042063078 +0000 UTC m=+5.546065701 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3f18fbf0-4d7d-4621-9260-199ef7cc0e0d-original-pull-secret") pod "global-pull-secret-syncer-cfb52" (UID: "3f18fbf0-4d7d-4621-9260-199ef7cc0e0d") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:40:52.243328 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:52.243286    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvc8l\" (UniqueName: \"kubernetes.io/projected/bf919600-b3d9-49db-b561-1e02cb809474-kube-api-access-qvc8l\") pod \"network-check-target-ngfbr\" (UID: \"bf919600-b3d9-49db-b561-1e02cb809474\") " pod="openshift-network-diagnostics/network-check-target-ngfbr"
Sep 17 17:40:52.243509 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:52.243436    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:40:52.243509 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:52.243459    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:40:52.243509 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:52.243473    2569 projected.go:196] Error preparing data for projected volume kube-api-access-qvc8l for pod openshift-network-diagnostics/network-check-target-ngfbr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:40:52.243678 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:52.243539    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf919600-b3d9-49db-b561-1e02cb809474-kube-api-access-qvc8l podName:bf919600-b3d9-49db-b561-1e02cb809474 nodeName:}" failed. No retries permitted until 2026-09-17 17:40:53.243520349 +0000 UTC m=+5.747522968 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-qvc8l" (UniqueName: "kubernetes.io/projected/bf919600-b3d9-49db-b561-1e02cb809474-kube-api-access-qvc8l") pod "network-check-target-ngfbr" (UID: "bf919600-b3d9-49db-b561-1e02cb809474") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:40:52.472361 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:52.471885    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42dd1d14_0348_40e6_824e_90b1763291f1.slice/crio-2fc49bd286522644648aae4ba1731e863cc2e9255c6e4b96106419b928294f5f WatchSource:0}: Error finding container 2fc49bd286522644648aae4ba1731e863cc2e9255c6e4b96106419b928294f5f: Status 404 returned error can't find the container with id 2fc49bd286522644648aae4ba1731e863cc2e9255c6e4b96106419b928294f5f
Sep 17 17:40:52.473023 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:52.472019    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 17:35:51 +0000 UTC" deadline="2028-04-06 19:22:04.988340438 +0000 UTC"
Sep 17 17:40:52.473023 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:52.473019    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13609h41m12.515331939s"
Sep 17 17:40:52.477486 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:52.477457    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b398422_1bbe_4f75_a7e6_a641cce3f5e8.slice/crio-6887ec0831e1c28005acd3ef22dde68f703a67127bb902f61292dc743a805384 WatchSource:0}: Error finding container 6887ec0831e1c28005acd3ef22dde68f703a67127bb902f61292dc743a805384: Status 404 returned error can't find the container with id 6887ec0831e1c28005acd3ef22dde68f703a67127bb902f61292dc743a805384
Sep 17 17:40:52.478535 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:52.478486    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod372b5295_ddc7_4999_9b48_c6cfb84d8e83.slice/crio-5629b7634e2d930cd0e7281045c68669d8562ee667ab57b390b97fedf14ced24 WatchSource:0}: Error finding container 5629b7634e2d930cd0e7281045c68669d8562ee667ab57b390b97fedf14ced24: Status 404 returned error can't find the container with id 5629b7634e2d930cd0e7281045c68669d8562ee667ab57b390b97fedf14ced24
Sep 17 17:40:52.480157 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:52.480133    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3809c4cd_e5d9_4647_95f8_1abbeefd438e.slice/crio-df93718c768178e9da3f01c92bdf90d00d4d3588d7cd4e6bf1f105a1cb9f62a4 WatchSource:0}: Error finding container df93718c768178e9da3f01c92bdf90d00d4d3588d7cd4e6bf1f105a1cb9f62a4: Status 404 returned error can't find the container with id df93718c768178e9da3f01c92bdf90d00d4d3588d7cd4e6bf1f105a1cb9f62a4
Sep 17 17:40:52.480508 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:40:52.480490    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc71d7ac_eae9_4ef5_a2f8_f0292bdc51b6.slice/crio-bf9c35f31ca7dc50cdf831e9fb4a434f7daa2c5d5341b08da902065ef620e0f4 WatchSource:0}: Error finding container bf9c35f31ca7dc50cdf831e9fb4a434f7daa2c5d5341b08da902065ef620e0f4: Status 404 returned error can't find the container with id bf9c35f31ca7dc50cdf831e9fb4a434f7daa2c5d5341b08da902065ef620e0f4
Sep 17 17:40:52.515065 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:52.515036    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nbmpx" event={"ID":"1b398422-1bbe-4f75-a7e6-a641cce3f5e8","Type":"ContainerStarted","Data":"6887ec0831e1c28005acd3ef22dde68f703a67127bb902f61292dc743a805384"}
Sep 17 17:40:52.516126 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:52.516101    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn" event={"ID":"6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff","Type":"ContainerStarted","Data":"85993878f8e3f0928bf270bd18aab4ae6bfbb3d5003d86a9d927e277825d99ac"}
Sep 17 17:40:52.517234 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:52.517210    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mwh5v" event={"ID":"911990d2-9984-4b04-979c-1380a8bdb7a2","Type":"ContainerStarted","Data":"ee6ed59282bd9fea41ea2b2d22b6c342b634210deb72184a1c0444d8b061cf10"}
Sep 17 17:40:52.518402 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:52.518383    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qn9k7" event={"ID":"42dd1d14-0348-40e6-824e-90b1763291f1","Type":"ContainerStarted","Data":"2fc49bd286522644648aae4ba1731e863cc2e9255c6e4b96106419b928294f5f"}
Sep 17 17:40:52.519595 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:52.519573    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qn6lp" event={"ID":"372b5295-ddc7-4999-9b48-c6cfb84d8e83","Type":"ContainerStarted","Data":"5629b7634e2d930cd0e7281045c68669d8562ee667ab57b390b97fedf14ced24"}
Sep 17 17:40:52.522373 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:52.522349    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9psk" event={"ID":"3809c4cd-e5d9-4647-95f8-1abbeefd438e","Type":"ContainerStarted","Data":"df93718c768178e9da3f01c92bdf90d00d4d3588d7cd4e6bf1f105a1cb9f62a4"}
Sep 17 17:40:52.524925 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:52.524902    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-csv2f" event={"ID":"d40ba619-04c7-482d-b46f-ec561e79987c","Type":"ContainerStarted","Data":"e2197301aa7427e94153110e4c58b6f8a72cbdef76694a06a4e84ff630a739b6"}
Sep 17 17:40:52.527006 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:52.526980    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fjfrk" event={"ID":"f847c3c6-f2ab-43a0-b467-27db788f5b9f","Type":"ContainerStarted","Data":"03b9329c87cfc0c33a942f68c9b2fbd06a9c7bfa57d0c7bf41982854aa802e1e"}
Sep 17 17:40:52.528071 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:52.528049    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hfw2c" event={"ID":"bc71d7ac-eae9-4ef5-a2f8-f0292bdc51b6","Type":"ContainerStarted","Data":"bf9c35f31ca7dc50cdf831e9fb4a434f7daa2c5d5341b08da902065ef620e0f4"}
Sep 17 17:40:53.049758 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:53.049614    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caf785f7-c156-4b6f-8c82-4c1efe5cd710-metrics-certs\") pod \"network-metrics-daemon-rzvhs\" (UID: \"caf785f7-c156-4b6f-8c82-4c1efe5cd710\") " pod="openshift-multus/network-metrics-daemon-rzvhs"
Sep 17 17:40:53.049758 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:53.049676    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3f18fbf0-4d7d-4621-9260-199ef7cc0e0d-original-pull-secret\") pod \"global-pull-secret-syncer-cfb52\" (UID: \"3f18fbf0-4d7d-4621-9260-199ef7cc0e0d\") " pod="kube-system/global-pull-secret-syncer-cfb52"
Sep 17 17:40:53.050002 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:53.049809    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:40:53.050002 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:53.049884    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f18fbf0-4d7d-4621-9260-199ef7cc0e0d-original-pull-secret podName:3f18fbf0-4d7d-4621-9260-199ef7cc0e0d nodeName:}" failed. No retries permitted until 2026-09-17 17:40:55.049855768 +0000 UTC m=+7.553858387 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3f18fbf0-4d7d-4621-9260-199ef7cc0e0d-original-pull-secret") pod "global-pull-secret-syncer-cfb52" (UID: "3f18fbf0-4d7d-4621-9260-199ef7cc0e0d") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:40:53.050313 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:53.050295    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:40:53.050389 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:53.050351    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/caf785f7-c156-4b6f-8c82-4c1efe5cd710-metrics-certs podName:caf785f7-c156-4b6f-8c82-4c1efe5cd710 nodeName:}" failed. No retries permitted until 2026-09-17 17:40:55.050335456 +0000 UTC m=+7.554338081 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/caf785f7-c156-4b6f-8c82-4c1efe5cd710-metrics-certs") pod "network-metrics-daemon-rzvhs" (UID: "caf785f7-c156-4b6f-8c82-4c1efe5cd710") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:40:53.251377 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:53.251339    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvc8l\" (UniqueName: \"kubernetes.io/projected/bf919600-b3d9-49db-b561-1e02cb809474-kube-api-access-qvc8l\") pod \"network-check-target-ngfbr\" (UID: \"bf919600-b3d9-49db-b561-1e02cb809474\") " pod="openshift-network-diagnostics/network-check-target-ngfbr"
Sep 17 17:40:53.251595 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:53.251566    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:40:53.251595 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:53.251593    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:40:53.251724 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:53.251605    2569 projected.go:196] Error preparing data for projected volume kube-api-access-qvc8l for pod openshift-network-diagnostics/network-check-target-ngfbr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:40:53.251724 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:53.251672    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf919600-b3d9-49db-b561-1e02cb809474-kube-api-access-qvc8l podName:bf919600-b3d9-49db-b561-1e02cb809474 nodeName:}" failed. No retries permitted until 2026-09-17 17:40:55.251653205 +0000 UTC m=+7.755655815 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-qvc8l" (UniqueName: "kubernetes.io/projected/bf919600-b3d9-49db-b561-1e02cb809474-kube-api-access-qvc8l") pod "network-check-target-ngfbr" (UID: "bf919600-b3d9-49db-b561-1e02cb809474") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:40:53.473790 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:53.473688    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 17:35:51 +0000 UTC" deadline="2028-04-15 21:32:41.271829337 +0000 UTC"
Sep 17 17:40:53.473790 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:53.473737    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13827h51m47.798097189s"
Sep 17 17:40:53.505546 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:53.505514    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ngfbr"
Sep 17 17:40:53.505713 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:53.505640    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ngfbr" podUID="bf919600-b3d9-49db-b561-1e02cb809474"
Sep 17 17:40:53.506215 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:53.506184    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cfb52"
Sep 17 17:40:53.506412 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:53.506379    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cfb52" podUID="3f18fbf0-4d7d-4621-9260-199ef7cc0e0d"
Sep 17 17:40:53.506517 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:53.506494    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rzvhs"
Sep 17 17:40:53.506661 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:53.506626    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rzvhs" podUID="caf785f7-c156-4b6f-8c82-4c1efe5cd710"
Sep 17 17:40:53.540434 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:53.540394    2569 generic.go:356] "Generic (PLEG): container finished" podID="273a3c53d0c1db5ff2ab130636ac4e73" containerID="158b4561856c903a296d79cddcaf39ee39d5670030d17a99101ffe23bb713e3e" exitCode=0
Sep 17 17:40:53.540622 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:53.540491    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.ec2.internal" event={"ID":"273a3c53d0c1db5ff2ab130636ac4e73","Type":"ContainerDied","Data":"158b4561856c903a296d79cddcaf39ee39d5670030d17a99101ffe23bb713e3e"}
Sep 17 17:40:53.554503 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:53.553958    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-14.ec2.internal" event={"ID":"73796eafa3737317490091fb30921865","Type":"ContainerStarted","Data":"fedff3aae6d615cd0e2aae2b645666ec95a5ccf249c039afa7f3c4a8373234bb"}
Sep 17 17:40:53.586446 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:53.586382    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-14.ec2.internal" podStartSLOduration=3.586359377 podStartE2EDuration="3.586359377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:40:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:40:53.586201379 +0000 UTC m=+6.090204004" watchObservedRunningTime="2026-09-17 17:40:53.586359377 +0000 UTC m=+6.090362018"
Sep 17 17:40:54.579799 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:54.579339    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.ec2.internal" event={"ID":"273a3c53d0c1db5ff2ab130636ac4e73","Type":"ContainerStarted","Data":"801eec487419f5b14a61a58bd02914a6fc0eb2a04871670a9bc94a9051dad2d5"}
Sep 17 17:40:55.068485 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:55.068393    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caf785f7-c156-4b6f-8c82-4c1efe5cd710-metrics-certs\") pod \"network-metrics-daemon-rzvhs\" (UID: \"caf785f7-c156-4b6f-8c82-4c1efe5cd710\") " pod="openshift-multus/network-metrics-daemon-rzvhs"
Sep 17 17:40:55.068485 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:55.068452    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3f18fbf0-4d7d-4621-9260-199ef7cc0e0d-original-pull-secret\") pod \"global-pull-secret-syncer-cfb52\" (UID: \"3f18fbf0-4d7d-4621-9260-199ef7cc0e0d\") " pod="kube-system/global-pull-secret-syncer-cfb52"
Sep 17 17:40:55.068709 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:55.068595    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:40:55.068709 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:55.068656    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f18fbf0-4d7d-4621-9260-199ef7cc0e0d-original-pull-secret podName:3f18fbf0-4d7d-4621-9260-199ef7cc0e0d nodeName:}" failed. No retries permitted until 2026-09-17 17:40:59.068638997 +0000 UTC m=+11.572641618 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3f18fbf0-4d7d-4621-9260-199ef7cc0e0d-original-pull-secret") pod "global-pull-secret-syncer-cfb52" (UID: "3f18fbf0-4d7d-4621-9260-199ef7cc0e0d") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:40:55.069118 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:55.069094    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:40:55.069255 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:55.069151    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/caf785f7-c156-4b6f-8c82-4c1efe5cd710-metrics-certs podName:caf785f7-c156-4b6f-8c82-4c1efe5cd710 nodeName:}" failed. No retries permitted until 2026-09-17 17:40:59.06913642 +0000 UTC m=+11.573139022 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/caf785f7-c156-4b6f-8c82-4c1efe5cd710-metrics-certs") pod "network-metrics-daemon-rzvhs" (UID: "caf785f7-c156-4b6f-8c82-4c1efe5cd710") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:40:55.270390 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:55.270349    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvc8l\" (UniqueName: \"kubernetes.io/projected/bf919600-b3d9-49db-b561-1e02cb809474-kube-api-access-qvc8l\") pod \"network-check-target-ngfbr\" (UID: \"bf919600-b3d9-49db-b561-1e02cb809474\") " pod="openshift-network-diagnostics/network-check-target-ngfbr"
Sep 17 17:40:55.270577 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:55.270554    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:40:55.270577 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:55.270573    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:40:55.270689 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:55.270585    2569 projected.go:196] Error preparing data for projected volume kube-api-access-qvc8l for pod openshift-network-diagnostics/network-check-target-ngfbr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:40:55.270689 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:55.270645    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf919600-b3d9-49db-b561-1e02cb809474-kube-api-access-qvc8l podName:bf919600-b3d9-49db-b561-1e02cb809474 nodeName:}" failed. No retries permitted until 2026-09-17 17:40:59.27062816 +0000 UTC m=+11.774630766 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-qvc8l" (UniqueName: "kubernetes.io/projected/bf919600-b3d9-49db-b561-1e02cb809474-kube-api-access-qvc8l") pod "network-check-target-ngfbr" (UID: "bf919600-b3d9-49db-b561-1e02cb809474") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:40:55.505114 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:55.505020    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cfb52"
Sep 17 17:40:55.505268 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:55.505170    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cfb52" podUID="3f18fbf0-4d7d-4621-9260-199ef7cc0e0d"
Sep 17 17:40:55.505563 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:55.505546    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rzvhs"
Sep 17 17:40:55.505674 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:55.505654    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rzvhs" podUID="caf785f7-c156-4b6f-8c82-4c1efe5cd710"
Sep 17 17:40:55.505743 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:55.505717    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ngfbr"
Sep 17 17:40:55.505799 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:55.505788    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ngfbr" podUID="bf919600-b3d9-49db-b561-1e02cb809474"
Sep 17 17:40:57.503234 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:57.503197    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ngfbr"
Sep 17 17:40:57.503742 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:57.503339    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ngfbr" podUID="bf919600-b3d9-49db-b561-1e02cb809474"
Sep 17 17:40:57.503742 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:57.503717    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cfb52"
Sep 17 17:40:57.503847 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:57.503821    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cfb52" podUID="3f18fbf0-4d7d-4621-9260-199ef7cc0e0d"
Sep 17 17:40:57.503938 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:57.503913    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rzvhs"
Sep 17 17:40:57.504067 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:57.504044    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rzvhs" podUID="caf785f7-c156-4b6f-8c82-4c1efe5cd710"
Sep 17 17:40:59.101048 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:59.100947    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caf785f7-c156-4b6f-8c82-4c1efe5cd710-metrics-certs\") pod \"network-metrics-daemon-rzvhs\" (UID: \"caf785f7-c156-4b6f-8c82-4c1efe5cd710\") " pod="openshift-multus/network-metrics-daemon-rzvhs"
Sep 17 17:40:59.101048 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:59.101012    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3f18fbf0-4d7d-4621-9260-199ef7cc0e0d-original-pull-secret\") pod \"global-pull-secret-syncer-cfb52\" (UID: \"3f18fbf0-4d7d-4621-9260-199ef7cc0e0d\") " pod="kube-system/global-pull-secret-syncer-cfb52"
Sep 17 17:40:59.101694 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:59.101152    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:40:59.101694 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:59.101153    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:40:59.101694 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:59.101224    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/caf785f7-c156-4b6f-8c82-4c1efe5cd710-metrics-certs podName:caf785f7-c156-4b6f-8c82-4c1efe5cd710 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:07.101203199 +0000 UTC m=+19.605205816 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/caf785f7-c156-4b6f-8c82-4c1efe5cd710-metrics-certs") pod "network-metrics-daemon-rzvhs" (UID: "caf785f7-c156-4b6f-8c82-4c1efe5cd710") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:40:59.101694 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:59.101243    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f18fbf0-4d7d-4621-9260-199ef7cc0e0d-original-pull-secret podName:3f18fbf0-4d7d-4621-9260-199ef7cc0e0d nodeName:}" failed. No retries permitted until 2026-09-17 17:41:07.101233678 +0000 UTC m=+19.605236281 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3f18fbf0-4d7d-4621-9260-199ef7cc0e0d-original-pull-secret") pod "global-pull-secret-syncer-cfb52" (UID: "3f18fbf0-4d7d-4621-9260-199ef7cc0e0d") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:40:59.302635 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:59.302592    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvc8l\" (UniqueName: \"kubernetes.io/projected/bf919600-b3d9-49db-b561-1e02cb809474-kube-api-access-qvc8l\") pod \"network-check-target-ngfbr\" (UID: \"bf919600-b3d9-49db-b561-1e02cb809474\") " pod="openshift-network-diagnostics/network-check-target-ngfbr"
Sep 17 17:40:59.302824 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:59.302811    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:40:59.302890 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:59.302833    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:40:59.302890 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:59.302846    2569 projected.go:196] Error preparing data for projected volume kube-api-access-qvc8l for pod openshift-network-diagnostics/network-check-target-ngfbr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:40:59.302978 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:59.302909    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf919600-b3d9-49db-b561-1e02cb809474-kube-api-access-qvc8l podName:bf919600-b3d9-49db-b561-1e02cb809474 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:07.302891721 +0000 UTC m=+19.806894330 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-qvc8l" (UniqueName: "kubernetes.io/projected/bf919600-b3d9-49db-b561-1e02cb809474-kube-api-access-qvc8l") pod "network-check-target-ngfbr" (UID: "bf919600-b3d9-49db-b561-1e02cb809474") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:40:59.503014 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:59.502936    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cfb52"
Sep 17 17:40:59.503014 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:59.502937    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rzvhs"
Sep 17 17:40:59.504384 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:59.504163    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cfb52" podUID="3f18fbf0-4d7d-4621-9260-199ef7cc0e0d"
Sep 17 17:40:59.504384 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:59.504213    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rzvhs" podUID="caf785f7-c156-4b6f-8c82-4c1efe5cd710"
Sep 17 17:40:59.504384 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:40:59.504268    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ngfbr"
Sep 17 17:40:59.504384 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:40:59.504346    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ngfbr" podUID="bf919600-b3d9-49db-b561-1e02cb809474"
Sep 17 17:41:01.503554 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:01.503192    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rzvhs"
Sep 17 17:41:01.503554 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:41:01.503348    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rzvhs" podUID="caf785f7-c156-4b6f-8c82-4c1efe5cd710"
Sep 17 17:41:01.503554 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:01.503193    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cfb52"
Sep 17 17:41:01.503554 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:41:01.503445    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cfb52" podUID="3f18fbf0-4d7d-4621-9260-199ef7cc0e0d"
Sep 17 17:41:01.503554 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:01.503192    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ngfbr"
Sep 17 17:41:01.503554 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:41:01.503515    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ngfbr" podUID="bf919600-b3d9-49db-b561-1e02cb809474"
Sep 17 17:41:03.503065 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:03.502987    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cfb52"
Sep 17 17:41:03.503065 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:03.503017    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ngfbr"
Sep 17 17:41:03.503676 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:03.502992    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rzvhs"
Sep 17 17:41:03.503676 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:41:03.503200    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cfb52" podUID="3f18fbf0-4d7d-4621-9260-199ef7cc0e0d"
Sep 17 17:41:03.503676 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:41:03.503323    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ngfbr" podUID="bf919600-b3d9-49db-b561-1e02cb809474"
Sep 17 17:41:03.503676 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:41:03.503423    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rzvhs" podUID="caf785f7-c156-4b6f-8c82-4c1efe5cd710"
Sep 17 17:41:05.503791 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:05.503690    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cfb52"
Sep 17 17:41:05.503791 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:05.503735    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ngfbr"
Sep 17 17:41:05.504344 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:41:05.503846    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cfb52" podUID="3f18fbf0-4d7d-4621-9260-199ef7cc0e0d"
Sep 17 17:41:05.504344 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:05.503691    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rzvhs"
Sep 17 17:41:05.504344 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:41:05.504322    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rzvhs" podUID="caf785f7-c156-4b6f-8c82-4c1efe5cd710"
Sep 17 17:41:05.504497 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:41:05.504428    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ngfbr" podUID="bf919600-b3d9-49db-b561-1e02cb809474"
Sep 17 17:41:07.160344 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:07.160307    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caf785f7-c156-4b6f-8c82-4c1efe5cd710-metrics-certs\") pod \"network-metrics-daemon-rzvhs\" (UID: \"caf785f7-c156-4b6f-8c82-4c1efe5cd710\") " pod="openshift-multus/network-metrics-daemon-rzvhs"
Sep 17 17:41:07.160344 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:07.160352    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3f18fbf0-4d7d-4621-9260-199ef7cc0e0d-original-pull-secret\") pod \"global-pull-secret-syncer-cfb52\" (UID: \"3f18fbf0-4d7d-4621-9260-199ef7cc0e0d\") " pod="kube-system/global-pull-secret-syncer-cfb52"
Sep 17 17:41:07.160738 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:41:07.160442    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:07.160738 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:41:07.160445    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:07.160738 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:41:07.160491    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f18fbf0-4d7d-4621-9260-199ef7cc0e0d-original-pull-secret podName:3f18fbf0-4d7d-4621-9260-199ef7cc0e0d nodeName:}" failed. No retries permitted until 2026-09-17 17:41:23.160477934 +0000 UTC m=+35.664480537 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3f18fbf0-4d7d-4621-9260-199ef7cc0e0d-original-pull-secret") pod "global-pull-secret-syncer-cfb52" (UID: "3f18fbf0-4d7d-4621-9260-199ef7cc0e0d") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:07.160738 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:41:07.160505    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/caf785f7-c156-4b6f-8c82-4c1efe5cd710-metrics-certs podName:caf785f7-c156-4b6f-8c82-4c1efe5cd710 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:23.160498715 +0000 UTC m=+35.664501317 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/caf785f7-c156-4b6f-8c82-4c1efe5cd710-metrics-certs") pod "network-metrics-daemon-rzvhs" (UID: "caf785f7-c156-4b6f-8c82-4c1efe5cd710") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:07.362148 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:07.362104    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvc8l\" (UniqueName: \"kubernetes.io/projected/bf919600-b3d9-49db-b561-1e02cb809474-kube-api-access-qvc8l\") pod \"network-check-target-ngfbr\" (UID: \"bf919600-b3d9-49db-b561-1e02cb809474\") " pod="openshift-network-diagnostics/network-check-target-ngfbr"
Sep 17 17:41:07.362316 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:41:07.362292    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:41:07.362316 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:41:07.362315    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:41:07.362399 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:41:07.362327    2569 projected.go:196] Error preparing data for projected volume kube-api-access-qvc8l for pod openshift-network-diagnostics/network-check-target-ngfbr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:07.362399 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:41:07.362390    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf919600-b3d9-49db-b561-1e02cb809474-kube-api-access-qvc8l podName:bf919600-b3d9-49db-b561-1e02cb809474 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:23.362373928 +0000 UTC m=+35.866376547 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-qvc8l" (UniqueName: "kubernetes.io/projected/bf919600-b3d9-49db-b561-1e02cb809474-kube-api-access-qvc8l") pod "network-check-target-ngfbr" (UID: "bf919600-b3d9-49db-b561-1e02cb809474") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:07.502985 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:07.502901    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rzvhs"
Sep 17 17:41:07.503180 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:41:07.503051    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rzvhs" podUID="caf785f7-c156-4b6f-8c82-4c1efe5cd710"
Sep 17 17:41:07.503180 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:07.503105    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cfb52"
Sep 17 17:41:07.503180 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:41:07.503173    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cfb52" podUID="3f18fbf0-4d7d-4621-9260-199ef7cc0e0d"
Sep 17 17:41:07.503362 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:07.503211    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ngfbr"
Sep 17 17:41:07.503362 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:41:07.503272    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ngfbr" podUID="bf919600-b3d9-49db-b561-1e02cb809474"
Sep 17 17:41:09.503672 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:09.503638    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ngfbr"
Sep 17 17:41:09.504027 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:09.503709    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rzvhs"
Sep 17 17:41:09.504027 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:41:09.503716    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ngfbr" podUID="bf919600-b3d9-49db-b561-1e02cb809474"
Sep 17 17:41:09.504027 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:41:09.503818    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rzvhs" podUID="caf785f7-c156-4b6f-8c82-4c1efe5cd710"
Sep 17 17:41:09.504027 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:09.503862    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cfb52"
Sep 17 17:41:09.504027 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:41:09.503908    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cfb52" podUID="3f18fbf0-4d7d-4621-9260-199ef7cc0e0d"
Sep 17 17:41:10.615384 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:10.615130    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nbmpx" event={"ID":"1b398422-1bbe-4f75-a7e6-a641cce3f5e8","Type":"ContainerStarted","Data":"6be474fe3133d877276318fc3d045f102a1b5e46c4ba4c1f27f0bf164f9f873c"}
Sep 17 17:41:10.617949 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:10.617930    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k4mqn_6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff/ovn-acl-logging/0.log"
Sep 17 17:41:10.618283 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:10.618263    2569 generic.go:356] "Generic (PLEG): container finished" podID="6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff" containerID="dca1616be506a904cb142c899e511cd1e41e8f3cbe0a3dc35a562672ff9a13aa" exitCode=1
Sep 17 17:41:10.618358 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:10.618314    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn" event={"ID":"6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff","Type":"ContainerStarted","Data":"647cb899caf69ee2c191b8fc6cde54ba5fb667f5236735d1f3a926c8c77a47bb"}
Sep 17 17:41:10.618411 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:10.618393    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn" event={"ID":"6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff","Type":"ContainerStarted","Data":"91f44b1374e8d9f74a2a57f2a9cc33c8f58b87d1abfab53588a7bd5413740995"}
Sep 17 17:41:10.618468 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:10.618445    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn" event={"ID":"6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff","Type":"ContainerStarted","Data":"9dc646fb286ef2c54b3e259d987e49ff46ed3cdbc43bd19ea37fff86410005d3"}
Sep 17 17:41:10.618517 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:10.618494    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn" event={"ID":"6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff","Type":"ContainerStarted","Data":"0622371da099eb65662ad6d2995704da6c5c8fd0dd26e4e2a7f8a7b380317cc8"}
Sep 17 17:41:10.618565 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:10.618541    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn" event={"ID":"6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff","Type":"ContainerDied","Data":"dca1616be506a904cb142c899e511cd1e41e8f3cbe0a3dc35a562672ff9a13aa"}
Sep 17 17:41:10.618613 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:10.618592    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn" event={"ID":"6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff","Type":"ContainerStarted","Data":"732f8d5e1385289068730fc76795877100ddef086c058b9ab1beb94e0e826444"}
Sep 17 17:41:10.619656 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:10.619634    2569 generic.go:356] "Generic (PLEG): container finished" podID="911990d2-9984-4b04-979c-1380a8bdb7a2" containerID="ddf3c44dac08f2fa8283e8f708ea42c5a8ef28c99b7df1a7d688e978d2fbef72" exitCode=0
Sep 17 17:41:10.619748 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:10.619659    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mwh5v" event={"ID":"911990d2-9984-4b04-979c-1380a8bdb7a2","Type":"ContainerDied","Data":"ddf3c44dac08f2fa8283e8f708ea42c5a8ef28c99b7df1a7d688e978d2fbef72"}
Sep 17 17:41:10.621157 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:10.621066    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qn9k7" event={"ID":"42dd1d14-0348-40e6-824e-90b1763291f1","Type":"ContainerStarted","Data":"05bac8d3ae4c17e7ac6ce45fb064717cf52e70706042719d839a056126df36a8"}
Sep 17 17:41:10.622522 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:10.622503    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qn6lp" event={"ID":"372b5295-ddc7-4999-9b48-c6cfb84d8e83","Type":"ContainerStarted","Data":"c62d6a63e8865f7aac8bbf7a22255f895338dd9c7318cac6d54f6fb804d2c941"}
Sep 17 17:41:10.623944 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:10.623927    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9psk" event={"ID":"3809c4cd-e5d9-4647-95f8-1abbeefd438e","Type":"ContainerStarted","Data":"eb2fb3c141dfa45d73ac630e14d4f5817cd2ce78fcbb528e631a3079d47a8e5d"}
Sep 17 17:41:10.625369 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:10.625349    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-csv2f" event={"ID":"d40ba619-04c7-482d-b46f-ec561e79987c","Type":"ContainerStarted","Data":"43b21bf9452651f748c9bb71affd149aef2f7508605fce5c873312893c2d873b"}
Sep 17 17:41:10.627118 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:10.627100    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fjfrk" event={"ID":"f847c3c6-f2ab-43a0-b467-27db788f5b9f","Type":"ContainerStarted","Data":"3fda5ac6da36074ed7c2e34bc213f619a31926265d16cda873861770f952b29f"}
Sep 17 17:41:10.631494 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:10.631460    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.ec2.internal" podStartSLOduration=20.631449938 podStartE2EDuration="20.631449938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:40:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:40:54.598449602 +0000 UTC m=+7.102452229" watchObservedRunningTime="2026-09-17 17:41:10.631449938 +0000 UTC m=+23.135452562"
Sep 17 17:41:10.631739 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:10.631720    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-nbmpx" podStartSLOduration=4.492980136 podStartE2EDuration="21.631716313s" totalImagesPullingTime="17.138736177s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:40:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:10.631319444 +0000 UTC m=+23.135322068" watchObservedRunningTime="2026-09-17 17:41:10.631716313 +0000 UTC m=+23.135718938"
Sep 17 17:41:10.651455 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:10.651414    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-qn6lp" podStartSLOduration=4.552363436 podStartE2EDuration="21.651397148s" totalImagesPullingTime="17.099033712s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:40:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:10.650754763 +0000 UTC m=+23.154757388" watchObservedRunningTime="2026-09-17 17:41:10.651397148 +0000 UTC m=+23.155399776"
Sep 17 17:41:10.668535 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:10.668495    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-fjfrk" podStartSLOduration=4.221007237 podStartE2EDuration="21.668482249s" totalImagesPullingTime="17.447475012s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:40:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:10.668156562 +0000 UTC m=+23.172159188" watchObservedRunningTime="2026-09-17 17:41:10.668482249 +0000 UTC m=+23.172484906"
Sep 17 17:41:10.685612 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:10.685564    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-csv2f" podStartSLOduration=4.517881779 podStartE2EDuration="21.685544335s" totalImagesPullingTime="17.167662556s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:40:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:10.685172129 +0000 UTC m=+23.189174766" watchObservedRunningTime="2026-09-17 17:41:10.685544335 +0000 UTC m=+23.189546962"
Sep 17 17:41:10.700400 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:10.700361    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-qn9k7" podStartSLOduration=4.535596811 podStartE2EDuration="21.700347456s" totalImagesPullingTime="17.164750645s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:40:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:10.699853501 +0000 UTC m=+23.203856156" watchObservedRunningTime="2026-09-17 17:41:10.700347456 +0000 UTC m=+23.204350080"
Sep 17 17:41:11.289940 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:11.289911    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-qn6lp"
Sep 17 17:41:11.290575 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:11.290556    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-qn6lp"
Sep 17 17:41:11.382978 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:11.382954    2569 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 17:41:11.503149 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:11.503112    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ngfbr"
Sep 17 17:41:11.503321 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:11.503121    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rzvhs"
Sep 17 17:41:11.503321 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:41:11.503225    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ngfbr" podUID="bf919600-b3d9-49db-b561-1e02cb809474"
Sep 17 17:41:11.503410 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:41:11.503329    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rzvhs" podUID="caf785f7-c156-4b6f-8c82-4c1efe5cd710"
Sep 17 17:41:11.503410 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:11.503122    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cfb52"
Sep 17 17:41:11.503479 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:41:11.503437    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cfb52" podUID="3f18fbf0-4d7d-4621-9260-199ef7cc0e0d"
Sep 17 17:41:11.517638 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:11.517563    2569 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T17:41:11.382974425Z","UUID":"cbd89386-4e53-4258-a4ec-1298c2b2a98b","Handler":null,"Name":"","Endpoint":""}
Sep 17 17:41:11.518917 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:11.518902    2569 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 17:41:11.519001 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:11.518935    2569 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 17:41:11.631336 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:11.631134    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9psk" event={"ID":"3809c4cd-e5d9-4647-95f8-1abbeefd438e","Type":"ContainerStarted","Data":"cab14bfe802a56fc198f64943150f640c6ada9ff8f81f634839c5b8315e370d1"}
Sep 17 17:41:11.632648 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:11.632602    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hfw2c" event={"ID":"bc71d7ac-eae9-4ef5-a2f8-f0292bdc51b6","Type":"ContainerStarted","Data":"5ebbdee61ddc6bee27497fcf5ed71db6c515d76529a13c3b0de64c25e29f0a3d"}
Sep 17 17:41:11.648392 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:11.648347    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-hfw2c" podStartSLOduration=5.483099603 podStartE2EDuration="22.648330047s" totalImagesPullingTime="17.165230444s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:40:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:11.648262013 +0000 UTC m=+24.152264659" watchObservedRunningTime="2026-09-17 17:41:11.648330047 +0000 UTC m=+24.152332673"
Sep 17 17:41:12.637837 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:12.637800    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9psk" event={"ID":"3809c4cd-e5d9-4647-95f8-1abbeefd438e","Type":"ContainerStarted","Data":"9c7213c05917f8ac09bb0e2f58a6ee7f4c5dec0b204c37c684f66bb5fd612583"}
Sep 17 17:41:12.641762 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:12.641737    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k4mqn_6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff/ovn-acl-logging/0.log"
Sep 17 17:41:12.642201 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:12.642184    2569 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 17:41:12.642345 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:12.642198    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn" event={"ID":"6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff","Type":"ContainerStarted","Data":"dfcab243fc881b51672cfd60a68801cb6cf7f82e782c64e62698df7cacdb542b"}
Sep 17 17:41:12.666454 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:12.666410    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9psk" podStartSLOduration=4.138437091 podStartE2EDuration="23.666395264s" totalImagesPullingTime="19.527958173s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:40:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:12.664290208 +0000 UTC m=+25.168292833" watchObservedRunningTime="2026-09-17 17:41:12.666395264 +0000 UTC m=+25.170397888"
Sep 17 17:41:13.503532 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:13.503463    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ngfbr"
Sep 17 17:41:13.503532 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:13.503506    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rzvhs"
Sep 17 17:41:13.503743 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:13.503472    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cfb52"
Sep 17 17:41:13.503743 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:41:13.503593    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ngfbr" podUID="bf919600-b3d9-49db-b561-1e02cb809474"
Sep 17 17:41:13.503743 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:41:13.503677    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cfb52" podUID="3f18fbf0-4d7d-4621-9260-199ef7cc0e0d"
Sep 17 17:41:13.503867 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:41:13.503755    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rzvhs" podUID="caf785f7-c156-4b6f-8c82-4c1efe5cd710"
Sep 17 17:41:15.503432 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:15.503270    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rzvhs"
Sep 17 17:41:15.504063 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:15.503273    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cfb52"
Sep 17 17:41:15.504063 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:41:15.503518    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rzvhs" podUID="caf785f7-c156-4b6f-8c82-4c1efe5cd710"
Sep 17 17:41:15.504063 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:15.503312    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ngfbr"
Sep 17 17:41:15.504063 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:41:15.503631    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cfb52" podUID="3f18fbf0-4d7d-4621-9260-199ef7cc0e0d"
Sep 17 17:41:15.504063 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:41:15.503640    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ngfbr" podUID="bf919600-b3d9-49db-b561-1e02cb809474"
Sep 17 17:41:15.652919 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:15.652856    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k4mqn_6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff/ovn-acl-logging/0.log"
Sep 17 17:41:15.653292 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:15.653265    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn" event={"ID":"6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff","Type":"ContainerStarted","Data":"d7a9cb6c2ec8af64546cd440c4f4b3b978dda26c6c37fda32d41910f07a9a8ce"}
Sep 17 17:41:15.653594 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:15.653569    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn"
Sep 17 17:41:15.653697 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:15.653642    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn"
Sep 17 17:41:15.653816 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:15.653800    2569 scope.go:117] "RemoveContainer" containerID="dca1616be506a904cb142c899e511cd1e41e8f3cbe0a3dc35a562672ff9a13aa"
Sep 17 17:41:15.655316 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:15.655295    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mwh5v" event={"ID":"911990d2-9984-4b04-979c-1380a8bdb7a2","Type":"ContainerStarted","Data":"c19b83c34ee77e0930591cfd5765d3af057345cc5737f6543b5bae792cb77bc4"}
Sep 17 17:41:15.670237 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:15.670212    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn"
Sep 17 17:41:16.661504 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:16.661477    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k4mqn_6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff/ovn-acl-logging/0.log"
Sep 17 17:41:16.661944 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:16.661849    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn" event={"ID":"6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff","Type":"ContainerStarted","Data":"7999862389b2e181409d7d5061161f43fce84c1883f4bfa22048ace68398a86a"}
Sep 17 17:41:16.662201 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:16.662183    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn"
Sep 17 17:41:16.664260 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:16.664232    2569 generic.go:356] "Generic (PLEG): container finished" podID="911990d2-9984-4b04-979c-1380a8bdb7a2" containerID="c19b83c34ee77e0930591cfd5765d3af057345cc5737f6543b5bae792cb77bc4" exitCode=0
Sep 17 17:41:16.664373 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:16.664274    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mwh5v" event={"ID":"911990d2-9984-4b04-979c-1380a8bdb7a2","Type":"ContainerDied","Data":"c19b83c34ee77e0930591cfd5765d3af057345cc5737f6543b5bae792cb77bc4"}
Sep 17 17:41:16.679113 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:16.679086    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn"
Sep 17 17:41:16.697578 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:16.697528    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn" podStartSLOduration=10.515869901 podStartE2EDuration="27.69751243s" totalImagesPullingTime="17.181642529s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:40:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:16.695719842 +0000 UTC m=+29.199722466" watchObservedRunningTime="2026-09-17 17:41:16.69751243 +0000 UTC m=+29.201515072"
Sep 17 17:41:17.092877 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:17.092679    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cfb52"]
Sep 17 17:41:17.093058 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:17.092961    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cfb52"
Sep 17 17:41:17.093058 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:41:17.093043    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cfb52" podUID="3f18fbf0-4d7d-4621-9260-199ef7cc0e0d"
Sep 17 17:41:17.096362 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:17.096330    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ngfbr"]
Sep 17 17:41:17.096498 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:17.096445    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ngfbr"
Sep 17 17:41:17.096583 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:41:17.096540    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ngfbr" podUID="bf919600-b3d9-49db-b561-1e02cb809474"
Sep 17 17:41:17.096931 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:17.096912    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rzvhs"]
Sep 17 17:41:17.097028 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:17.097011    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rzvhs"
Sep 17 17:41:17.097123 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:41:17.097106    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rzvhs" podUID="caf785f7-c156-4b6f-8c82-4c1efe5cd710"
Sep 17 17:41:18.503346 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:18.503306    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ngfbr"
Sep 17 17:41:18.503346 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:18.503306    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rzvhs"
Sep 17 17:41:18.503790 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:18.503332    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cfb52"
Sep 17 17:41:18.503790 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:41:18.503422    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ngfbr" podUID="bf919600-b3d9-49db-b561-1e02cb809474"
Sep 17 17:41:18.503790 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:41:18.503494    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cfb52" podUID="3f18fbf0-4d7d-4621-9260-199ef7cc0e0d"
Sep 17 17:41:18.503790 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:41:18.503584    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rzvhs" podUID="caf785f7-c156-4b6f-8c82-4c1efe5cd710"
Sep 17 17:41:18.671733 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:18.671699    2569 generic.go:356] "Generic (PLEG): container finished" podID="911990d2-9984-4b04-979c-1380a8bdb7a2" containerID="8d7fc47cf868395745bb92ddb2853da4a7b1b601e916c3e37b82c412e2ec888f" exitCode=0
Sep 17 17:41:18.671890 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:18.671791    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mwh5v" event={"ID":"911990d2-9984-4b04-979c-1380a8bdb7a2","Type":"ContainerDied","Data":"8d7fc47cf868395745bb92ddb2853da4a7b1b601e916c3e37b82c412e2ec888f"}
Sep 17 17:41:19.677506 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:19.677479    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mwh5v" event={"ID":"911990d2-9984-4b04-979c-1380a8bdb7a2","Type":"ContainerStarted","Data":"ab268646a1e1a996d46983bc5cccecf677d9cc27d5e091332f87e5273f4398dd"}
Sep 17 17:41:20.503087 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:20.503054    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ngfbr"
Sep 17 17:41:20.503261 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:20.503054    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cfb52"
Sep 17 17:41:20.503261 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:41:20.503195    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ngfbr" podUID="bf919600-b3d9-49db-b561-1e02cb809474"
Sep 17 17:41:20.503261 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:20.503054    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rzvhs"
Sep 17 17:41:20.503261 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:41:20.503250    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cfb52" podUID="3f18fbf0-4d7d-4621-9260-199ef7cc0e0d"
Sep 17 17:41:20.503482 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:41:20.503325    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rzvhs" podUID="caf785f7-c156-4b6f-8c82-4c1efe5cd710"
Sep 17 17:41:20.682731 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:20.682688    2569 generic.go:356] "Generic (PLEG): container finished" podID="911990d2-9984-4b04-979c-1380a8bdb7a2" containerID="ab268646a1e1a996d46983bc5cccecf677d9cc27d5e091332f87e5273f4398dd" exitCode=0
Sep 17 17:41:20.682731 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:20.682734    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mwh5v" event={"ID":"911990d2-9984-4b04-979c-1380a8bdb7a2","Type":"ContainerDied","Data":"ab268646a1e1a996d46983bc5cccecf677d9cc27d5e091332f87e5273f4398dd"}
Sep 17 17:41:21.329352 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:21.329312    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-qn6lp"
Sep 17 17:41:21.329531 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:21.329485    2569 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 17:41:21.329986 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:21.329956    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qn6lp"
Sep 17 17:41:22.492209 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:22.492174    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-14.ec2.internal" event="NodeReady"
Sep 17 17:41:22.492648 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:22.492322    2569 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 17:41:22.503014 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:22.502934    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ngfbr"
Sep 17 17:41:22.503014 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:22.502969    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cfb52"
Sep 17 17:41:22.503261 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:22.503220    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rzvhs"
Sep 17 17:41:22.506175 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:22.506147    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 17:41:22.507611 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:22.507580    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 17:41:22.507736 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:22.507690    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 17:41:22.507736 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:22.507711    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-b8q2r\""
Sep 17 17:41:22.507846 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:22.507811    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nqf6f\""
Sep 17 17:41:22.507940 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:22.507918    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 17:41:22.543485 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:22.543452    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-6lkgm"]
Sep 17 17:41:22.558897 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:22.558866    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-78r7n"]
Sep 17 17:41:22.559108 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:22.559062    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6lkgm"
Sep 17 17:41:22.562885 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:22.562854    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4db82\""
Sep 17 17:41:22.563158 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:22.563132    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 17:41:22.563158 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:22.563150    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 17:41:22.573243 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:22.573222    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6lkgm"]
Sep 17 17:41:22.573360 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:22.573257    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-78r7n"]
Sep 17 17:41:22.573405 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:22.573356    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-78r7n"
Sep 17 17:41:22.576511 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:22.576464    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-2krvt\""
Sep 17 17:41:22.576621 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:22.576513    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 17:41:22.576621 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:22.576530    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 17:41:22.576975 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:22.576957    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 17:41:22.666424 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:22.666331    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/023f4b84-91bb-4103-97cb-5123dc08e459-config-volume\") pod \"dns-default-6lkgm\" (UID: \"023f4b84-91bb-4103-97cb-5123dc08e459\") " pod="openshift-dns/dns-default-6lkgm"
Sep 17 17:41:22.666592 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:22.666455    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvlc6\" (UniqueName: \"kubernetes.io/projected/5b1aeab9-ca3c-4ae4-82dc-fae81696a23a-kube-api-access-zvlc6\") pod \"ingress-canary-78r7n\" (UID: \"5b1aeab9-ca3c-4ae4-82dc-fae81696a23a\") " pod="openshift-ingress-canary/ingress-canary-78r7n"
Sep 17 17:41:22.666592 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:22.666519    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b1aeab9-ca3c-4ae4-82dc-fae81696a23a-cert\") pod \"ingress-canary-78r7n\" (UID: \"5b1aeab9-ca3c-4ae4-82dc-fae81696a23a\") " pod="openshift-ingress-canary/ingress-canary-78r7n"
Sep 17 17:41:22.666592 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:22.666546    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/023f4b84-91bb-4103-97cb-5123dc08e459-metrics-tls\") pod \"dns-default-6lkgm\" (UID: \"023f4b84-91bb-4103-97cb-5123dc08e459\") " pod="openshift-dns/dns-default-6lkgm"
Sep 17 17:41:22.666592 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:22.666572    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v7vw\" (UniqueName: \"kubernetes.io/projected/023f4b84-91bb-4103-97cb-5123dc08e459-kube-api-access-8v7vw\") pod \"dns-default-6lkgm\" (UID: \"023f4b84-91bb-4103-97cb-5123dc08e459\") " pod="openshift-dns/dns-default-6lkgm"
Sep 17 17:41:22.666785 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:22.666618    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/023f4b84-91bb-4103-97cb-5123dc08e459-tmp-dir\") pod \"dns-default-6lkgm\" (UID: \"023f4b84-91bb-4103-97cb-5123dc08e459\") " pod="openshift-dns/dns-default-6lkgm"
Sep 17 17:41:22.766953 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:22.766917    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/023f4b84-91bb-4103-97cb-5123dc08e459-config-volume\") pod \"dns-default-6lkgm\" (UID: \"023f4b84-91bb-4103-97cb-5123dc08e459\") " pod="openshift-dns/dns-default-6lkgm"
Sep 17 17:41:22.767166 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:22.766980    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvlc6\" (UniqueName: \"kubernetes.io/projected/5b1aeab9-ca3c-4ae4-82dc-fae81696a23a-kube-api-access-zvlc6\") pod \"ingress-canary-78r7n\" (UID: \"5b1aeab9-ca3c-4ae4-82dc-fae81696a23a\") " pod="openshift-ingress-canary/ingress-canary-78r7n"
Sep 17 17:41:22.767166 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:22.767021    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b1aeab9-ca3c-4ae4-82dc-fae81696a23a-cert\") pod \"ingress-canary-78r7n\" (UID: \"5b1aeab9-ca3c-4ae4-82dc-fae81696a23a\") " pod="openshift-ingress-canary/ingress-canary-78r7n"
Sep 17 17:41:22.767166 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:22.767042    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/023f4b84-91bb-4103-97cb-5123dc08e459-metrics-tls\") pod \"dns-default-6lkgm\" (UID: \"023f4b84-91bb-4103-97cb-5123dc08e459\") " pod="openshift-dns/dns-default-6lkgm"
Sep 17 17:41:22.767166 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:22.767068    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v7vw\" (UniqueName: \"kubernetes.io/projected/023f4b84-91bb-4103-97cb-5123dc08e459-kube-api-access-8v7vw\") pod \"dns-default-6lkgm\" (UID: \"023f4b84-91bb-4103-97cb-5123dc08e459\") " pod="openshift-dns/dns-default-6lkgm"
Sep 17 17:41:22.767166 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:22.767117    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/023f4b84-91bb-4103-97cb-5123dc08e459-tmp-dir\") pod \"dns-default-6lkgm\" (UID: \"023f4b84-91bb-4103-97cb-5123dc08e459\") " pod="openshift-dns/dns-default-6lkgm"
Sep 17 17:41:22.767427 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:41:22.767178    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:41:22.767427 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:41:22.767258    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/023f4b84-91bb-4103-97cb-5123dc08e459-metrics-tls podName:023f4b84-91bb-4103-97cb-5123dc08e459 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:23.267236466 +0000 UTC m=+35.771239100 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/023f4b84-91bb-4103-97cb-5123dc08e459-metrics-tls") pod "dns-default-6lkgm" (UID: "023f4b84-91bb-4103-97cb-5123dc08e459") : secret "dns-default-metrics-tls" not found
Sep 17 17:41:22.767427 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:22.767409    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/023f4b84-91bb-4103-97cb-5123dc08e459-tmp-dir\") pod \"dns-default-6lkgm\" (UID: \"023f4b84-91bb-4103-97cb-5123dc08e459\") " pod="openshift-dns/dns-default-6lkgm"
Sep 17 17:41:22.767579 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:22.767562    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/023f4b84-91bb-4103-97cb-5123dc08e459-config-volume\") pod \"dns-default-6lkgm\" (UID: \"023f4b84-91bb-4103-97cb-5123dc08e459\") " pod="openshift-dns/dns-default-6lkgm"
Sep 17 17:41:22.771540 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:22.771345    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b1aeab9-ca3c-4ae4-82dc-fae81696a23a-cert\") pod \"ingress-canary-78r7n\" (UID: \"5b1aeab9-ca3c-4ae4-82dc-fae81696a23a\") " pod="openshift-ingress-canary/ingress-canary-78r7n"
Sep 17 17:41:22.777123 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:22.777096    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v7vw\" (UniqueName: \"kubernetes.io/projected/023f4b84-91bb-4103-97cb-5123dc08e459-kube-api-access-8v7vw\") pod \"dns-default-6lkgm\" (UID: \"023f4b84-91bb-4103-97cb-5123dc08e459\") " pod="openshift-dns/dns-default-6lkgm"
Sep 17 17:41:22.777325 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:22.777306    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvlc6\" (UniqueName: \"kubernetes.io/projected/5b1aeab9-ca3c-4ae4-82dc-fae81696a23a-kube-api-access-zvlc6\") pod \"ingress-canary-78r7n\" (UID: \"5b1aeab9-ca3c-4ae4-82dc-fae81696a23a\") " pod="openshift-ingress-canary/ingress-canary-78r7n"
Sep 17 17:41:22.883220 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:22.883176    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-78r7n"
Sep 17 17:41:23.170190 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:23.170154    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caf785f7-c156-4b6f-8c82-4c1efe5cd710-metrics-certs\") pod \"network-metrics-daemon-rzvhs\" (UID: \"caf785f7-c156-4b6f-8c82-4c1efe5cd710\") " pod="openshift-multus/network-metrics-daemon-rzvhs"
Sep 17 17:41:23.170401 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:23.170222    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3f18fbf0-4d7d-4621-9260-199ef7cc0e0d-original-pull-secret\") pod \"global-pull-secret-syncer-cfb52\" (UID: \"3f18fbf0-4d7d-4621-9260-199ef7cc0e0d\") " pod="kube-system/global-pull-secret-syncer-cfb52"
Sep 17 17:41:23.170401 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:41:23.170281    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 17:41:23.170401 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:41:23.170351    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/caf785f7-c156-4b6f-8c82-4c1efe5cd710-metrics-certs podName:caf785f7-c156-4b6f-8c82-4c1efe5cd710 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:55.170335597 +0000 UTC m=+67.674338224 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/caf785f7-c156-4b6f-8c82-4c1efe5cd710-metrics-certs") pod "network-metrics-daemon-rzvhs" (UID: "caf785f7-c156-4b6f-8c82-4c1efe5cd710") : secret "metrics-daemon-secret" not found
Sep 17 17:41:23.173896 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:23.173873    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3f18fbf0-4d7d-4621-9260-199ef7cc0e0d-original-pull-secret\") pod \"global-pull-secret-syncer-cfb52\" (UID: \"3f18fbf0-4d7d-4621-9260-199ef7cc0e0d\") " pod="kube-system/global-pull-secret-syncer-cfb52"
Sep 17 17:41:23.199474 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:23.199429    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-78r7n"]
Sep 17 17:41:23.203366 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:41:23.203339    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b1aeab9_ca3c_4ae4_82dc_fae81696a23a.slice/crio-bfc43d82a7d9e5237edda4efec929b20076821ee5ddbd5a5c72470de42f437de WatchSource:0}: Error finding container bfc43d82a7d9e5237edda4efec929b20076821ee5ddbd5a5c72470de42f437de: Status 404 returned error can't find the container with id bfc43d82a7d9e5237edda4efec929b20076821ee5ddbd5a5c72470de42f437de
Sep 17 17:41:23.271517 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:23.271415    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/023f4b84-91bb-4103-97cb-5123dc08e459-metrics-tls\") pod \"dns-default-6lkgm\" (UID: \"023f4b84-91bb-4103-97cb-5123dc08e459\") " pod="openshift-dns/dns-default-6lkgm"
Sep 17 17:41:23.271671 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:41:23.271567    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:41:23.271671 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:41:23.271632    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/023f4b84-91bb-4103-97cb-5123dc08e459-metrics-tls podName:023f4b84-91bb-4103-97cb-5123dc08e459 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:24.271617207 +0000 UTC m=+36.775619810 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/023f4b84-91bb-4103-97cb-5123dc08e459-metrics-tls") pod "dns-default-6lkgm" (UID: "023f4b84-91bb-4103-97cb-5123dc08e459") : secret "dns-default-metrics-tls" not found
Sep 17 17:41:23.372526 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:23.372490    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvc8l\" (UniqueName: \"kubernetes.io/projected/bf919600-b3d9-49db-b561-1e02cb809474-kube-api-access-qvc8l\") pod \"network-check-target-ngfbr\" (UID: \"bf919600-b3d9-49db-b561-1e02cb809474\") " pod="openshift-network-diagnostics/network-check-target-ngfbr"
Sep 17 17:41:23.375723 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:23.375699    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvc8l\" (UniqueName: \"kubernetes.io/projected/bf919600-b3d9-49db-b561-1e02cb809474-kube-api-access-qvc8l\") pod \"network-check-target-ngfbr\" (UID: \"bf919600-b3d9-49db-b561-1e02cb809474\") " pod="openshift-network-diagnostics/network-check-target-ngfbr"
Sep 17 17:41:23.415580 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:23.415537    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ngfbr"
Sep 17 17:41:23.423861 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:23.423827    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cfb52"
Sep 17 17:41:23.691415 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:23.691376    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-78r7n" event={"ID":"5b1aeab9-ca3c-4ae4-82dc-fae81696a23a","Type":"ContainerStarted","Data":"bfc43d82a7d9e5237edda4efec929b20076821ee5ddbd5a5c72470de42f437de"}
Sep 17 17:41:23.721014 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:23.720984    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ngfbr"]
Sep 17 17:41:23.723319 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:41:23.723289    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf919600_b3d9_49db_b561_1e02cb809474.slice/crio-91ce7aa4f77b848b89b913ff9140d029352171e5c1edb8e0d019318313c695f6 WatchSource:0}: Error finding container 91ce7aa4f77b848b89b913ff9140d029352171e5c1edb8e0d019318313c695f6: Status 404 returned error can't find the container with id 91ce7aa4f77b848b89b913ff9140d029352171e5c1edb8e0d019318313c695f6
Sep 17 17:41:23.746238 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:23.746201    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cfb52"]
Sep 17 17:41:23.750172 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:41:23.750140    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f18fbf0_4d7d_4621_9260_199ef7cc0e0d.slice/crio-aaf26c18f15b6831a599846de613c40727279757fd21ba85a9cb02ac4eb550f8 WatchSource:0}: Error finding container aaf26c18f15b6831a599846de613c40727279757fd21ba85a9cb02ac4eb550f8: Status 404 returned error can't find the container with id aaf26c18f15b6831a599846de613c40727279757fd21ba85a9cb02ac4eb550f8
Sep 17 17:41:24.278914 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:24.278872    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/023f4b84-91bb-4103-97cb-5123dc08e459-metrics-tls\") pod \"dns-default-6lkgm\" (UID: \"023f4b84-91bb-4103-97cb-5123dc08e459\") " pod="openshift-dns/dns-default-6lkgm"
Sep 17 17:41:24.279275 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:41:24.279047    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:41:24.279275 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:41:24.279137    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/023f4b84-91bb-4103-97cb-5123dc08e459-metrics-tls podName:023f4b84-91bb-4103-97cb-5123dc08e459 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:26.2791142 +0000 UTC m=+38.783116808 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/023f4b84-91bb-4103-97cb-5123dc08e459-metrics-tls") pod "dns-default-6lkgm" (UID: "023f4b84-91bb-4103-97cb-5123dc08e459") : secret "dns-default-metrics-tls" not found
Sep 17 17:41:24.697885 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:24.697779    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cfb52" event={"ID":"3f18fbf0-4d7d-4621-9260-199ef7cc0e0d","Type":"ContainerStarted","Data":"aaf26c18f15b6831a599846de613c40727279757fd21ba85a9cb02ac4eb550f8"}
Sep 17 17:41:24.699853 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:24.699792    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ngfbr" event={"ID":"bf919600-b3d9-49db-b561-1e02cb809474","Type":"ContainerStarted","Data":"91ce7aa4f77b848b89b913ff9140d029352171e5c1edb8e0d019318313c695f6"}
Sep 17 17:41:26.297438 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:26.297394    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/023f4b84-91bb-4103-97cb-5123dc08e459-metrics-tls\") pod \"dns-default-6lkgm\" (UID: \"023f4b84-91bb-4103-97cb-5123dc08e459\") " pod="openshift-dns/dns-default-6lkgm"
Sep 17 17:41:26.297925 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:41:26.297584    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:41:26.297925 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:41:26.297672    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/023f4b84-91bb-4103-97cb-5123dc08e459-metrics-tls podName:023f4b84-91bb-4103-97cb-5123dc08e459 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:30.297642664 +0000 UTC m=+42.801645267 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/023f4b84-91bb-4103-97cb-5123dc08e459-metrics-tls") pod "dns-default-6lkgm" (UID: "023f4b84-91bb-4103-97cb-5123dc08e459") : secret "dns-default-metrics-tls" not found
Sep 17 17:41:30.331631 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:30.331358    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/023f4b84-91bb-4103-97cb-5123dc08e459-metrics-tls\") pod \"dns-default-6lkgm\" (UID: \"023f4b84-91bb-4103-97cb-5123dc08e459\") " pod="openshift-dns/dns-default-6lkgm"
Sep 17 17:41:30.331631 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:41:30.331501    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:41:30.332174 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:41:30.331636    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/023f4b84-91bb-4103-97cb-5123dc08e459-metrics-tls podName:023f4b84-91bb-4103-97cb-5123dc08e459 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:38.331616112 +0000 UTC m=+50.835618715 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/023f4b84-91bb-4103-97cb-5123dc08e459-metrics-tls") pod "dns-default-6lkgm" (UID: "023f4b84-91bb-4103-97cb-5123dc08e459") : secret "dns-default-metrics-tls" not found
Sep 17 17:41:30.717118 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:30.717069    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-78r7n" event={"ID":"5b1aeab9-ca3c-4ae4-82dc-fae81696a23a","Type":"ContainerStarted","Data":"24c7facb1eae49239e00c6af308bf13b3944f6467934c06ef85818d7ab51a516"}
Sep 17 17:41:30.718669 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:30.718643    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cfb52" event={"ID":"3f18fbf0-4d7d-4621-9260-199ef7cc0e0d","Type":"ContainerStarted","Data":"e32151e0c85bd13e2e1342678e54468dbb710aa96ab848aec4e11f3d39529e49"}
Sep 17 17:41:30.721402 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:30.721364    2569 generic.go:356] "Generic (PLEG): container finished" podID="911990d2-9984-4b04-979c-1380a8bdb7a2" containerID="1bb5100077615d4ad3ea2e16785470d964e9da06f20f435618f03a42fd981794" exitCode=0
Sep 17 17:41:30.721530 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:30.721445    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mwh5v" event={"ID":"911990d2-9984-4b04-979c-1380a8bdb7a2","Type":"ContainerDied","Data":"1bb5100077615d4ad3ea2e16785470d964e9da06f20f435618f03a42fd981794"}
Sep 17 17:41:30.722971 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:30.722949    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ngfbr" event={"ID":"bf919600-b3d9-49db-b561-1e02cb809474","Type":"ContainerStarted","Data":"0ff2b212a6d05cb7c26a31064de222b9327b9081a6d6e9ad577060a2edeb8ceb"}
Sep 17 17:41:30.723125 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:30.723109    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ngfbr"
Sep 17 17:41:30.763712 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:30.763655    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-cfb52" podStartSLOduration=35.748977128 podStartE2EDuration="41.763636207s" totalImagesPullingTime="6.014659079s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:40:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:30.762711363 +0000 UTC m=+43.266713990" watchObservedRunningTime="2026-09-17 17:41:30.763636207 +0000 UTC m=+43.267638831"
Sep 17 17:41:30.764413 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:30.764375    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-78r7n" podStartSLOduration=2.219169869 podStartE2EDuration="8.764361028s" totalImagesPullingTime="6.545191159s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:41:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:30.741485424 +0000 UTC m=+43.245488048" watchObservedRunningTime="2026-09-17 17:41:30.764361028 +0000 UTC m=+43.268363702"
Sep 17 17:41:30.813276 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:30.813222    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-ngfbr" podStartSLOduration=35.782570551 podStartE2EDuration="41.813202565s" totalImagesPullingTime="6.030632014s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:40:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:30.812054587 +0000 UTC m=+43.316057213" watchObservedRunningTime="2026-09-17 17:41:30.813202565 +0000 UTC m=+43.317205189"
Sep 17 17:41:31.728335 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:31.728303    2569 generic.go:356] "Generic (PLEG): container finished" podID="911990d2-9984-4b04-979c-1380a8bdb7a2" containerID="9bd5e61674cc34961f0a6d2c9bbac8b4ecfaada47a6c68bdbd1163d58b2bf780" exitCode=0
Sep 17 17:41:31.728813 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:31.728396    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mwh5v" event={"ID":"911990d2-9984-4b04-979c-1380a8bdb7a2","Type":"ContainerDied","Data":"9bd5e61674cc34961f0a6d2c9bbac8b4ecfaada47a6c68bdbd1163d58b2bf780"}
Sep 17 17:41:32.733963 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:32.733926    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mwh5v" event={"ID":"911990d2-9984-4b04-979c-1380a8bdb7a2","Type":"ContainerStarted","Data":"4bb4cef8ed7432bb77aa3e926984060f4a6fd33a748bcae9e98429164037d5dc"}
Sep 17 17:41:32.762036 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:32.761990    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-mwh5v" podStartSLOduration=10.371136067 podStartE2EDuration="43.761972436s" totalImagesPullingTime="33.390836369s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:40:49 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:32.761374472 +0000 UTC m=+45.265377111" watchObservedRunningTime="2026-09-17 17:41:32.761972436 +0000 UTC m=+45.265975060"
Sep 17 17:41:38.387359 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:38.387314    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/023f4b84-91bb-4103-97cb-5123dc08e459-metrics-tls\") pod \"dns-default-6lkgm\" (UID: \"023f4b84-91bb-4103-97cb-5123dc08e459\") " pod="openshift-dns/dns-default-6lkgm"
Sep 17 17:41:38.387748 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:41:38.387436    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:41:38.387748 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:41:38.387497    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/023f4b84-91bb-4103-97cb-5123dc08e459-metrics-tls podName:023f4b84-91bb-4103-97cb-5123dc08e459 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:54.387483327 +0000 UTC m=+66.891485930 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/023f4b84-91bb-4103-97cb-5123dc08e459-metrics-tls") pod "dns-default-6lkgm" (UID: "023f4b84-91bb-4103-97cb-5123dc08e459") : secret "dns-default-metrics-tls" not found
Sep 17 17:41:48.691958 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:48.691929    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k4mqn"
Sep 17 17:41:54.392945 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:54.392902    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/023f4b84-91bb-4103-97cb-5123dc08e459-metrics-tls\") pod \"dns-default-6lkgm\" (UID: \"023f4b84-91bb-4103-97cb-5123dc08e459\") " pod="openshift-dns/dns-default-6lkgm"
Sep 17 17:41:54.393412 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:41:54.393059    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:41:54.393412 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:41:54.393136    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/023f4b84-91bb-4103-97cb-5123dc08e459-metrics-tls podName:023f4b84-91bb-4103-97cb-5123dc08e459 nodeName:}" failed. No retries permitted until 2026-09-17 17:42:26.393120673 +0000 UTC m=+98.897123276 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/023f4b84-91bb-4103-97cb-5123dc08e459-metrics-tls") pod "dns-default-6lkgm" (UID: "023f4b84-91bb-4103-97cb-5123dc08e459") : secret "dns-default-metrics-tls" not found
Sep 17 17:41:55.199061 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:41:55.199014    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caf785f7-c156-4b6f-8c82-4c1efe5cd710-metrics-certs\") pod \"network-metrics-daemon-rzvhs\" (UID: \"caf785f7-c156-4b6f-8c82-4c1efe5cd710\") " pod="openshift-multus/network-metrics-daemon-rzvhs"
Sep 17 17:41:55.199257 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:41:55.199175    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 17:41:55.199257 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:41:55.199239    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/caf785f7-c156-4b6f-8c82-4c1efe5cd710-metrics-certs podName:caf785f7-c156-4b6f-8c82-4c1efe5cd710 nodeName:}" failed. No retries permitted until 2026-09-17 17:42:59.199222018 +0000 UTC m=+131.703224633 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/caf785f7-c156-4b6f-8c82-4c1efe5cd710-metrics-certs") pod "network-metrics-daemon-rzvhs" (UID: "caf785f7-c156-4b6f-8c82-4c1efe5cd710") : secret "metrics-daemon-secret" not found
Sep 17 17:42:00.450864 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.450827    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-g9gvb"]
Sep 17 17:42:00.453632 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.453612    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-g9gvb"
Sep 17 17:42:00.456168 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.456142    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-p2w8r\""
Sep 17 17:42:00.465759 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.465739    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-g9gvb"]
Sep 17 17:42:00.573974 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.573936    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-sx8ns"]
Sep 17 17:42:00.576696 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.576679    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-sx8ns"
Sep 17 17:42:00.580662 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.580638    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 17:42:00.580797 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.580686    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 17 17:42:00.580797 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.580694    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-v26b9\""
Sep 17 17:42:00.581004 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.580990    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 17 17:42:00.581198 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.581186    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 17:42:00.604701 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.604672    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-55f9595458-br8n7"]
Sep 17 17:42:00.611116 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.611088    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-t7pqb"]
Sep 17 17:42:00.611259 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.611158    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-55f9595458-br8n7"
Sep 17 17:42:00.613709 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.613689    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-sx8ns"]
Sep 17 17:42:00.613806 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.613767    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-t7pqb"
Sep 17 17:42:00.624171 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.624149    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 17 17:42:00.624288 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.624196    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:42:00.624399 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.624384    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 17 17:42:00.624399 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.624391    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 17 17:42:00.624602 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.624583    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-g8k7c\""
Sep 17 17:42:00.632261 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.632236    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-628xg\""
Sep 17 17:42:00.632370 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.632236    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 17 17:42:00.632938 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.632916    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 17 17:42:00.633023 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.633005    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 17 17:42:00.634660 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.634646    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 17 17:42:00.635236 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.635223    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 17 17:42:00.635431 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.635415    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5xnf\" (UniqueName: \"kubernetes.io/projected/65730b96-9ce9-4d61-9fd6-1ee7b9e6ebf7-kube-api-access-j5xnf\") pod \"network-check-source-7cbc45fddc-g9gvb\" (UID: \"65730b96-9ce9-4d61-9fd6-1ee7b9e6ebf7\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-g9gvb"
Sep 17 17:42:00.637514 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.637500    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 17 17:42:00.643609 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.643590    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 17 17:42:00.645765 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.645748    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 17 17:42:00.656187 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.656168    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-t7pqb"]
Sep 17 17:42:00.659768 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.659748    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-55f9595458-br8n7"]
Sep 17 17:42:00.736245 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.736142    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3de5fb99-4d6c-4241-a85b-902f36b0a49a-snapshots\") pod \"insights-operator-6f9c644dd9-sx8ns\" (UID: \"3de5fb99-4d6c-4241-a85b-902f36b0a49a\") " pod="openshift-insights/insights-operator-6f9c644dd9-sx8ns"
Sep 17 17:42:00.736245 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.736187    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k86r\" (UniqueName: \"kubernetes.io/projected/efc7ba59-612a-4b8a-9935-f2228ec44b30-kube-api-access-5k86r\") pod \"router-default-55f9595458-br8n7\" (UID: \"efc7ba59-612a-4b8a-9935-f2228ec44b30\") " pod="openshift-ingress/router-default-55f9595458-br8n7"
Sep 17 17:42:00.736245 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.736210    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b8d76e28-4655-4d21-98d8-0324e4756aaf-config\") pod \"console-operator-fd4bb46c-t7pqb\" (UID: \"b8d76e28-4655-4d21-98d8-0324e4756aaf\") " pod="openshift-console-operator/console-operator-fd4bb46c-t7pqb"
Sep 17 17:42:00.736245 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.736230    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b8d76e28-4655-4d21-98d8-0324e4756aaf-serving-cert\") pod \"console-operator-fd4bb46c-t7pqb\" (UID: \"b8d76e28-4655-4d21-98d8-0324e4756aaf\") " pod="openshift-console-operator/console-operator-fd4bb46c-t7pqb"
Sep 17 17:42:00.736573 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.736253    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b8d76e28-4655-4d21-98d8-0324e4756aaf-trusted-ca\") pod \"console-operator-fd4bb46c-t7pqb\" (UID: \"b8d76e28-4655-4d21-98d8-0324e4756aaf\") " pod="openshift-console-operator/console-operator-fd4bb46c-t7pqb"
Sep 17 17:42:00.736573 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.736273    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nk6n\" (UniqueName: \"kubernetes.io/projected/b8d76e28-4655-4d21-98d8-0324e4756aaf-kube-api-access-4nk6n\") pod \"console-operator-fd4bb46c-t7pqb\" (UID: \"b8d76e28-4655-4d21-98d8-0324e4756aaf\") " pod="openshift-console-operator/console-operator-fd4bb46c-t7pqb"
Sep 17 17:42:00.736573 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.736299    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efc7ba59-612a-4b8a-9935-f2228ec44b30-metrics-certs\") pod \"router-default-55f9595458-br8n7\" (UID: \"efc7ba59-612a-4b8a-9935-f2228ec44b30\") " pod="openshift-ingress/router-default-55f9595458-br8n7"
Sep 17 17:42:00.736573 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.736334    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3de5fb99-4d6c-4241-a85b-902f36b0a49a-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-sx8ns\" (UID: \"3de5fb99-4d6c-4241-a85b-902f36b0a49a\") " pod="openshift-insights/insights-operator-6f9c644dd9-sx8ns"
Sep 17 17:42:00.736573 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.736355    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nshlx\" (UniqueName: \"kubernetes.io/projected/3de5fb99-4d6c-4241-a85b-902f36b0a49a-kube-api-access-nshlx\") pod \"insights-operator-6f9c644dd9-sx8ns\" (UID: \"3de5fb99-4d6c-4241-a85b-902f36b0a49a\") " pod="openshift-insights/insights-operator-6f9c644dd9-sx8ns"
Sep 17 17:42:00.736573 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.736384    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/efc7ba59-612a-4b8a-9935-f2228ec44b30-stats-auth\") pod \"router-default-55f9595458-br8n7\" (UID: \"efc7ba59-612a-4b8a-9935-f2228ec44b30\") " pod="openshift-ingress/router-default-55f9595458-br8n7"
Sep 17 17:42:00.736573 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.736411    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efc7ba59-612a-4b8a-9935-f2228ec44b30-service-ca-bundle\") pod \"router-default-55f9595458-br8n7\" (UID: \"efc7ba59-612a-4b8a-9935-f2228ec44b30\") " pod="openshift-ingress/router-default-55f9595458-br8n7"
Sep 17 17:42:00.736573 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.736451    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/efc7ba59-612a-4b8a-9935-f2228ec44b30-default-certificate\") pod \"router-default-55f9595458-br8n7\" (UID: \"efc7ba59-612a-4b8a-9935-f2228ec44b30\") " pod="openshift-ingress/router-default-55f9595458-br8n7"
Sep 17 17:42:00.736573 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.736514    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3de5fb99-4d6c-4241-a85b-902f36b0a49a-serving-cert\") pod \"insights-operator-6f9c644dd9-sx8ns\" (UID: \"3de5fb99-4d6c-4241-a85b-902f36b0a49a\") " pod="openshift-insights/insights-operator-6f9c644dd9-sx8ns"
Sep 17 17:42:00.736893 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.736582    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3de5fb99-4d6c-4241-a85b-902f36b0a49a-tmp\") pod \"insights-operator-6f9c644dd9-sx8ns\" (UID: \"3de5fb99-4d6c-4241-a85b-902f36b0a49a\") " pod="openshift-insights/insights-operator-6f9c644dd9-sx8ns"
Sep 17 17:42:00.736893 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.736621    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3de5fb99-4d6c-4241-a85b-902f36b0a49a-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-sx8ns\" (UID: \"3de5fb99-4d6c-4241-a85b-902f36b0a49a\") " pod="openshift-insights/insights-operator-6f9c644dd9-sx8ns"
Sep 17 17:42:00.736893 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.736652    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5xnf\" (UniqueName: \"kubernetes.io/projected/65730b96-9ce9-4d61-9fd6-1ee7b9e6ebf7-kube-api-access-j5xnf\") pod \"network-check-source-7cbc45fddc-g9gvb\" (UID: \"65730b96-9ce9-4d61-9fd6-1ee7b9e6ebf7\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-g9gvb"
Sep 17 17:42:00.757455 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.757420    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5xnf\" (UniqueName: \"kubernetes.io/projected/65730b96-9ce9-4d61-9fd6-1ee7b9e6ebf7-kube-api-access-j5xnf\") pod \"network-check-source-7cbc45fddc-g9gvb\" (UID: \"65730b96-9ce9-4d61-9fd6-1ee7b9e6ebf7\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-g9gvb"
Sep 17 17:42:00.762246 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.762188    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-g9gvb"
Sep 17 17:42:00.837637 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.837603    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/efc7ba59-612a-4b8a-9935-f2228ec44b30-default-certificate\") pod \"router-default-55f9595458-br8n7\" (UID: \"efc7ba59-612a-4b8a-9935-f2228ec44b30\") " pod="openshift-ingress/router-default-55f9595458-br8n7"
Sep 17 17:42:00.837820 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.837657    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3de5fb99-4d6c-4241-a85b-902f36b0a49a-serving-cert\") pod \"insights-operator-6f9c644dd9-sx8ns\" (UID: \"3de5fb99-4d6c-4241-a85b-902f36b0a49a\") " pod="openshift-insights/insights-operator-6f9c644dd9-sx8ns"
Sep 17 17:42:00.837820 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.837691    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3de5fb99-4d6c-4241-a85b-902f36b0a49a-tmp\") pod \"insights-operator-6f9c644dd9-sx8ns\" (UID: \"3de5fb99-4d6c-4241-a85b-902f36b0a49a\") " pod="openshift-insights/insights-operator-6f9c644dd9-sx8ns"
Sep 17 17:42:00.837820 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.837707    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3de5fb99-4d6c-4241-a85b-902f36b0a49a-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-sx8ns\" (UID: \"3de5fb99-4d6c-4241-a85b-902f36b0a49a\") " pod="openshift-insights/insights-operator-6f9c644dd9-sx8ns"
Sep 17 17:42:00.837820 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.837726    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3de5fb99-4d6c-4241-a85b-902f36b0a49a-snapshots\") pod \"insights-operator-6f9c644dd9-sx8ns\" (UID: \"3de5fb99-4d6c-4241-a85b-902f36b0a49a\") " pod="openshift-insights/insights-operator-6f9c644dd9-sx8ns"
Sep 17 17:42:00.837820 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.837740    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k86r\" (UniqueName: \"kubernetes.io/projected/efc7ba59-612a-4b8a-9935-f2228ec44b30-kube-api-access-5k86r\") pod \"router-default-55f9595458-br8n7\" (UID: \"efc7ba59-612a-4b8a-9935-f2228ec44b30\") " pod="openshift-ingress/router-default-55f9595458-br8n7"
Sep 17 17:42:00.837820 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.837755    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b8d76e28-4655-4d21-98d8-0324e4756aaf-config\") pod \"console-operator-fd4bb46c-t7pqb\" (UID: \"b8d76e28-4655-4d21-98d8-0324e4756aaf\") " pod="openshift-console-operator/console-operator-fd4bb46c-t7pqb"
Sep 17 17:42:00.837820 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.837778    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b8d76e28-4655-4d21-98d8-0324e4756aaf-serving-cert\") pod \"console-operator-fd4bb46c-t7pqb\" (UID: \"b8d76e28-4655-4d21-98d8-0324e4756aaf\") " pod="openshift-console-operator/console-operator-fd4bb46c-t7pqb"
Sep 17 17:42:00.837820 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.837799    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b8d76e28-4655-4d21-98d8-0324e4756aaf-trusted-ca\") pod \"console-operator-fd4bb46c-t7pqb\" (UID: \"b8d76e28-4655-4d21-98d8-0324e4756aaf\") " pod="openshift-console-operator/console-operator-fd4bb46c-t7pqb"
Sep 17 17:42:00.837820 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.837822    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nk6n\" (UniqueName: \"kubernetes.io/projected/b8d76e28-4655-4d21-98d8-0324e4756aaf-kube-api-access-4nk6n\") pod \"console-operator-fd4bb46c-t7pqb\" (UID: \"b8d76e28-4655-4d21-98d8-0324e4756aaf\") " pod="openshift-console-operator/console-operator-fd4bb46c-t7pqb"
Sep 17 17:42:00.838302 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.837848    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efc7ba59-612a-4b8a-9935-f2228ec44b30-metrics-certs\") pod \"router-default-55f9595458-br8n7\" (UID: \"efc7ba59-612a-4b8a-9935-f2228ec44b30\") " pod="openshift-ingress/router-default-55f9595458-br8n7"
Sep 17 17:42:00.838302 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.837874    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3de5fb99-4d6c-4241-a85b-902f36b0a49a-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-sx8ns\" (UID: \"3de5fb99-4d6c-4241-a85b-902f36b0a49a\") " pod="openshift-insights/insights-operator-6f9c644dd9-sx8ns"
Sep 17 17:42:00.838302 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.837898    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nshlx\" (UniqueName: \"kubernetes.io/projected/3de5fb99-4d6c-4241-a85b-902f36b0a49a-kube-api-access-nshlx\") pod \"insights-operator-6f9c644dd9-sx8ns\" (UID: \"3de5fb99-4d6c-4241-a85b-902f36b0a49a\") " pod="openshift-insights/insights-operator-6f9c644dd9-sx8ns"
Sep 17 17:42:00.838302 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.837928    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/efc7ba59-612a-4b8a-9935-f2228ec44b30-stats-auth\") pod \"router-default-55f9595458-br8n7\" (UID: \"efc7ba59-612a-4b8a-9935-f2228ec44b30\") " pod="openshift-ingress/router-default-55f9595458-br8n7"
Sep 17 17:42:00.838302 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.837955    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efc7ba59-612a-4b8a-9935-f2228ec44b30-service-ca-bundle\") pod \"router-default-55f9595458-br8n7\" (UID: \"efc7ba59-612a-4b8a-9935-f2228ec44b30\") " pod="openshift-ingress/router-default-55f9595458-br8n7"
Sep 17 17:42:00.838302 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.838163    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3de5fb99-4d6c-4241-a85b-902f36b0a49a-tmp\") pod \"insights-operator-6f9c644dd9-sx8ns\" (UID: \"3de5fb99-4d6c-4241-a85b-902f36b0a49a\") " pod="openshift-insights/insights-operator-6f9c644dd9-sx8ns"
Sep 17 17:42:00.838302 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:42:00.838178    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/efc7ba59-612a-4b8a-9935-f2228ec44b30-service-ca-bundle podName:efc7ba59-612a-4b8a-9935-f2228ec44b30 nodeName:}" failed. No retries permitted until 2026-09-17 17:42:01.338070605 +0000 UTC m=+73.842073209 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/efc7ba59-612a-4b8a-9935-f2228ec44b30-service-ca-bundle") pod "router-default-55f9595458-br8n7" (UID: "efc7ba59-612a-4b8a-9935-f2228ec44b30") : configmap references non-existent config key: service-ca.crt
Sep 17 17:42:00.838302 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:42:00.838197    2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 17:42:00.838302 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:42:00.838264    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efc7ba59-612a-4b8a-9935-f2228ec44b30-metrics-certs podName:efc7ba59-612a-4b8a-9935-f2228ec44b30 nodeName:}" failed. No retries permitted until 2026-09-17 17:42:01.338247534 +0000 UTC m=+73.842250153 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/efc7ba59-612a-4b8a-9935-f2228ec44b30-metrics-certs") pod "router-default-55f9595458-br8n7" (UID: "efc7ba59-612a-4b8a-9935-f2228ec44b30") : secret "router-metrics-certs-default" not found
Sep 17 17:42:00.838755 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.838476    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3de5fb99-4d6c-4241-a85b-902f36b0a49a-snapshots\") pod \"insights-operator-6f9c644dd9-sx8ns\" (UID: \"3de5fb99-4d6c-4241-a85b-902f36b0a49a\") " pod="openshift-insights/insights-operator-6f9c644dd9-sx8ns"
Sep 17 17:42:00.838755 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.838569    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b8d76e28-4655-4d21-98d8-0324e4756aaf-config\") pod \"console-operator-fd4bb46c-t7pqb\" (UID: \"b8d76e28-4655-4d21-98d8-0324e4756aaf\") " pod="openshift-console-operator/console-operator-fd4bb46c-t7pqb"
Sep 17 17:42:00.839322 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.839294    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3de5fb99-4d6c-4241-a85b-902f36b0a49a-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-sx8ns\" (UID: \"3de5fb99-4d6c-4241-a85b-902f36b0a49a\") " pod="openshift-insights/insights-operator-6f9c644dd9-sx8ns"
Sep 17 17:42:00.839518 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.839466    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b8d76e28-4655-4d21-98d8-0324e4756aaf-trusted-ca\") pod \"console-operator-fd4bb46c-t7pqb\" (UID: \"b8d76e28-4655-4d21-98d8-0324e4756aaf\") " pod="openshift-console-operator/console-operator-fd4bb46c-t7pqb"
Sep 17 17:42:00.839603 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.839519    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3de5fb99-4d6c-4241-a85b-902f36b0a49a-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-sx8ns\" (UID: \"3de5fb99-4d6c-4241-a85b-902f36b0a49a\") " pod="openshift-insights/insights-operator-6f9c644dd9-sx8ns"
Sep 17 17:42:00.841069 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.841042    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/efc7ba59-612a-4b8a-9935-f2228ec44b30-default-certificate\") pod \"router-default-55f9595458-br8n7\" (UID: \"efc7ba59-612a-4b8a-9935-f2228ec44b30\") " pod="openshift-ingress/router-default-55f9595458-br8n7"
Sep 17 17:42:00.841178 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.841096    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3de5fb99-4d6c-4241-a85b-902f36b0a49a-serving-cert\") pod \"insights-operator-6f9c644dd9-sx8ns\" (UID: \"3de5fb99-4d6c-4241-a85b-902f36b0a49a\") " pod="openshift-insights/insights-operator-6f9c644dd9-sx8ns"
Sep 17 17:42:00.841413 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.841393    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b8d76e28-4655-4d21-98d8-0324e4756aaf-serving-cert\") pod \"console-operator-fd4bb46c-t7pqb\" (UID: \"b8d76e28-4655-4d21-98d8-0324e4756aaf\") " pod="openshift-console-operator/console-operator-fd4bb46c-t7pqb"
Sep 17 17:42:00.841474 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.841415    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/efc7ba59-612a-4b8a-9935-f2228ec44b30-stats-auth\") pod \"router-default-55f9595458-br8n7\" (UID: \"efc7ba59-612a-4b8a-9935-f2228ec44b30\") " pod="openshift-ingress/router-default-55f9595458-br8n7"
Sep 17 17:42:00.854229 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.854204    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nshlx\" (UniqueName: \"kubernetes.io/projected/3de5fb99-4d6c-4241-a85b-902f36b0a49a-kube-api-access-nshlx\") pod \"insights-operator-6f9c644dd9-sx8ns\" (UID: \"3de5fb99-4d6c-4241-a85b-902f36b0a49a\") " pod="openshift-insights/insights-operator-6f9c644dd9-sx8ns"
Sep 17 17:42:00.854371 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.854248    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nk6n\" (UniqueName: \"kubernetes.io/projected/b8d76e28-4655-4d21-98d8-0324e4756aaf-kube-api-access-4nk6n\") pod \"console-operator-fd4bb46c-t7pqb\" (UID: \"b8d76e28-4655-4d21-98d8-0324e4756aaf\") " pod="openshift-console-operator/console-operator-fd4bb46c-t7pqb"
Sep 17 17:42:00.855349 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.855330    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k86r\" (UniqueName: \"kubernetes.io/projected/efc7ba59-612a-4b8a-9935-f2228ec44b30-kube-api-access-5k86r\") pod \"router-default-55f9595458-br8n7\" (UID: \"efc7ba59-612a-4b8a-9935-f2228ec44b30\") " pod="openshift-ingress/router-default-55f9595458-br8n7"
Sep 17 17:42:00.885344 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.885309    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-sx8ns"
Sep 17 17:42:00.925828 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:00.925790    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-t7pqb"
Sep 17 17:42:01.031389 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:01.031296    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-g9gvb"]
Sep 17 17:42:01.035142 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:42:01.035110    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65730b96_9ce9_4d61_9fd6_1ee7b9e6ebf7.slice/crio-6d53122c31f5d9a3152bf7f066bfdc22ac4d15ce1ee43daf4f027caa3f0dadea WatchSource:0}: Error finding container 6d53122c31f5d9a3152bf7f066bfdc22ac4d15ce1ee43daf4f027caa3f0dadea: Status 404 returned error can't find the container with id 6d53122c31f5d9a3152bf7f066bfdc22ac4d15ce1ee43daf4f027caa3f0dadea
Sep 17 17:42:01.160384 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:01.160351    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-sx8ns"]
Sep 17 17:42:01.164184 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:42:01.164155    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3de5fb99_4d6c_4241_a85b_902f36b0a49a.slice/crio-6d4085fd25a5df9dbad0c30742242b50e5c772266f653d0ad80ecc21f11faf6f WatchSource:0}: Error finding container 6d4085fd25a5df9dbad0c30742242b50e5c772266f653d0ad80ecc21f11faf6f: Status 404 returned error can't find the container with id 6d4085fd25a5df9dbad0c30742242b50e5c772266f653d0ad80ecc21f11faf6f
Sep 17 17:42:01.200270 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:01.200238    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-t7pqb"]
Sep 17 17:42:01.202949 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:42:01.202911    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8d76e28_4655_4d21_98d8_0324e4756aaf.slice/crio-cfe78a588de6e522425d5d0ded60a5f24eb95ae020639b9d56b08dce4694834e WatchSource:0}: Error finding container cfe78a588de6e522425d5d0ded60a5f24eb95ae020639b9d56b08dce4694834e: Status 404 returned error can't find the container with id cfe78a588de6e522425d5d0ded60a5f24eb95ae020639b9d56b08dce4694834e
Sep 17 17:42:01.342713 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:01.342628    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efc7ba59-612a-4b8a-9935-f2228ec44b30-metrics-certs\") pod \"router-default-55f9595458-br8n7\" (UID: \"efc7ba59-612a-4b8a-9935-f2228ec44b30\") " pod="openshift-ingress/router-default-55f9595458-br8n7"
Sep 17 17:42:01.342713 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:01.342677    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efc7ba59-612a-4b8a-9935-f2228ec44b30-service-ca-bundle\") pod \"router-default-55f9595458-br8n7\" (UID: \"efc7ba59-612a-4b8a-9935-f2228ec44b30\") " pod="openshift-ingress/router-default-55f9595458-br8n7"
Sep 17 17:42:01.342907 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:42:01.342776    2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 17:42:01.342907 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:42:01.342786    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/efc7ba59-612a-4b8a-9935-f2228ec44b30-service-ca-bundle podName:efc7ba59-612a-4b8a-9935-f2228ec44b30 nodeName:}" failed. No retries permitted until 2026-09-17 17:42:02.342771246 +0000 UTC m=+74.846773848 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/efc7ba59-612a-4b8a-9935-f2228ec44b30-service-ca-bundle") pod "router-default-55f9595458-br8n7" (UID: "efc7ba59-612a-4b8a-9935-f2228ec44b30") : configmap references non-existent config key: service-ca.crt
Sep 17 17:42:01.342907 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:42:01.342902    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efc7ba59-612a-4b8a-9935-f2228ec44b30-metrics-certs podName:efc7ba59-612a-4b8a-9935-f2228ec44b30 nodeName:}" failed. No retries permitted until 2026-09-17 17:42:02.342889953 +0000 UTC m=+74.846892560 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/efc7ba59-612a-4b8a-9935-f2228ec44b30-metrics-certs") pod "router-default-55f9595458-br8n7" (UID: "efc7ba59-612a-4b8a-9935-f2228ec44b30") : secret "router-metrics-certs-default" not found
Sep 17 17:42:01.731678 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:01.731324    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-ngfbr"
Sep 17 17:42:01.805410 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:01.805324    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-t7pqb" event={"ID":"b8d76e28-4655-4d21-98d8-0324e4756aaf","Type":"ContainerStarted","Data":"cfe78a588de6e522425d5d0ded60a5f24eb95ae020639b9d56b08dce4694834e"}
Sep 17 17:42:01.807828 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:01.807749    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-g9gvb" event={"ID":"65730b96-9ce9-4d61-9fd6-1ee7b9e6ebf7","Type":"ContainerStarted","Data":"d57d4aabe571eabc3ec31e50174d84e183ca4c98b858b7247689c69e01dd6f4d"}
Sep 17 17:42:01.807828 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:01.807802    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-g9gvb" event={"ID":"65730b96-9ce9-4d61-9fd6-1ee7b9e6ebf7","Type":"ContainerStarted","Data":"6d53122c31f5d9a3152bf7f066bfdc22ac4d15ce1ee43daf4f027caa3f0dadea"}
Sep 17 17:42:01.814330 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:01.810797    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-sx8ns" event={"ID":"3de5fb99-4d6c-4241-a85b-902f36b0a49a","Type":"ContainerStarted","Data":"6d4085fd25a5df9dbad0c30742242b50e5c772266f653d0ad80ecc21f11faf6f"}
Sep 17 17:42:02.350384 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:02.350342    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efc7ba59-612a-4b8a-9935-f2228ec44b30-metrics-certs\") pod \"router-default-55f9595458-br8n7\" (UID: \"efc7ba59-612a-4b8a-9935-f2228ec44b30\") " pod="openshift-ingress/router-default-55f9595458-br8n7"
Sep 17 17:42:02.350561 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:02.350394    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efc7ba59-612a-4b8a-9935-f2228ec44b30-service-ca-bundle\") pod \"router-default-55f9595458-br8n7\" (UID: \"efc7ba59-612a-4b8a-9935-f2228ec44b30\") " pod="openshift-ingress/router-default-55f9595458-br8n7"
Sep 17 17:42:02.350561 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:42:02.350482    2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 17:42:02.350561 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:42:02.350537    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/efc7ba59-612a-4b8a-9935-f2228ec44b30-service-ca-bundle podName:efc7ba59-612a-4b8a-9935-f2228ec44b30 nodeName:}" failed. No retries permitted until 2026-09-17 17:42:04.35052327 +0000 UTC m=+76.854525873 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/efc7ba59-612a-4b8a-9935-f2228ec44b30-service-ca-bundle") pod "router-default-55f9595458-br8n7" (UID: "efc7ba59-612a-4b8a-9935-f2228ec44b30") : configmap references non-existent config key: service-ca.crt
Sep 17 17:42:02.350561 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:42:02.350552    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efc7ba59-612a-4b8a-9935-f2228ec44b30-metrics-certs podName:efc7ba59-612a-4b8a-9935-f2228ec44b30 nodeName:}" failed. No retries permitted until 2026-09-17 17:42:04.350546034 +0000 UTC m=+76.854548638 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/efc7ba59-612a-4b8a-9935-f2228ec44b30-metrics-certs") pod "router-default-55f9595458-br8n7" (UID: "efc7ba59-612a-4b8a-9935-f2228ec44b30") : secret "router-metrics-certs-default" not found
Sep 17 17:42:03.818261 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:03.818150    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-sx8ns" event={"ID":"3de5fb99-4d6c-4241-a85b-902f36b0a49a","Type":"ContainerStarted","Data":"14d9eed890188872efdc2709a84be9e6a715e2e842b2a4d847f515f1bc9c0f92"}
Sep 17 17:42:03.819859 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:03.819837    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-t7pqb_b8d76e28-4655-4d21-98d8-0324e4756aaf/console-operator/0.log"
Sep 17 17:42:03.819982 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:03.819875    2569 generic.go:356] "Generic (PLEG): container finished" podID="b8d76e28-4655-4d21-98d8-0324e4756aaf" containerID="cb32211b455b02973f13a317cedc77145b74e66655841c3dcba2dcf5dfed7e04" exitCode=255
Sep 17 17:42:03.819982 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:03.819904    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-t7pqb" event={"ID":"b8d76e28-4655-4d21-98d8-0324e4756aaf","Type":"ContainerDied","Data":"cb32211b455b02973f13a317cedc77145b74e66655841c3dcba2dcf5dfed7e04"}
Sep 17 17:42:03.820145 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:03.820131    2569 scope.go:117] "RemoveContainer" containerID="cb32211b455b02973f13a317cedc77145b74e66655841c3dcba2dcf5dfed7e04"
Sep 17 17:42:03.839485 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:03.839432    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-6f9c644dd9-sx8ns" podStartSLOduration=1.437231903 podStartE2EDuration="3.839417413s" totalImagesPullingTime="2.40218551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:42:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:03.838545779 +0000 UTC m=+76.342548441" watchObservedRunningTime="2026-09-17 17:42:03.839417413 +0000 UTC m=+76.343420038"
Sep 17 17:42:03.839708 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:03.839681    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-g9gvb" podStartSLOduration=3.839671736 podStartE2EDuration="3.839671736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:42:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:01.829292549 +0000 UTC m=+74.333295202" watchObservedRunningTime="2026-09-17 17:42:03.839671736 +0000 UTC m=+76.343674406"
Sep 17 17:42:04.364716 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:04.364675    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efc7ba59-612a-4b8a-9935-f2228ec44b30-metrics-certs\") pod \"router-default-55f9595458-br8n7\" (UID: \"efc7ba59-612a-4b8a-9935-f2228ec44b30\") " pod="openshift-ingress/router-default-55f9595458-br8n7"
Sep 17 17:42:04.364716 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:04.364728    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efc7ba59-612a-4b8a-9935-f2228ec44b30-service-ca-bundle\") pod \"router-default-55f9595458-br8n7\" (UID: \"efc7ba59-612a-4b8a-9935-f2228ec44b30\") " pod="openshift-ingress/router-default-55f9595458-br8n7"
Sep 17 17:42:04.364956 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:42:04.364826    2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 17:42:04.364956 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:42:04.364854    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/efc7ba59-612a-4b8a-9935-f2228ec44b30-service-ca-bundle podName:efc7ba59-612a-4b8a-9935-f2228ec44b30 nodeName:}" failed. No retries permitted until 2026-09-17 17:42:08.364839468 +0000 UTC m=+80.868842074 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/efc7ba59-612a-4b8a-9935-f2228ec44b30-service-ca-bundle") pod "router-default-55f9595458-br8n7" (UID: "efc7ba59-612a-4b8a-9935-f2228ec44b30") : configmap references non-existent config key: service-ca.crt
Sep 17 17:42:04.364956 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:42:04.364883    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efc7ba59-612a-4b8a-9935-f2228ec44b30-metrics-certs podName:efc7ba59-612a-4b8a-9935-f2228ec44b30 nodeName:}" failed. No retries permitted until 2026-09-17 17:42:08.364869089 +0000 UTC m=+80.868871691 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/efc7ba59-612a-4b8a-9935-f2228ec44b30-metrics-certs") pod "router-default-55f9595458-br8n7" (UID: "efc7ba59-612a-4b8a-9935-f2228ec44b30") : secret "router-metrics-certs-default" not found
Sep 17 17:42:04.824968 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:04.824940    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-t7pqb_b8d76e28-4655-4d21-98d8-0324e4756aaf/console-operator/1.log"
Sep 17 17:42:04.825434 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:04.825416    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-t7pqb_b8d76e28-4655-4d21-98d8-0324e4756aaf/console-operator/0.log"
Sep 17 17:42:04.825507 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:04.825461    2569 generic.go:356] "Generic (PLEG): container finished" podID="b8d76e28-4655-4d21-98d8-0324e4756aaf" containerID="02a9f57b767e8fe7332dbe06ceca6b2cf7db74d180cd618c5216d88c3fbeec02" exitCode=255
Sep 17 17:42:04.825562 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:04.825540    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-t7pqb" event={"ID":"b8d76e28-4655-4d21-98d8-0324e4756aaf","Type":"ContainerDied","Data":"02a9f57b767e8fe7332dbe06ceca6b2cf7db74d180cd618c5216d88c3fbeec02"}
Sep 17 17:42:04.825606 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:04.825600    2569 scope.go:117] "RemoveContainer" containerID="cb32211b455b02973f13a317cedc77145b74e66655841c3dcba2dcf5dfed7e04"
Sep 17 17:42:04.825821 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:04.825792    2569 scope.go:117] "RemoveContainer" containerID="02a9f57b767e8fe7332dbe06ceca6b2cf7db74d180cd618c5216d88c3fbeec02"
Sep 17 17:42:04.825993 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:42:04.825975    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-t7pqb_openshift-console-operator(b8d76e28-4655-4d21-98d8-0324e4756aaf)\"" pod="openshift-console-operator/console-operator-fd4bb46c-t7pqb" podUID="b8d76e28-4655-4d21-98d8-0324e4756aaf"
Sep 17 17:42:04.881726 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:04.881697    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-ggclv"]
Sep 17 17:42:04.885727 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:04.885709    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ggclv"
Sep 17 17:42:04.897978 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:42:04.897949    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-rbac-proxy\" is forbidden: User \"system:node:ip-10-0-131-14.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-131-14.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" type="*v1.ConfigMap"
Sep 17 17:42:04.897978 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:42:04.897949    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"insights-runtime-extractor-sa-dockercfg-cb5pj\" is forbidden: User \"system:node:ip-10-0-131-14.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-131-14.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-cb5pj\"" type="*v1.Secret"
Sep 17 17:42:04.897978 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:42:04.897949    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"insights-runtime-extractor-tls\" is forbidden: User \"system:node:ip-10-0-131-14.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-131-14.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" type="*v1.Secret"
Sep 17 17:42:04.933253 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:04.933214    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ggclv"]
Sep 17 17:42:04.967714 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:04.967650    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/642e943b-2b5e-410d-8a32-c1eaf28b5fb4-data-volume\") pod \"insights-runtime-extractor-ggclv\" (UID: \"642e943b-2b5e-410d-8a32-c1eaf28b5fb4\") " pod="openshift-insights/insights-runtime-extractor-ggclv"
Sep 17 17:42:04.967903 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:04.967734    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/642e943b-2b5e-410d-8a32-c1eaf28b5fb4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ggclv\" (UID: \"642e943b-2b5e-410d-8a32-c1eaf28b5fb4\") " pod="openshift-insights/insights-runtime-extractor-ggclv"
Sep 17 17:42:04.967903 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:04.967771    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/642e943b-2b5e-410d-8a32-c1eaf28b5fb4-crio-socket\") pod \"insights-runtime-extractor-ggclv\" (UID: \"642e943b-2b5e-410d-8a32-c1eaf28b5fb4\") " pod="openshift-insights/insights-runtime-extractor-ggclv"
Sep 17 17:42:04.967903 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:04.967802    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/642e943b-2b5e-410d-8a32-c1eaf28b5fb4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ggclv\" (UID: \"642e943b-2b5e-410d-8a32-c1eaf28b5fb4\") " pod="openshift-insights/insights-runtime-extractor-ggclv"
Sep 17 17:42:04.967903 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:04.967824    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln24p\" (UniqueName: \"kubernetes.io/projected/642e943b-2b5e-410d-8a32-c1eaf28b5fb4-kube-api-access-ln24p\") pod \"insights-runtime-extractor-ggclv\" (UID: \"642e943b-2b5e-410d-8a32-c1eaf28b5fb4\") " pod="openshift-insights/insights-runtime-extractor-ggclv"
Sep 17 17:42:05.068329 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:05.068285    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/642e943b-2b5e-410d-8a32-c1eaf28b5fb4-data-volume\") pod \"insights-runtime-extractor-ggclv\" (UID: \"642e943b-2b5e-410d-8a32-c1eaf28b5fb4\") " pod="openshift-insights/insights-runtime-extractor-ggclv"
Sep 17 17:42:05.068329 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:05.068326    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/642e943b-2b5e-410d-8a32-c1eaf28b5fb4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ggclv\" (UID: \"642e943b-2b5e-410d-8a32-c1eaf28b5fb4\") " pod="openshift-insights/insights-runtime-extractor-ggclv"
Sep 17 17:42:05.068543 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:05.068345    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/642e943b-2b5e-410d-8a32-c1eaf28b5fb4-crio-socket\") pod \"insights-runtime-extractor-ggclv\" (UID: \"642e943b-2b5e-410d-8a32-c1eaf28b5fb4\") " pod="openshift-insights/insights-runtime-extractor-ggclv"
Sep 17 17:42:05.068543 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:05.068419    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/642e943b-2b5e-410d-8a32-c1eaf28b5fb4-crio-socket\") pod \"insights-runtime-extractor-ggclv\" (UID: \"642e943b-2b5e-410d-8a32-c1eaf28b5fb4\") " pod="openshift-insights/insights-runtime-extractor-ggclv"
Sep 17 17:42:05.068543 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:05.068451    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/642e943b-2b5e-410d-8a32-c1eaf28b5fb4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ggclv\" (UID: \"642e943b-2b5e-410d-8a32-c1eaf28b5fb4\") " pod="openshift-insights/insights-runtime-extractor-ggclv"
Sep 17 17:42:05.068543 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:05.068490    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln24p\" (UniqueName: \"kubernetes.io/projected/642e943b-2b5e-410d-8a32-c1eaf28b5fb4-kube-api-access-ln24p\") pod \"insights-runtime-extractor-ggclv\" (UID: \"642e943b-2b5e-410d-8a32-c1eaf28b5fb4\") " pod="openshift-insights/insights-runtime-extractor-ggclv"
Sep 17 17:42:05.068702 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:05.068683    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/642e943b-2b5e-410d-8a32-c1eaf28b5fb4-data-volume\") pod \"insights-runtime-extractor-ggclv\" (UID: \"642e943b-2b5e-410d-8a32-c1eaf28b5fb4\") " pod="openshift-insights/insights-runtime-extractor-ggclv"
Sep 17 17:42:05.079767 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:05.079696    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln24p\" (UniqueName: \"kubernetes.io/projected/642e943b-2b5e-410d-8a32-c1eaf28b5fb4-kube-api-access-ln24p\") pod \"insights-runtime-extractor-ggclv\" (UID: \"642e943b-2b5e-410d-8a32-c1eaf28b5fb4\") " pod="openshift-insights/insights-runtime-extractor-ggclv"
Sep 17 17:42:05.830789 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:05.830762    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-t7pqb_b8d76e28-4655-4d21-98d8-0324e4756aaf/console-operator/1.log"
Sep 17 17:42:05.831194 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:05.831095    2569 scope.go:117] "RemoveContainer" containerID="02a9f57b767e8fe7332dbe06ceca6b2cf7db74d180cd618c5216d88c3fbeec02"
Sep 17 17:42:05.831258 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:42:05.831239    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-t7pqb_openshift-console-operator(b8d76e28-4655-4d21-98d8-0324e4756aaf)\"" pod="openshift-console-operator/console-operator-fd4bb46c-t7pqb" podUID="b8d76e28-4655-4d21-98d8-0324e4756aaf"
Sep 17 17:42:05.979312 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:05.979274    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 17:42:05.988740 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:42:05.988715    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:42:05.988865 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:42:05.988786    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/642e943b-2b5e-410d-8a32-c1eaf28b5fb4-insights-runtime-extractor-tls podName:642e943b-2b5e-410d-8a32-c1eaf28b5fb4 nodeName:}" failed. No retries permitted until 2026-09-17 17:42:06.488762998 +0000 UTC m=+78.992765604 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/642e943b-2b5e-410d-8a32-c1eaf28b5fb4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ggclv" (UID: "642e943b-2b5e-410d-8a32-c1eaf28b5fb4") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:42:06.009903 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:06.009877    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-cb5pj\""
Sep 17 17:42:06.068977 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:42:06.068943    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: failed to sync configmap cache: timed out waiting for the condition
Sep 17 17:42:06.069150 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:42:06.069027    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/642e943b-2b5e-410d-8a32-c1eaf28b5fb4-kube-rbac-proxy-cm podName:642e943b-2b5e-410d-8a32-c1eaf28b5fb4 nodeName:}" failed. No retries permitted until 2026-09-17 17:42:06.569005516 +0000 UTC m=+79.073008120 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/642e943b-2b5e-410d-8a32-c1eaf28b5fb4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ggclv" (UID: "642e943b-2b5e-410d-8a32-c1eaf28b5fb4") : failed to sync configmap cache: timed out waiting for the condition
Sep 17 17:42:06.182728 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:06.182654    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-x8vvv"]
Sep 17 17:42:06.185449 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:06.185429    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-x8vvv"
Sep 17 17:42:06.188297 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:06.188273    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-wt4fr\""
Sep 17 17:42:06.188420 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:06.188275    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 17 17:42:06.188420 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:06.188274    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 17 17:42:06.197368 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:06.197344    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-x8vvv"]
Sep 17 17:42:06.279592 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:06.279548    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgfl7\" (UniqueName: \"kubernetes.io/projected/d0fa8652-da03-451f-bc9d-bf02e2264e2f-kube-api-access-hgfl7\") pod \"migrator-76794cbd6c-x8vvv\" (UID: \"d0fa8652-da03-451f-bc9d-bf02e2264e2f\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-x8vvv"
Sep 17 17:42:06.300670 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:06.300645    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 17:42:06.380812 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:06.380776    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgfl7\" (UniqueName: \"kubernetes.io/projected/d0fa8652-da03-451f-bc9d-bf02e2264e2f-kube-api-access-hgfl7\") pod \"migrator-76794cbd6c-x8vvv\" (UID: \"d0fa8652-da03-451f-bc9d-bf02e2264e2f\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-x8vvv"
Sep 17 17:42:06.390287 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:06.390258    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgfl7\" (UniqueName: \"kubernetes.io/projected/d0fa8652-da03-451f-bc9d-bf02e2264e2f-kube-api-access-hgfl7\") pod \"migrator-76794cbd6c-x8vvv\" (UID: \"d0fa8652-da03-451f-bc9d-bf02e2264e2f\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-x8vvv"
Sep 17 17:42:06.494494 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:06.494451    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-x8vvv"
Sep 17 17:42:06.582861 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:06.582830    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/642e943b-2b5e-410d-8a32-c1eaf28b5fb4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ggclv\" (UID: \"642e943b-2b5e-410d-8a32-c1eaf28b5fb4\") " pod="openshift-insights/insights-runtime-extractor-ggclv"
Sep 17 17:42:06.582861 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:06.582874    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/642e943b-2b5e-410d-8a32-c1eaf28b5fb4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ggclv\" (UID: \"642e943b-2b5e-410d-8a32-c1eaf28b5fb4\") " pod="openshift-insights/insights-runtime-extractor-ggclv"
Sep 17 17:42:06.583091 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:42:06.582957    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:42:06.583091 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:42:06.582978    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:42:06.583091 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:42:06.583026    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/642e943b-2b5e-410d-8a32-c1eaf28b5fb4-kube-rbac-proxy-cm podName:642e943b-2b5e-410d-8a32-c1eaf28b5fb4 nodeName:}" failed. No retries permitted until 2026-09-17 17:42:07.583009372 +0000 UTC m=+80.087011975 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/642e943b-2b5e-410d-8a32-c1eaf28b5fb4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ggclv" (UID: "642e943b-2b5e-410d-8a32-c1eaf28b5fb4") : configmap "kube-rbac-proxy" not found
Sep 17 17:42:06.583091 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:42:06.583041    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/642e943b-2b5e-410d-8a32-c1eaf28b5fb4-insights-runtime-extractor-tls podName:642e943b-2b5e-410d-8a32-c1eaf28b5fb4 nodeName:}" failed. No retries permitted until 2026-09-17 17:42:07.583033819 +0000 UTC m=+80.087036422 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/642e943b-2b5e-410d-8a32-c1eaf28b5fb4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ggclv" (UID: "642e943b-2b5e-410d-8a32-c1eaf28b5fb4") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:42:06.766900 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:06.766819    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-x8vvv"]
Sep 17 17:42:06.769960 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:42:06.769930    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0fa8652_da03_451f_bc9d_bf02e2264e2f.slice/crio-aa74178e6fbf965b4b0abc2c8032016261f471d04c8430673f80cc143cccf048 WatchSource:0}: Error finding container aa74178e6fbf965b4b0abc2c8032016261f471d04c8430673f80cc143cccf048: Status 404 returned error can't find the container with id aa74178e6fbf965b4b0abc2c8032016261f471d04c8430673f80cc143cccf048
Sep 17 17:42:06.790926 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:06.790899    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qn9k7_42dd1d14-0348-40e6-824e-90b1763291f1/dns-node-resolver/0.log"
Sep 17 17:42:06.835326 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:06.835291    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-x8vvv" event={"ID":"d0fa8652-da03-451f-bc9d-bf02e2264e2f","Type":"ContainerStarted","Data":"aa74178e6fbf965b4b0abc2c8032016261f471d04c8430673f80cc143cccf048"}
Sep 17 17:42:07.589831 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:07.589796    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/642e943b-2b5e-410d-8a32-c1eaf28b5fb4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ggclv\" (UID: \"642e943b-2b5e-410d-8a32-c1eaf28b5fb4\") " pod="openshift-insights/insights-runtime-extractor-ggclv"
Sep 17 17:42:07.589976 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:07.589845    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/642e943b-2b5e-410d-8a32-c1eaf28b5fb4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ggclv\" (UID: \"642e943b-2b5e-410d-8a32-c1eaf28b5fb4\") " pod="openshift-insights/insights-runtime-extractor-ggclv"
Sep 17 17:42:07.589976 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:42:07.589926    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:42:07.589976 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:42:07.589949    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:42:07.590113 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:42:07.590002    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/642e943b-2b5e-410d-8a32-c1eaf28b5fb4-kube-rbac-proxy-cm podName:642e943b-2b5e-410d-8a32-c1eaf28b5fb4 nodeName:}" failed. No retries permitted until 2026-09-17 17:42:09.589984643 +0000 UTC m=+82.093987246 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/642e943b-2b5e-410d-8a32-c1eaf28b5fb4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ggclv" (UID: "642e943b-2b5e-410d-8a32-c1eaf28b5fb4") : configmap "kube-rbac-proxy" not found
Sep 17 17:42:07.590113 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:42:07.590017    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/642e943b-2b5e-410d-8a32-c1eaf28b5fb4-insights-runtime-extractor-tls podName:642e943b-2b5e-410d-8a32-c1eaf28b5fb4 nodeName:}" failed. No retries permitted until 2026-09-17 17:42:09.590011398 +0000 UTC m=+82.094014000 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/642e943b-2b5e-410d-8a32-c1eaf28b5fb4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ggclv" (UID: "642e943b-2b5e-410d-8a32-c1eaf28b5fb4") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:42:07.592511 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:07.592484    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nbmpx_1b398422-1bbe-4f75-a7e6-a641cce3f5e8/node-ca/0.log"
Sep 17 17:42:08.395364 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:08.395332    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efc7ba59-612a-4b8a-9935-f2228ec44b30-metrics-certs\") pod \"router-default-55f9595458-br8n7\" (UID: \"efc7ba59-612a-4b8a-9935-f2228ec44b30\") " pod="openshift-ingress/router-default-55f9595458-br8n7"
Sep 17 17:42:08.395782 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:08.395385    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efc7ba59-612a-4b8a-9935-f2228ec44b30-service-ca-bundle\") pod \"router-default-55f9595458-br8n7\" (UID: \"efc7ba59-612a-4b8a-9935-f2228ec44b30\") " pod="openshift-ingress/router-default-55f9595458-br8n7"
Sep 17 17:42:08.395782 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:42:08.395500    2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 17:42:08.395782 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:42:08.395543    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/efc7ba59-612a-4b8a-9935-f2228ec44b30-service-ca-bundle podName:efc7ba59-612a-4b8a-9935-f2228ec44b30 nodeName:}" failed. No retries permitted until 2026-09-17 17:42:16.395525475 +0000 UTC m=+88.899528096 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/efc7ba59-612a-4b8a-9935-f2228ec44b30-service-ca-bundle") pod "router-default-55f9595458-br8n7" (UID: "efc7ba59-612a-4b8a-9935-f2228ec44b30") : configmap references non-existent config key: service-ca.crt
Sep 17 17:42:08.395782 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:42:08.395605    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efc7ba59-612a-4b8a-9935-f2228ec44b30-metrics-certs podName:efc7ba59-612a-4b8a-9935-f2228ec44b30 nodeName:}" failed. No retries permitted until 2026-09-17 17:42:16.395593641 +0000 UTC m=+88.899596250 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/efc7ba59-612a-4b8a-9935-f2228ec44b30-metrics-certs") pod "router-default-55f9595458-br8n7" (UID: "efc7ba59-612a-4b8a-9935-f2228ec44b30") : secret "router-metrics-certs-default" not found
Sep 17 17:42:08.396685 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:08.396663    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-78r7n_5b1aeab9-ca3c-4ae4-82dc-fae81696a23a/serve-healthcheck-canary/0.log"
Sep 17 17:42:08.843408 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:08.843368    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-x8vvv" event={"ID":"d0fa8652-da03-451f-bc9d-bf02e2264e2f","Type":"ContainerStarted","Data":"0f94879c1b88088169fa3d0b2347050330a839e7c5d6f8b00a3499a2a54ba5dd"}
Sep 17 17:42:08.843627 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:08.843468    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-x8vvv" event={"ID":"d0fa8652-da03-451f-bc9d-bf02e2264e2f","Type":"ContainerStarted","Data":"4258a3dc3bf98734376225961fbe8769d2ccabcd096b17ea78089578d0e09fe2"}
Sep 17 17:42:08.866092 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:08.866036    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-x8vvv" podStartSLOduration=1.806208662 podStartE2EDuration="2.866019807s" totalImagesPullingTime="1.059811145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:42:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:08.865674288 +0000 UTC m=+81.369676913" watchObservedRunningTime="2026-09-17 17:42:08.866019807 +0000 UTC m=+81.370022431"
Sep 17 17:42:09.603256 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:09.603214    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/642e943b-2b5e-410d-8a32-c1eaf28b5fb4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ggclv\" (UID: \"642e943b-2b5e-410d-8a32-c1eaf28b5fb4\") " pod="openshift-insights/insights-runtime-extractor-ggclv"
Sep 17 17:42:09.603256 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:09.603265    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/642e943b-2b5e-410d-8a32-c1eaf28b5fb4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ggclv\" (UID: \"642e943b-2b5e-410d-8a32-c1eaf28b5fb4\") " pod="openshift-insights/insights-runtime-extractor-ggclv"
Sep 17 17:42:09.603728 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:42:09.603361    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:42:09.603728 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:42:09.603387    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:42:09.603728 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:42:09.603439    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/642e943b-2b5e-410d-8a32-c1eaf28b5fb4-insights-runtime-extractor-tls podName:642e943b-2b5e-410d-8a32-c1eaf28b5fb4 nodeName:}" failed. No retries permitted until 2026-09-17 17:42:13.603425241 +0000 UTC m=+86.107427845 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/642e943b-2b5e-410d-8a32-c1eaf28b5fb4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ggclv" (UID: "642e943b-2b5e-410d-8a32-c1eaf28b5fb4") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:42:09.603728 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:42:09.603516    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/642e943b-2b5e-410d-8a32-c1eaf28b5fb4-kube-rbac-proxy-cm podName:642e943b-2b5e-410d-8a32-c1eaf28b5fb4 nodeName:}" failed. No retries permitted until 2026-09-17 17:42:13.603471486 +0000 UTC m=+86.107474186 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/642e943b-2b5e-410d-8a32-c1eaf28b5fb4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ggclv" (UID: "642e943b-2b5e-410d-8a32-c1eaf28b5fb4") : configmap "kube-rbac-proxy" not found
Sep 17 17:42:10.926443 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:10.926389    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-t7pqb"
Sep 17 17:42:10.926443 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:10.926463    2569 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-t7pqb"
Sep 17 17:42:10.926883 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:10.926796    2569 scope.go:117] "RemoveContainer" containerID="02a9f57b767e8fe7332dbe06ceca6b2cf7db74d180cd618c5216d88c3fbeec02"
Sep 17 17:42:10.926957 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:42:10.926937    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-t7pqb_openshift-console-operator(b8d76e28-4655-4d21-98d8-0324e4756aaf)\"" pod="openshift-console-operator/console-operator-fd4bb46c-t7pqb" podUID="b8d76e28-4655-4d21-98d8-0324e4756aaf"
Sep 17 17:42:13.634948 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:13.634908    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/642e943b-2b5e-410d-8a32-c1eaf28b5fb4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ggclv\" (UID: \"642e943b-2b5e-410d-8a32-c1eaf28b5fb4\") " pod="openshift-insights/insights-runtime-extractor-ggclv"
Sep 17 17:42:13.634948 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:13.634963    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/642e943b-2b5e-410d-8a32-c1eaf28b5fb4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ggclv\" (UID: \"642e943b-2b5e-410d-8a32-c1eaf28b5fb4\") " pod="openshift-insights/insights-runtime-extractor-ggclv"
Sep 17 17:42:13.635428 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:42:13.635039    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:42:13.635428 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:42:13.635057    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:42:13.635428 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:42:13.635137    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/642e943b-2b5e-410d-8a32-c1eaf28b5fb4-insights-runtime-extractor-tls podName:642e943b-2b5e-410d-8a32-c1eaf28b5fb4 nodeName:}" failed. No retries permitted until 2026-09-17 17:42:21.635119429 +0000 UTC m=+94.139122033 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/642e943b-2b5e-410d-8a32-c1eaf28b5fb4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ggclv" (UID: "642e943b-2b5e-410d-8a32-c1eaf28b5fb4") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:42:13.635428 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:42:13.635201    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/642e943b-2b5e-410d-8a32-c1eaf28b5fb4-kube-rbac-proxy-cm podName:642e943b-2b5e-410d-8a32-c1eaf28b5fb4 nodeName:}" failed. No retries permitted until 2026-09-17 17:42:21.635182231 +0000 UTC m=+94.139184857 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/642e943b-2b5e-410d-8a32-c1eaf28b5fb4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ggclv" (UID: "642e943b-2b5e-410d-8a32-c1eaf28b5fb4") : configmap "kube-rbac-proxy" not found
Sep 17 17:42:16.457045 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:16.457011    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efc7ba59-612a-4b8a-9935-f2228ec44b30-service-ca-bundle\") pod \"router-default-55f9595458-br8n7\" (UID: \"efc7ba59-612a-4b8a-9935-f2228ec44b30\") " pod="openshift-ingress/router-default-55f9595458-br8n7"
Sep 17 17:42:16.457492 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:16.457120    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efc7ba59-612a-4b8a-9935-f2228ec44b30-metrics-certs\") pod \"router-default-55f9595458-br8n7\" (UID: \"efc7ba59-612a-4b8a-9935-f2228ec44b30\") " pod="openshift-ingress/router-default-55f9595458-br8n7"
Sep 17 17:42:16.457492 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:42:16.457225    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/efc7ba59-612a-4b8a-9935-f2228ec44b30-service-ca-bundle podName:efc7ba59-612a-4b8a-9935-f2228ec44b30 nodeName:}" failed. No retries permitted until 2026-09-17 17:42:32.45719918 +0000 UTC m=+104.961201793 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/efc7ba59-612a-4b8a-9935-f2228ec44b30-service-ca-bundle") pod "router-default-55f9595458-br8n7" (UID: "efc7ba59-612a-4b8a-9935-f2228ec44b30") : configmap references non-existent config key: service-ca.crt
Sep 17 17:42:16.459414 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:16.459397    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efc7ba59-612a-4b8a-9935-f2228ec44b30-metrics-certs\") pod \"router-default-55f9595458-br8n7\" (UID: \"efc7ba59-612a-4b8a-9935-f2228ec44b30\") " pod="openshift-ingress/router-default-55f9595458-br8n7"
Sep 17 17:42:21.695242 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:21.695199    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/642e943b-2b5e-410d-8a32-c1eaf28b5fb4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ggclv\" (UID: \"642e943b-2b5e-410d-8a32-c1eaf28b5fb4\") " pod="openshift-insights/insights-runtime-extractor-ggclv"
Sep 17 17:42:21.695707 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:21.695254    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/642e943b-2b5e-410d-8a32-c1eaf28b5fb4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ggclv\" (UID: \"642e943b-2b5e-410d-8a32-c1eaf28b5fb4\") " pod="openshift-insights/insights-runtime-extractor-ggclv"
Sep 17 17:42:21.695784 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:21.695764    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/642e943b-2b5e-410d-8a32-c1eaf28b5fb4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ggclv\" (UID: \"642e943b-2b5e-410d-8a32-c1eaf28b5fb4\") " pod="openshift-insights/insights-runtime-extractor-ggclv"
Sep 17 17:42:21.697537 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:21.697521    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/642e943b-2b5e-410d-8a32-c1eaf28b5fb4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ggclv\" (UID: \"642e943b-2b5e-410d-8a32-c1eaf28b5fb4\") " pod="openshift-insights/insights-runtime-extractor-ggclv"
Sep 17 17:42:21.993976 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:21.993945    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ggclv"
Sep 17 17:42:22.265861 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:22.265830    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ggclv"]
Sep 17 17:42:22.267647 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:42:22.267616    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod642e943b_2b5e_410d_8a32_c1eaf28b5fb4.slice/crio-657d31c63a7cf29541d30934a32e2c365f8fbfc34965b4679781da96802c77ec WatchSource:0}: Error finding container 657d31c63a7cf29541d30934a32e2c365f8fbfc34965b4679781da96802c77ec: Status 404 returned error can't find the container with id 657d31c63a7cf29541d30934a32e2c365f8fbfc34965b4679781da96802c77ec
Sep 17 17:42:22.885865 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:22.885829    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ggclv" event={"ID":"642e943b-2b5e-410d-8a32-c1eaf28b5fb4","Type":"ContainerStarted","Data":"a72e7eb47b8770e9745ba042e306bdd64f2441724bb3a8a52aa81941cde3a6ca"}
Sep 17 17:42:22.886337 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:22.885889    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ggclv" event={"ID":"642e943b-2b5e-410d-8a32-c1eaf28b5fb4","Type":"ContainerStarted","Data":"657d31c63a7cf29541d30934a32e2c365f8fbfc34965b4679781da96802c77ec"}
Sep 17 17:42:23.890877 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:23.890842    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ggclv" event={"ID":"642e943b-2b5e-410d-8a32-c1eaf28b5fb4","Type":"ContainerStarted","Data":"9c2f43ee6398da60e35c33b0a3e7474bdd859dd1a644ff77c699577a3668e4c2"}
Sep 17 17:42:24.896091 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:24.896039    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ggclv" event={"ID":"642e943b-2b5e-410d-8a32-c1eaf28b5fb4","Type":"ContainerStarted","Data":"9d3c43a9be1efcc6ced94eed4626caf6506791a2b6c5dd29fb140e046938e619"}
Sep 17 17:42:24.914293 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:24.914235    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-ggclv" podStartSLOduration=18.93447491 podStartE2EDuration="20.914216625s" totalImagesPullingTime="1.979741715s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:42:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:24.913122039 +0000 UTC m=+97.417124661" watchObservedRunningTime="2026-09-17 17:42:24.914216625 +0000 UTC m=+97.418219250"
Sep 17 17:42:25.503483 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:25.503445    2569 scope.go:117] "RemoveContainer" containerID="02a9f57b767e8fe7332dbe06ceca6b2cf7db74d180cd618c5216d88c3fbeec02"
Sep 17 17:42:25.900733 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:25.900656    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-t7pqb_b8d76e28-4655-4d21-98d8-0324e4756aaf/console-operator/2.log"
Sep 17 17:42:25.901183 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:25.901144    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-t7pqb_b8d76e28-4655-4d21-98d8-0324e4756aaf/console-operator/1.log"
Sep 17 17:42:25.901183 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:25.901175    2569 generic.go:356] "Generic (PLEG): container finished" podID="b8d76e28-4655-4d21-98d8-0324e4756aaf" containerID="22aa916511d1b418b854d97971fb1a3ea13605026410231a2396dbe66dd54824" exitCode=255
Sep 17 17:42:25.901267 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:25.901247    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-t7pqb" event={"ID":"b8d76e28-4655-4d21-98d8-0324e4756aaf","Type":"ContainerDied","Data":"22aa916511d1b418b854d97971fb1a3ea13605026410231a2396dbe66dd54824"}
Sep 17 17:42:25.901316 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:25.901306    2569 scope.go:117] "RemoveContainer" containerID="02a9f57b767e8fe7332dbe06ceca6b2cf7db74d180cd618c5216d88c3fbeec02"
Sep 17 17:42:25.901653 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:25.901613    2569 scope.go:117] "RemoveContainer" containerID="22aa916511d1b418b854d97971fb1a3ea13605026410231a2396dbe66dd54824"
Sep 17 17:42:25.901800 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:42:25.901781    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-t7pqb_openshift-console-operator(b8d76e28-4655-4d21-98d8-0324e4756aaf)\"" pod="openshift-console-operator/console-operator-fd4bb46c-t7pqb" podUID="b8d76e28-4655-4d21-98d8-0324e4756aaf"
Sep 17 17:42:26.434796 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:26.434742    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/023f4b84-91bb-4103-97cb-5123dc08e459-metrics-tls\") pod \"dns-default-6lkgm\" (UID: \"023f4b84-91bb-4103-97cb-5123dc08e459\") " pod="openshift-dns/dns-default-6lkgm"
Sep 17 17:42:26.437061 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:26.437039    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/023f4b84-91bb-4103-97cb-5123dc08e459-metrics-tls\") pod \"dns-default-6lkgm\" (UID: \"023f4b84-91bb-4103-97cb-5123dc08e459\") " pod="openshift-dns/dns-default-6lkgm"
Sep 17 17:42:26.474071 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:26.474036    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4db82\""
Sep 17 17:42:26.481906 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:26.481880    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6lkgm"
Sep 17 17:42:26.758275 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:26.758244    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6lkgm"]
Sep 17 17:42:26.760908 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:42:26.760869    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod023f4b84_91bb_4103_97cb_5123dc08e459.slice/crio-95efb353455603ad1d0efb3e0a78bdef04d58d83075727987ba022139763892e WatchSource:0}: Error finding container 95efb353455603ad1d0efb3e0a78bdef04d58d83075727987ba022139763892e: Status 404 returned error can't find the container with id 95efb353455603ad1d0efb3e0a78bdef04d58d83075727987ba022139763892e
Sep 17 17:42:26.906548 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:26.906506    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6lkgm" event={"ID":"023f4b84-91bb-4103-97cb-5123dc08e459","Type":"ContainerStarted","Data":"95efb353455603ad1d0efb3e0a78bdef04d58d83075727987ba022139763892e"}
Sep 17 17:42:26.908017 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:26.907997    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-t7pqb_b8d76e28-4655-4d21-98d8-0324e4756aaf/console-operator/2.log"
Sep 17 17:42:28.916238 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:28.916204    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6lkgm" event={"ID":"023f4b84-91bb-4103-97cb-5123dc08e459","Type":"ContainerStarted","Data":"1963c1a0f1ce132bcde9de9980d1acd0082e0f5cee1da1a8a8878ad21af9fac9"}
Sep 17 17:42:28.916597 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:28.916259    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6lkgm" event={"ID":"023f4b84-91bb-4103-97cb-5123dc08e459","Type":"ContainerStarted","Data":"7de728029c3a73393296b1a7580aafde39e8ac557cb056fadc23c38f37c3f184"}
Sep 17 17:42:28.916597 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:28.916331    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-6lkgm"
Sep 17 17:42:28.934714 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:28.934654    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-6lkgm" podStartSLOduration=65.532155181 podStartE2EDuration="1m6.934635468s" totalImagesPullingTime="1.402480287s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:41:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:28.933992153 +0000 UTC m=+101.437994781" watchObservedRunningTime="2026-09-17 17:42:28.934635468 +0000 UTC m=+101.438638093"
Sep 17 17:42:30.926804 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:30.926773    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-t7pqb"
Sep 17 17:42:30.927210 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:30.926820    2569 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-t7pqb"
Sep 17 17:42:30.927210 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:30.927126    2569 scope.go:117] "RemoveContainer" containerID="22aa916511d1b418b854d97971fb1a3ea13605026410231a2396dbe66dd54824"
Sep 17 17:42:30.927284 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:42:30.927267    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-t7pqb_openshift-console-operator(b8d76e28-4655-4d21-98d8-0324e4756aaf)\"" pod="openshift-console-operator/console-operator-fd4bb46c-t7pqb" podUID="b8d76e28-4655-4d21-98d8-0324e4756aaf"
Sep 17 17:42:31.252900 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:31.252863    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b88778f9-qc9tc"]
Sep 17 17:42:31.255889 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:31.255865    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b88778f9-qc9tc"
Sep 17 17:42:31.259528 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:31.259503    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 17 17:42:31.260134 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:31.260113    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 17 17:42:31.260345 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:31.260328    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 17 17:42:31.260563 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:31.260549    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 17 17:42:31.269426 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:31.269399    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b88778f9-qc9tc"]
Sep 17 17:42:31.368322 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:31.368285    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/978886ca-f497-4c29-97f7-d5683ab1f862-tmp\") pod \"klusterlet-addon-workmgr-7b88778f9-qc9tc\" (UID: \"978886ca-f497-4c29-97f7-d5683ab1f862\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b88778f9-qc9tc"
Sep 17 17:42:31.368470 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:31.368400    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/978886ca-f497-4c29-97f7-d5683ab1f862-klusterlet-config\") pod \"klusterlet-addon-workmgr-7b88778f9-qc9tc\" (UID: \"978886ca-f497-4c29-97f7-d5683ab1f862\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b88778f9-qc9tc"
Sep 17 17:42:31.368470 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:31.368437    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v66vt\" (UniqueName: \"kubernetes.io/projected/978886ca-f497-4c29-97f7-d5683ab1f862-kube-api-access-v66vt\") pod \"klusterlet-addon-workmgr-7b88778f9-qc9tc\" (UID: \"978886ca-f497-4c29-97f7-d5683ab1f862\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b88778f9-qc9tc"
Sep 17 17:42:31.469435 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:31.469395    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/978886ca-f497-4c29-97f7-d5683ab1f862-klusterlet-config\") pod \"klusterlet-addon-workmgr-7b88778f9-qc9tc\" (UID: \"978886ca-f497-4c29-97f7-d5683ab1f862\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b88778f9-qc9tc"
Sep 17 17:42:31.469612 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:31.469456    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v66vt\" (UniqueName: \"kubernetes.io/projected/978886ca-f497-4c29-97f7-d5683ab1f862-kube-api-access-v66vt\") pod \"klusterlet-addon-workmgr-7b88778f9-qc9tc\" (UID: \"978886ca-f497-4c29-97f7-d5683ab1f862\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b88778f9-qc9tc"
Sep 17 17:42:31.469612 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:31.469485    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/978886ca-f497-4c29-97f7-d5683ab1f862-tmp\") pod \"klusterlet-addon-workmgr-7b88778f9-qc9tc\" (UID: \"978886ca-f497-4c29-97f7-d5683ab1f862\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b88778f9-qc9tc"
Sep 17 17:42:31.469871 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:31.469854    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/978886ca-f497-4c29-97f7-d5683ab1f862-tmp\") pod \"klusterlet-addon-workmgr-7b88778f9-qc9tc\" (UID: \"978886ca-f497-4c29-97f7-d5683ab1f862\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b88778f9-qc9tc"
Sep 17 17:42:31.471859 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:31.471841    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/978886ca-f497-4c29-97f7-d5683ab1f862-klusterlet-config\") pod \"klusterlet-addon-workmgr-7b88778f9-qc9tc\" (UID: \"978886ca-f497-4c29-97f7-d5683ab1f862\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b88778f9-qc9tc"
Sep 17 17:42:31.479436 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:31.479409    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v66vt\" (UniqueName: \"kubernetes.io/projected/978886ca-f497-4c29-97f7-d5683ab1f862-kube-api-access-v66vt\") pod \"klusterlet-addon-workmgr-7b88778f9-qc9tc\" (UID: \"978886ca-f497-4c29-97f7-d5683ab1f862\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b88778f9-qc9tc"
Sep 17 17:42:31.564859 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:31.564765    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b88778f9-qc9tc"
Sep 17 17:42:31.831263 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:31.831184    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b88778f9-qc9tc"]
Sep 17 17:42:31.833913 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:42:31.833885    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod978886ca_f497_4c29_97f7_d5683ab1f862.slice/crio-32f1aa258958a98dfa75dfcfabd3727937d2fac39fbb791378ad76a8726727b8 WatchSource:0}: Error finding container 32f1aa258958a98dfa75dfcfabd3727937d2fac39fbb791378ad76a8726727b8: Status 404 returned error can't find the container with id 32f1aa258958a98dfa75dfcfabd3727937d2fac39fbb791378ad76a8726727b8
Sep 17 17:42:31.926651 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:31.926612    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b88778f9-qc9tc" event={"ID":"978886ca-f497-4c29-97f7-d5683ab1f862","Type":"ContainerStarted","Data":"32f1aa258958a98dfa75dfcfabd3727937d2fac39fbb791378ad76a8726727b8"}
Sep 17 17:42:32.477595 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:32.477556    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efc7ba59-612a-4b8a-9935-f2228ec44b30-service-ca-bundle\") pod \"router-default-55f9595458-br8n7\" (UID: \"efc7ba59-612a-4b8a-9935-f2228ec44b30\") " pod="openshift-ingress/router-default-55f9595458-br8n7"
Sep 17 17:42:32.478248 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:32.478229    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efc7ba59-612a-4b8a-9935-f2228ec44b30-service-ca-bundle\") pod \"router-default-55f9595458-br8n7\" (UID: \"efc7ba59-612a-4b8a-9935-f2228ec44b30\") " pod="openshift-ingress/router-default-55f9595458-br8n7"
Sep 17 17:42:32.721318 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:32.721274    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-55f9595458-br8n7"
Sep 17 17:42:33.000621 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:33.000457    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-55f9595458-br8n7"]
Sep 17 17:42:33.004352 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:42:33.004316    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefc7ba59_612a_4b8a_9935_f2228ec44b30.slice/crio-5cbc259932e0c7910294036cbe80d8b33b230c1361462deec0558f999239a741 WatchSource:0}: Error finding container 5cbc259932e0c7910294036cbe80d8b33b230c1361462deec0558f999239a741: Status 404 returned error can't find the container with id 5cbc259932e0c7910294036cbe80d8b33b230c1361462deec0558f999239a741
Sep 17 17:42:33.936320 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:33.936063    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-55f9595458-br8n7" event={"ID":"efc7ba59-612a-4b8a-9935-f2228ec44b30","Type":"ContainerStarted","Data":"8a1d0642b12fe485a5ae8ef90a4fc7cdca6aa1ffff2e8c8a7da0f7b024d97ae1"}
Sep 17 17:42:33.936805 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:33.936350    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-55f9595458-br8n7" event={"ID":"efc7ba59-612a-4b8a-9935-f2228ec44b30","Type":"ContainerStarted","Data":"5cbc259932e0c7910294036cbe80d8b33b230c1361462deec0558f999239a741"}
Sep 17 17:42:33.971045 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:33.970983    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-55f9595458-br8n7" podStartSLOduration=33.970961093 podStartE2EDuration="33.970961093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:42:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:33.970437126 +0000 UTC m=+106.474439752" watchObservedRunningTime="2026-09-17 17:42:33.970961093 +0000 UTC m=+106.474963720"
Sep 17 17:42:34.721452 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:34.721411    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-55f9595458-br8n7"
Sep 17 17:42:34.724571 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:34.724543    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-55f9595458-br8n7"
Sep 17 17:42:34.939938 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:34.939903    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-55f9595458-br8n7"
Sep 17 17:42:34.941379 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:34.941358    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-55f9595458-br8n7"
Sep 17 17:42:35.944721 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:35.944683    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b88778f9-qc9tc" event={"ID":"978886ca-f497-4c29-97f7-d5683ab1f862","Type":"ContainerStarted","Data":"0cfe823a1c6f15d5c144a034241531953966ba799e7e43b1a5e4ab47873b466a"}
Sep 17 17:42:35.945231 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:35.944779    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b88778f9-qc9tc"
Sep 17 17:42:35.946493 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:35.946476    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b88778f9-qc9tc"
Sep 17 17:42:35.963416 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:35.963374    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b88778f9-qc9tc" podStartSLOduration=1.00836874 podStartE2EDuration="4.963358587s" totalImagesPullingTime="3.954989847s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:42:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:35.962195214 +0000 UTC m=+108.466197839" watchObservedRunningTime="2026-09-17 17:42:35.963358587 +0000 UTC m=+108.467361215"
Sep 17 17:42:36.792547 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:36.792512    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-7m22j"]
Sep 17 17:42:36.795438 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:36.795423    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-7m22j"
Sep 17 17:42:36.800802 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:36.800781    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 17 17:42:36.800932 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:36.800787    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-4tfz8\""
Sep 17 17:42:36.805287 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:36.805266    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-7m22j"]
Sep 17 17:42:36.918034 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:36.917978    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b54d572b-971f-4e9b-9f00-c1b8b3eb9188-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-7m22j\" (UID: \"b54d572b-971f-4e9b-9f00-c1b8b3eb9188\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-7m22j"
Sep 17 17:42:37.019827 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:37.019738    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b54d572b-971f-4e9b-9f00-c1b8b3eb9188-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-7m22j\" (UID: \"b54d572b-971f-4e9b-9f00-c1b8b3eb9188\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-7m22j"
Sep 17 17:42:37.024543 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:37.024471    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b54d572b-971f-4e9b-9f00-c1b8b3eb9188-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-7m22j\" (UID: \"b54d572b-971f-4e9b-9f00-c1b8b3eb9188\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-7m22j"
Sep 17 17:42:37.103751 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:37.103663    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-7m22j"
Sep 17 17:42:37.372022 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:37.371936    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-7m22j"]
Sep 17 17:42:37.374705 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:42:37.374679    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb54d572b_971f_4e9b_9f00_c1b8b3eb9188.slice/crio-c08d6c6ef283f146dc7b77789fb29532207478c15686ddf90c8f7ada65cc3eda WatchSource:0}: Error finding container c08d6c6ef283f146dc7b77789fb29532207478c15686ddf90c8f7ada65cc3eda: Status 404 returned error can't find the container with id c08d6c6ef283f146dc7b77789fb29532207478c15686ddf90c8f7ada65cc3eda
Sep 17 17:42:37.953142 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:37.953104    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-7m22j" event={"ID":"b54d572b-971f-4e9b-9f00-c1b8b3eb9188","Type":"ContainerStarted","Data":"c08d6c6ef283f146dc7b77789fb29532207478c15686ddf90c8f7ada65cc3eda"}
Sep 17 17:42:38.921548 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:38.921518    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6lkgm"
Sep 17 17:42:38.958535 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:38.958504    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-7m22j" event={"ID":"b54d572b-971f-4e9b-9f00-c1b8b3eb9188","Type":"ContainerStarted","Data":"b3e824493399c2290a5307774dceaa27ccb0a8cf5c5f68a083b5ffd29fc09d08"}
Sep 17 17:42:38.958728 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:38.958680    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-7m22j"
Sep 17 17:42:38.963893 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:38.963867    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-7m22j"
Sep 17 17:42:38.979584 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:38.979535    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-7m22j" podStartSLOduration=2.003894352 podStartE2EDuration="2.97951795s" totalImagesPullingTime="975.623598ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:42:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:38.97870424 +0000 UTC m=+111.482706866" watchObservedRunningTime="2026-09-17 17:42:38.97951795 +0000 UTC m=+111.483520575"
Sep 17 17:42:39.866183 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:39.866146    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-c7xwv"]
Sep 17 17:42:39.869398 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:39.869380    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-c7xwv"
Sep 17 17:42:39.872219 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:39.872194    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 17 17:42:39.873436 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:39.873415    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-zrq7m\""
Sep 17 17:42:39.873561 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:39.873447    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 17 17:42:39.873561 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:39.873462    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 17:42:39.873666 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:39.873583    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 17:42:39.873726 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:39.873710    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 17:42:39.879804 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:39.879774    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-c7xwv"]
Sep 17 17:42:39.943435 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:39.943402    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4c85ebb0-dc17-4937-9dca-9d87547620f3-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-c7xwv\" (UID: \"4c85ebb0-dc17-4937-9dca-9d87547620f3\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-c7xwv"
Sep 17 17:42:39.943435 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:39.943439    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtxfk\" (UniqueName: \"kubernetes.io/projected/4c85ebb0-dc17-4937-9dca-9d87547620f3-kube-api-access-xtxfk\") pod \"prometheus-operator-7bd5d56d6-c7xwv\" (UID: \"4c85ebb0-dc17-4937-9dca-9d87547620f3\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-c7xwv"
Sep 17 17:42:39.943843 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:39.943467    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4c85ebb0-dc17-4937-9dca-9d87547620f3-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-c7xwv\" (UID: \"4c85ebb0-dc17-4937-9dca-9d87547620f3\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-c7xwv"
Sep 17 17:42:39.943843 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:39.943495    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4c85ebb0-dc17-4937-9dca-9d87547620f3-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-c7xwv\" (UID: \"4c85ebb0-dc17-4937-9dca-9d87547620f3\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-c7xwv"
Sep 17 17:42:40.044774 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:40.044741    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4c85ebb0-dc17-4937-9dca-9d87547620f3-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-c7xwv\" (UID: \"4c85ebb0-dc17-4937-9dca-9d87547620f3\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-c7xwv"
Sep 17 17:42:40.044958 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:40.044787    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4c85ebb0-dc17-4937-9dca-9d87547620f3-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-c7xwv\" (UID: \"4c85ebb0-dc17-4937-9dca-9d87547620f3\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-c7xwv"
Sep 17 17:42:40.044958 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:40.044857    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4c85ebb0-dc17-4937-9dca-9d87547620f3-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-c7xwv\" (UID: \"4c85ebb0-dc17-4937-9dca-9d87547620f3\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-c7xwv"
Sep 17 17:42:40.044958 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:40.044878    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtxfk\" (UniqueName: \"kubernetes.io/projected/4c85ebb0-dc17-4937-9dca-9d87547620f3-kube-api-access-xtxfk\") pod \"prometheus-operator-7bd5d56d6-c7xwv\" (UID: \"4c85ebb0-dc17-4937-9dca-9d87547620f3\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-c7xwv"
Sep 17 17:42:40.045641 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:40.045619    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4c85ebb0-dc17-4937-9dca-9d87547620f3-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-c7xwv\" (UID: \"4c85ebb0-dc17-4937-9dca-9d87547620f3\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-c7xwv"
Sep 17 17:42:40.047359 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:40.047324    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4c85ebb0-dc17-4937-9dca-9d87547620f3-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-c7xwv\" (UID: \"4c85ebb0-dc17-4937-9dca-9d87547620f3\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-c7xwv"
Sep 17 17:42:40.047472 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:40.047397    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4c85ebb0-dc17-4937-9dca-9d87547620f3-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-c7xwv\" (UID: \"4c85ebb0-dc17-4937-9dca-9d87547620f3\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-c7xwv"
Sep 17 17:42:40.052870 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:40.052847    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtxfk\" (UniqueName: \"kubernetes.io/projected/4c85ebb0-dc17-4937-9dca-9d87547620f3-kube-api-access-xtxfk\") pod \"prometheus-operator-7bd5d56d6-c7xwv\" (UID: \"4c85ebb0-dc17-4937-9dca-9d87547620f3\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-c7xwv"
Sep 17 17:42:40.178937 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:40.178856    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-c7xwv"
Sep 17 17:42:40.449744 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:40.449715    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-c7xwv"]
Sep 17 17:42:40.451789 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:42:40.451762    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c85ebb0_dc17_4937_9dca_9d87547620f3.slice/crio-10ff47a0da2a9810ed2bc351fa1b8fbaa1ce27bff31443f05459105d9d9c1d39 WatchSource:0}: Error finding container 10ff47a0da2a9810ed2bc351fa1b8fbaa1ce27bff31443f05459105d9d9c1d39: Status 404 returned error can't find the container with id 10ff47a0da2a9810ed2bc351fa1b8fbaa1ce27bff31443f05459105d9d9c1d39
Sep 17 17:42:40.966391 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:40.966354    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-c7xwv" event={"ID":"4c85ebb0-dc17-4937-9dca-9d87547620f3","Type":"ContainerStarted","Data":"10ff47a0da2a9810ed2bc351fa1b8fbaa1ce27bff31443f05459105d9d9c1d39"}
Sep 17 17:42:45.503703 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:45.503669    2569 scope.go:117] "RemoveContainer" containerID="22aa916511d1b418b854d97971fb1a3ea13605026410231a2396dbe66dd54824"
Sep 17 17:42:45.504161 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:42:45.503874    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-t7pqb_openshift-console-operator(b8d76e28-4655-4d21-98d8-0324e4756aaf)\"" pod="openshift-console-operator/console-operator-fd4bb46c-t7pqb" podUID="b8d76e28-4655-4d21-98d8-0324e4756aaf"
Sep 17 17:42:45.984468 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:45.984425    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-c7xwv" event={"ID":"4c85ebb0-dc17-4937-9dca-9d87547620f3","Type":"ContainerStarted","Data":"f8d26b0b2d039268b31d46680c2b0e8c5488e9e418d796cad154143f689b23ad"}
Sep 17 17:42:45.984653 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:45.984487    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-c7xwv" event={"ID":"4c85ebb0-dc17-4937-9dca-9d87547620f3","Type":"ContainerStarted","Data":"0b526292f0f650cd4b8e032b321aaa94dae14a4bac435de0518604c5e10d9082"}
Sep 17 17:42:46.004205 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:46.004152    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-c7xwv" podStartSLOduration=2.503362921 podStartE2EDuration="7.004133907s" totalImagesPullingTime="4.500770986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:42:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:46.003710646 +0000 UTC m=+118.507713271" watchObservedRunningTime="2026-09-17 17:42:46.004133907 +0000 UTC m=+118.508136532"
Sep 17 17:42:48.494930 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:48.494890    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-kfnnp"]
Sep 17 17:42:48.498397 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:48.498374    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kfnnp"
Sep 17 17:42:48.501844 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:48.501811    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rvws2\""
Sep 17 17:42:48.502053 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:48.502027    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 17:42:48.502417 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:48.502396    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 17:42:48.502826 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:48.502806    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 17:42:48.621426 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:48.621395    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c5650e98-c78e-42cf-8c37-f83e002791aa-node-exporter-textfile\") pod \"node-exporter-kfnnp\" (UID: \"c5650e98-c78e-42cf-8c37-f83e002791aa\") " pod="openshift-monitoring/node-exporter-kfnnp"
Sep 17 17:42:48.621426 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:48.621430    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c5650e98-c78e-42cf-8c37-f83e002791aa-node-exporter-tls\") pod \"node-exporter-kfnnp\" (UID: \"c5650e98-c78e-42cf-8c37-f83e002791aa\") " pod="openshift-monitoring/node-exporter-kfnnp"
Sep 17 17:42:48.621632 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:48.621449    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c5650e98-c78e-42cf-8c37-f83e002791aa-root\") pod \"node-exporter-kfnnp\" (UID: \"c5650e98-c78e-42cf-8c37-f83e002791aa\") " pod="openshift-monitoring/node-exporter-kfnnp"
Sep 17 17:42:48.621632 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:48.621540    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c5650e98-c78e-42cf-8c37-f83e002791aa-node-exporter-accelerators-collector-config\") pod \"node-exporter-kfnnp\" (UID: \"c5650e98-c78e-42cf-8c37-f83e002791aa\") " pod="openshift-monitoring/node-exporter-kfnnp"
Sep 17 17:42:48.621632 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:48.621577    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c5650e98-c78e-42cf-8c37-f83e002791aa-metrics-client-ca\") pod \"node-exporter-kfnnp\" (UID: \"c5650e98-c78e-42cf-8c37-f83e002791aa\") " pod="openshift-monitoring/node-exporter-kfnnp"
Sep 17 17:42:48.621632 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:48.621596    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c5650e98-c78e-42cf-8c37-f83e002791aa-sys\") pod \"node-exporter-kfnnp\" (UID: \"c5650e98-c78e-42cf-8c37-f83e002791aa\") " pod="openshift-monitoring/node-exporter-kfnnp"
Sep 17 17:42:48.621632 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:48.621611    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c5650e98-c78e-42cf-8c37-f83e002791aa-node-exporter-wtmp\") pod \"node-exporter-kfnnp\" (UID: \"c5650e98-c78e-42cf-8c37-f83e002791aa\") " pod="openshift-monitoring/node-exporter-kfnnp"
Sep 17 17:42:48.621789 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:48.621657    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c5650e98-c78e-42cf-8c37-f83e002791aa-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kfnnp\" (UID: \"c5650e98-c78e-42cf-8c37-f83e002791aa\") " pod="openshift-monitoring/node-exporter-kfnnp"
Sep 17 17:42:48.621789 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:48.621716    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m9r5\" (UniqueName: \"kubernetes.io/projected/c5650e98-c78e-42cf-8c37-f83e002791aa-kube-api-access-8m9r5\") pod \"node-exporter-kfnnp\" (UID: \"c5650e98-c78e-42cf-8c37-f83e002791aa\") " pod="openshift-monitoring/node-exporter-kfnnp"
Sep 17 17:42:48.722833 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:48.722802    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c5650e98-c78e-42cf-8c37-f83e002791aa-node-exporter-accelerators-collector-config\") pod \"node-exporter-kfnnp\" (UID: \"c5650e98-c78e-42cf-8c37-f83e002791aa\") " pod="openshift-monitoring/node-exporter-kfnnp"
Sep 17 17:42:48.722833 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:48.722835    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c5650e98-c78e-42cf-8c37-f83e002791aa-metrics-client-ca\") pod \"node-exporter-kfnnp\" (UID: \"c5650e98-c78e-42cf-8c37-f83e002791aa\") " pod="openshift-monitoring/node-exporter-kfnnp"
Sep 17 17:42:48.723062 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:48.722853    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c5650e98-c78e-42cf-8c37-f83e002791aa-sys\") pod \"node-exporter-kfnnp\" (UID: \"c5650e98-c78e-42cf-8c37-f83e002791aa\") " pod="openshift-monitoring/node-exporter-kfnnp"
Sep 17 17:42:48.723062 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:48.722869    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c5650e98-c78e-42cf-8c37-f83e002791aa-node-exporter-wtmp\") pod \"node-exporter-kfnnp\" (UID: \"c5650e98-c78e-42cf-8c37-f83e002791aa\") " pod="openshift-monitoring/node-exporter-kfnnp"
Sep 17 17:42:48.723062 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:48.722894    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c5650e98-c78e-42cf-8c37-f83e002791aa-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kfnnp\" (UID: \"c5650e98-c78e-42cf-8c37-f83e002791aa\") " pod="openshift-monitoring/node-exporter-kfnnp"
Sep 17 17:42:48.723062 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:48.722947    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m9r5\" (UniqueName: \"kubernetes.io/projected/c5650e98-c78e-42cf-8c37-f83e002791aa-kube-api-access-8m9r5\") pod \"node-exporter-kfnnp\" (UID: \"c5650e98-c78e-42cf-8c37-f83e002791aa\") " pod="openshift-monitoring/node-exporter-kfnnp"
Sep 17 17:42:48.723062 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:48.722954    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c5650e98-c78e-42cf-8c37-f83e002791aa-sys\") pod \"node-exporter-kfnnp\" (UID: \"c5650e98-c78e-42cf-8c37-f83e002791aa\") " pod="openshift-monitoring/node-exporter-kfnnp"
Sep 17 17:42:48.723062 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:48.722990    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c5650e98-c78e-42cf-8c37-f83e002791aa-node-exporter-textfile\") pod \"node-exporter-kfnnp\" (UID: \"c5650e98-c78e-42cf-8c37-f83e002791aa\") " pod="openshift-monitoring/node-exporter-kfnnp"
Sep 17 17:42:48.723062 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:48.723035    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c5650e98-c78e-42cf-8c37-f83e002791aa-node-exporter-wtmp\") pod \"node-exporter-kfnnp\" (UID: \"c5650e98-c78e-42cf-8c37-f83e002791aa\") " pod="openshift-monitoring/node-exporter-kfnnp"
Sep 17 17:42:48.723412 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:48.723142    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c5650e98-c78e-42cf-8c37-f83e002791aa-node-exporter-tls\") pod \"node-exporter-kfnnp\" (UID: \"c5650e98-c78e-42cf-8c37-f83e002791aa\") " pod="openshift-monitoring/node-exporter-kfnnp"
Sep 17 17:42:48.723412 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:48.723175    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c5650e98-c78e-42cf-8c37-f83e002791aa-root\") pod \"node-exporter-kfnnp\" (UID: \"c5650e98-c78e-42cf-8c37-f83e002791aa\") " pod="openshift-monitoring/node-exporter-kfnnp"
Sep 17 17:42:48.723412 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:48.723263    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c5650e98-c78e-42cf-8c37-f83e002791aa-root\") pod \"node-exporter-kfnnp\" (UID: \"c5650e98-c78e-42cf-8c37-f83e002791aa\") " pod="openshift-monitoring/node-exporter-kfnnp"
Sep 17 17:42:48.723412 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:48.723346    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c5650e98-c78e-42cf-8c37-f83e002791aa-node-exporter-textfile\") pod \"node-exporter-kfnnp\" (UID: \"c5650e98-c78e-42cf-8c37-f83e002791aa\") " pod="openshift-monitoring/node-exporter-kfnnp"
Sep 17 17:42:48.723582 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:48.723486    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c5650e98-c78e-42cf-8c37-f83e002791aa-metrics-client-ca\") pod \"node-exporter-kfnnp\" (UID: \"c5650e98-c78e-42cf-8c37-f83e002791aa\") " pod="openshift-monitoring/node-exporter-kfnnp"
Sep 17 17:42:48.723582 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:48.723517    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c5650e98-c78e-42cf-8c37-f83e002791aa-node-exporter-accelerators-collector-config\") pod \"node-exporter-kfnnp\" (UID: \"c5650e98-c78e-42cf-8c37-f83e002791aa\") " pod="openshift-monitoring/node-exporter-kfnnp"
Sep 17 17:42:48.725535 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:48.725502    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c5650e98-c78e-42cf-8c37-f83e002791aa-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kfnnp\" (UID: \"c5650e98-c78e-42cf-8c37-f83e002791aa\") " pod="openshift-monitoring/node-exporter-kfnnp"
Sep 17 17:42:48.725656 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:48.725495    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c5650e98-c78e-42cf-8c37-f83e002791aa-node-exporter-tls\") pod \"node-exporter-kfnnp\" (UID: \"c5650e98-c78e-42cf-8c37-f83e002791aa\") " pod="openshift-monitoring/node-exporter-kfnnp"
Sep 17 17:42:48.731676 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:48.731656    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m9r5\" (UniqueName: \"kubernetes.io/projected/c5650e98-c78e-42cf-8c37-f83e002791aa-kube-api-access-8m9r5\") pod \"node-exporter-kfnnp\" (UID: \"c5650e98-c78e-42cf-8c37-f83e002791aa\") " pod="openshift-monitoring/node-exporter-kfnnp"
Sep 17 17:42:48.808528 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:48.808441    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kfnnp"
Sep 17 17:42:48.819987 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:42:48.819957    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5650e98_c78e_42cf_8c37_f83e002791aa.slice/crio-75005c463b66d4b19550678445ac7240489b61a57f2953b3e7e5d93e8003631c WatchSource:0}: Error finding container 75005c463b66d4b19550678445ac7240489b61a57f2953b3e7e5d93e8003631c: Status 404 returned error can't find the container with id 75005c463b66d4b19550678445ac7240489b61a57f2953b3e7e5d93e8003631c
Sep 17 17:42:48.996061 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:48.996019    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kfnnp" event={"ID":"c5650e98-c78e-42cf-8c37-f83e002791aa","Type":"ContainerStarted","Data":"75005c463b66d4b19550678445ac7240489b61a57f2953b3e7e5d93e8003631c"}
Sep 17 17:42:50.001187 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:50.001144    2569 generic.go:356] "Generic (PLEG): container finished" podID="c5650e98-c78e-42cf-8c37-f83e002791aa" containerID="669a495c1195a580e15982db66c5037a704dbd97f490a8f4ad2e51f7547fb56b" exitCode=0
Sep 17 17:42:50.001608 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:50.001194    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kfnnp" event={"ID":"c5650e98-c78e-42cf-8c37-f83e002791aa","Type":"ContainerDied","Data":"669a495c1195a580e15982db66c5037a704dbd97f490a8f4ad2e51f7547fb56b"}
Sep 17 17:42:51.006325 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:51.006289    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kfnnp" event={"ID":"c5650e98-c78e-42cf-8c37-f83e002791aa","Type":"ContainerStarted","Data":"fa9d8120f8a393fa258e2a0c6b03fbe0beca7f0754cac8b3327e92ff076e462a"}
Sep 17 17:42:51.006748 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:51.006342    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kfnnp" event={"ID":"c5650e98-c78e-42cf-8c37-f83e002791aa","Type":"ContainerStarted","Data":"f8602788588d5ae040fe8156dfc1299cd209f60bb0f192415ffe67e04683b947"}
Sep 17 17:42:51.030202 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:51.030141    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-kfnnp" podStartSLOduration=1.953829887 podStartE2EDuration="3.03012407s" totalImagesPullingTime="1.014664438s" totalInitContainerRuntime="61.629745ms" isStatefulPod=true podCreationTimestamp="2026-09-17 17:42:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:51.028058922 +0000 UTC m=+123.532061546" watchObservedRunningTime="2026-09-17 17:42:51.03012407 +0000 UTC m=+123.534126739"
Sep 17 17:42:52.856018 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:52.855977    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-74d659dd7f-k5rqx"]
Sep 17 17:42:52.859067 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:52.859049    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-74d659dd7f-k5rqx"
Sep 17 17:42:52.862287 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:52.862246    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-xdkjh\""
Sep 17 17:42:52.863998 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:52.863974    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 17 17:42:52.864235 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:52.864212    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 17 17:42:52.864447 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:52.864426    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 17 17:42:52.864519 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:52.863989    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-64hbch8o0d914\""
Sep 17 17:42:52.864713 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:52.864697    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 17 17:42:52.875458 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:52.875437    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-74d659dd7f-k5rqx"]
Sep 17 17:42:52.962380 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:52.962342    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f91a3eda-11bd-4add-a5e7-a5491083081e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-74d659dd7f-k5rqx\" (UID: \"f91a3eda-11bd-4add-a5e7-a5491083081e\") " pod="openshift-monitoring/metrics-server-74d659dd7f-k5rqx"
Sep 17 17:42:52.962380 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:52.962380    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f91a3eda-11bd-4add-a5e7-a5491083081e-secret-metrics-server-tls\") pod \"metrics-server-74d659dd7f-k5rqx\" (UID: \"f91a3eda-11bd-4add-a5e7-a5491083081e\") " pod="openshift-monitoring/metrics-server-74d659dd7f-k5rqx"
Sep 17 17:42:52.962628 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:52.962454    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f91a3eda-11bd-4add-a5e7-a5491083081e-metrics-server-audit-profiles\") pod \"metrics-server-74d659dd7f-k5rqx\" (UID: \"f91a3eda-11bd-4add-a5e7-a5491083081e\") " pod="openshift-monitoring/metrics-server-74d659dd7f-k5rqx"
Sep 17 17:42:52.962628 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:52.962523    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f91a3eda-11bd-4add-a5e7-a5491083081e-client-ca-bundle\") pod \"metrics-server-74d659dd7f-k5rqx\" (UID: \"f91a3eda-11bd-4add-a5e7-a5491083081e\") " pod="openshift-monitoring/metrics-server-74d659dd7f-k5rqx"
Sep 17 17:42:52.962628 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:52.962545    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrcm6\" (UniqueName: \"kubernetes.io/projected/f91a3eda-11bd-4add-a5e7-a5491083081e-kube-api-access-jrcm6\") pod \"metrics-server-74d659dd7f-k5rqx\" (UID: \"f91a3eda-11bd-4add-a5e7-a5491083081e\") " pod="openshift-monitoring/metrics-server-74d659dd7f-k5rqx"
Sep 17 17:42:52.962788 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:52.962627    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f91a3eda-11bd-4add-a5e7-a5491083081e-secret-metrics-server-client-certs\") pod \"metrics-server-74d659dd7f-k5rqx\" (UID: \"f91a3eda-11bd-4add-a5e7-a5491083081e\") " pod="openshift-monitoring/metrics-server-74d659dd7f-k5rqx"
Sep 17 17:42:52.962788 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:52.962700    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f91a3eda-11bd-4add-a5e7-a5491083081e-audit-log\") pod \"metrics-server-74d659dd7f-k5rqx\" (UID: \"f91a3eda-11bd-4add-a5e7-a5491083081e\") " pod="openshift-monitoring/metrics-server-74d659dd7f-k5rqx"
Sep 17 17:42:53.064069 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:53.064032    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f91a3eda-11bd-4add-a5e7-a5491083081e-client-ca-bundle\") pod \"metrics-server-74d659dd7f-k5rqx\" (UID: \"f91a3eda-11bd-4add-a5e7-a5491083081e\") " pod="openshift-monitoring/metrics-server-74d659dd7f-k5rqx"
Sep 17 17:42:53.064069 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:53.064070    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrcm6\" (UniqueName: \"kubernetes.io/projected/f91a3eda-11bd-4add-a5e7-a5491083081e-kube-api-access-jrcm6\") pod \"metrics-server-74d659dd7f-k5rqx\" (UID: \"f91a3eda-11bd-4add-a5e7-a5491083081e\") " pod="openshift-monitoring/metrics-server-74d659dd7f-k5rqx"
Sep 17 17:42:53.064312 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:53.064271    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f91a3eda-11bd-4add-a5e7-a5491083081e-secret-metrics-server-client-certs\") pod \"metrics-server-74d659dd7f-k5rqx\" (UID: \"f91a3eda-11bd-4add-a5e7-a5491083081e\") " pod="openshift-monitoring/metrics-server-74d659dd7f-k5rqx"
Sep 17 17:42:53.064356 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:53.064318    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f91a3eda-11bd-4add-a5e7-a5491083081e-audit-log\") pod \"metrics-server-74d659dd7f-k5rqx\" (UID: \"f91a3eda-11bd-4add-a5e7-a5491083081e\") " pod="openshift-monitoring/metrics-server-74d659dd7f-k5rqx"
Sep 17 17:42:53.064398 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:53.064370    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f91a3eda-11bd-4add-a5e7-a5491083081e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-74d659dd7f-k5rqx\" (UID: \"f91a3eda-11bd-4add-a5e7-a5491083081e\") " pod="openshift-monitoring/metrics-server-74d659dd7f-k5rqx"
Sep 17 17:42:53.064518 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:53.064491    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f91a3eda-11bd-4add-a5e7-a5491083081e-secret-metrics-server-tls\") pod \"metrics-server-74d659dd7f-k5rqx\" (UID: \"f91a3eda-11bd-4add-a5e7-a5491083081e\") " pod="openshift-monitoring/metrics-server-74d659dd7f-k5rqx"
Sep 17 17:42:53.064586 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:53.064568    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f91a3eda-11bd-4add-a5e7-a5491083081e-metrics-server-audit-profiles\") pod \"metrics-server-74d659dd7f-k5rqx\" (UID: \"f91a3eda-11bd-4add-a5e7-a5491083081e\") " pod="openshift-monitoring/metrics-server-74d659dd7f-k5rqx"
Sep 17 17:42:53.064809 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:53.064786    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f91a3eda-11bd-4add-a5e7-a5491083081e-audit-log\") pod \"metrics-server-74d659dd7f-k5rqx\" (UID: \"f91a3eda-11bd-4add-a5e7-a5491083081e\") " pod="openshift-monitoring/metrics-server-74d659dd7f-k5rqx"
Sep 17 17:42:53.065231 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:53.065205    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f91a3eda-11bd-4add-a5e7-a5491083081e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-74d659dd7f-k5rqx\" (UID: \"f91a3eda-11bd-4add-a5e7-a5491083081e\") " pod="openshift-monitoring/metrics-server-74d659dd7f-k5rqx"
Sep 17 17:42:53.065759 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:53.065732    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f91a3eda-11bd-4add-a5e7-a5491083081e-metrics-server-audit-profiles\") pod \"metrics-server-74d659dd7f-k5rqx\" (UID: \"f91a3eda-11bd-4add-a5e7-a5491083081e\") " pod="openshift-monitoring/metrics-server-74d659dd7f-k5rqx"
Sep 17 17:42:53.067122 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:53.067099    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f91a3eda-11bd-4add-a5e7-a5491083081e-secret-metrics-server-tls\") pod \"metrics-server-74d659dd7f-k5rqx\" (UID: \"f91a3eda-11bd-4add-a5e7-a5491083081e\") " pod="openshift-monitoring/metrics-server-74d659dd7f-k5rqx"
Sep 17 17:42:53.067361 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:53.067343    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f91a3eda-11bd-4add-a5e7-a5491083081e-secret-metrics-server-client-certs\") pod \"metrics-server-74d659dd7f-k5rqx\" (UID: \"f91a3eda-11bd-4add-a5e7-a5491083081e\") " pod="openshift-monitoring/metrics-server-74d659dd7f-k5rqx"
Sep 17 17:42:53.067403 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:53.067350    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f91a3eda-11bd-4add-a5e7-a5491083081e-client-ca-bundle\") pod \"metrics-server-74d659dd7f-k5rqx\" (UID: \"f91a3eda-11bd-4add-a5e7-a5491083081e\") " pod="openshift-monitoring/metrics-server-74d659dd7f-k5rqx"
Sep 17 17:42:53.073005 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:53.072983    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrcm6\" (UniqueName: \"kubernetes.io/projected/f91a3eda-11bd-4add-a5e7-a5491083081e-kube-api-access-jrcm6\") pod \"metrics-server-74d659dd7f-k5rqx\" (UID: \"f91a3eda-11bd-4add-a5e7-a5491083081e\") " pod="openshift-monitoring/metrics-server-74d659dd7f-k5rqx"
Sep 17 17:42:53.171169 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:53.171059    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-74d659dd7f-k5rqx"
Sep 17 17:42:53.447377 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:53.447340    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-74d659dd7f-k5rqx"]
Sep 17 17:42:53.450212 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:42:53.450186    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf91a3eda_11bd_4add_a5e7_a5491083081e.slice/crio-83c975fcd26bbf18f69e602175bdde857df27203f82dc60760a879cb4e368a5a WatchSource:0}: Error finding container 83c975fcd26bbf18f69e602175bdde857df27203f82dc60760a879cb4e368a5a: Status 404 returned error can't find the container with id 83c975fcd26bbf18f69e602175bdde857df27203f82dc60760a879cb4e368a5a
Sep 17 17:42:54.018388 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.018347    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-74d659dd7f-k5rqx" event={"ID":"f91a3eda-11bd-4add-a5e7-a5491083081e","Type":"ContainerStarted","Data":"83c975fcd26bbf18f69e602175bdde857df27203f82dc60760a879cb4e368a5a"}
Sep 17 17:42:54.593818 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.593767    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 17:42:54.599777 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.599738    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:54.604484 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.604450    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 17 17:42:54.604900 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.604677    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 17 17:42:54.604900 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.604706    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 17 17:42:54.604900 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.604787    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-76wkd\""
Sep 17 17:42:54.604900 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.604845    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 17 17:42:54.608222 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.608200    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 17 17:42:54.608800 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.608492    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 17 17:42:54.608800 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.608646    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 17 17:42:54.608800 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.608731    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 17 17:42:54.608800 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.608784    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-ba8usoq20r88c\""
Sep 17 17:42:54.609010 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.608902    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 17 17:42:54.610057 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.609573    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 17 17:42:54.610057 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.609606    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 17 17:42:54.610057 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.609769    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 17 17:42:54.610057 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.609817    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 17 17:42:54.611306 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.611282    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 17 17:42:54.615784 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.615761    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 17:42:54.782111 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.782060    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/be8bc2cc-9e94-4a44-8897-5f609be6f263-config-out\") pod \"prometheus-k8s-0\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:54.782307 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.782114    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be8bc2cc-9e94-4a44-8897-5f609be6f263-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:54.782307 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.782219    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/be8bc2cc-9e94-4a44-8897-5f609be6f263-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:54.782307 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.782276    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/be8bc2cc-9e94-4a44-8897-5f609be6f263-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:54.782440 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.782329    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/be8bc2cc-9e94-4a44-8897-5f609be6f263-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:54.782440 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.782361    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/be8bc2cc-9e94-4a44-8897-5f609be6f263-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:54.782440 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.782422    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/be8bc2cc-9e94-4a44-8897-5f609be6f263-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:54.782552 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.782449    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/be8bc2cc-9e94-4a44-8897-5f609be6f263-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:54.782552 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.782476    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be8bc2cc-9e94-4a44-8897-5f609be6f263-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:54.782552 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.782504    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be8bc2cc-9e94-4a44-8897-5f609be6f263-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:54.782552 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.782541    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/be8bc2cc-9e94-4a44-8897-5f609be6f263-web-config\") pod \"prometheus-k8s-0\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:54.782674 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.782638    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/be8bc2cc-9e94-4a44-8897-5f609be6f263-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:54.782723 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.782700    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h568j\" (UniqueName: \"kubernetes.io/projected/be8bc2cc-9e94-4a44-8897-5f609be6f263-kube-api-access-h568j\") pod \"prometheus-k8s-0\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:54.782779 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.782762    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/be8bc2cc-9e94-4a44-8897-5f609be6f263-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:54.782825 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.782798    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/be8bc2cc-9e94-4a44-8897-5f609be6f263-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:54.782871 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.782826    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/be8bc2cc-9e94-4a44-8897-5f609be6f263-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:54.782871 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.782863    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/be8bc2cc-9e94-4a44-8897-5f609be6f263-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:54.782960 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.782890    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/be8bc2cc-9e94-4a44-8897-5f609be6f263-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:54.782960 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.782935    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/be8bc2cc-9e94-4a44-8897-5f609be6f263-config\") pod \"prometheus-k8s-0\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:54.783034 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.782959    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be8bc2cc-9e94-4a44-8897-5f609be6f263-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:54.884200 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.884127    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/be8bc2cc-9e94-4a44-8897-5f609be6f263-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:54.884200 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.884165    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be8bc2cc-9e94-4a44-8897-5f609be6f263-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:54.884410 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.884202    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be8bc2cc-9e94-4a44-8897-5f609be6f263-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:54.884410 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.884228    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/be8bc2cc-9e94-4a44-8897-5f609be6f263-web-config\") pod \"prometheus-k8s-0\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:54.884410 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.884255    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/be8bc2cc-9e94-4a44-8897-5f609be6f263-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:54.884410 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.884283    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h568j\" (UniqueName: \"kubernetes.io/projected/be8bc2cc-9e94-4a44-8897-5f609be6f263-kube-api-access-h568j\") pod \"prometheus-k8s-0\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:54.884410 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.884331    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/be8bc2cc-9e94-4a44-8897-5f609be6f263-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:54.884410 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.884356    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/be8bc2cc-9e94-4a44-8897-5f609be6f263-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:54.884410 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.884384    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/be8bc2cc-9e94-4a44-8897-5f609be6f263-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:54.884835 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.884468    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/be8bc2cc-9e94-4a44-8897-5f609be6f263-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:54.884835 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.884517    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/be8bc2cc-9e94-4a44-8897-5f609be6f263-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:54.884835 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.884543    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/be8bc2cc-9e94-4a44-8897-5f609be6f263-config\") pod \"prometheus-k8s-0\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:54.884835 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.884565    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be8bc2cc-9e94-4a44-8897-5f609be6f263-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:54.884835 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.884616    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/be8bc2cc-9e94-4a44-8897-5f609be6f263-config-out\") pod \"prometheus-k8s-0\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:54.884835 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.884644    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be8bc2cc-9e94-4a44-8897-5f609be6f263-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:54.884835 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.884714    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/be8bc2cc-9e94-4a44-8897-5f609be6f263-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:54.884835 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.884767    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/be8bc2cc-9e94-4a44-8897-5f609be6f263-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:54.884835 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.884813    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/be8bc2cc-9e94-4a44-8897-5f609be6f263-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:54.885760 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.884837    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/be8bc2cc-9e94-4a44-8897-5f609be6f263-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:54.885760 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.884874    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/be8bc2cc-9e94-4a44-8897-5f609be6f263-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:54.885760 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.885360    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be8bc2cc-9e94-4a44-8897-5f609be6f263-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:54.885760 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.885722    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be8bc2cc-9e94-4a44-8897-5f609be6f263-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:54.890290 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.887389    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/be8bc2cc-9e94-4a44-8897-5f609be6f263-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:54.890290 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.887461    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/be8bc2cc-9e94-4a44-8897-5f609be6f263-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:54.890290 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.887682    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/be8bc2cc-9e94-4a44-8897-5f609be6f263-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:54.890290 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.887773    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/be8bc2cc-9e94-4a44-8897-5f609be6f263-config-out\") pod \"prometheus-k8s-0\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:54.890290 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.888376    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be8bc2cc-9e94-4a44-8897-5f609be6f263-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:54.890290 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.889798    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/be8bc2cc-9e94-4a44-8897-5f609be6f263-web-config\") pod \"prometheus-k8s-0\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:54.890290 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.889963    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/be8bc2cc-9e94-4a44-8897-5f609be6f263-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:54.890290 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.890167    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/be8bc2cc-9e94-4a44-8897-5f609be6f263-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:54.890290 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.890201    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/be8bc2cc-9e94-4a44-8897-5f609be6f263-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:54.890290 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.890227    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/be8bc2cc-9e94-4a44-8897-5f609be6f263-config\") pod \"prometheus-k8s-0\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:54.890912 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.890730    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/be8bc2cc-9e94-4a44-8897-5f609be6f263-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:54.891334 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.891306    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be8bc2cc-9e94-4a44-8897-5f609be6f263-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:54.891792 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.891764    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/be8bc2cc-9e94-4a44-8897-5f609be6f263-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:54.892370 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.892345    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/be8bc2cc-9e94-4a44-8897-5f609be6f263-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:54.892581 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.892559    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/be8bc2cc-9e94-4a44-8897-5f609be6f263-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:54.893003 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.892964    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/be8bc2cc-9e94-4a44-8897-5f609be6f263-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:54.893224 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.893205    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/be8bc2cc-9e94-4a44-8897-5f609be6f263-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:54.894657 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.894636    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h568j\" (UniqueName: \"kubernetes.io/projected/be8bc2cc-9e94-4a44-8897-5f609be6f263-kube-api-access-h568j\") pod \"prometheus-k8s-0\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:54.912192 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:54.912161    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:42:55.229055 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:55.229009    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 17:42:55.232679 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:42:55.232630    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe8bc2cc_9e94_4a44_8897_5f609be6f263.slice/crio-2bc5b30923d3344dc09b32b358b457e6372d4cb4c587a8a42e61b7624956b409 WatchSource:0}: Error finding container 2bc5b30923d3344dc09b32b358b457e6372d4cb4c587a8a42e61b7624956b409: Status 404 returned error can't find the container with id 2bc5b30923d3344dc09b32b358b457e6372d4cb4c587a8a42e61b7624956b409
Sep 17 17:42:56.029244 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:56.029204    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-74d659dd7f-k5rqx" event={"ID":"f91a3eda-11bd-4add-a5e7-a5491083081e","Type":"ContainerStarted","Data":"5f35d4c904aa88d5da25ba52cfb6a3338ac9e81f7e0f00d296ebc7b1e539d10d"}
Sep 17 17:42:56.030791 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:56.030760    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"be8bc2cc-9e94-4a44-8897-5f609be6f263","Type":"ContainerStarted","Data":"2bc5b30923d3344dc09b32b358b457e6372d4cb4c587a8a42e61b7624956b409"}
Sep 17 17:42:56.049834 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:56.049753    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-74d659dd7f-k5rqx" podStartSLOduration=2.54738788 podStartE2EDuration="4.049730886s" totalImagesPullingTime="1.502343006s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:42:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:56.049454032 +0000 UTC m=+128.553456671" watchObservedRunningTime="2026-09-17 17:42:56.049730886 +0000 UTC m=+128.553733529"
Sep 17 17:42:57.036394 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:57.036362    2569 generic.go:356] "Generic (PLEG): container finished" podID="be8bc2cc-9e94-4a44-8897-5f609be6f263" containerID="e5a3cf85c8af59236d517bee97a54e44de29a2df04684bf1ff488f09706b582b" exitCode=0
Sep 17 17:42:57.036785 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:57.036453    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"be8bc2cc-9e94-4a44-8897-5f609be6f263","Type":"ContainerDied","Data":"e5a3cf85c8af59236d517bee97a54e44de29a2df04684bf1ff488f09706b582b"}
Sep 17 17:42:59.224980 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:59.224916    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caf785f7-c156-4b6f-8c82-4c1efe5cd710-metrics-certs\") pod \"network-metrics-daemon-rzvhs\" (UID: \"caf785f7-c156-4b6f-8c82-4c1efe5cd710\") " pod="openshift-multus/network-metrics-daemon-rzvhs"
Sep 17 17:42:59.227293 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:59.227259    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caf785f7-c156-4b6f-8c82-4c1efe5cd710-metrics-certs\") pod \"network-metrics-daemon-rzvhs\" (UID: \"caf785f7-c156-4b6f-8c82-4c1efe5cd710\") " pod="openshift-multus/network-metrics-daemon-rzvhs"
Sep 17 17:42:59.436668 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:59.436626    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nqf6f\""
Sep 17 17:42:59.443651 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:42:59.443409    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rzvhs"
Sep 17 17:43:00.503215 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:43:00.503181    2569 scope.go:117] "RemoveContainer" containerID="22aa916511d1b418b854d97971fb1a3ea13605026410231a2396dbe66dd54824"
Sep 17 17:43:00.823413 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:43:00.823384    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rzvhs"]
Sep 17 17:43:00.825728 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:43:00.825702    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaf785f7_c156_4b6f_8c82_4c1efe5cd710.slice/crio-be54e702c6be49701680dd736ded3c4b08017d8d74dc001e3ce5eb00d30a92de WatchSource:0}: Error finding container be54e702c6be49701680dd736ded3c4b08017d8d74dc001e3ce5eb00d30a92de: Status 404 returned error can't find the container with id be54e702c6be49701680dd736ded3c4b08017d8d74dc001e3ce5eb00d30a92de
Sep 17 17:43:01.060585 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:43:01.060494    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"be8bc2cc-9e94-4a44-8897-5f609be6f263","Type":"ContainerStarted","Data":"9b7a76e1f93f7ef9ef7bda1eac4a49895a1b437ddcc96410e1b7bada627a3f37"}
Sep 17 17:43:01.060585 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:43:01.060576    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"be8bc2cc-9e94-4a44-8897-5f609be6f263","Type":"ContainerStarted","Data":"fe0d6bdab4b9d938fa796d9f6e97c6fb7eb77f65fb8201ad2e5c766bd882d209"}
Sep 17 17:43:01.061798 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:43:01.061760    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rzvhs" event={"ID":"caf785f7-c156-4b6f-8c82-4c1efe5cd710","Type":"ContainerStarted","Data":"be54e702c6be49701680dd736ded3c4b08017d8d74dc001e3ce5eb00d30a92de"}
Sep 17 17:43:01.063596 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:43:01.063575    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-t7pqb_b8d76e28-4655-4d21-98d8-0324e4756aaf/console-operator/2.log"
Sep 17 17:43:01.063729 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:43:01.063631    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-t7pqb" event={"ID":"b8d76e28-4655-4d21-98d8-0324e4756aaf","Type":"ContainerStarted","Data":"e0b6e6bb61c9edda3da9091382f7922e0e85b165eed209f0a17d5da4e132c701"}
Sep 17 17:43:01.063951 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:43:01.063911    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-t7pqb"
Sep 17 17:43:01.084395 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:43:01.084344    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-fd4bb46c-t7pqb" podStartSLOduration=58.723058064 podStartE2EDuration="1m1.08432854s" totalImagesPullingTime="2.361270476s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:42:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:43:01.082198552 +0000 UTC m=+133.586201179" watchObservedRunningTime="2026-09-17 17:43:01.08432854 +0000 UTC m=+133.588331164"
Sep 17 17:43:01.894239 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:43:01.894203    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-fd4bb46c-t7pqb"
Sep 17 17:43:02.073057 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:43:02.072956    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rzvhs" event={"ID":"caf785f7-c156-4b6f-8c82-4c1efe5cd710","Type":"ContainerStarted","Data":"4d8174af93ea362da2cf6ae65379e3eaeae3817f853510e571613e3854276c39"}
Sep 17 17:43:03.078058 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:43:03.078019    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rzvhs" event={"ID":"caf785f7-c156-4b6f-8c82-4c1efe5cd710","Type":"ContainerStarted","Data":"e988bf6a49c86383476c50b5d06248e03fbef6e7a6944a049f2ce4e681fae917"}
Sep 17 17:43:03.081254 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:43:03.081223    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"be8bc2cc-9e94-4a44-8897-5f609be6f263","Type":"ContainerStarted","Data":"2487e275e38dce21f3825fd5db5b46221622e3ae3802a36eb425b11eee4f9784"}
Sep 17 17:43:03.081404 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:43:03.081291    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"be8bc2cc-9e94-4a44-8897-5f609be6f263","Type":"ContainerStarted","Data":"8a007a950ec61fb6320aca30282a7954a0b85a6594e9bb76840a5536faaee77d"}
Sep 17 17:43:03.081404 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:43:03.081333    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"be8bc2cc-9e94-4a44-8897-5f609be6f263","Type":"ContainerStarted","Data":"55389ceace2ba5a54fd4e46d98faf5125e5d06745751f9fd25dd455992d5ad70"}
Sep 17 17:43:03.081404 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:43:03.081374    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"be8bc2cc-9e94-4a44-8897-5f609be6f263","Type":"ContainerStarted","Data":"1e0b5a02b2dca736fd6e082460f2f75a8aa4816923ec2a3db10922e626726244"}
Sep 17 17:43:03.095738 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:43:03.095688    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-rzvhs" podStartSLOduration=133.062732054 podStartE2EDuration="2m14.095667788s" totalImagesPullingTime="1.032935734s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:40:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:43:03.094609242 +0000 UTC m=+135.598611880" watchObservedRunningTime="2026-09-17 17:43:03.095667788 +0000 UTC m=+135.599670416"
Sep 17 17:43:03.128645 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:43:03.128584    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.556305853 podStartE2EDuration="9.128565291s" totalImagesPullingTime="6.55387203s" totalInitContainerRuntime="18.387408ms" isStatefulPod=false podCreationTimestamp="2026-09-17 17:42:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:43:03.126362616 +0000 UTC m=+135.630365302" watchObservedRunningTime="2026-09-17 17:43:03.128565291 +0000 UTC m=+135.632567915"
Sep 17 17:43:04.912506 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:43:04.912470    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:13.171323 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:43:13.171281    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-74d659dd7f-k5rqx"
Sep 17 17:43:13.171820 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:43:13.171351    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-74d659dd7f-k5rqx"
Sep 17 17:43:29.177879 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:43:29.177835    2569 generic.go:356] "Generic (PLEG): container finished" podID="3de5fb99-4d6c-4241-a85b-902f36b0a49a" containerID="14d9eed890188872efdc2709a84be9e6a715e2e842b2a4d847f515f1bc9c0f92" exitCode=0
Sep 17 17:43:29.178284 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:43:29.177906    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-sx8ns" event={"ID":"3de5fb99-4d6c-4241-a85b-902f36b0a49a","Type":"ContainerDied","Data":"14d9eed890188872efdc2709a84be9e6a715e2e842b2a4d847f515f1bc9c0f92"}
Sep 17 17:43:29.178284 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:43:29.178282    2569 scope.go:117] "RemoveContainer" containerID="14d9eed890188872efdc2709a84be9e6a715e2e842b2a4d847f515f1bc9c0f92"
Sep 17 17:43:30.184059 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:43:30.184029    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-sx8ns" event={"ID":"3de5fb99-4d6c-4241-a85b-902f36b0a49a","Type":"ContainerStarted","Data":"11100847fd0c382546999abf00a57a3d3072ecd40c0aa1e93a542e142496ba6d"}
Sep 17 17:43:32.845742 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:43:32.845668    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-74d659dd7f-k5rqx_f91a3eda-11bd-4add-a5e7-a5491083081e/metrics-server/0.log"
Sep 17 17:43:33.177216 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:43:33.177148    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-74d659dd7f-k5rqx"
Sep 17 17:43:33.180983 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:43:33.180964    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-74d659dd7f-k5rqx"
Sep 17 17:43:34.449209 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:43:34.449178    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kfnnp_c5650e98-c78e-42cf-8c37-f83e002791aa/init-textfile/0.log"
Sep 17 17:43:34.650438 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:43:34.650388    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kfnnp_c5650e98-c78e-42cf-8c37-f83e002791aa/node-exporter/0.log"
Sep 17 17:43:34.845804 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:43:34.845692    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kfnnp_c5650e98-c78e-42cf-8c37-f83e002791aa/kube-rbac-proxy/0.log"
Sep 17 17:43:35.646542 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:43:35.646512    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_be8bc2cc-9e94-4a44-8897-5f609be6f263/init-config-reloader/0.log"
Sep 17 17:43:35.849648 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:43:35.849613    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_be8bc2cc-9e94-4a44-8897-5f609be6f263/prometheus/0.log"
Sep 17 17:43:36.046881 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:43:36.046854    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_be8bc2cc-9e94-4a44-8897-5f609be6f263/config-reloader/0.log"
Sep 17 17:43:36.246899 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:43:36.246869    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_be8bc2cc-9e94-4a44-8897-5f609be6f263/thanos-sidecar/0.log"
Sep 17 17:43:36.446947 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:43:36.446888    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_be8bc2cc-9e94-4a44-8897-5f609be6f263/kube-rbac-proxy-web/0.log"
Sep 17 17:43:36.646496 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:43:36.646466    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_be8bc2cc-9e94-4a44-8897-5f609be6f263/kube-rbac-proxy/0.log"
Sep 17 17:43:36.845862 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:43:36.845792    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_be8bc2cc-9e94-4a44-8897-5f609be6f263/kube-rbac-proxy-thanos/0.log"
Sep 17 17:43:37.047058 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:43:37.047025    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-c7xwv_4c85ebb0-dc17-4937-9dca-9d87547620f3/prometheus-operator/0.log"
Sep 17 17:43:37.245707 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:43:37.245641    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-c7xwv_4c85ebb0-dc17-4937-9dca-9d87547620f3/kube-rbac-proxy/0.log"
Sep 17 17:43:37.446177 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:43:37.446147    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5c6747dc7b-7m22j_b54d572b-971f-4e9b-9f00-c1b8b3eb9188/prometheus-operator-admission-webhook/0.log"
Sep 17 17:43:39.045843 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:43:39.045817    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-t7pqb_b8d76e28-4655-4d21-98d8-0324e4756aaf/console-operator/2.log"
Sep 17 17:43:39.247909 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:43:39.247883    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-t7pqb_b8d76e28-4655-4d21-98d8-0324e4756aaf/console-operator/3.log"
Sep 17 17:43:40.046506 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:43:40.046449    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-55f9595458-br8n7_efc7ba59-612a-4b8a-9935-f2228ec44b30/router/0.log"
Sep 17 17:43:40.446292 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:43:40.446260    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-78r7n_5b1aeab9-ca3c-4ae4-82dc-fae81696a23a/serve-healthcheck-canary/0.log"
Sep 17 17:43:54.912874 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:43:54.912823    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:54.928885 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:43:54.928855    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:55.304962 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:43:55.304938    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:44:13.066506 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.066472    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 17:44:13.067390 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.067351    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="be8bc2cc-9e94-4a44-8897-5f609be6f263" containerName="prometheus" containerID="cri-o://fe0d6bdab4b9d938fa796d9f6e97c6fb7eb77f65fb8201ad2e5c766bd882d209" gracePeriod=600
Sep 17 17:44:13.067656 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.067614    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="be8bc2cc-9e94-4a44-8897-5f609be6f263" containerName="kube-rbac-proxy" containerID="cri-o://8a007a950ec61fb6320aca30282a7954a0b85a6594e9bb76840a5536faaee77d" gracePeriod=600
Sep 17 17:44:13.067823 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.067636    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="be8bc2cc-9e94-4a44-8897-5f609be6f263" containerName="kube-rbac-proxy-thanos" containerID="cri-o://2487e275e38dce21f3825fd5db5b46221622e3ae3802a36eb425b11eee4f9784" gracePeriod=600
Sep 17 17:44:13.067912 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.067645    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="be8bc2cc-9e94-4a44-8897-5f609be6f263" containerName="config-reloader" containerID="cri-o://9b7a76e1f93f7ef9ef7bda1eac4a49895a1b437ddcc96410e1b7bada627a3f37" gracePeriod=600
Sep 17 17:44:13.067982 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.067643    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="be8bc2cc-9e94-4a44-8897-5f609be6f263" containerName="kube-rbac-proxy-web" containerID="cri-o://55389ceace2ba5a54fd4e46d98faf5125e5d06745751f9fd25dd455992d5ad70" gracePeriod=600
Sep 17 17:44:13.067982 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.067677    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="be8bc2cc-9e94-4a44-8897-5f609be6f263" containerName="thanos-sidecar" containerID="cri-o://1e0b5a02b2dca736fd6e082460f2f75a8aa4816923ec2a3db10922e626726244" gracePeriod=600
Sep 17 17:44:13.329881 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.329842    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:44:13.363228 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.363148    2569 generic.go:356] "Generic (PLEG): container finished" podID="be8bc2cc-9e94-4a44-8897-5f609be6f263" containerID="2487e275e38dce21f3825fd5db5b46221622e3ae3802a36eb425b11eee4f9784" exitCode=0
Sep 17 17:44:13.363228 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.363181    2569 generic.go:356] "Generic (PLEG): container finished" podID="be8bc2cc-9e94-4a44-8897-5f609be6f263" containerID="8a007a950ec61fb6320aca30282a7954a0b85a6594e9bb76840a5536faaee77d" exitCode=0
Sep 17 17:44:13.363228 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.363193    2569 generic.go:356] "Generic (PLEG): container finished" podID="be8bc2cc-9e94-4a44-8897-5f609be6f263" containerID="55389ceace2ba5a54fd4e46d98faf5125e5d06745751f9fd25dd455992d5ad70" exitCode=0
Sep 17 17:44:13.363228 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.363201    2569 generic.go:356] "Generic (PLEG): container finished" podID="be8bc2cc-9e94-4a44-8897-5f609be6f263" containerID="1e0b5a02b2dca736fd6e082460f2f75a8aa4816923ec2a3db10922e626726244" exitCode=0
Sep 17 17:44:13.363228 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.363209    2569 generic.go:356] "Generic (PLEG): container finished" podID="be8bc2cc-9e94-4a44-8897-5f609be6f263" containerID="9b7a76e1f93f7ef9ef7bda1eac4a49895a1b437ddcc96410e1b7bada627a3f37" exitCode=0
Sep 17 17:44:13.363228 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.363219    2569 generic.go:356] "Generic (PLEG): container finished" podID="be8bc2cc-9e94-4a44-8897-5f609be6f263" containerID="fe0d6bdab4b9d938fa796d9f6e97c6fb7eb77f65fb8201ad2e5c766bd882d209" exitCode=0
Sep 17 17:44:13.363795 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.363243    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:44:13.363795 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.363232    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"be8bc2cc-9e94-4a44-8897-5f609be6f263","Type":"ContainerDied","Data":"2487e275e38dce21f3825fd5db5b46221622e3ae3802a36eb425b11eee4f9784"}
Sep 17 17:44:13.363795 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.363364    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"be8bc2cc-9e94-4a44-8897-5f609be6f263","Type":"ContainerDied","Data":"8a007a950ec61fb6320aca30282a7954a0b85a6594e9bb76840a5536faaee77d"}
Sep 17 17:44:13.363795 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.363409    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"be8bc2cc-9e94-4a44-8897-5f609be6f263","Type":"ContainerDied","Data":"55389ceace2ba5a54fd4e46d98faf5125e5d06745751f9fd25dd455992d5ad70"}
Sep 17 17:44:13.363795 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.363452    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"be8bc2cc-9e94-4a44-8897-5f609be6f263","Type":"ContainerDied","Data":"1e0b5a02b2dca736fd6e082460f2f75a8aa4816923ec2a3db10922e626726244"}
Sep 17 17:44:13.363795 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.363493    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"be8bc2cc-9e94-4a44-8897-5f609be6f263","Type":"ContainerDied","Data":"9b7a76e1f93f7ef9ef7bda1eac4a49895a1b437ddcc96410e1b7bada627a3f37"}
Sep 17 17:44:13.363795 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.363535    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"be8bc2cc-9e94-4a44-8897-5f609be6f263","Type":"ContainerDied","Data":"fe0d6bdab4b9d938fa796d9f6e97c6fb7eb77f65fb8201ad2e5c766bd882d209"}
Sep 17 17:44:13.363795 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.363575    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"be8bc2cc-9e94-4a44-8897-5f609be6f263","Type":"ContainerDied","Data":"2bc5b30923d3344dc09b32b358b457e6372d4cb4c587a8a42e61b7624956b409"}
Sep 17 17:44:13.363795 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.363624    2569 scope.go:117] "RemoveContainer" containerID="2487e275e38dce21f3825fd5db5b46221622e3ae3802a36eb425b11eee4f9784"
Sep 17 17:44:13.379493 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.379372    2569 scope.go:117] "RemoveContainer" containerID="8a007a950ec61fb6320aca30282a7954a0b85a6594e9bb76840a5536faaee77d"
Sep 17 17:44:13.390976 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.390954    2569 scope.go:117] "RemoveContainer" containerID="55389ceace2ba5a54fd4e46d98faf5125e5d06745751f9fd25dd455992d5ad70"
Sep 17 17:44:13.401605 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.401575    2569 scope.go:117] "RemoveContainer" containerID="1e0b5a02b2dca736fd6e082460f2f75a8aa4816923ec2a3db10922e626726244"
Sep 17 17:44:13.409651 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.409625    2569 scope.go:117] "RemoveContainer" containerID="9b7a76e1f93f7ef9ef7bda1eac4a49895a1b437ddcc96410e1b7bada627a3f37"
Sep 17 17:44:13.416670 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.416653    2569 scope.go:117] "RemoveContainer" containerID="fe0d6bdab4b9d938fa796d9f6e97c6fb7eb77f65fb8201ad2e5c766bd882d209"
Sep 17 17:44:13.424610 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.424591    2569 scope.go:117] "RemoveContainer" containerID="e5a3cf85c8af59236d517bee97a54e44de29a2df04684bf1ff488f09706b582b"
Sep 17 17:44:13.431589 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.431571    2569 scope.go:117] "RemoveContainer" containerID="2487e275e38dce21f3825fd5db5b46221622e3ae3802a36eb425b11eee4f9784"
Sep 17 17:44:13.431894 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:44:13.431873    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2487e275e38dce21f3825fd5db5b46221622e3ae3802a36eb425b11eee4f9784\": container with ID starting with 2487e275e38dce21f3825fd5db5b46221622e3ae3802a36eb425b11eee4f9784 not found: ID does not exist" containerID="2487e275e38dce21f3825fd5db5b46221622e3ae3802a36eb425b11eee4f9784"
Sep 17 17:44:13.431941 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.431905    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2487e275e38dce21f3825fd5db5b46221622e3ae3802a36eb425b11eee4f9784"} err="failed to get container status \"2487e275e38dce21f3825fd5db5b46221622e3ae3802a36eb425b11eee4f9784\": rpc error: code = NotFound desc = could not find container \"2487e275e38dce21f3825fd5db5b46221622e3ae3802a36eb425b11eee4f9784\": container with ID starting with 2487e275e38dce21f3825fd5db5b46221622e3ae3802a36eb425b11eee4f9784 not found: ID does not exist"
Sep 17 17:44:13.431941 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.431938    2569 scope.go:117] "RemoveContainer" containerID="8a007a950ec61fb6320aca30282a7954a0b85a6594e9bb76840a5536faaee77d"
Sep 17 17:44:13.432225 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:44:13.432208    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a007a950ec61fb6320aca30282a7954a0b85a6594e9bb76840a5536faaee77d\": container with ID starting with 8a007a950ec61fb6320aca30282a7954a0b85a6594e9bb76840a5536faaee77d not found: ID does not exist" containerID="8a007a950ec61fb6320aca30282a7954a0b85a6594e9bb76840a5536faaee77d"
Sep 17 17:44:13.432284 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.432231    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a007a950ec61fb6320aca30282a7954a0b85a6594e9bb76840a5536faaee77d"} err="failed to get container status \"8a007a950ec61fb6320aca30282a7954a0b85a6594e9bb76840a5536faaee77d\": rpc error: code = NotFound desc = could not find container \"8a007a950ec61fb6320aca30282a7954a0b85a6594e9bb76840a5536faaee77d\": container with ID starting with 8a007a950ec61fb6320aca30282a7954a0b85a6594e9bb76840a5536faaee77d not found: ID does not exist"
Sep 17 17:44:13.432284 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.432247    2569 scope.go:117] "RemoveContainer" containerID="55389ceace2ba5a54fd4e46d98faf5125e5d06745751f9fd25dd455992d5ad70"
Sep 17 17:44:13.432487 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:44:13.432470    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55389ceace2ba5a54fd4e46d98faf5125e5d06745751f9fd25dd455992d5ad70\": container with ID starting with 55389ceace2ba5a54fd4e46d98faf5125e5d06745751f9fd25dd455992d5ad70 not found: ID does not exist" containerID="55389ceace2ba5a54fd4e46d98faf5125e5d06745751f9fd25dd455992d5ad70"
Sep 17 17:44:13.432527 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.432492    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55389ceace2ba5a54fd4e46d98faf5125e5d06745751f9fd25dd455992d5ad70"} err="failed to get container status \"55389ceace2ba5a54fd4e46d98faf5125e5d06745751f9fd25dd455992d5ad70\": rpc error: code = NotFound desc = could not find container \"55389ceace2ba5a54fd4e46d98faf5125e5d06745751f9fd25dd455992d5ad70\": container with ID starting with 55389ceace2ba5a54fd4e46d98faf5125e5d06745751f9fd25dd455992d5ad70 not found: ID does not exist"
Sep 17 17:44:13.432527 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.432507    2569 scope.go:117] "RemoveContainer" containerID="1e0b5a02b2dca736fd6e082460f2f75a8aa4816923ec2a3db10922e626726244"
Sep 17 17:44:13.432730 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:44:13.432712    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e0b5a02b2dca736fd6e082460f2f75a8aa4816923ec2a3db10922e626726244\": container with ID starting with 1e0b5a02b2dca736fd6e082460f2f75a8aa4816923ec2a3db10922e626726244 not found: ID does not exist" containerID="1e0b5a02b2dca736fd6e082460f2f75a8aa4816923ec2a3db10922e626726244"
Sep 17 17:44:13.432778 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.432734    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e0b5a02b2dca736fd6e082460f2f75a8aa4816923ec2a3db10922e626726244"} err="failed to get container status \"1e0b5a02b2dca736fd6e082460f2f75a8aa4816923ec2a3db10922e626726244\": rpc error: code = NotFound desc = could not find container \"1e0b5a02b2dca736fd6e082460f2f75a8aa4816923ec2a3db10922e626726244\": container with ID starting with 1e0b5a02b2dca736fd6e082460f2f75a8aa4816923ec2a3db10922e626726244 not found: ID does not exist"
Sep 17 17:44:13.432778 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.432746    2569 scope.go:117] "RemoveContainer" containerID="9b7a76e1f93f7ef9ef7bda1eac4a49895a1b437ddcc96410e1b7bada627a3f37"
Sep 17 17:44:13.432957 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:44:13.432941    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b7a76e1f93f7ef9ef7bda1eac4a49895a1b437ddcc96410e1b7bada627a3f37\": container with ID starting with 9b7a76e1f93f7ef9ef7bda1eac4a49895a1b437ddcc96410e1b7bada627a3f37 not found: ID does not exist" containerID="9b7a76e1f93f7ef9ef7bda1eac4a49895a1b437ddcc96410e1b7bada627a3f37"
Sep 17 17:44:13.433000 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.432962    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b7a76e1f93f7ef9ef7bda1eac4a49895a1b437ddcc96410e1b7bada627a3f37"} err="failed to get container status \"9b7a76e1f93f7ef9ef7bda1eac4a49895a1b437ddcc96410e1b7bada627a3f37\": rpc error: code = NotFound desc = could not find container \"9b7a76e1f93f7ef9ef7bda1eac4a49895a1b437ddcc96410e1b7bada627a3f37\": container with ID starting with 9b7a76e1f93f7ef9ef7bda1eac4a49895a1b437ddcc96410e1b7bada627a3f37 not found: ID does not exist"
Sep 17 17:44:13.433000 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.432974    2569 scope.go:117] "RemoveContainer" containerID="fe0d6bdab4b9d938fa796d9f6e97c6fb7eb77f65fb8201ad2e5c766bd882d209"
Sep 17 17:44:13.433227 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:44:13.433212    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe0d6bdab4b9d938fa796d9f6e97c6fb7eb77f65fb8201ad2e5c766bd882d209\": container with ID starting with fe0d6bdab4b9d938fa796d9f6e97c6fb7eb77f65fb8201ad2e5c766bd882d209 not found: ID does not exist" containerID="fe0d6bdab4b9d938fa796d9f6e97c6fb7eb77f65fb8201ad2e5c766bd882d209"
Sep 17 17:44:13.433287 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.433229    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe0d6bdab4b9d938fa796d9f6e97c6fb7eb77f65fb8201ad2e5c766bd882d209"} err="failed to get container status \"fe0d6bdab4b9d938fa796d9f6e97c6fb7eb77f65fb8201ad2e5c766bd882d209\": rpc error: code = NotFound desc = could not find container \"fe0d6bdab4b9d938fa796d9f6e97c6fb7eb77f65fb8201ad2e5c766bd882d209\": container with ID starting with fe0d6bdab4b9d938fa796d9f6e97c6fb7eb77f65fb8201ad2e5c766bd882d209 not found: ID does not exist"
Sep 17 17:44:13.433287 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.433242    2569 scope.go:117] "RemoveContainer" containerID="e5a3cf85c8af59236d517bee97a54e44de29a2df04684bf1ff488f09706b582b"
Sep 17 17:44:13.433481 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:44:13.433464    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5a3cf85c8af59236d517bee97a54e44de29a2df04684bf1ff488f09706b582b\": container with ID starting with e5a3cf85c8af59236d517bee97a54e44de29a2df04684bf1ff488f09706b582b not found: ID does not exist" containerID="e5a3cf85c8af59236d517bee97a54e44de29a2df04684bf1ff488f09706b582b"
Sep 17 17:44:13.433526 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.433484    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5a3cf85c8af59236d517bee97a54e44de29a2df04684bf1ff488f09706b582b"} err="failed to get container status \"e5a3cf85c8af59236d517bee97a54e44de29a2df04684bf1ff488f09706b582b\": rpc error: code = NotFound desc = could not find container \"e5a3cf85c8af59236d517bee97a54e44de29a2df04684bf1ff488f09706b582b\": container with ID starting with e5a3cf85c8af59236d517bee97a54e44de29a2df04684bf1ff488f09706b582b not found: ID does not exist"
Sep 17 17:44:13.433526 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.433496    2569 scope.go:117] "RemoveContainer" containerID="2487e275e38dce21f3825fd5db5b46221622e3ae3802a36eb425b11eee4f9784"
Sep 17 17:44:13.433717 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.433698    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2487e275e38dce21f3825fd5db5b46221622e3ae3802a36eb425b11eee4f9784"} err="failed to get container status \"2487e275e38dce21f3825fd5db5b46221622e3ae3802a36eb425b11eee4f9784\": rpc error: code = NotFound desc = could not find container \"2487e275e38dce21f3825fd5db5b46221622e3ae3802a36eb425b11eee4f9784\": container with ID starting with 2487e275e38dce21f3825fd5db5b46221622e3ae3802a36eb425b11eee4f9784 not found: ID does not exist"
Sep 17 17:44:13.433762 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.433718    2569 scope.go:117] "RemoveContainer" containerID="8a007a950ec61fb6320aca30282a7954a0b85a6594e9bb76840a5536faaee77d"
Sep 17 17:44:13.433937 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.433919    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a007a950ec61fb6320aca30282a7954a0b85a6594e9bb76840a5536faaee77d"} err="failed to get container status \"8a007a950ec61fb6320aca30282a7954a0b85a6594e9bb76840a5536faaee77d\": rpc error: code = NotFound desc = could not find container \"8a007a950ec61fb6320aca30282a7954a0b85a6594e9bb76840a5536faaee77d\": container with ID starting with 8a007a950ec61fb6320aca30282a7954a0b85a6594e9bb76840a5536faaee77d not found: ID does not exist"
Sep 17 17:44:13.434004 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.433939    2569 scope.go:117] "RemoveContainer" containerID="55389ceace2ba5a54fd4e46d98faf5125e5d06745751f9fd25dd455992d5ad70"
Sep 17 17:44:13.434219 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.434196    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55389ceace2ba5a54fd4e46d98faf5125e5d06745751f9fd25dd455992d5ad70"} err="failed to get container status \"55389ceace2ba5a54fd4e46d98faf5125e5d06745751f9fd25dd455992d5ad70\": rpc error: code = NotFound desc = could not find container \"55389ceace2ba5a54fd4e46d98faf5125e5d06745751f9fd25dd455992d5ad70\": container with ID starting with 55389ceace2ba5a54fd4e46d98faf5125e5d06745751f9fd25dd455992d5ad70 not found: ID does not exist"
Sep 17 17:44:13.434219 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.434219    2569 scope.go:117] "RemoveContainer" containerID="1e0b5a02b2dca736fd6e082460f2f75a8aa4816923ec2a3db10922e626726244"
Sep 17 17:44:13.434450 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.434432    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e0b5a02b2dca736fd6e082460f2f75a8aa4816923ec2a3db10922e626726244"} err="failed to get container status \"1e0b5a02b2dca736fd6e082460f2f75a8aa4816923ec2a3db10922e626726244\": rpc error: code = NotFound desc = could not find container \"1e0b5a02b2dca736fd6e082460f2f75a8aa4816923ec2a3db10922e626726244\": container with ID starting with 1e0b5a02b2dca736fd6e082460f2f75a8aa4816923ec2a3db10922e626726244 not found: ID does not exist"
Sep 17 17:44:13.434493 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.434452    2569 scope.go:117] "RemoveContainer" containerID="9b7a76e1f93f7ef9ef7bda1eac4a49895a1b437ddcc96410e1b7bada627a3f37"
Sep 17 17:44:13.434676 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.434659    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b7a76e1f93f7ef9ef7bda1eac4a49895a1b437ddcc96410e1b7bada627a3f37"} err="failed to get container status \"9b7a76e1f93f7ef9ef7bda1eac4a49895a1b437ddcc96410e1b7bada627a3f37\": rpc error: code = NotFound desc = could not find container \"9b7a76e1f93f7ef9ef7bda1eac4a49895a1b437ddcc96410e1b7bada627a3f37\": container with ID starting with 9b7a76e1f93f7ef9ef7bda1eac4a49895a1b437ddcc96410e1b7bada627a3f37 not found: ID does not exist"
Sep 17 17:44:13.434676 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.434675    2569 scope.go:117] "RemoveContainer" containerID="fe0d6bdab4b9d938fa796d9f6e97c6fb7eb77f65fb8201ad2e5c766bd882d209"
Sep 17 17:44:13.434873 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.434857    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe0d6bdab4b9d938fa796d9f6e97c6fb7eb77f65fb8201ad2e5c766bd882d209"} err="failed to get container status \"fe0d6bdab4b9d938fa796d9f6e97c6fb7eb77f65fb8201ad2e5c766bd882d209\": rpc error: code = NotFound desc = could not find container \"fe0d6bdab4b9d938fa796d9f6e97c6fb7eb77f65fb8201ad2e5c766bd882d209\": container with ID starting with fe0d6bdab4b9d938fa796d9f6e97c6fb7eb77f65fb8201ad2e5c766bd882d209 not found: ID does not exist"
Sep 17 17:44:13.434920 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.434873    2569 scope.go:117] "RemoveContainer" containerID="e5a3cf85c8af59236d517bee97a54e44de29a2df04684bf1ff488f09706b582b"
Sep 17 17:44:13.435069 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.435054    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5a3cf85c8af59236d517bee97a54e44de29a2df04684bf1ff488f09706b582b"} err="failed to get container status \"e5a3cf85c8af59236d517bee97a54e44de29a2df04684bf1ff488f09706b582b\": rpc error: code = NotFound desc = could not find container \"e5a3cf85c8af59236d517bee97a54e44de29a2df04684bf1ff488f09706b582b\": container with ID starting with e5a3cf85c8af59236d517bee97a54e44de29a2df04684bf1ff488f09706b582b not found: ID does not exist"
Sep 17 17:44:13.435189 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.435069    2569 scope.go:117] "RemoveContainer" containerID="2487e275e38dce21f3825fd5db5b46221622e3ae3802a36eb425b11eee4f9784"
Sep 17 17:44:13.435348 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.435330    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2487e275e38dce21f3825fd5db5b46221622e3ae3802a36eb425b11eee4f9784"} err="failed to get container status \"2487e275e38dce21f3825fd5db5b46221622e3ae3802a36eb425b11eee4f9784\": rpc error: code = NotFound desc = could not find container \"2487e275e38dce21f3825fd5db5b46221622e3ae3802a36eb425b11eee4f9784\": container with ID starting with 2487e275e38dce21f3825fd5db5b46221622e3ae3802a36eb425b11eee4f9784 not found: ID does not exist"
Sep 17 17:44:13.435389 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.435349    2569 scope.go:117] "RemoveContainer" containerID="8a007a950ec61fb6320aca30282a7954a0b85a6594e9bb76840a5536faaee77d"
Sep 17 17:44:13.435548 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.435529    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a007a950ec61fb6320aca30282a7954a0b85a6594e9bb76840a5536faaee77d"} err="failed to get container status \"8a007a950ec61fb6320aca30282a7954a0b85a6594e9bb76840a5536faaee77d\": rpc error: code = NotFound desc = could not find container \"8a007a950ec61fb6320aca30282a7954a0b85a6594e9bb76840a5536faaee77d\": container with ID starting with 8a007a950ec61fb6320aca30282a7954a0b85a6594e9bb76840a5536faaee77d not found: ID does not exist"
Sep 17 17:44:13.435601 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.435549    2569 scope.go:117] "RemoveContainer" containerID="55389ceace2ba5a54fd4e46d98faf5125e5d06745751f9fd25dd455992d5ad70"
Sep 17 17:44:13.435804 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.435775    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55389ceace2ba5a54fd4e46d98faf5125e5d06745751f9fd25dd455992d5ad70"} err="failed to get container status \"55389ceace2ba5a54fd4e46d98faf5125e5d06745751f9fd25dd455992d5ad70\": rpc error: code = NotFound desc = could not find container \"55389ceace2ba5a54fd4e46d98faf5125e5d06745751f9fd25dd455992d5ad70\": container with ID starting with 55389ceace2ba5a54fd4e46d98faf5125e5d06745751f9fd25dd455992d5ad70 not found: ID does not exist"
Sep 17 17:44:13.435804 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.435801    2569 scope.go:117] "RemoveContainer" containerID="1e0b5a02b2dca736fd6e082460f2f75a8aa4816923ec2a3db10922e626726244"
Sep 17 17:44:13.436087 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.436056    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e0b5a02b2dca736fd6e082460f2f75a8aa4816923ec2a3db10922e626726244"} err="failed to get container status \"1e0b5a02b2dca736fd6e082460f2f75a8aa4816923ec2a3db10922e626726244\": rpc error: code = NotFound desc = could not find container \"1e0b5a02b2dca736fd6e082460f2f75a8aa4816923ec2a3db10922e626726244\": container with ID starting with 1e0b5a02b2dca736fd6e082460f2f75a8aa4816923ec2a3db10922e626726244 not found: ID does not exist"
Sep 17 17:44:13.436139 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.436088    2569 scope.go:117] "RemoveContainer" containerID="9b7a76e1f93f7ef9ef7bda1eac4a49895a1b437ddcc96410e1b7bada627a3f37"
Sep 17 17:44:13.436337 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.436317    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b7a76e1f93f7ef9ef7bda1eac4a49895a1b437ddcc96410e1b7bada627a3f37"} err="failed to get container status \"9b7a76e1f93f7ef9ef7bda1eac4a49895a1b437ddcc96410e1b7bada627a3f37\": rpc error: code = NotFound desc = could not find container \"9b7a76e1f93f7ef9ef7bda1eac4a49895a1b437ddcc96410e1b7bada627a3f37\": container with ID starting with 9b7a76e1f93f7ef9ef7bda1eac4a49895a1b437ddcc96410e1b7bada627a3f37 not found: ID does not exist"
Sep 17 17:44:13.436337 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.436336    2569 scope.go:117] "RemoveContainer" containerID="fe0d6bdab4b9d938fa796d9f6e97c6fb7eb77f65fb8201ad2e5c766bd882d209"
Sep 17 17:44:13.436565 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.436548    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe0d6bdab4b9d938fa796d9f6e97c6fb7eb77f65fb8201ad2e5c766bd882d209"} err="failed to get container status \"fe0d6bdab4b9d938fa796d9f6e97c6fb7eb77f65fb8201ad2e5c766bd882d209\": rpc error: code = NotFound desc = could not find container \"fe0d6bdab4b9d938fa796d9f6e97c6fb7eb77f65fb8201ad2e5c766bd882d209\": container with ID starting with fe0d6bdab4b9d938fa796d9f6e97c6fb7eb77f65fb8201ad2e5c766bd882d209 not found: ID does not exist"
Sep 17 17:44:13.436611 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.436566    2569 scope.go:117] "RemoveContainer" containerID="e5a3cf85c8af59236d517bee97a54e44de29a2df04684bf1ff488f09706b582b"
Sep 17 17:44:13.436822 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.436793    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5a3cf85c8af59236d517bee97a54e44de29a2df04684bf1ff488f09706b582b"} err="failed to get container status \"e5a3cf85c8af59236d517bee97a54e44de29a2df04684bf1ff488f09706b582b\": rpc error: code = NotFound desc = could not find container \"e5a3cf85c8af59236d517bee97a54e44de29a2df04684bf1ff488f09706b582b\": container with ID starting with e5a3cf85c8af59236d517bee97a54e44de29a2df04684bf1ff488f09706b582b not found: ID does not exist"
Sep 17 17:44:13.436822 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.436819    2569 scope.go:117] "RemoveContainer" containerID="2487e275e38dce21f3825fd5db5b46221622e3ae3802a36eb425b11eee4f9784"
Sep 17 17:44:13.437045 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.437025    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2487e275e38dce21f3825fd5db5b46221622e3ae3802a36eb425b11eee4f9784"} err="failed to get container status \"2487e275e38dce21f3825fd5db5b46221622e3ae3802a36eb425b11eee4f9784\": rpc error: code = NotFound desc = could not find container \"2487e275e38dce21f3825fd5db5b46221622e3ae3802a36eb425b11eee4f9784\": container with ID starting with 2487e275e38dce21f3825fd5db5b46221622e3ae3802a36eb425b11eee4f9784 not found: ID does not exist"
Sep 17 17:44:13.437110 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.437047    2569 scope.go:117] "RemoveContainer" containerID="8a007a950ec61fb6320aca30282a7954a0b85a6594e9bb76840a5536faaee77d"
Sep 17 17:44:13.437328 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.437302    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a007a950ec61fb6320aca30282a7954a0b85a6594e9bb76840a5536faaee77d"} err="failed to get container status \"8a007a950ec61fb6320aca30282a7954a0b85a6594e9bb76840a5536faaee77d\": rpc error: code = NotFound desc = could not find container \"8a007a950ec61fb6320aca30282a7954a0b85a6594e9bb76840a5536faaee77d\": container with ID starting with 8a007a950ec61fb6320aca30282a7954a0b85a6594e9bb76840a5536faaee77d not found: ID does not exist"
Sep 17 17:44:13.437383 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.437328    2569 scope.go:117] "RemoveContainer" containerID="55389ceace2ba5a54fd4e46d98faf5125e5d06745751f9fd25dd455992d5ad70"
Sep 17 17:44:13.437596 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.437569    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55389ceace2ba5a54fd4e46d98faf5125e5d06745751f9fd25dd455992d5ad70"} err="failed to get container status \"55389ceace2ba5a54fd4e46d98faf5125e5d06745751f9fd25dd455992d5ad70\": rpc error: code = NotFound desc = could not find container \"55389ceace2ba5a54fd4e46d98faf5125e5d06745751f9fd25dd455992d5ad70\": container with ID starting with 55389ceace2ba5a54fd4e46d98faf5125e5d06745751f9fd25dd455992d5ad70 not found: ID does not exist"
Sep 17 17:44:13.437596 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.437588    2569 scope.go:117] "RemoveContainer" containerID="1e0b5a02b2dca736fd6e082460f2f75a8aa4816923ec2a3db10922e626726244"
Sep 17 17:44:13.437841 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.437823    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e0b5a02b2dca736fd6e082460f2f75a8aa4816923ec2a3db10922e626726244"} err="failed to get container status \"1e0b5a02b2dca736fd6e082460f2f75a8aa4816923ec2a3db10922e626726244\": rpc error: code = NotFound desc = could not find container \"1e0b5a02b2dca736fd6e082460f2f75a8aa4816923ec2a3db10922e626726244\": container with ID starting with 1e0b5a02b2dca736fd6e082460f2f75a8aa4816923ec2a3db10922e626726244 not found: ID does not exist"
Sep 17 17:44:13.437885 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.437841    2569 scope.go:117] "RemoveContainer" containerID="9b7a76e1f93f7ef9ef7bda1eac4a49895a1b437ddcc96410e1b7bada627a3f37"
Sep 17 17:44:13.438128 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.438102    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b7a76e1f93f7ef9ef7bda1eac4a49895a1b437ddcc96410e1b7bada627a3f37"} err="failed to get container status \"9b7a76e1f93f7ef9ef7bda1eac4a49895a1b437ddcc96410e1b7bada627a3f37\": rpc error: code = NotFound desc = could not find container \"9b7a76e1f93f7ef9ef7bda1eac4a49895a1b437ddcc96410e1b7bada627a3f37\": container with ID starting with 9b7a76e1f93f7ef9ef7bda1eac4a49895a1b437ddcc96410e1b7bada627a3f37 not found: ID does not exist"
Sep 17 17:44:13.438200 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.438129    2569 scope.go:117] "RemoveContainer" containerID="fe0d6bdab4b9d938fa796d9f6e97c6fb7eb77f65fb8201ad2e5c766bd882d209"
Sep 17 17:44:13.438380 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.438352    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe0d6bdab4b9d938fa796d9f6e97c6fb7eb77f65fb8201ad2e5c766bd882d209"} err="failed to get container status \"fe0d6bdab4b9d938fa796d9f6e97c6fb7eb77f65fb8201ad2e5c766bd882d209\": rpc error: code = NotFound desc = could not find container \"fe0d6bdab4b9d938fa796d9f6e97c6fb7eb77f65fb8201ad2e5c766bd882d209\": container with ID starting with fe0d6bdab4b9d938fa796d9f6e97c6fb7eb77f65fb8201ad2e5c766bd882d209 not found: ID does not exist"
Sep 17 17:44:13.438428 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.438380    2569 scope.go:117] "RemoveContainer" containerID="e5a3cf85c8af59236d517bee97a54e44de29a2df04684bf1ff488f09706b582b"
Sep 17 17:44:13.438576 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.438560    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5a3cf85c8af59236d517bee97a54e44de29a2df04684bf1ff488f09706b582b"} err="failed to get container status \"e5a3cf85c8af59236d517bee97a54e44de29a2df04684bf1ff488f09706b582b\": rpc error: code = NotFound desc = could not find container \"e5a3cf85c8af59236d517bee97a54e44de29a2df04684bf1ff488f09706b582b\": container with ID starting with e5a3cf85c8af59236d517bee97a54e44de29a2df04684bf1ff488f09706b582b not found: ID does not exist"
Sep 17 17:44:13.438617 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.438576    2569 scope.go:117] "RemoveContainer" containerID="2487e275e38dce21f3825fd5db5b46221622e3ae3802a36eb425b11eee4f9784"
Sep 17 17:44:13.438773 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.438758    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2487e275e38dce21f3825fd5db5b46221622e3ae3802a36eb425b11eee4f9784"} err="failed to get container status \"2487e275e38dce21f3825fd5db5b46221622e3ae3802a36eb425b11eee4f9784\": rpc error: code = NotFound desc = could not find container \"2487e275e38dce21f3825fd5db5b46221622e3ae3802a36eb425b11eee4f9784\": container with ID starting with 2487e275e38dce21f3825fd5db5b46221622e3ae3802a36eb425b11eee4f9784 not found: ID does not exist"
Sep 17 17:44:13.438816 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.438773    2569 scope.go:117] "RemoveContainer" containerID="8a007a950ec61fb6320aca30282a7954a0b85a6594e9bb76840a5536faaee77d"
Sep 17 17:44:13.438956 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.438941    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a007a950ec61fb6320aca30282a7954a0b85a6594e9bb76840a5536faaee77d"} err="failed to get container status \"8a007a950ec61fb6320aca30282a7954a0b85a6594e9bb76840a5536faaee77d\": rpc error: code = NotFound desc = could not find container \"8a007a950ec61fb6320aca30282a7954a0b85a6594e9bb76840a5536faaee77d\": container with ID starting with 8a007a950ec61fb6320aca30282a7954a0b85a6594e9bb76840a5536faaee77d not found: ID does not exist"
Sep 17 17:44:13.438997 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.438956    2569 scope.go:117] "RemoveContainer" containerID="55389ceace2ba5a54fd4e46d98faf5125e5d06745751f9fd25dd455992d5ad70"
Sep 17 17:44:13.439160 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.439144    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55389ceace2ba5a54fd4e46d98faf5125e5d06745751f9fd25dd455992d5ad70"} err="failed to get container status \"55389ceace2ba5a54fd4e46d98faf5125e5d06745751f9fd25dd455992d5ad70\": rpc error: code = NotFound desc = could not find container \"55389ceace2ba5a54fd4e46d98faf5125e5d06745751f9fd25dd455992d5ad70\": container with ID starting with 55389ceace2ba5a54fd4e46d98faf5125e5d06745751f9fd25dd455992d5ad70 not found: ID does not exist"
Sep 17 17:44:13.439211 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.439159    2569 scope.go:117] "RemoveContainer" containerID="1e0b5a02b2dca736fd6e082460f2f75a8aa4816923ec2a3db10922e626726244"
Sep 17 17:44:13.439359 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.439345    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e0b5a02b2dca736fd6e082460f2f75a8aa4816923ec2a3db10922e626726244"} err="failed to get container status \"1e0b5a02b2dca736fd6e082460f2f75a8aa4816923ec2a3db10922e626726244\": rpc error: code = NotFound desc = could not find container \"1e0b5a02b2dca736fd6e082460f2f75a8aa4816923ec2a3db10922e626726244\": container with ID starting with 1e0b5a02b2dca736fd6e082460f2f75a8aa4816923ec2a3db10922e626726244 not found: ID does not exist"
Sep 17 17:44:13.439414 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.439359    2569 scope.go:117] "RemoveContainer" containerID="9b7a76e1f93f7ef9ef7bda1eac4a49895a1b437ddcc96410e1b7bada627a3f37"
Sep 17 17:44:13.439546 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.439531    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b7a76e1f93f7ef9ef7bda1eac4a49895a1b437ddcc96410e1b7bada627a3f37"} err="failed to get container status \"9b7a76e1f93f7ef9ef7bda1eac4a49895a1b437ddcc96410e1b7bada627a3f37\": rpc error: code = NotFound desc = could not find container \"9b7a76e1f93f7ef9ef7bda1eac4a49895a1b437ddcc96410e1b7bada627a3f37\": container with ID starting with 9b7a76e1f93f7ef9ef7bda1eac4a49895a1b437ddcc96410e1b7bada627a3f37 not found: ID does not exist"
Sep 17 17:44:13.439584 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.439546    2569 scope.go:117] "RemoveContainer" containerID="fe0d6bdab4b9d938fa796d9f6e97c6fb7eb77f65fb8201ad2e5c766bd882d209"
Sep 17 17:44:13.439752 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.439735    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe0d6bdab4b9d938fa796d9f6e97c6fb7eb77f65fb8201ad2e5c766bd882d209"} err="failed to get container status \"fe0d6bdab4b9d938fa796d9f6e97c6fb7eb77f65fb8201ad2e5c766bd882d209\": rpc error: code = NotFound desc = could not find container \"fe0d6bdab4b9d938fa796d9f6e97c6fb7eb77f65fb8201ad2e5c766bd882d209\": container with ID starting with fe0d6bdab4b9d938fa796d9f6e97c6fb7eb77f65fb8201ad2e5c766bd882d209 not found: ID does not exist"
Sep 17 17:44:13.439787 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.439752    2569 scope.go:117] "RemoveContainer" containerID="e5a3cf85c8af59236d517bee97a54e44de29a2df04684bf1ff488f09706b582b"
Sep 17 17:44:13.439935 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.439914    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5a3cf85c8af59236d517bee97a54e44de29a2df04684bf1ff488f09706b582b"} err="failed to get container status \"e5a3cf85c8af59236d517bee97a54e44de29a2df04684bf1ff488f09706b582b\": rpc error: code = NotFound desc = could not find container \"e5a3cf85c8af59236d517bee97a54e44de29a2df04684bf1ff488f09706b582b\": container with ID starting with e5a3cf85c8af59236d517bee97a54e44de29a2df04684bf1ff488f09706b582b not found: ID does not exist"
Sep 17 17:44:13.439977 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.439938    2569 scope.go:117] "RemoveContainer" containerID="2487e275e38dce21f3825fd5db5b46221622e3ae3802a36eb425b11eee4f9784"
Sep 17 17:44:13.440198 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.440173    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2487e275e38dce21f3825fd5db5b46221622e3ae3802a36eb425b11eee4f9784"} err="failed to get container status \"2487e275e38dce21f3825fd5db5b46221622e3ae3802a36eb425b11eee4f9784\": rpc error: code = NotFound desc = could not find container \"2487e275e38dce21f3825fd5db5b46221622e3ae3802a36eb425b11eee4f9784\": container with ID starting with 2487e275e38dce21f3825fd5db5b46221622e3ae3802a36eb425b11eee4f9784 not found: ID does not exist"
Sep 17 17:44:13.440198 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.440197    2569 scope.go:117] "RemoveContainer" containerID="8a007a950ec61fb6320aca30282a7954a0b85a6594e9bb76840a5536faaee77d"
Sep 17 17:44:13.440441 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.440420    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a007a950ec61fb6320aca30282a7954a0b85a6594e9bb76840a5536faaee77d"} err="failed to get container status \"8a007a950ec61fb6320aca30282a7954a0b85a6594e9bb76840a5536faaee77d\": rpc error: code = NotFound desc = could not find container \"8a007a950ec61fb6320aca30282a7954a0b85a6594e9bb76840a5536faaee77d\": container with ID starting with 8a007a950ec61fb6320aca30282a7954a0b85a6594e9bb76840a5536faaee77d not found: ID does not exist"
Sep 17 17:44:13.440486 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.440441    2569 scope.go:117] "RemoveContainer" containerID="55389ceace2ba5a54fd4e46d98faf5125e5d06745751f9fd25dd455992d5ad70"
Sep 17 17:44:13.440645 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.440629    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55389ceace2ba5a54fd4e46d98faf5125e5d06745751f9fd25dd455992d5ad70"} err="failed to get container status \"55389ceace2ba5a54fd4e46d98faf5125e5d06745751f9fd25dd455992d5ad70\": rpc error: code = NotFound desc = could not find container \"55389ceace2ba5a54fd4e46d98faf5125e5d06745751f9fd25dd455992d5ad70\": container with ID starting with 55389ceace2ba5a54fd4e46d98faf5125e5d06745751f9fd25dd455992d5ad70 not found: ID does not exist"
Sep 17 17:44:13.440645 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.440644    2569 scope.go:117] "RemoveContainer" containerID="1e0b5a02b2dca736fd6e082460f2f75a8aa4816923ec2a3db10922e626726244"
Sep 17 17:44:13.440837 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.440823    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e0b5a02b2dca736fd6e082460f2f75a8aa4816923ec2a3db10922e626726244"} err="failed to get container status \"1e0b5a02b2dca736fd6e082460f2f75a8aa4816923ec2a3db10922e626726244\": rpc error: code = NotFound desc = could not find container \"1e0b5a02b2dca736fd6e082460f2f75a8aa4816923ec2a3db10922e626726244\": container with ID starting with 1e0b5a02b2dca736fd6e082460f2f75a8aa4816923ec2a3db10922e626726244 not found: ID does not exist"
Sep 17 17:44:13.440881 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.440838    2569 scope.go:117] "RemoveContainer" containerID="9b7a76e1f93f7ef9ef7bda1eac4a49895a1b437ddcc96410e1b7bada627a3f37"
Sep 17 17:44:13.441055 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.441039    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b7a76e1f93f7ef9ef7bda1eac4a49895a1b437ddcc96410e1b7bada627a3f37"} err="failed to get container status \"9b7a76e1f93f7ef9ef7bda1eac4a49895a1b437ddcc96410e1b7bada627a3f37\": rpc error: code = NotFound desc = could not find container \"9b7a76e1f93f7ef9ef7bda1eac4a49895a1b437ddcc96410e1b7bada627a3f37\": container with ID starting with 9b7a76e1f93f7ef9ef7bda1eac4a49895a1b437ddcc96410e1b7bada627a3f37 not found: ID does not exist"
Sep 17 17:44:13.441115 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.441057    2569 scope.go:117] "RemoveContainer" containerID="fe0d6bdab4b9d938fa796d9f6e97c6fb7eb77f65fb8201ad2e5c766bd882d209"
Sep 17 17:44:13.441287 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.441270    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe0d6bdab4b9d938fa796d9f6e97c6fb7eb77f65fb8201ad2e5c766bd882d209"} err="failed to get container status \"fe0d6bdab4b9d938fa796d9f6e97c6fb7eb77f65fb8201ad2e5c766bd882d209\": rpc error: code = NotFound desc = could not find container \"fe0d6bdab4b9d938fa796d9f6e97c6fb7eb77f65fb8201ad2e5c766bd882d209\": container with ID starting with fe0d6bdab4b9d938fa796d9f6e97c6fb7eb77f65fb8201ad2e5c766bd882d209 not found: ID does not exist"
Sep 17 17:44:13.441333 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.441288    2569 scope.go:117] "RemoveContainer" containerID="e5a3cf85c8af59236d517bee97a54e44de29a2df04684bf1ff488f09706b582b"
Sep 17 17:44:13.441506 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.441478    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5a3cf85c8af59236d517bee97a54e44de29a2df04684bf1ff488f09706b582b"} err="failed to get container status \"e5a3cf85c8af59236d517bee97a54e44de29a2df04684bf1ff488f09706b582b\": rpc error: code = NotFound desc = could not find container \"e5a3cf85c8af59236d517bee97a54e44de29a2df04684bf1ff488f09706b582b\": container with ID starting with e5a3cf85c8af59236d517bee97a54e44de29a2df04684bf1ff488f09706b582b not found: ID does not exist"
Sep 17 17:44:13.456711 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.456692    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/be8bc2cc-9e94-4a44-8897-5f609be6f263-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"be8bc2cc-9e94-4a44-8897-5f609be6f263\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") "
Sep 17 17:44:13.456774 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.456726    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/be8bc2cc-9e94-4a44-8897-5f609be6f263-config\") pod \"be8bc2cc-9e94-4a44-8897-5f609be6f263\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") "
Sep 17 17:44:13.456774 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.456749    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be8bc2cc-9e94-4a44-8897-5f609be6f263-configmap-metrics-client-ca\") pod \"be8bc2cc-9e94-4a44-8897-5f609be6f263\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") "
Sep 17 17:44:13.456837 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.456776    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/be8bc2cc-9e94-4a44-8897-5f609be6f263-secret-prometheus-k8s-tls\") pod \"be8bc2cc-9e94-4a44-8897-5f609be6f263\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") "
Sep 17 17:44:13.456953 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.456931    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be8bc2cc-9e94-4a44-8897-5f609be6f263-prometheus-trusted-ca-bundle\") pod \"be8bc2cc-9e94-4a44-8897-5f609be6f263\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") "
Sep 17 17:44:13.457010 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.456982    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/be8bc2cc-9e94-4a44-8897-5f609be6f263-secret-grpc-tls\") pod \"be8bc2cc-9e94-4a44-8897-5f609be6f263\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") "
Sep 17 17:44:13.457061 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.457008    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/be8bc2cc-9e94-4a44-8897-5f609be6f263-prometheus-k8s-rulefiles-0\") pod \"be8bc2cc-9e94-4a44-8897-5f609be6f263\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") "
Sep 17 17:44:13.457061 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.457039    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be8bc2cc-9e94-4a44-8897-5f609be6f263-configmap-kubelet-serving-ca-bundle\") pod \"be8bc2cc-9e94-4a44-8897-5f609be6f263\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") "
Sep 17 17:44:13.457191 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.457105    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/be8bc2cc-9e94-4a44-8897-5f609be6f263-thanos-prometheus-http-client-file\") pod \"be8bc2cc-9e94-4a44-8897-5f609be6f263\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") "
Sep 17 17:44:13.457191 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.457141    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be8bc2cc-9e94-4a44-8897-5f609be6f263-configmap-serving-certs-ca-bundle\") pod \"be8bc2cc-9e94-4a44-8897-5f609be6f263\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") "
Sep 17 17:44:13.457191 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.457182    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/be8bc2cc-9e94-4a44-8897-5f609be6f263-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"be8bc2cc-9e94-4a44-8897-5f609be6f263\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") "
Sep 17 17:44:13.457343 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.457220    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/be8bc2cc-9e94-4a44-8897-5f609be6f263-config-out\") pod \"be8bc2cc-9e94-4a44-8897-5f609be6f263\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") "
Sep 17 17:44:13.457343 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.457247    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/be8bc2cc-9e94-4a44-8897-5f609be6f263-tls-assets\") pod \"be8bc2cc-9e94-4a44-8897-5f609be6f263\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") "
Sep 17 17:44:13.457343 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.457244    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be8bc2cc-9e94-4a44-8897-5f609be6f263-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "be8bc2cc-9e94-4a44-8897-5f609be6f263" (UID: "be8bc2cc-9e94-4a44-8897-5f609be6f263"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:44:13.457343 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.457273    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/be8bc2cc-9e94-4a44-8897-5f609be6f263-prometheus-k8s-rulefiles-1\") pod \"be8bc2cc-9e94-4a44-8897-5f609be6f263\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") "
Sep 17 17:44:13.457343 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.457303    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/be8bc2cc-9e94-4a44-8897-5f609be6f263-secret-kube-rbac-proxy\") pod \"be8bc2cc-9e94-4a44-8897-5f609be6f263\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") "
Sep 17 17:44:13.457343 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.457327    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/be8bc2cc-9e94-4a44-8897-5f609be6f263-secret-metrics-client-certs\") pod \"be8bc2cc-9e94-4a44-8897-5f609be6f263\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") "
Sep 17 17:44:13.457617 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.457358    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/be8bc2cc-9e94-4a44-8897-5f609be6f263-prometheus-k8s-rulefiles-2\") pod \"be8bc2cc-9e94-4a44-8897-5f609be6f263\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") "
Sep 17 17:44:13.457617 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.457383    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/be8bc2cc-9e94-4a44-8897-5f609be6f263-prometheus-k8s-db\") pod \"be8bc2cc-9e94-4a44-8897-5f609be6f263\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") "
Sep 17 17:44:13.457617 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.457439    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h568j\" (UniqueName: \"kubernetes.io/projected/be8bc2cc-9e94-4a44-8897-5f609be6f263-kube-api-access-h568j\") pod \"be8bc2cc-9e94-4a44-8897-5f609be6f263\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") "
Sep 17 17:44:13.457617 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.457465    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/be8bc2cc-9e94-4a44-8897-5f609be6f263-web-config\") pod \"be8bc2cc-9e94-4a44-8897-5f609be6f263\" (UID: \"be8bc2cc-9e94-4a44-8897-5f609be6f263\") "
Sep 17 17:44:13.457857 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.457726    2569 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be8bc2cc-9e94-4a44-8897-5f609be6f263-configmap-metrics-client-ca\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 17:44:13.459642 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.457371    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be8bc2cc-9e94-4a44-8897-5f609be6f263-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "be8bc2cc-9e94-4a44-8897-5f609be6f263" (UID: "be8bc2cc-9e94-4a44-8897-5f609be6f263"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:44:13.459642 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.459600    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be8bc2cc-9e94-4a44-8897-5f609be6f263-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "be8bc2cc-9e94-4a44-8897-5f609be6f263" (UID: "be8bc2cc-9e94-4a44-8897-5f609be6f263"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:44:13.459642 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.457573    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be8bc2cc-9e94-4a44-8897-5f609be6f263-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "be8bc2cc-9e94-4a44-8897-5f609be6f263" (UID: "be8bc2cc-9e94-4a44-8897-5f609be6f263"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:44:13.459642 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.457734    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be8bc2cc-9e94-4a44-8897-5f609be6f263-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "be8bc2cc-9e94-4a44-8897-5f609be6f263" (UID: "be8bc2cc-9e94-4a44-8897-5f609be6f263"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:44:13.460021 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.457952    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be8bc2cc-9e94-4a44-8897-5f609be6f263-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "be8bc2cc-9e94-4a44-8897-5f609be6f263" (UID: "be8bc2cc-9e94-4a44-8897-5f609be6f263"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:44:13.460021 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.458804    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be8bc2cc-9e94-4a44-8897-5f609be6f263-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "be8bc2cc-9e94-4a44-8897-5f609be6f263" (UID: "be8bc2cc-9e94-4a44-8897-5f609be6f263"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:44:13.460021 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.459011    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be8bc2cc-9e94-4a44-8897-5f609be6f263-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "be8bc2cc-9e94-4a44-8897-5f609be6f263" (UID: "be8bc2cc-9e94-4a44-8897-5f609be6f263"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:44:13.460456 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.460382    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be8bc2cc-9e94-4a44-8897-5f609be6f263-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "be8bc2cc-9e94-4a44-8897-5f609be6f263" (UID: "be8bc2cc-9e94-4a44-8897-5f609be6f263"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:44:13.460548 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.460515    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be8bc2cc-9e94-4a44-8897-5f609be6f263-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "be8bc2cc-9e94-4a44-8897-5f609be6f263" (UID: "be8bc2cc-9e94-4a44-8897-5f609be6f263"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:44:13.460693 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.460658    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be8bc2cc-9e94-4a44-8897-5f609be6f263-config" (OuterVolumeSpecName: "config") pod "be8bc2cc-9e94-4a44-8897-5f609be6f263" (UID: "be8bc2cc-9e94-4a44-8897-5f609be6f263"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:44:13.460805 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.460769    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be8bc2cc-9e94-4a44-8897-5f609be6f263-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "be8bc2cc-9e94-4a44-8897-5f609be6f263" (UID: "be8bc2cc-9e94-4a44-8897-5f609be6f263"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:44:13.460933 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.460906    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be8bc2cc-9e94-4a44-8897-5f609be6f263-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "be8bc2cc-9e94-4a44-8897-5f609be6f263" (UID: "be8bc2cc-9e94-4a44-8897-5f609be6f263"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:44:13.461194 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.461162    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be8bc2cc-9e94-4a44-8897-5f609be6f263-config-out" (OuterVolumeSpecName: "config-out") pod "be8bc2cc-9e94-4a44-8897-5f609be6f263" (UID: "be8bc2cc-9e94-4a44-8897-5f609be6f263"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:44:13.461297 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.461271    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be8bc2cc-9e94-4a44-8897-5f609be6f263-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "be8bc2cc-9e94-4a44-8897-5f609be6f263" (UID: "be8bc2cc-9e94-4a44-8897-5f609be6f263"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:44:13.462519 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.462489    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be8bc2cc-9e94-4a44-8897-5f609be6f263-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "be8bc2cc-9e94-4a44-8897-5f609be6f263" (UID: "be8bc2cc-9e94-4a44-8897-5f609be6f263"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:44:13.462624 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.462498    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be8bc2cc-9e94-4a44-8897-5f609be6f263-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "be8bc2cc-9e94-4a44-8897-5f609be6f263" (UID: "be8bc2cc-9e94-4a44-8897-5f609be6f263"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:44:13.462683 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.462640    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be8bc2cc-9e94-4a44-8897-5f609be6f263-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "be8bc2cc-9e94-4a44-8897-5f609be6f263" (UID: "be8bc2cc-9e94-4a44-8897-5f609be6f263"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:44:13.462864 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.462850    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be8bc2cc-9e94-4a44-8897-5f609be6f263-kube-api-access-h568j" (OuterVolumeSpecName: "kube-api-access-h568j") pod "be8bc2cc-9e94-4a44-8897-5f609be6f263" (UID: "be8bc2cc-9e94-4a44-8897-5f609be6f263"). InnerVolumeSpecName "kube-api-access-h568j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:44:13.470578 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.470554    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be8bc2cc-9e94-4a44-8897-5f609be6f263-web-config" (OuterVolumeSpecName: "web-config") pod "be8bc2cc-9e94-4a44-8897-5f609be6f263" (UID: "be8bc2cc-9e94-4a44-8897-5f609be6f263"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:44:13.558973 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.558933    2569 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/be8bc2cc-9e94-4a44-8897-5f609be6f263-secret-prometheus-k8s-tls\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 17:44:13.558973 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.558966    2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be8bc2cc-9e94-4a44-8897-5f609be6f263-prometheus-trusted-ca-bundle\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 17:44:13.558973 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.558978    2569 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/be8bc2cc-9e94-4a44-8897-5f609be6f263-secret-grpc-tls\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 17:44:13.559213 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.558987    2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/be8bc2cc-9e94-4a44-8897-5f609be6f263-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 17:44:13.559213 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.558998    2569 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be8bc2cc-9e94-4a44-8897-5f609be6f263-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 17:44:13.559213 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.559006    2569 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/be8bc2cc-9e94-4a44-8897-5f609be6f263-thanos-prometheus-http-client-file\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 17:44:13.559213 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.559014    2569 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be8bc2cc-9e94-4a44-8897-5f609be6f263-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 17:44:13.559213 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.559022    2569 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/be8bc2cc-9e94-4a44-8897-5f609be6f263-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 17:44:13.559213 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.559032    2569 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/be8bc2cc-9e94-4a44-8897-5f609be6f263-config-out\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 17:44:13.559213 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.559040    2569 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/be8bc2cc-9e94-4a44-8897-5f609be6f263-tls-assets\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 17:44:13.559213 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.559047    2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/be8bc2cc-9e94-4a44-8897-5f609be6f263-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 17:44:13.559213 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.559055    2569 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/be8bc2cc-9e94-4a44-8897-5f609be6f263-secret-kube-rbac-proxy\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 17:44:13.559213 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.559063    2569 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/be8bc2cc-9e94-4a44-8897-5f609be6f263-secret-metrics-client-certs\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 17:44:13.559213 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.559071    2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/be8bc2cc-9e94-4a44-8897-5f609be6f263-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 17:44:13.559213 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.559103    2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/be8bc2cc-9e94-4a44-8897-5f609be6f263-prometheus-k8s-db\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 17:44:13.559213 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.559118    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h568j\" (UniqueName: \"kubernetes.io/projected/be8bc2cc-9e94-4a44-8897-5f609be6f263-kube-api-access-h568j\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 17:44:13.559213 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.559131    2569 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/be8bc2cc-9e94-4a44-8897-5f609be6f263-web-config\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 17:44:13.559213 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.559142    2569 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/be8bc2cc-9e94-4a44-8897-5f609be6f263-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 17:44:13.559213 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.559150    2569 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/be8bc2cc-9e94-4a44-8897-5f609be6f263-config\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 17:44:13.697288 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.697264    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 17:44:13.701882 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.701848    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 17:44:13.754815 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.754780    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 17:44:13.755048 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.755035    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be8bc2cc-9e94-4a44-8897-5f609be6f263" containerName="prometheus"
Sep 17 17:44:13.755107 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.755050    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="be8bc2cc-9e94-4a44-8897-5f609be6f263" containerName="prometheus"
Sep 17 17:44:13.755107 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.755064    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be8bc2cc-9e94-4a44-8897-5f609be6f263" containerName="init-config-reloader"
Sep 17 17:44:13.755107 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.755069    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="be8bc2cc-9e94-4a44-8897-5f609be6f263" containerName="init-config-reloader"
Sep 17 17:44:13.755107 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.755096    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be8bc2cc-9e94-4a44-8897-5f609be6f263" containerName="config-reloader"
Sep 17 17:44:13.755107 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.755103    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="be8bc2cc-9e94-4a44-8897-5f609be6f263" containerName="config-reloader"
Sep 17 17:44:13.755256 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.755115    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be8bc2cc-9e94-4a44-8897-5f609be6f263" containerName="kube-rbac-proxy"
Sep 17 17:44:13.755256 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.755120    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="be8bc2cc-9e94-4a44-8897-5f609be6f263" containerName="kube-rbac-proxy"
Sep 17 17:44:13.755256 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.755129    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be8bc2cc-9e94-4a44-8897-5f609be6f263" containerName="kube-rbac-proxy-web"
Sep 17 17:44:13.755256 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.755134    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="be8bc2cc-9e94-4a44-8897-5f609be6f263" containerName="kube-rbac-proxy-web"
Sep 17 17:44:13.755256 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.755141    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be8bc2cc-9e94-4a44-8897-5f609be6f263" containerName="thanos-sidecar"
Sep 17 17:44:13.755256 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.755146    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="be8bc2cc-9e94-4a44-8897-5f609be6f263" containerName="thanos-sidecar"
Sep 17 17:44:13.755256 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.755153    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be8bc2cc-9e94-4a44-8897-5f609be6f263" containerName="kube-rbac-proxy-thanos"
Sep 17 17:44:13.755256 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.755157    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="be8bc2cc-9e94-4a44-8897-5f609be6f263" containerName="kube-rbac-proxy-thanos"
Sep 17 17:44:13.755256 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.755199    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="be8bc2cc-9e94-4a44-8897-5f609be6f263" containerName="thanos-sidecar"
Sep 17 17:44:13.755256 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.755205    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="be8bc2cc-9e94-4a44-8897-5f609be6f263" containerName="kube-rbac-proxy-web"
Sep 17 17:44:13.755256 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.755211    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="be8bc2cc-9e94-4a44-8897-5f609be6f263" containerName="kube-rbac-proxy"
Sep 17 17:44:13.755256 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.755216    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="be8bc2cc-9e94-4a44-8897-5f609be6f263" containerName="kube-rbac-proxy-thanos"
Sep 17 17:44:13.755256 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.755222    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="be8bc2cc-9e94-4a44-8897-5f609be6f263" containerName="prometheus"
Sep 17 17:44:13.755256 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.755227    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="be8bc2cc-9e94-4a44-8897-5f609be6f263" containerName="config-reloader"
Sep 17 17:44:13.760373 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.760354    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:44:13.768182 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.768159    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 17 17:44:13.768512 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.768493    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 17 17:44:13.768582 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.768517    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 17 17:44:13.769153 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.769135    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 17 17:44:13.769197 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.769154    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 17 17:44:13.769197 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.769170    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 17 17:44:13.769297 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.769250    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 17 17:44:13.769297 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.769273    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 17 17:44:13.769390 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.769279    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 17 17:44:13.769566 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.769548    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 17 17:44:13.769626 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.769549    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-ba8usoq20r88c\""
Sep 17 17:44:13.769952 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.769938    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 17 17:44:13.770016 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.769950    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 17 17:44:13.770196 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.770184    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-76wkd\""
Sep 17 17:44:13.786015 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.785994    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 17 17:44:13.800517 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.800484    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 17:44:13.802877 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.802858    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 17 17:44:13.861065 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.861030    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfe07e07-2803-45c5-b12c-460185459d32-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cfe07e07-2803-45c5-b12c-460185459d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:44:13.861065 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.861065    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cfe07e07-2803-45c5-b12c-460185459d32-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cfe07e07-2803-45c5-b12c-460185459d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:44:13.861335 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.861146    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cfe07e07-2803-45c5-b12c-460185459d32-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cfe07e07-2803-45c5-b12c-460185459d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:44:13.861335 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.861186    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sbvd\" (UniqueName: \"kubernetes.io/projected/cfe07e07-2803-45c5-b12c-460185459d32-kube-api-access-5sbvd\") pod \"prometheus-k8s-0\" (UID: \"cfe07e07-2803-45c5-b12c-460185459d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:44:13.861335 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.861224    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cfe07e07-2803-45c5-b12c-460185459d32-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cfe07e07-2803-45c5-b12c-460185459d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:44:13.861335 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.861255    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cfe07e07-2803-45c5-b12c-460185459d32-config-out\") pod \"prometheus-k8s-0\" (UID: \"cfe07e07-2803-45c5-b12c-460185459d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:44:13.861335 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.861279    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfe07e07-2803-45c5-b12c-460185459d32-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cfe07e07-2803-45c5-b12c-460185459d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:44:13.861335 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.861312    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cfe07e07-2803-45c5-b12c-460185459d32-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cfe07e07-2803-45c5-b12c-460185459d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:44:13.861607 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.861338    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cfe07e07-2803-45c5-b12c-460185459d32-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cfe07e07-2803-45c5-b12c-460185459d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:44:13.861607 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.861364    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cfe07e07-2803-45c5-b12c-460185459d32-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cfe07e07-2803-45c5-b12c-460185459d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:44:13.861607 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.861390    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cfe07e07-2803-45c5-b12c-460185459d32-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cfe07e07-2803-45c5-b12c-460185459d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:44:13.861607 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.861432    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cfe07e07-2803-45c5-b12c-460185459d32-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cfe07e07-2803-45c5-b12c-460185459d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:44:13.861607 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.861455    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cfe07e07-2803-45c5-b12c-460185459d32-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cfe07e07-2803-45c5-b12c-460185459d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:44:13.861607 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.861490    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cfe07e07-2803-45c5-b12c-460185459d32-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cfe07e07-2803-45c5-b12c-460185459d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:44:13.861607 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.861514    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cfe07e07-2803-45c5-b12c-460185459d32-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cfe07e07-2803-45c5-b12c-460185459d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:44:13.861607 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.861564    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfe07e07-2803-45c5-b12c-460185459d32-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cfe07e07-2803-45c5-b12c-460185459d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:44:13.861607 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.861583    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/cfe07e07-2803-45c5-b12c-460185459d32-config\") pod \"prometheus-k8s-0\" (UID: \"cfe07e07-2803-45c5-b12c-460185459d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:44:13.861607 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.861605    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cfe07e07-2803-45c5-b12c-460185459d32-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cfe07e07-2803-45c5-b12c-460185459d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:44:13.861970 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.861679    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cfe07e07-2803-45c5-b12c-460185459d32-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cfe07e07-2803-45c5-b12c-460185459d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:44:13.861970 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.861722    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cfe07e07-2803-45c5-b12c-460185459d32-web-config\") pod \"prometheus-k8s-0\" (UID: \"cfe07e07-2803-45c5-b12c-460185459d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:44:13.962521 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.962433    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cfe07e07-2803-45c5-b12c-460185459d32-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cfe07e07-2803-45c5-b12c-460185459d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:44:13.962521 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.962474    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cfe07e07-2803-45c5-b12c-460185459d32-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cfe07e07-2803-45c5-b12c-460185459d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:44:13.962521 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.962499    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cfe07e07-2803-45c5-b12c-460185459d32-web-config\") pod \"prometheus-k8s-0\" (UID: \"cfe07e07-2803-45c5-b12c-460185459d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:44:13.962521 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.962522    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfe07e07-2803-45c5-b12c-460185459d32-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cfe07e07-2803-45c5-b12c-460185459d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:44:13.962844 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.962541    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cfe07e07-2803-45c5-b12c-460185459d32-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cfe07e07-2803-45c5-b12c-460185459d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:44:13.962844 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.962565    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cfe07e07-2803-45c5-b12c-460185459d32-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cfe07e07-2803-45c5-b12c-460185459d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:44:13.962844 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.962586    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sbvd\" (UniqueName: \"kubernetes.io/projected/cfe07e07-2803-45c5-b12c-460185459d32-kube-api-access-5sbvd\") pod \"prometheus-k8s-0\" (UID: \"cfe07e07-2803-45c5-b12c-460185459d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:44:13.962844 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.962628    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cfe07e07-2803-45c5-b12c-460185459d32-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cfe07e07-2803-45c5-b12c-460185459d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:44:13.962844 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.962660    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cfe07e07-2803-45c5-b12c-460185459d32-config-out\") pod \"prometheus-k8s-0\" (UID: \"cfe07e07-2803-45c5-b12c-460185459d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:44:13.962844 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.962683    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfe07e07-2803-45c5-b12c-460185459d32-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cfe07e07-2803-45c5-b12c-460185459d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:44:13.962844 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.962719    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cfe07e07-2803-45c5-b12c-460185459d32-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cfe07e07-2803-45c5-b12c-460185459d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:44:13.962844 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.962744    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cfe07e07-2803-45c5-b12c-460185459d32-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cfe07e07-2803-45c5-b12c-460185459d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:44:13.962844 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.962763    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cfe07e07-2803-45c5-b12c-460185459d32-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cfe07e07-2803-45c5-b12c-460185459d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:44:13.962844 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.962783    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cfe07e07-2803-45c5-b12c-460185459d32-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cfe07e07-2803-45c5-b12c-460185459d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:44:13.962844 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.962811    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cfe07e07-2803-45c5-b12c-460185459d32-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cfe07e07-2803-45c5-b12c-460185459d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:44:13.962844 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.962833    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cfe07e07-2803-45c5-b12c-460185459d32-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cfe07e07-2803-45c5-b12c-460185459d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:44:13.963437 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.962869    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cfe07e07-2803-45c5-b12c-460185459d32-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cfe07e07-2803-45c5-b12c-460185459d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:44:13.963437 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.962893    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cfe07e07-2803-45c5-b12c-460185459d32-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cfe07e07-2803-45c5-b12c-460185459d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:44:13.963437 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.962933    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfe07e07-2803-45c5-b12c-460185459d32-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cfe07e07-2803-45c5-b12c-460185459d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:44:13.963437 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.962962    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/cfe07e07-2803-45c5-b12c-460185459d32-config\") pod \"prometheus-k8s-0\" (UID: \"cfe07e07-2803-45c5-b12c-460185459d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:44:13.963437 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.963337    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cfe07e07-2803-45c5-b12c-460185459d32-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cfe07e07-2803-45c5-b12c-460185459d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:44:13.963688 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.963500    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cfe07e07-2803-45c5-b12c-460185459d32-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cfe07e07-2803-45c5-b12c-460185459d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:44:13.963688 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.963651    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfe07e07-2803-45c5-b12c-460185459d32-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cfe07e07-2803-45c5-b12c-460185459d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:44:13.964017 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.963983    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cfe07e07-2803-45c5-b12c-460185459d32-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cfe07e07-2803-45c5-b12c-460185459d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:44:13.965748 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.965718    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cfe07e07-2803-45c5-b12c-460185459d32-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cfe07e07-2803-45c5-b12c-460185459d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:44:13.965864 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.965787    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cfe07e07-2803-45c5-b12c-460185459d32-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cfe07e07-2803-45c5-b12c-460185459d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:44:13.965936 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.965920    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/cfe07e07-2803-45c5-b12c-460185459d32-config\") pod \"prometheus-k8s-0\" (UID: \"cfe07e07-2803-45c5-b12c-460185459d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:44:13.966001 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.965981    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cfe07e07-2803-45c5-b12c-460185459d32-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cfe07e07-2803-45c5-b12c-460185459d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:44:13.966064 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.966026    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cfe07e07-2803-45c5-b12c-460185459d32-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cfe07e07-2803-45c5-b12c-460185459d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:44:13.966348 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.966327    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cfe07e07-2803-45c5-b12c-460185459d32-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cfe07e07-2803-45c5-b12c-460185459d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:44:13.966455 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.966387    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cfe07e07-2803-45c5-b12c-460185459d32-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cfe07e07-2803-45c5-b12c-460185459d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:44:13.966455 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.966438    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfe07e07-2803-45c5-b12c-460185459d32-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cfe07e07-2803-45c5-b12c-460185459d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:44:13.966607 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.966580    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfe07e07-2803-45c5-b12c-460185459d32-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cfe07e07-2803-45c5-b12c-460185459d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:44:13.967017 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.966999    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cfe07e07-2803-45c5-b12c-460185459d32-web-config\") pod \"prometheus-k8s-0\" (UID: \"cfe07e07-2803-45c5-b12c-460185459d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:44:13.968416 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.968388    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cfe07e07-2803-45c5-b12c-460185459d32-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cfe07e07-2803-45c5-b12c-460185459d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:44:13.968491 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.968474    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cfe07e07-2803-45c5-b12c-460185459d32-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cfe07e07-2803-45c5-b12c-460185459d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:44:13.968556 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.968540    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cfe07e07-2803-45c5-b12c-460185459d32-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cfe07e07-2803-45c5-b12c-460185459d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:44:13.968590 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.968544    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cfe07e07-2803-45c5-b12c-460185459d32-config-out\") pod \"prometheus-k8s-0\" (UID: \"cfe07e07-2803-45c5-b12c-460185459d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:44:13.968711 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.968695    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cfe07e07-2803-45c5-b12c-460185459d32-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cfe07e07-2803-45c5-b12c-460185459d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:44:13.971368 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:13.971346    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sbvd\" (UniqueName: \"kubernetes.io/projected/cfe07e07-2803-45c5-b12c-460185459d32-kube-api-access-5sbvd\") pod \"prometheus-k8s-0\" (UID: \"cfe07e07-2803-45c5-b12c-460185459d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:44:14.069656 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:14.069603    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:44:14.350354 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:14.350320    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 17:44:14.352503 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:44:14.352475    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfe07e07_2803_45c5_b12c_460185459d32.slice/crio-af515e65e0e8422128080c85b5320eb7f1f9d7991289d167168045abd0d8aa97 WatchSource:0}: Error finding container af515e65e0e8422128080c85b5320eb7f1f9d7991289d167168045abd0d8aa97: Status 404 returned error can't find the container with id af515e65e0e8422128080c85b5320eb7f1f9d7991289d167168045abd0d8aa97
Sep 17 17:44:14.369967 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:14.369939    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cfe07e07-2803-45c5-b12c-460185459d32","Type":"ContainerStarted","Data":"af515e65e0e8422128080c85b5320eb7f1f9d7991289d167168045abd0d8aa97"}
Sep 17 17:44:15.375405 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:15.375311    2569 generic.go:356] "Generic (PLEG): container finished" podID="cfe07e07-2803-45c5-b12c-460185459d32" containerID="7fab7fd0ec4aeef09cc4749e30cc9db338a1b492f39189081134dde91473fd41" exitCode=0
Sep 17 17:44:15.375405 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:15.375391    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cfe07e07-2803-45c5-b12c-460185459d32","Type":"ContainerDied","Data":"7fab7fd0ec4aeef09cc4749e30cc9db338a1b492f39189081134dde91473fd41"}
Sep 17 17:44:15.509229 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:15.509199    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be8bc2cc-9e94-4a44-8897-5f609be6f263" path="/var/lib/kubelet/pods/be8bc2cc-9e94-4a44-8897-5f609be6f263/volumes"
Sep 17 17:44:16.383745 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:16.383708    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cfe07e07-2803-45c5-b12c-460185459d32","Type":"ContainerStarted","Data":"a84644b811a9d64de9fb5cd10c4162900b7e6c761cf589b6bfb0022583a13f0e"}
Sep 17 17:44:16.384223 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:16.383775    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cfe07e07-2803-45c5-b12c-460185459d32","Type":"ContainerStarted","Data":"0a8be16c1be7bce1f84a6421843d593ecdf4c4d0b8795c1335b7f34d03d80506"}
Sep 17 17:44:16.384223 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:16.383806    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cfe07e07-2803-45c5-b12c-460185459d32","Type":"ContainerStarted","Data":"38c26454cbf61d2323114ca5e0046f84711318d5157e3b9527c8778388935310"}
Sep 17 17:44:16.384223 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:16.383832    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cfe07e07-2803-45c5-b12c-460185459d32","Type":"ContainerStarted","Data":"2d4f40a013bc94dde7e89fc7a16c0c21065d53ccdb55b0486c2f0a8b3d626319"}
Sep 17 17:44:16.384223 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:16.383859    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cfe07e07-2803-45c5-b12c-460185459d32","Type":"ContainerStarted","Data":"981fe2347abfedacff3978079ee06b9d6a9794b4b804a40ea70b75ad99363fdb"}
Sep 17 17:44:16.384223 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:16.383885    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cfe07e07-2803-45c5-b12c-460185459d32","Type":"ContainerStarted","Data":"911b96dad5403707f90523562661020b440b8e438c206fc1a257d5cb838365e4"}
Sep 17 17:44:16.415248 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:16.415196    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.395727631 podStartE2EDuration="3.415178317s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.450686ms" isStatefulPod=false podCreationTimestamp="2026-09-17 17:44:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:44:16.41341145 +0000 UTC m=+208.917414112" watchObservedRunningTime="2026-09-17 17:44:16.415178317 +0000 UTC m=+208.919180941"
Sep 17 17:44:19.070287 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:44:19.070245    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:45:14.070139 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:45:14.070105    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:45:14.086028 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:45:14.086002    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:45:14.621328 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:45:14.621301    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:45:49.429910 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:45:49.429886    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-t7pqb_b8d76e28-4655-4d21-98d8-0324e4756aaf/console-operator/2.log"
Sep 17 17:45:49.431641 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:45:49.431618    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-t7pqb_b8d76e28-4655-4d21-98d8-0324e4756aaf/console-operator/2.log"
Sep 17 17:45:49.438219 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:45:49.438191    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k4mqn_6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff/ovn-acl-logging/0.log"
Sep 17 17:45:49.440260 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:45:49.440234    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k4mqn_6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff/ovn-acl-logging/0.log"
Sep 17 17:45:49.444325 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:45:49.444306    2569 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 17:49:54.387771 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:49:54.387689    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-bz49k"]
Sep 17 17:49:54.391045 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:49:54.391028    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-bz49k"
Sep 17 17:49:54.394407 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:49:54.394388    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 17:49:54.395564 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:49:54.395540    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 17 17:49:54.395564 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:49:54.395542    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 17 17:49:54.395704 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:49:54.395549    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 17:49:54.395704 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:49:54.395597    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-dcrjv\""
Sep 17 17:49:54.400156 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:49:54.400135    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-bz49k"]
Sep 17 17:49:54.457889 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:49:54.457845    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pg2s\" (UniqueName: \"kubernetes.io/projected/4d2cfbc9-260a-4bb3-98d0-f70eea33c6a4-kube-api-access-6pg2s\") pod \"s3-init-bz49k\" (UID: \"4d2cfbc9-260a-4bb3-98d0-f70eea33c6a4\") " pod="kserve/s3-init-bz49k"
Sep 17 17:49:54.457889 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:49:54.457888    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d2cfbc9-260a-4bb3-98d0-f70eea33c6a4-model-cache\") pod \"s3-init-bz49k\" (UID: \"4d2cfbc9-260a-4bb3-98d0-f70eea33c6a4\") " pod="kserve/s3-init-bz49k"
Sep 17 17:49:54.559244 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:49:54.559193    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pg2s\" (UniqueName: \"kubernetes.io/projected/4d2cfbc9-260a-4bb3-98d0-f70eea33c6a4-kube-api-access-6pg2s\") pod \"s3-init-bz49k\" (UID: \"4d2cfbc9-260a-4bb3-98d0-f70eea33c6a4\") " pod="kserve/s3-init-bz49k"
Sep 17 17:49:54.559244 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:49:54.559251    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d2cfbc9-260a-4bb3-98d0-f70eea33c6a4-model-cache\") pod \"s3-init-bz49k\" (UID: \"4d2cfbc9-260a-4bb3-98d0-f70eea33c6a4\") " pod="kserve/s3-init-bz49k"
Sep 17 17:49:54.559656 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:49:54.559636    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d2cfbc9-260a-4bb3-98d0-f70eea33c6a4-model-cache\") pod \"s3-init-bz49k\" (UID: \"4d2cfbc9-260a-4bb3-98d0-f70eea33c6a4\") " pod="kserve/s3-init-bz49k"
Sep 17 17:49:54.568954 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:49:54.568931    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pg2s\" (UniqueName: \"kubernetes.io/projected/4d2cfbc9-260a-4bb3-98d0-f70eea33c6a4-kube-api-access-6pg2s\") pod \"s3-init-bz49k\" (UID: \"4d2cfbc9-260a-4bb3-98d0-f70eea33c6a4\") " pod="kserve/s3-init-bz49k"
Sep 17 17:49:54.718172 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:49:54.718126    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-bz49k"
Sep 17 17:49:54.990426 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:49:54.990400    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-bz49k"]
Sep 17 17:49:54.992906 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:49:54.992873    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d2cfbc9_260a_4bb3_98d0_f70eea33c6a4.slice/crio-ec5fba1178502fe48f9ffadd65b495b63266c6fb9033fce360144ee3767c276e WatchSource:0}: Error finding container ec5fba1178502fe48f9ffadd65b495b63266c6fb9033fce360144ee3767c276e: Status 404 returned error can't find the container with id ec5fba1178502fe48f9ffadd65b495b63266c6fb9033fce360144ee3767c276e
Sep 17 17:49:54.994906 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:49:54.994893    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:49:55.671014 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:49:55.670968    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-bz49k" event={"ID":"4d2cfbc9-260a-4bb3-98d0-f70eea33c6a4","Type":"ContainerStarted","Data":"ec5fba1178502fe48f9ffadd65b495b63266c6fb9033fce360144ee3767c276e"}
Sep 17 17:49:59.690623 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:49:59.690557    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-bz49k" event={"ID":"4d2cfbc9-260a-4bb3-98d0-f70eea33c6a4","Type":"ContainerStarted","Data":"ca1e584c23155909c537cc264960d1fb05127c7e6fa1ba60bbc26bbdd3a23e4f"}
Sep 17 17:50:01.700498 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:50:01.700459    2569 generic.go:356] "Generic (PLEG): container finished" podID="4d2cfbc9-260a-4bb3-98d0-f70eea33c6a4" containerID="ca1e584c23155909c537cc264960d1fb05127c7e6fa1ba60bbc26bbdd3a23e4f" exitCode=0
Sep 17 17:50:01.700981 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:50:01.700527    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-bz49k" event={"ID":"4d2cfbc9-260a-4bb3-98d0-f70eea33c6a4","Type":"ContainerDied","Data":"ca1e584c23155909c537cc264960d1fb05127c7e6fa1ba60bbc26bbdd3a23e4f"}
Sep 17 17:50:06.724706 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:50:06.724667    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-bz49k" event={"ID":"4d2cfbc9-260a-4bb3-98d0-f70eea33c6a4","Type":"ContainerStarted","Data":"7d6094440a0c15901b8d49b1954719e604f2ba24e148093ce9c4bcf836607217"}
Sep 17 17:50:06.774860 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:50:06.774804    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-bz49k" podStartSLOduration=2.609992192 podStartE2EDuration="12.774785547s" totalImagesPullingTime="8.176683088s" totalInitContainerRuntime="1.988110267s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:49:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:50:06.77352169 +0000 UTC m=+559.277524322" watchObservedRunningTime="2026-09-17 17:50:06.774785547 +0000 UTC m=+559.278788177"
Sep 17 17:50:49.466198 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:50:49.466167    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-t7pqb_b8d76e28-4655-4d21-98d8-0324e4756aaf/console-operator/2.log"
Sep 17 17:50:49.466821 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:50:49.466430    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-t7pqb_b8d76e28-4655-4d21-98d8-0324e4756aaf/console-operator/2.log"
Sep 17 17:50:49.472202 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:50:49.472176    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k4mqn_6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff/ovn-acl-logging/0.log"
Sep 17 17:50:49.472677 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:50:49.472659    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k4mqn_6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff/ovn-acl-logging/0.log"
Sep 17 17:51:11.982999 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:51:11.982962    2569 generic.go:356] "Generic (PLEG): container finished" podID="4d2cfbc9-260a-4bb3-98d0-f70eea33c6a4" containerID="7d6094440a0c15901b8d49b1954719e604f2ba24e148093ce9c4bcf836607217" exitCode=0
Sep 17 17:51:11.983428 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:51:11.983029    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-bz49k" event={"ID":"4d2cfbc9-260a-4bb3-98d0-f70eea33c6a4","Type":"ContainerDied","Data":"7d6094440a0c15901b8d49b1954719e604f2ba24e148093ce9c4bcf836607217"}
Sep 17 17:51:13.111183 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:51:13.111157    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-bz49k"
Sep 17 17:51:13.208886 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:51:13.208845    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d2cfbc9-260a-4bb3-98d0-f70eea33c6a4-model-cache\") pod \"4d2cfbc9-260a-4bb3-98d0-f70eea33c6a4\" (UID: \"4d2cfbc9-260a-4bb3-98d0-f70eea33c6a4\") "
Sep 17 17:51:13.209107 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:51:13.208983    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6pg2s\" (UniqueName: \"kubernetes.io/projected/4d2cfbc9-260a-4bb3-98d0-f70eea33c6a4-kube-api-access-6pg2s\") pod \"4d2cfbc9-260a-4bb3-98d0-f70eea33c6a4\" (UID: \"4d2cfbc9-260a-4bb3-98d0-f70eea33c6a4\") "
Sep 17 17:51:13.211344 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:51:13.211306    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d2cfbc9-260a-4bb3-98d0-f70eea33c6a4-kube-api-access-6pg2s" (OuterVolumeSpecName: "kube-api-access-6pg2s") pod "4d2cfbc9-260a-4bb3-98d0-f70eea33c6a4" (UID: "4d2cfbc9-260a-4bb3-98d0-f70eea33c6a4"). InnerVolumeSpecName "kube-api-access-6pg2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:51:13.227798 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:51:13.227750    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d2cfbc9-260a-4bb3-98d0-f70eea33c6a4-model-cache" (OuterVolumeSpecName: "model-cache") pod "4d2cfbc9-260a-4bb3-98d0-f70eea33c6a4" (UID: "4d2cfbc9-260a-4bb3-98d0-f70eea33c6a4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:51:13.309931 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:51:13.309840    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pg2s\" (UniqueName: \"kubernetes.io/projected/4d2cfbc9-260a-4bb3-98d0-f70eea33c6a4-kube-api-access-6pg2s\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 17:51:13.309931 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:51:13.309875    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d2cfbc9-260a-4bb3-98d0-f70eea33c6a4-model-cache\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 17:51:13.992146 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:51:13.992116    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-bz49k"
Sep 17 17:51:13.992333 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:51:13.992115    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-bz49k" event={"ID":"4d2cfbc9-260a-4bb3-98d0-f70eea33c6a4","Type":"ContainerDied","Data":"ec5fba1178502fe48f9ffadd65b495b63266c6fb9033fce360144ee3767c276e"}
Sep 17 17:51:13.992333 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:51:13.992230    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec5fba1178502fe48f9ffadd65b495b63266c6fb9033fce360144ee3767c276e"
Sep 17 17:51:50.349137 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:51:50.349104    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-zdrc5"]
Sep 17 17:51:50.349595 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:51:50.349401    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d2cfbc9-260a-4bb3-98d0-f70eea33c6a4" containerName="s3-init"
Sep 17 17:51:50.349595 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:51:50.349412    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d2cfbc9-260a-4bb3-98d0-f70eea33c6a4" containerName="s3-init"
Sep 17 17:51:50.349595 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:51:50.349423    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d2cfbc9-260a-4bb3-98d0-f70eea33c6a4" containerName="download-hf-model"
Sep 17 17:51:50.349595 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:51:50.349429    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d2cfbc9-260a-4bb3-98d0-f70eea33c6a4" containerName="download-hf-model"
Sep 17 17:51:50.349595 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:51:50.349472    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d2cfbc9-260a-4bb3-98d0-f70eea33c6a4" containerName="s3-init"
Sep 17 17:51:50.352347 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:51:50.352329    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-zdrc5"
Sep 17 17:51:50.355815 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:51:50.355793    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 17 17:51:50.355962 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:51:50.355865    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Sep 17 17:51:50.356690 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:51:50.356671    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-dcrjv\""
Sep 17 17:51:50.356799 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:51:50.356713    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 17:51:50.357042 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:51:50.357027    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 17:51:50.362258 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:51:50.362234    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-zdrc5"]
Sep 17 17:51:50.413390 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:51:50.413355    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk52l\" (UniqueName: \"kubernetes.io/projected/5ddb7c37-dbbc-49e3-afaf-6dc48ac7b76c-kube-api-access-mk52l\") pod \"s3-tls-init-custom-zdrc5\" (UID: \"5ddb7c37-dbbc-49e3-afaf-6dc48ac7b76c\") " pod="kserve/s3-tls-init-custom-zdrc5"
Sep 17 17:51:50.413581 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:51:50.413462    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ddb7c37-dbbc-49e3-afaf-6dc48ac7b76c-model-cache\") pod \"s3-tls-init-custom-zdrc5\" (UID: \"5ddb7c37-dbbc-49e3-afaf-6dc48ac7b76c\") " pod="kserve/s3-tls-init-custom-zdrc5"
Sep 17 17:51:50.514936 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:51:50.514891    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk52l\" (UniqueName: \"kubernetes.io/projected/5ddb7c37-dbbc-49e3-afaf-6dc48ac7b76c-kube-api-access-mk52l\") pod \"s3-tls-init-custom-zdrc5\" (UID: \"5ddb7c37-dbbc-49e3-afaf-6dc48ac7b76c\") " pod="kserve/s3-tls-init-custom-zdrc5"
Sep 17 17:51:50.515153 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:51:50.514991    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ddb7c37-dbbc-49e3-afaf-6dc48ac7b76c-model-cache\") pod \"s3-tls-init-custom-zdrc5\" (UID: \"5ddb7c37-dbbc-49e3-afaf-6dc48ac7b76c\") " pod="kserve/s3-tls-init-custom-zdrc5"
Sep 17 17:51:50.515366 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:51:50.515348    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ddb7c37-dbbc-49e3-afaf-6dc48ac7b76c-model-cache\") pod \"s3-tls-init-custom-zdrc5\" (UID: \"5ddb7c37-dbbc-49e3-afaf-6dc48ac7b76c\") " pod="kserve/s3-tls-init-custom-zdrc5"
Sep 17 17:51:50.524189 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:51:50.524162    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk52l\" (UniqueName: \"kubernetes.io/projected/5ddb7c37-dbbc-49e3-afaf-6dc48ac7b76c-kube-api-access-mk52l\") pod \"s3-tls-init-custom-zdrc5\" (UID: \"5ddb7c37-dbbc-49e3-afaf-6dc48ac7b76c\") " pod="kserve/s3-tls-init-custom-zdrc5"
Sep 17 17:51:50.680580 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:51:50.680491    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-zdrc5"
Sep 17 17:51:50.952357 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:51:50.952323    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-zdrc5"]
Sep 17 17:51:50.955573 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:51:50.955548    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5ddb7c37_dbbc_49e3_afaf_6dc48ac7b76c.slice/crio-36e83b7f5d2a4d09ba08ac1994bd3d8727523d2849181d5b929c43ac4c922325 WatchSource:0}: Error finding container 36e83b7f5d2a4d09ba08ac1994bd3d8727523d2849181d5b929c43ac4c922325: Status 404 returned error can't find the container with id 36e83b7f5d2a4d09ba08ac1994bd3d8727523d2849181d5b929c43ac4c922325
Sep 17 17:51:51.146544 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:51:51.146505    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-zdrc5" event={"ID":"5ddb7c37-dbbc-49e3-afaf-6dc48ac7b76c","Type":"ContainerStarted","Data":"6764317c5781c30591d405bc3b76ee816b7b60e4853e5e89e6732bd123ed5457"}
Sep 17 17:51:51.146544 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:51:51.146548    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-zdrc5" event={"ID":"5ddb7c37-dbbc-49e3-afaf-6dc48ac7b76c","Type":"ContainerStarted","Data":"36e83b7f5d2a4d09ba08ac1994bd3d8727523d2849181d5b929c43ac4c922325"}
Sep 17 17:51:53.157559 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:51:53.157466    2569 generic.go:356] "Generic (PLEG): container finished" podID="5ddb7c37-dbbc-49e3-afaf-6dc48ac7b76c" containerID="6764317c5781c30591d405bc3b76ee816b7b60e4853e5e89e6732bd123ed5457" exitCode=0
Sep 17 17:51:53.158039 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:51:53.157546    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-zdrc5" event={"ID":"5ddb7c37-dbbc-49e3-afaf-6dc48ac7b76c","Type":"ContainerDied","Data":"6764317c5781c30591d405bc3b76ee816b7b60e4853e5e89e6732bd123ed5457"}
Sep 17 17:51:54.163187 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:51:54.163153    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-zdrc5" event={"ID":"5ddb7c37-dbbc-49e3-afaf-6dc48ac7b76c","Type":"ContainerStarted","Data":"5ed8f0777bbd1bb16db329ba43ee01d10bb6abe7687ff3f35bd1fbb97b802b09"}
Sep 17 17:51:54.183235 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:51:54.183181    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-custom-zdrc5" podStartSLOduration=2.377996865 podStartE2EDuration="4.183162602s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.805165737s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:51:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:51:54.182283075 +0000 UTC m=+666.686285712" watchObservedRunningTime="2026-09-17 17:51:54.183162602 +0000 UTC m=+666.687165226"
Sep 17 17:52:00.187049 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:00.187009    2569 generic.go:356] "Generic (PLEG): container finished" podID="5ddb7c37-dbbc-49e3-afaf-6dc48ac7b76c" containerID="5ed8f0777bbd1bb16db329ba43ee01d10bb6abe7687ff3f35bd1fbb97b802b09" exitCode=0
Sep 17 17:52:00.187495 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:00.187094    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-zdrc5" event={"ID":"5ddb7c37-dbbc-49e3-afaf-6dc48ac7b76c","Type":"ContainerDied","Data":"5ed8f0777bbd1bb16db329ba43ee01d10bb6abe7687ff3f35bd1fbb97b802b09"}
Sep 17 17:52:01.331431 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:01.331405    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-zdrc5"
Sep 17 17:52:01.401268 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:01.401151    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mk52l\" (UniqueName: \"kubernetes.io/projected/5ddb7c37-dbbc-49e3-afaf-6dc48ac7b76c-kube-api-access-mk52l\") pod \"5ddb7c37-dbbc-49e3-afaf-6dc48ac7b76c\" (UID: \"5ddb7c37-dbbc-49e3-afaf-6dc48ac7b76c\") "
Sep 17 17:52:01.401473 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:01.401331    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ddb7c37-dbbc-49e3-afaf-6dc48ac7b76c-model-cache\") pod \"5ddb7c37-dbbc-49e3-afaf-6dc48ac7b76c\" (UID: \"5ddb7c37-dbbc-49e3-afaf-6dc48ac7b76c\") "
Sep 17 17:52:01.403481 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:01.403445    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ddb7c37-dbbc-49e3-afaf-6dc48ac7b76c-kube-api-access-mk52l" (OuterVolumeSpecName: "kube-api-access-mk52l") pod "5ddb7c37-dbbc-49e3-afaf-6dc48ac7b76c" (UID: "5ddb7c37-dbbc-49e3-afaf-6dc48ac7b76c"). InnerVolumeSpecName "kube-api-access-mk52l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:52:01.424663 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:01.424628    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ddb7c37-dbbc-49e3-afaf-6dc48ac7b76c-model-cache" (OuterVolumeSpecName: "model-cache") pod "5ddb7c37-dbbc-49e3-afaf-6dc48ac7b76c" (UID: "5ddb7c37-dbbc-49e3-afaf-6dc48ac7b76c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:52:01.502695 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:01.502655    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mk52l\" (UniqueName: \"kubernetes.io/projected/5ddb7c37-dbbc-49e3-afaf-6dc48ac7b76c-kube-api-access-mk52l\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 17:52:01.502695 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:01.502685    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ddb7c37-dbbc-49e3-afaf-6dc48ac7b76c-model-cache\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 17:52:02.196280 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:02.196244    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-zdrc5" event={"ID":"5ddb7c37-dbbc-49e3-afaf-6dc48ac7b76c","Type":"ContainerDied","Data":"36e83b7f5d2a4d09ba08ac1994bd3d8727523d2849181d5b929c43ac4c922325"}
Sep 17 17:52:02.196280 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:02.196275    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-zdrc5"
Sep 17 17:52:02.196280 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:02.196286    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36e83b7f5d2a4d09ba08ac1994bd3d8727523d2849181d5b929c43ac4c922325"
Sep 17 17:52:05.472095 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:05.472043    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-97l7q"]
Sep 17 17:52:05.472555 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:05.472400    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ddb7c37-dbbc-49e3-afaf-6dc48ac7b76c" containerName="download-hf-model"
Sep 17 17:52:05.472555 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:05.472413    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ddb7c37-dbbc-49e3-afaf-6dc48ac7b76c" containerName="download-hf-model"
Sep 17 17:52:05.472555 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:05.472426    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ddb7c37-dbbc-49e3-afaf-6dc48ac7b76c" containerName="s3-tls-init-custom"
Sep 17 17:52:05.472555 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:05.472431    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ddb7c37-dbbc-49e3-afaf-6dc48ac7b76c" containerName="s3-tls-init-custom"
Sep 17 17:52:05.472555 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:05.472496    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ddb7c37-dbbc-49e3-afaf-6dc48ac7b76c" containerName="s3-tls-init-custom"
Sep 17 17:52:05.475335 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:05.475318    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-97l7q"
Sep 17 17:52:05.478186 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:05.478158    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 17 17:52:05.478338 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:05.478186    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-dcrjv\""
Sep 17 17:52:05.478338 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:05.478209    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 17:52:05.478338 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:05.478168    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Sep 17 17:52:05.478338 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:05.478196    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 17:52:05.483661 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:05.483639    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-97l7q"]
Sep 17 17:52:05.533781 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:05.533722    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24m5m\" (UniqueName: \"kubernetes.io/projected/cac6dd97-231b-4c91-b35e-051390a8bc12-kube-api-access-24m5m\") pod \"s3-tls-init-serving-97l7q\" (UID: \"cac6dd97-231b-4c91-b35e-051390a8bc12\") " pod="kserve/s3-tls-init-serving-97l7q"
Sep 17 17:52:05.533980 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:05.533809    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cac6dd97-231b-4c91-b35e-051390a8bc12-model-cache\") pod \"s3-tls-init-serving-97l7q\" (UID: \"cac6dd97-231b-4c91-b35e-051390a8bc12\") " pod="kserve/s3-tls-init-serving-97l7q"
Sep 17 17:52:05.635033 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:05.634988    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24m5m\" (UniqueName: \"kubernetes.io/projected/cac6dd97-231b-4c91-b35e-051390a8bc12-kube-api-access-24m5m\") pod \"s3-tls-init-serving-97l7q\" (UID: \"cac6dd97-231b-4c91-b35e-051390a8bc12\") " pod="kserve/s3-tls-init-serving-97l7q"
Sep 17 17:52:05.635238 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:05.635060    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cac6dd97-231b-4c91-b35e-051390a8bc12-model-cache\") pod \"s3-tls-init-serving-97l7q\" (UID: \"cac6dd97-231b-4c91-b35e-051390a8bc12\") " pod="kserve/s3-tls-init-serving-97l7q"
Sep 17 17:52:05.635469 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:05.635450    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cac6dd97-231b-4c91-b35e-051390a8bc12-model-cache\") pod \"s3-tls-init-serving-97l7q\" (UID: \"cac6dd97-231b-4c91-b35e-051390a8bc12\") " pod="kserve/s3-tls-init-serving-97l7q"
Sep 17 17:52:05.644180 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:05.644159    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24m5m\" (UniqueName: \"kubernetes.io/projected/cac6dd97-231b-4c91-b35e-051390a8bc12-kube-api-access-24m5m\") pod \"s3-tls-init-serving-97l7q\" (UID: \"cac6dd97-231b-4c91-b35e-051390a8bc12\") " pod="kserve/s3-tls-init-serving-97l7q"
Sep 17 17:52:05.800114 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:05.799986    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-97l7q"
Sep 17 17:52:06.086474 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:06.086385    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-97l7q"]
Sep 17 17:52:06.089583 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:52:06.089552    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcac6dd97_231b_4c91_b35e_051390a8bc12.slice/crio-66f623279caed33981e065c275c58d559be5c37495a4a65efbc842f796f4a9bf WatchSource:0}: Error finding container 66f623279caed33981e065c275c58d559be5c37495a4a65efbc842f796f4a9bf: Status 404 returned error can't find the container with id 66f623279caed33981e065c275c58d559be5c37495a4a65efbc842f796f4a9bf
Sep 17 17:52:06.214704 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:06.214665    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-97l7q" event={"ID":"cac6dd97-231b-4c91-b35e-051390a8bc12","Type":"ContainerStarted","Data":"b92ca8b5bd8da85b00094ce836862e68e33c930ca40800051b0a7dfcb951dab0"}
Sep 17 17:52:06.214704 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:06.214710    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-97l7q" event={"ID":"cac6dd97-231b-4c91-b35e-051390a8bc12","Type":"ContainerStarted","Data":"66f623279caed33981e065c275c58d559be5c37495a4a65efbc842f796f4a9bf"}
Sep 17 17:52:08.225754 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:08.225718    2569 generic.go:356] "Generic (PLEG): container finished" podID="cac6dd97-231b-4c91-b35e-051390a8bc12" containerID="b92ca8b5bd8da85b00094ce836862e68e33c930ca40800051b0a7dfcb951dab0" exitCode=0
Sep 17 17:52:08.226188 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:08.225796    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-97l7q" event={"ID":"cac6dd97-231b-4c91-b35e-051390a8bc12","Type":"ContainerDied","Data":"b92ca8b5bd8da85b00094ce836862e68e33c930ca40800051b0a7dfcb951dab0"}
Sep 17 17:52:09.231655 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:09.231621    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-97l7q" event={"ID":"cac6dd97-231b-4c91-b35e-051390a8bc12","Type":"ContainerStarted","Data":"1acb26338802b68b3e66da353d8f43d353b3cc16517b7b9d0d2b4bd5f96277dc"}
Sep 17 17:52:09.249527 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:09.249471    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-97l7q" podStartSLOduration=2.5255125339999998 podStartE2EDuration="4.249454157s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.723941623s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:52:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:52:09.248634551 +0000 UTC m=+681.752637177" watchObservedRunningTime="2026-09-17 17:52:09.249454157 +0000 UTC m=+681.753456781"
Sep 17 17:52:16.261030 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:16.260991    2569 generic.go:356] "Generic (PLEG): container finished" podID="cac6dd97-231b-4c91-b35e-051390a8bc12" containerID="1acb26338802b68b3e66da353d8f43d353b3cc16517b7b9d0d2b4bd5f96277dc" exitCode=0
Sep 17 17:52:16.261520 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:16.261062    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-97l7q" event={"ID":"cac6dd97-231b-4c91-b35e-051390a8bc12","Type":"ContainerDied","Data":"1acb26338802b68b3e66da353d8f43d353b3cc16517b7b9d0d2b4bd5f96277dc"}
Sep 17 17:52:17.400362 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:17.400330    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-97l7q"
Sep 17 17:52:17.538914 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:17.538823    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cac6dd97-231b-4c91-b35e-051390a8bc12-model-cache\") pod \"cac6dd97-231b-4c91-b35e-051390a8bc12\" (UID: \"cac6dd97-231b-4c91-b35e-051390a8bc12\") "
Sep 17 17:52:17.538914 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:17.538884    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-24m5m\" (UniqueName: \"kubernetes.io/projected/cac6dd97-231b-4c91-b35e-051390a8bc12-kube-api-access-24m5m\") pod \"cac6dd97-231b-4c91-b35e-051390a8bc12\" (UID: \"cac6dd97-231b-4c91-b35e-051390a8bc12\") "
Sep 17 17:52:17.541029 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:17.540996    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cac6dd97-231b-4c91-b35e-051390a8bc12-kube-api-access-24m5m" (OuterVolumeSpecName: "kube-api-access-24m5m") pod "cac6dd97-231b-4c91-b35e-051390a8bc12" (UID: "cac6dd97-231b-4c91-b35e-051390a8bc12"). InnerVolumeSpecName "kube-api-access-24m5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:52:17.562134 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:17.562095    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cac6dd97-231b-4c91-b35e-051390a8bc12-model-cache" (OuterVolumeSpecName: "model-cache") pod "cac6dd97-231b-4c91-b35e-051390a8bc12" (UID: "cac6dd97-231b-4c91-b35e-051390a8bc12"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:52:17.639599 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:17.639561    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cac6dd97-231b-4c91-b35e-051390a8bc12-model-cache\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 17:52:17.639599 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:17.639591    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24m5m\" (UniqueName: \"kubernetes.io/projected/cac6dd97-231b-4c91-b35e-051390a8bc12-kube-api-access-24m5m\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 17:52:18.271784 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:18.271748    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-97l7q" event={"ID":"cac6dd97-231b-4c91-b35e-051390a8bc12","Type":"ContainerDied","Data":"66f623279caed33981e065c275c58d559be5c37495a4a65efbc842f796f4a9bf"}
Sep 17 17:52:18.271784 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:18.271788    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66f623279caed33981e065c275c58d559be5c37495a4a65efbc842f796f4a9bf"
Sep 17 17:52:18.271784 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:18.271789    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-97l7q"
Sep 17 17:52:30.050178 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:30.050139    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-3e69ce-predictor-58cdd6dc8c-fmq2d"]
Sep 17 17:52:30.050791 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:30.050545    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cac6dd97-231b-4c91-b35e-051390a8bc12" containerName="download-hf-model"
Sep 17 17:52:30.050791 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:30.050562    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="cac6dd97-231b-4c91-b35e-051390a8bc12" containerName="download-hf-model"
Sep 17 17:52:30.050791 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:30.050592    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cac6dd97-231b-4c91-b35e-051390a8bc12" containerName="s3-tls-init-serving"
Sep 17 17:52:30.050791 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:30.050600    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="cac6dd97-231b-4c91-b35e-051390a8bc12" containerName="s3-tls-init-serving"
Sep 17 17:52:30.050791 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:30.050686    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="cac6dd97-231b-4c91-b35e-051390a8bc12" containerName="s3-tls-init-serving"
Sep 17 17:52:30.055095 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:30.055052    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-3e69ce-predictor-58cdd6dc8c-fmq2d"
Sep 17 17:52:30.058068 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:30.058003    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-3e69ce-predictor-serving-cert\""
Sep 17 17:52:30.058068 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:30.058059    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:52:30.058256 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:30.058093    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:52:30.058256 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:30.058103    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-3e69ce\""
Sep 17 17:52:30.059450 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:30.059430    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-3e69ce-kube-rbac-proxy-sar-config\""
Sep 17 17:52:30.059530 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:30.059461    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:52:30.059530 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:30.059523    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-3e69ce-dockercfg-z4qfx\""
Sep 17 17:52:30.065121 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:30.065098    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-3e69ce-predictor-58cdd6dc8c-fmq2d"]
Sep 17 17:52:30.143817 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:30.143786    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fef8b62e-a60a-4058-b8cf-d2475c2484ff-cabundle-cert\") pod \"isvc-init-fail-3e69ce-predictor-58cdd6dc8c-fmq2d\" (UID: \"fef8b62e-a60a-4058-b8cf-d2475c2484ff\") " pod="kserve-ci-e2e-test/isvc-init-fail-3e69ce-predictor-58cdd6dc8c-fmq2d"
Sep 17 17:52:30.144009 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:30.143836    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fef8b62e-a60a-4058-b8cf-d2475c2484ff-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-3e69ce-predictor-58cdd6dc8c-fmq2d\" (UID: \"fef8b62e-a60a-4058-b8cf-d2475c2484ff\") " pod="kserve-ci-e2e-test/isvc-init-fail-3e69ce-predictor-58cdd6dc8c-fmq2d"
Sep 17 17:52:30.144009 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:30.143876    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fef8b62e-a60a-4058-b8cf-d2475c2484ff-proxy-tls\") pod \"isvc-init-fail-3e69ce-predictor-58cdd6dc8c-fmq2d\" (UID: \"fef8b62e-a60a-4058-b8cf-d2475c2484ff\") " pod="kserve-ci-e2e-test/isvc-init-fail-3e69ce-predictor-58cdd6dc8c-fmq2d"
Sep 17 17:52:30.144009 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:30.143900    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fef8b62e-a60a-4058-b8cf-d2475c2484ff-kserve-provision-location\") pod \"isvc-init-fail-3e69ce-predictor-58cdd6dc8c-fmq2d\" (UID: \"fef8b62e-a60a-4058-b8cf-d2475c2484ff\") " pod="kserve-ci-e2e-test/isvc-init-fail-3e69ce-predictor-58cdd6dc8c-fmq2d"
Sep 17 17:52:30.144009 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:30.143968    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttdkt\" (UniqueName: \"kubernetes.io/projected/fef8b62e-a60a-4058-b8cf-d2475c2484ff-kube-api-access-ttdkt\") pod \"isvc-init-fail-3e69ce-predictor-58cdd6dc8c-fmq2d\" (UID: \"fef8b62e-a60a-4058-b8cf-d2475c2484ff\") " pod="kserve-ci-e2e-test/isvc-init-fail-3e69ce-predictor-58cdd6dc8c-fmq2d"
Sep 17 17:52:30.245043 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:30.245005    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fef8b62e-a60a-4058-b8cf-d2475c2484ff-cabundle-cert\") pod \"isvc-init-fail-3e69ce-predictor-58cdd6dc8c-fmq2d\" (UID: \"fef8b62e-a60a-4058-b8cf-d2475c2484ff\") " pod="kserve-ci-e2e-test/isvc-init-fail-3e69ce-predictor-58cdd6dc8c-fmq2d"
Sep 17 17:52:30.245043 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:30.245048    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fef8b62e-a60a-4058-b8cf-d2475c2484ff-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-3e69ce-predictor-58cdd6dc8c-fmq2d\" (UID: \"fef8b62e-a60a-4058-b8cf-d2475c2484ff\") " pod="kserve-ci-e2e-test/isvc-init-fail-3e69ce-predictor-58cdd6dc8c-fmq2d"
Sep 17 17:52:30.245325 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:30.245088    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fef8b62e-a60a-4058-b8cf-d2475c2484ff-proxy-tls\") pod \"isvc-init-fail-3e69ce-predictor-58cdd6dc8c-fmq2d\" (UID: \"fef8b62e-a60a-4058-b8cf-d2475c2484ff\") " pod="kserve-ci-e2e-test/isvc-init-fail-3e69ce-predictor-58cdd6dc8c-fmq2d"
Sep 17 17:52:30.245325 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:30.245105    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fef8b62e-a60a-4058-b8cf-d2475c2484ff-kserve-provision-location\") pod \"isvc-init-fail-3e69ce-predictor-58cdd6dc8c-fmq2d\" (UID: \"fef8b62e-a60a-4058-b8cf-d2475c2484ff\") " pod="kserve-ci-e2e-test/isvc-init-fail-3e69ce-predictor-58cdd6dc8c-fmq2d"
Sep 17 17:52:30.245325 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:30.245135    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttdkt\" (UniqueName: \"kubernetes.io/projected/fef8b62e-a60a-4058-b8cf-d2475c2484ff-kube-api-access-ttdkt\") pod \"isvc-init-fail-3e69ce-predictor-58cdd6dc8c-fmq2d\" (UID: \"fef8b62e-a60a-4058-b8cf-d2475c2484ff\") " pod="kserve-ci-e2e-test/isvc-init-fail-3e69ce-predictor-58cdd6dc8c-fmq2d"
Sep 17 17:52:30.245580 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:30.245559    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fef8b62e-a60a-4058-b8cf-d2475c2484ff-kserve-provision-location\") pod \"isvc-init-fail-3e69ce-predictor-58cdd6dc8c-fmq2d\" (UID: \"fef8b62e-a60a-4058-b8cf-d2475c2484ff\") " pod="kserve-ci-e2e-test/isvc-init-fail-3e69ce-predictor-58cdd6dc8c-fmq2d"
Sep 17 17:52:30.245818 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:30.245795    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fef8b62e-a60a-4058-b8cf-d2475c2484ff-cabundle-cert\") pod \"isvc-init-fail-3e69ce-predictor-58cdd6dc8c-fmq2d\" (UID: \"fef8b62e-a60a-4058-b8cf-d2475c2484ff\") " pod="kserve-ci-e2e-test/isvc-init-fail-3e69ce-predictor-58cdd6dc8c-fmq2d"
Sep 17 17:52:30.245894 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:30.245819    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fef8b62e-a60a-4058-b8cf-d2475c2484ff-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-3e69ce-predictor-58cdd6dc8c-fmq2d\" (UID: \"fef8b62e-a60a-4058-b8cf-d2475c2484ff\") " pod="kserve-ci-e2e-test/isvc-init-fail-3e69ce-predictor-58cdd6dc8c-fmq2d"
Sep 17 17:52:30.247696 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:30.247675    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fef8b62e-a60a-4058-b8cf-d2475c2484ff-proxy-tls\") pod \"isvc-init-fail-3e69ce-predictor-58cdd6dc8c-fmq2d\" (UID: \"fef8b62e-a60a-4058-b8cf-d2475c2484ff\") " pod="kserve-ci-e2e-test/isvc-init-fail-3e69ce-predictor-58cdd6dc8c-fmq2d"
Sep 17 17:52:30.254222 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:30.254195    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttdkt\" (UniqueName: \"kubernetes.io/projected/fef8b62e-a60a-4058-b8cf-d2475c2484ff-kube-api-access-ttdkt\") pod \"isvc-init-fail-3e69ce-predictor-58cdd6dc8c-fmq2d\" (UID: \"fef8b62e-a60a-4058-b8cf-d2475c2484ff\") " pod="kserve-ci-e2e-test/isvc-init-fail-3e69ce-predictor-58cdd6dc8c-fmq2d"
Sep 17 17:52:30.366908 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:30.366814    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-3e69ce-predictor-58cdd6dc8c-fmq2d"
Sep 17 17:52:30.640504 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:30.640482    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-3e69ce-predictor-58cdd6dc8c-fmq2d"]
Sep 17 17:52:30.642527 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:52:30.642498    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfef8b62e_a60a_4058_b8cf_d2475c2484ff.slice/crio-dc8bd4d4f2fa90f47e9c42417439f4736d2b008a226ba2ad032cbbced6fa08f3 WatchSource:0}: Error finding container dc8bd4d4f2fa90f47e9c42417439f4736d2b008a226ba2ad032cbbced6fa08f3: Status 404 returned error can't find the container with id dc8bd4d4f2fa90f47e9c42417439f4736d2b008a226ba2ad032cbbced6fa08f3
Sep 17 17:52:31.324180 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:31.324137    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-3e69ce-predictor-58cdd6dc8c-fmq2d" event={"ID":"fef8b62e-a60a-4058-b8cf-d2475c2484ff","Type":"ContainerStarted","Data":"533adffb3b825280a3ff3ea4b703d3031dbb7ea6d6abc9dfdc99945eeb2b4b9a"}
Sep 17 17:52:31.324581 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:31.324206    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-3e69ce-predictor-58cdd6dc8c-fmq2d" event={"ID":"fef8b62e-a60a-4058-b8cf-d2475c2484ff","Type":"ContainerStarted","Data":"dc8bd4d4f2fa90f47e9c42417439f4736d2b008a226ba2ad032cbbced6fa08f3"}
Sep 17 17:52:36.344775 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:36.344744    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-3e69ce-predictor-58cdd6dc8c-fmq2d_fef8b62e-a60a-4058-b8cf-d2475c2484ff/storage-initializer/0.log"
Sep 17 17:52:36.345216 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:36.344781    2569 generic.go:356] "Generic (PLEG): container finished" podID="fef8b62e-a60a-4058-b8cf-d2475c2484ff" containerID="533adffb3b825280a3ff3ea4b703d3031dbb7ea6d6abc9dfdc99945eeb2b4b9a" exitCode=1
Sep 17 17:52:36.345216 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:36.344859    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-3e69ce-predictor-58cdd6dc8c-fmq2d" event={"ID":"fef8b62e-a60a-4058-b8cf-d2475c2484ff","Type":"ContainerDied","Data":"533adffb3b825280a3ff3ea4b703d3031dbb7ea6d6abc9dfdc99945eeb2b4b9a"}
Sep 17 17:52:37.350564 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:37.350538    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-3e69ce-predictor-58cdd6dc8c-fmq2d_fef8b62e-a60a-4058-b8cf-d2475c2484ff/storage-initializer/0.log"
Sep 17 17:52:37.350945 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:37.350645    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-3e69ce-predictor-58cdd6dc8c-fmq2d" event={"ID":"fef8b62e-a60a-4058-b8cf-d2475c2484ff","Type":"ContainerStarted","Data":"6021b9d700952ffb14b0f8c0c8648690f792acb63fd8198a18eade083fc3390a"}
Sep 17 17:52:39.972311 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:39.972274    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-3e69ce-predictor-58cdd6dc8c-fmq2d"]
Sep 17 17:52:39.972776 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:39.972546    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-3e69ce-predictor-58cdd6dc8c-fmq2d" podUID="fef8b62e-a60a-4058-b8cf-d2475c2484ff" containerName="storage-initializer" containerID="cri-o://6021b9d700952ffb14b0f8c0c8648690f792acb63fd8198a18eade083fc3390a" gracePeriod=30
Sep 17 17:52:41.516057 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:41.516033    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-3e69ce-predictor-58cdd6dc8c-fmq2d_fef8b62e-a60a-4058-b8cf-d2475c2484ff/storage-initializer/1.log"
Sep 17 17:52:41.516497 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:41.516482    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-3e69ce-predictor-58cdd6dc8c-fmq2d_fef8b62e-a60a-4058-b8cf-d2475c2484ff/storage-initializer/0.log"
Sep 17 17:52:41.518860 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:41.518844    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-3e69ce-predictor-58cdd6dc8c-fmq2d"
Sep 17 17:52:41.644969 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:41.644881    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fef8b62e-a60a-4058-b8cf-d2475c2484ff-kserve-provision-location\") pod \"fef8b62e-a60a-4058-b8cf-d2475c2484ff\" (UID: \"fef8b62e-a60a-4058-b8cf-d2475c2484ff\") "
Sep 17 17:52:41.644969 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:41.644959    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fef8b62e-a60a-4058-b8cf-d2475c2484ff-cabundle-cert\") pod \"fef8b62e-a60a-4058-b8cf-d2475c2484ff\" (UID: \"fef8b62e-a60a-4058-b8cf-d2475c2484ff\") "
Sep 17 17:52:41.645215 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:41.644983    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ttdkt\" (UniqueName: \"kubernetes.io/projected/fef8b62e-a60a-4058-b8cf-d2475c2484ff-kube-api-access-ttdkt\") pod \"fef8b62e-a60a-4058-b8cf-d2475c2484ff\" (UID: \"fef8b62e-a60a-4058-b8cf-d2475c2484ff\") "
Sep 17 17:52:41.645215 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:41.645032    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fef8b62e-a60a-4058-b8cf-d2475c2484ff-proxy-tls\") pod \"fef8b62e-a60a-4058-b8cf-d2475c2484ff\" (UID: \"fef8b62e-a60a-4058-b8cf-d2475c2484ff\") "
Sep 17 17:52:41.645215 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:41.645107    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fef8b62e-a60a-4058-b8cf-d2475c2484ff-kube-rbac-proxy-sar-config\") pod \"fef8b62e-a60a-4058-b8cf-d2475c2484ff\" (UID: \"fef8b62e-a60a-4058-b8cf-d2475c2484ff\") "
Sep 17 17:52:41.645376 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:41.645234    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fef8b62e-a60a-4058-b8cf-d2475c2484ff-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fef8b62e-a60a-4058-b8cf-d2475c2484ff" (UID: "fef8b62e-a60a-4058-b8cf-d2475c2484ff"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:52:41.645435 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:41.645375    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fef8b62e-a60a-4058-b8cf-d2475c2484ff-kserve-provision-location\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 17:52:41.645488 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:41.645427    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fef8b62e-a60a-4058-b8cf-d2475c2484ff-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fef8b62e-a60a-4058-b8cf-d2475c2484ff" (UID: "fef8b62e-a60a-4058-b8cf-d2475c2484ff"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:52:41.645535 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:41.645504    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fef8b62e-a60a-4058-b8cf-d2475c2484ff-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fef8b62e-a60a-4058-b8cf-d2475c2484ff" (UID: "fef8b62e-a60a-4058-b8cf-d2475c2484ff"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:52:41.647266 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:41.647230    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fef8b62e-a60a-4058-b8cf-d2475c2484ff-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fef8b62e-a60a-4058-b8cf-d2475c2484ff" (UID: "fef8b62e-a60a-4058-b8cf-d2475c2484ff"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:52:41.647374 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:41.647350    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fef8b62e-a60a-4058-b8cf-d2475c2484ff-kube-api-access-ttdkt" (OuterVolumeSpecName: "kube-api-access-ttdkt") pod "fef8b62e-a60a-4058-b8cf-d2475c2484ff" (UID: "fef8b62e-a60a-4058-b8cf-d2475c2484ff"). InnerVolumeSpecName "kube-api-access-ttdkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:52:41.745787 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:41.745744    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fef8b62e-a60a-4058-b8cf-d2475c2484ff-cabundle-cert\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 17:52:41.745787 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:41.745777    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttdkt\" (UniqueName: \"kubernetes.io/projected/fef8b62e-a60a-4058-b8cf-d2475c2484ff-kube-api-access-ttdkt\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 17:52:41.745787 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:41.745787    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fef8b62e-a60a-4058-b8cf-d2475c2484ff-proxy-tls\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 17:52:41.745787 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:41.745798    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fef8b62e-a60a-4058-b8cf-d2475c2484ff-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 17:52:42.370300 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:42.370271    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-3e69ce-predictor-58cdd6dc8c-fmq2d_fef8b62e-a60a-4058-b8cf-d2475c2484ff/storage-initializer/1.log"
Sep 17 17:52:42.370720 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:42.370703    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-3e69ce-predictor-58cdd6dc8c-fmq2d_fef8b62e-a60a-4058-b8cf-d2475c2484ff/storage-initializer/0.log"
Sep 17 17:52:42.370770 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:42.370742    2569 generic.go:356] "Generic (PLEG): container finished" podID="fef8b62e-a60a-4058-b8cf-d2475c2484ff" containerID="6021b9d700952ffb14b0f8c0c8648690f792acb63fd8198a18eade083fc3390a" exitCode=1
Sep 17 17:52:42.370812 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:42.370797    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-3e69ce-predictor-58cdd6dc8c-fmq2d"
Sep 17 17:52:42.370890 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:42.370803    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-3e69ce-predictor-58cdd6dc8c-fmq2d" event={"ID":"fef8b62e-a60a-4058-b8cf-d2475c2484ff","Type":"ContainerDied","Data":"6021b9d700952ffb14b0f8c0c8648690f792acb63fd8198a18eade083fc3390a"}
Sep 17 17:52:42.370960 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:42.370940    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-3e69ce-predictor-58cdd6dc8c-fmq2d" event={"ID":"fef8b62e-a60a-4058-b8cf-d2475c2484ff","Type":"ContainerDied","Data":"dc8bd4d4f2fa90f47e9c42417439f4736d2b008a226ba2ad032cbbced6fa08f3"}
Sep 17 17:52:42.371000 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:42.370987    2569 scope.go:117] "RemoveContainer" containerID="6021b9d700952ffb14b0f8c0c8648690f792acb63fd8198a18eade083fc3390a"
Sep 17 17:52:42.380986 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:42.380968    2569 scope.go:117] "RemoveContainer" containerID="533adffb3b825280a3ff3ea4b703d3031dbb7ea6d6abc9dfdc99945eeb2b4b9a"
Sep 17 17:52:42.388465 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:42.388446    2569 scope.go:117] "RemoveContainer" containerID="6021b9d700952ffb14b0f8c0c8648690f792acb63fd8198a18eade083fc3390a"
Sep 17 17:52:42.388731 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:52:42.388712    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6021b9d700952ffb14b0f8c0c8648690f792acb63fd8198a18eade083fc3390a\": container with ID starting with 6021b9d700952ffb14b0f8c0c8648690f792acb63fd8198a18eade083fc3390a not found: ID does not exist" containerID="6021b9d700952ffb14b0f8c0c8648690f792acb63fd8198a18eade083fc3390a"
Sep 17 17:52:42.388778 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:42.388740    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6021b9d700952ffb14b0f8c0c8648690f792acb63fd8198a18eade083fc3390a"} err="failed to get container status \"6021b9d700952ffb14b0f8c0c8648690f792acb63fd8198a18eade083fc3390a\": rpc error: code = NotFound desc = could not find container \"6021b9d700952ffb14b0f8c0c8648690f792acb63fd8198a18eade083fc3390a\": container with ID starting with 6021b9d700952ffb14b0f8c0c8648690f792acb63fd8198a18eade083fc3390a not found: ID does not exist"
Sep 17 17:52:42.388778 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:42.388757    2569 scope.go:117] "RemoveContainer" containerID="533adffb3b825280a3ff3ea4b703d3031dbb7ea6d6abc9dfdc99945eeb2b4b9a"
Sep 17 17:52:42.388990 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:52:42.388972    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"533adffb3b825280a3ff3ea4b703d3031dbb7ea6d6abc9dfdc99945eeb2b4b9a\": container with ID starting with 533adffb3b825280a3ff3ea4b703d3031dbb7ea6d6abc9dfdc99945eeb2b4b9a not found: ID does not exist" containerID="533adffb3b825280a3ff3ea4b703d3031dbb7ea6d6abc9dfdc99945eeb2b4b9a"
Sep 17 17:52:42.389033 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:42.388997    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"533adffb3b825280a3ff3ea4b703d3031dbb7ea6d6abc9dfdc99945eeb2b4b9a"} err="failed to get container status \"533adffb3b825280a3ff3ea4b703d3031dbb7ea6d6abc9dfdc99945eeb2b4b9a\": rpc error: code = NotFound desc = could not find container \"533adffb3b825280a3ff3ea4b703d3031dbb7ea6d6abc9dfdc99945eeb2b4b9a\": container with ID starting with 533adffb3b825280a3ff3ea4b703d3031dbb7ea6d6abc9dfdc99945eeb2b4b9a not found: ID does not exist"
Sep 17 17:52:42.419978 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:42.419945    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-3e69ce-predictor-58cdd6dc8c-fmq2d"]
Sep 17 17:52:42.425593 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:42.425562    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-3e69ce-predictor-58cdd6dc8c-fmq2d"]
Sep 17 17:52:43.507372 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:43.507338    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fef8b62e-a60a-4058-b8cf-d2475c2484ff" path="/var/lib/kubelet/pods/fef8b62e-a60a-4058-b8cf-d2475c2484ff/volumes"
Sep 17 17:52:45.178181 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:45.178142    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lkc58"]
Sep 17 17:52:45.178624 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:45.178488    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fef8b62e-a60a-4058-b8cf-d2475c2484ff" containerName="storage-initializer"
Sep 17 17:52:45.178624 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:45.178501    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="fef8b62e-a60a-4058-b8cf-d2475c2484ff" containerName="storage-initializer"
Sep 17 17:52:45.178624 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:45.178598    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="fef8b62e-a60a-4058-b8cf-d2475c2484ff" containerName="storage-initializer"
Sep 17 17:52:45.178624 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:45.178612    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="fef8b62e-a60a-4058-b8cf-d2475c2484ff" containerName="storage-initializer"
Sep 17 17:52:45.178825 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:45.178687    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fef8b62e-a60a-4058-b8cf-d2475c2484ff" containerName="storage-initializer"
Sep 17 17:52:45.178825 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:45.178699    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="fef8b62e-a60a-4058-b8cf-d2475c2484ff" containerName="storage-initializer"
Sep 17 17:52:45.185168 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:45.185136    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lkc58"
Sep 17 17:52:45.188422 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:45.188396    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-kkx8f\""
Sep 17 17:52:45.188564 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:45.188396    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 17:52:45.188564 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:45.188507    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-predictor-serving-cert\""
Sep 17 17:52:45.188680 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:45.188594    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:52:45.188830 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:45.188804    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:52:45.188938 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:45.188851    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\""
Sep 17 17:52:45.189427 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:45.189408    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:52:45.193139 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:45.193119    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lkc58"]
Sep 17 17:52:45.378478 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:45.378413    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23095e57-36d9-44b4-926d-fc7de2158696-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-lkc58\" (UID: \"23095e57-36d9-44b4-926d-fc7de2158696\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lkc58"
Sep 17 17:52:45.378672 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:45.378498    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/23095e57-36d9-44b4-926d-fc7de2158696-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-lkc58\" (UID: \"23095e57-36d9-44b4-926d-fc7de2158696\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lkc58"
Sep 17 17:52:45.378672 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:45.378569    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/23095e57-36d9-44b4-926d-fc7de2158696-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-lkc58\" (UID: \"23095e57-36d9-44b4-926d-fc7de2158696\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lkc58"
Sep 17 17:52:45.378672 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:45.378611    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkdjd\" (UniqueName: \"kubernetes.io/projected/23095e57-36d9-44b4-926d-fc7de2158696-kube-api-access-gkdjd\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-lkc58\" (UID: \"23095e57-36d9-44b4-926d-fc7de2158696\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lkc58"
Sep 17 17:52:45.378672 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:45.378640    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23095e57-36d9-44b4-926d-fc7de2158696-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-lkc58\" (UID: \"23095e57-36d9-44b4-926d-fc7de2158696\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lkc58"
Sep 17 17:52:45.479348 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:45.479305    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23095e57-36d9-44b4-926d-fc7de2158696-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-lkc58\" (UID: \"23095e57-36d9-44b4-926d-fc7de2158696\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lkc58"
Sep 17 17:52:45.479570 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:45.479365    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23095e57-36d9-44b4-926d-fc7de2158696-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-lkc58\" (UID: \"23095e57-36d9-44b4-926d-fc7de2158696\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lkc58"
Sep 17 17:52:45.479570 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:45.479395    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/23095e57-36d9-44b4-926d-fc7de2158696-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-lkc58\" (UID: \"23095e57-36d9-44b4-926d-fc7de2158696\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lkc58"
Sep 17 17:52:45.479570 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:45.479442    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/23095e57-36d9-44b4-926d-fc7de2158696-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-lkc58\" (UID: \"23095e57-36d9-44b4-926d-fc7de2158696\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lkc58"
Sep 17 17:52:45.479570 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:45.479472    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkdjd\" (UniqueName: \"kubernetes.io/projected/23095e57-36d9-44b4-926d-fc7de2158696-kube-api-access-gkdjd\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-lkc58\" (UID: \"23095e57-36d9-44b4-926d-fc7de2158696\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lkc58"
Sep 17 17:52:45.479906 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:45.479876    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23095e57-36d9-44b4-926d-fc7de2158696-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-lkc58\" (UID: \"23095e57-36d9-44b4-926d-fc7de2158696\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lkc58"
Sep 17 17:52:45.480216 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:45.480199    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/23095e57-36d9-44b4-926d-fc7de2158696-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-lkc58\" (UID: \"23095e57-36d9-44b4-926d-fc7de2158696\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lkc58"
Sep 17 17:52:45.480273 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:45.480242    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/23095e57-36d9-44b4-926d-fc7de2158696-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-lkc58\" (UID: \"23095e57-36d9-44b4-926d-fc7de2158696\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lkc58"
Sep 17 17:52:45.481789 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:45.481772    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23095e57-36d9-44b4-926d-fc7de2158696-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-lkc58\" (UID: \"23095e57-36d9-44b4-926d-fc7de2158696\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lkc58"
Sep 17 17:52:45.488224 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:45.488202    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkdjd\" (UniqueName: \"kubernetes.io/projected/23095e57-36d9-44b4-926d-fc7de2158696-kube-api-access-gkdjd\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-lkc58\" (UID: \"23095e57-36d9-44b4-926d-fc7de2158696\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lkc58"
Sep 17 17:52:45.496993 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:45.496969    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lkc58"
Sep 17 17:52:45.773184 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:45.773157    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lkc58"]
Sep 17 17:52:45.775123 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:52:45.775091    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23095e57_36d9_44b4_926d_fc7de2158696.slice/crio-637cce4009d5c130821d815f0c6463b28a01c634cc02855b8de53568d3e438df WatchSource:0}: Error finding container 637cce4009d5c130821d815f0c6463b28a01c634cc02855b8de53568d3e438df: Status 404 returned error can't find the container with id 637cce4009d5c130821d815f0c6463b28a01c634cc02855b8de53568d3e438df
Sep 17 17:52:46.388509 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:46.388470    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lkc58" event={"ID":"23095e57-36d9-44b4-926d-fc7de2158696","Type":"ContainerStarted","Data":"6eace2fb7e054e727ad9736972d669a83911b49c04efce5be7e85b24318efacc"}
Sep 17 17:52:46.388975 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:46.388541    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lkc58" event={"ID":"23095e57-36d9-44b4-926d-fc7de2158696","Type":"ContainerStarted","Data":"637cce4009d5c130821d815f0c6463b28a01c634cc02855b8de53568d3e438df"}
Sep 17 17:52:50.405604 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:50.405568    2569 generic.go:356] "Generic (PLEG): container finished" podID="23095e57-36d9-44b4-926d-fc7de2158696" containerID="6eace2fb7e054e727ad9736972d669a83911b49c04efce5be7e85b24318efacc" exitCode=0
Sep 17 17:52:50.406010 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:52:50.405639    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lkc58" event={"ID":"23095e57-36d9-44b4-926d-fc7de2158696","Type":"ContainerDied","Data":"6eace2fb7e054e727ad9736972d669a83911b49c04efce5be7e85b24318efacc"}
Sep 17 17:53:13.524644 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:53:13.524604    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lkc58" event={"ID":"23095e57-36d9-44b4-926d-fc7de2158696","Type":"ContainerStarted","Data":"97d04c1c5d13268063492cbd1e03f3c7a69b9974cfaa8846b40d239be05dbe28"}
Sep 17 17:53:15.536033 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:53:15.536002    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lkc58" event={"ID":"23095e57-36d9-44b4-926d-fc7de2158696","Type":"ContainerStarted","Data":"3e429673497752530aa7151e1c2288da117ce6df193f0ff8554c02641068c854"}
Sep 17 17:53:15.536454 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:53:15.536158    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lkc58"
Sep 17 17:53:15.557296 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:53:15.557243    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lkc58" podStartSLOduration=2.487878053 podStartE2EDuration="30.557226215s" totalImagesPullingTime="24.400115593s" totalInitContainerRuntime="3.669232569s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:52:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:53:15.556398146 +0000 UTC m=+748.060400771" watchObservedRunningTime="2026-09-17 17:53:15.557226215 +0000 UTC m=+748.061228840"
Sep 17 17:53:16.540295 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:53:16.540260    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lkc58"
Sep 17 17:53:16.541684 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:53:16.541646    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lkc58" podUID="23095e57-36d9-44b4-926d-fc7de2158696" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 17:53:17.544192 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:53:17.544148    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lkc58" podUID="23095e57-36d9-44b4-926d-fc7de2158696" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 17:53:22.549956 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:53:22.549925    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lkc58"
Sep 17 17:53:22.550557 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:53:22.550527    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lkc58" podUID="23095e57-36d9-44b4-926d-fc7de2158696" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 17:53:32.551174 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:53:32.551126    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lkc58" podUID="23095e57-36d9-44b4-926d-fc7de2158696" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 17:53:42.550616 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:53:42.550571    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lkc58" podUID="23095e57-36d9-44b4-926d-fc7de2158696" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 17:53:52.550814 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:53:52.550767    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lkc58" podUID="23095e57-36d9-44b4-926d-fc7de2158696" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 17:54:02.551127 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:02.551018    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lkc58" podUID="23095e57-36d9-44b4-926d-fc7de2158696" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 17:54:12.551232 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:12.551189    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lkc58" podUID="23095e57-36d9-44b4-926d-fc7de2158696" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 17:54:22.551224 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:22.551180    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lkc58" podUID="23095e57-36d9-44b4-926d-fc7de2158696" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 17:54:30.503193 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:30.503149    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lkc58" podUID="23095e57-36d9-44b4-926d-fc7de2158696" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 17:54:40.503750 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:40.503714    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lkc58"
Sep 17 17:54:45.768613 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:45.768578    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lkc58"]
Sep 17 17:54:45.769062 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:45.768833    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lkc58" podUID="23095e57-36d9-44b4-926d-fc7de2158696" containerName="kserve-container" containerID="cri-o://97d04c1c5d13268063492cbd1e03f3c7a69b9974cfaa8846b40d239be05dbe28" gracePeriod=30
Sep 17 17:54:45.769062 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:45.768879    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lkc58" podUID="23095e57-36d9-44b4-926d-fc7de2158696" containerName="kube-rbac-proxy" containerID="cri-o://3e429673497752530aa7151e1c2288da117ce6df193f0ff8554c02641068c854" gracePeriod=30
Sep 17 17:54:45.896840 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:45.896809    2569 generic.go:356] "Generic (PLEG): container finished" podID="23095e57-36d9-44b4-926d-fc7de2158696" containerID="3e429673497752530aa7151e1c2288da117ce6df193f0ff8554c02641068c854" exitCode=2
Sep 17 17:54:45.896981 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:45.896885    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lkc58" event={"ID":"23095e57-36d9-44b4-926d-fc7de2158696","Type":"ContainerDied","Data":"3e429673497752530aa7151e1c2288da117ce6df193f0ff8554c02641068c854"}
Sep 17 17:54:47.544879 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:47.544827    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lkc58" podUID="23095e57-36d9-44b4-926d-fc7de2158696" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.24:8643/healthz\": dial tcp 10.133.0.24:8643: connect: connection refused"
Sep 17 17:54:50.504134 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:50.504058    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lkc58" podUID="23095e57-36d9-44b4-926d-fc7de2158696" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 17:54:50.980419 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:50.980384    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-x8g2s"]
Sep 17 17:54:50.983948 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:50.983931    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-x8g2s"
Sep 17 17:54:50.986751 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:50.986733    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\""
Sep 17 17:54:50.986938 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:50.986926    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-predictor-serving-cert\""
Sep 17 17:54:50.994480 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:50.994457    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-x8g2s"]
Sep 17 17:54:51.024546 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:51.024515    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e16f3122-b1c7-4722-9b7e-63f516e083ea-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-x8g2s\" (UID: \"e16f3122-b1c7-4722-9b7e-63f516e083ea\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-x8g2s"
Sep 17 17:54:51.024715 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:51.024559    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e16f3122-b1c7-4722-9b7e-63f516e083ea-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-x8g2s\" (UID: \"e16f3122-b1c7-4722-9b7e-63f516e083ea\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-x8g2s"
Sep 17 17:54:51.024715 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:51.024581    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvdvl\" (UniqueName: \"kubernetes.io/projected/e16f3122-b1c7-4722-9b7e-63f516e083ea-kube-api-access-fvdvl\") pod \"isvc-predictive-xgboost-predictor-695f978978-x8g2s\" (UID: \"e16f3122-b1c7-4722-9b7e-63f516e083ea\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-x8g2s"
Sep 17 17:54:51.024715 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:51.024620    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e16f3122-b1c7-4722-9b7e-63f516e083ea-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-x8g2s\" (UID: \"e16f3122-b1c7-4722-9b7e-63f516e083ea\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-x8g2s"
Sep 17 17:54:51.024715 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:51.024637    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e16f3122-b1c7-4722-9b7e-63f516e083ea-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-x8g2s\" (UID: \"e16f3122-b1c7-4722-9b7e-63f516e083ea\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-x8g2s"
Sep 17 17:54:51.125836 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:51.125789    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e16f3122-b1c7-4722-9b7e-63f516e083ea-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-x8g2s\" (UID: \"e16f3122-b1c7-4722-9b7e-63f516e083ea\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-x8g2s"
Sep 17 17:54:51.125836 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:51.125841    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e16f3122-b1c7-4722-9b7e-63f516e083ea-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-x8g2s\" (UID: \"e16f3122-b1c7-4722-9b7e-63f516e083ea\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-x8g2s"
Sep 17 17:54:51.126116 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:51.125870    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvdvl\" (UniqueName: \"kubernetes.io/projected/e16f3122-b1c7-4722-9b7e-63f516e083ea-kube-api-access-fvdvl\") pod \"isvc-predictive-xgboost-predictor-695f978978-x8g2s\" (UID: \"e16f3122-b1c7-4722-9b7e-63f516e083ea\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-x8g2s"
Sep 17 17:54:51.126116 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:51.125893    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e16f3122-b1c7-4722-9b7e-63f516e083ea-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-x8g2s\" (UID: \"e16f3122-b1c7-4722-9b7e-63f516e083ea\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-x8g2s"
Sep 17 17:54:51.126116 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:51.125913    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e16f3122-b1c7-4722-9b7e-63f516e083ea-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-x8g2s\" (UID: \"e16f3122-b1c7-4722-9b7e-63f516e083ea\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-x8g2s"
Sep 17 17:54:51.126286 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:54:51.126131    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-serving-cert: secret "isvc-predictive-xgboost-predictor-serving-cert" not found
Sep 17 17:54:51.126286 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:54:51.126220    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e16f3122-b1c7-4722-9b7e-63f516e083ea-proxy-tls podName:e16f3122-b1c7-4722-9b7e-63f516e083ea nodeName:}" failed. No retries permitted until 2026-09-17 17:54:51.626193931 +0000 UTC m=+844.130196549 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e16f3122-b1c7-4722-9b7e-63f516e083ea-proxy-tls") pod "isvc-predictive-xgboost-predictor-695f978978-x8g2s" (UID: "e16f3122-b1c7-4722-9b7e-63f516e083ea") : secret "isvc-predictive-xgboost-predictor-serving-cert" not found
Sep 17 17:54:51.126534 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:51.126510    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e16f3122-b1c7-4722-9b7e-63f516e083ea-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-x8g2s\" (UID: \"e16f3122-b1c7-4722-9b7e-63f516e083ea\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-x8g2s"
Sep 17 17:54:51.126590 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:51.126567    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e16f3122-b1c7-4722-9b7e-63f516e083ea-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-x8g2s\" (UID: \"e16f3122-b1c7-4722-9b7e-63f516e083ea\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-x8g2s"
Sep 17 17:54:51.126780 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:51.126762    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e16f3122-b1c7-4722-9b7e-63f516e083ea-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-x8g2s\" (UID: \"e16f3122-b1c7-4722-9b7e-63f516e083ea\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-x8g2s"
Sep 17 17:54:51.135525 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:51.135474    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvdvl\" (UniqueName: \"kubernetes.io/projected/e16f3122-b1c7-4722-9b7e-63f516e083ea-kube-api-access-fvdvl\") pod \"isvc-predictive-xgboost-predictor-695f978978-x8g2s\" (UID: \"e16f3122-b1c7-4722-9b7e-63f516e083ea\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-x8g2s"
Sep 17 17:54:51.215004 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:51.214976    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lkc58"
Sep 17 17:54:51.227065 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:51.227042    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/23095e57-36d9-44b4-926d-fc7de2158696-kube-rbac-proxy-sar-config\") pod \"23095e57-36d9-44b4-926d-fc7de2158696\" (UID: \"23095e57-36d9-44b4-926d-fc7de2158696\") "
Sep 17 17:54:51.227197 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:51.227115    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23095e57-36d9-44b4-926d-fc7de2158696-proxy-tls\") pod \"23095e57-36d9-44b4-926d-fc7de2158696\" (UID: \"23095e57-36d9-44b4-926d-fc7de2158696\") "
Sep 17 17:54:51.227197 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:51.227159    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/23095e57-36d9-44b4-926d-fc7de2158696-cabundle-cert\") pod \"23095e57-36d9-44b4-926d-fc7de2158696\" (UID: \"23095e57-36d9-44b4-926d-fc7de2158696\") "
Sep 17 17:54:51.227197 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:51.227186    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23095e57-36d9-44b4-926d-fc7de2158696-kserve-provision-location\") pod \"23095e57-36d9-44b4-926d-fc7de2158696\" (UID: \"23095e57-36d9-44b4-926d-fc7de2158696\") "
Sep 17 17:54:51.227365 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:51.227231    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gkdjd\" (UniqueName: \"kubernetes.io/projected/23095e57-36d9-44b4-926d-fc7de2158696-kube-api-access-gkdjd\") pod \"23095e57-36d9-44b4-926d-fc7de2158696\" (UID: \"23095e57-36d9-44b4-926d-fc7de2158696\") "
Sep 17 17:54:51.227496 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:51.227466    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23095e57-36d9-44b4-926d-fc7de2158696-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "23095e57-36d9-44b4-926d-fc7de2158696" (UID: "23095e57-36d9-44b4-926d-fc7de2158696"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:54:51.227496 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:51.227486    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23095e57-36d9-44b4-926d-fc7de2158696-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "23095e57-36d9-44b4-926d-fc7de2158696" (UID: "23095e57-36d9-44b4-926d-fc7de2158696"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:54:51.227588 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:51.227519    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23095e57-36d9-44b4-926d-fc7de2158696-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "23095e57-36d9-44b4-926d-fc7de2158696" (UID: "23095e57-36d9-44b4-926d-fc7de2158696"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:54:51.229415 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:51.229388    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23095e57-36d9-44b4-926d-fc7de2158696-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "23095e57-36d9-44b4-926d-fc7de2158696" (UID: "23095e57-36d9-44b4-926d-fc7de2158696"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:54:51.229415 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:51.229399    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23095e57-36d9-44b4-926d-fc7de2158696-kube-api-access-gkdjd" (OuterVolumeSpecName: "kube-api-access-gkdjd") pod "23095e57-36d9-44b4-926d-fc7de2158696" (UID: "23095e57-36d9-44b4-926d-fc7de2158696"). InnerVolumeSpecName "kube-api-access-gkdjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:54:51.327925 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:51.327877    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/23095e57-36d9-44b4-926d-fc7de2158696-cabundle-cert\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 17:54:51.327925 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:51.327915    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23095e57-36d9-44b4-926d-fc7de2158696-kserve-provision-location\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 17:54:51.327925 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:51.327931    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkdjd\" (UniqueName: \"kubernetes.io/projected/23095e57-36d9-44b4-926d-fc7de2158696-kube-api-access-gkdjd\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 17:54:51.328209 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:51.327943    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/23095e57-36d9-44b4-926d-fc7de2158696-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 17:54:51.328209 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:51.327955    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23095e57-36d9-44b4-926d-fc7de2158696-proxy-tls\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 17:54:51.629491 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:51.629447    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e16f3122-b1c7-4722-9b7e-63f516e083ea-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-x8g2s\" (UID: \"e16f3122-b1c7-4722-9b7e-63f516e083ea\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-x8g2s"
Sep 17 17:54:51.631876 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:51.631853    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e16f3122-b1c7-4722-9b7e-63f516e083ea-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-x8g2s\" (UID: \"e16f3122-b1c7-4722-9b7e-63f516e083ea\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-x8g2s"
Sep 17 17:54:51.894644 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:51.894537    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-x8g2s"
Sep 17 17:54:51.923705 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:51.923674    2569 generic.go:356] "Generic (PLEG): container finished" podID="23095e57-36d9-44b4-926d-fc7de2158696" containerID="97d04c1c5d13268063492cbd1e03f3c7a69b9974cfaa8846b40d239be05dbe28" exitCode=0
Sep 17 17:54:51.923853 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:51.923733    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lkc58" event={"ID":"23095e57-36d9-44b4-926d-fc7de2158696","Type":"ContainerDied","Data":"97d04c1c5d13268063492cbd1e03f3c7a69b9974cfaa8846b40d239be05dbe28"}
Sep 17 17:54:51.923853 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:51.923755    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lkc58"
Sep 17 17:54:51.923853 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:51.923784    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lkc58" event={"ID":"23095e57-36d9-44b4-926d-fc7de2158696","Type":"ContainerDied","Data":"637cce4009d5c130821d815f0c6463b28a01c634cc02855b8de53568d3e438df"}
Sep 17 17:54:51.923853 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:51.923811    2569 scope.go:117] "RemoveContainer" containerID="3e429673497752530aa7151e1c2288da117ce6df193f0ff8554c02641068c854"
Sep 17 17:54:51.933302 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:51.933274    2569 scope.go:117] "RemoveContainer" containerID="97d04c1c5d13268063492cbd1e03f3c7a69b9974cfaa8846b40d239be05dbe28"
Sep 17 17:54:51.943562 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:51.943520    2569 scope.go:117] "RemoveContainer" containerID="6eace2fb7e054e727ad9736972d669a83911b49c04efce5be7e85b24318efacc"
Sep 17 17:54:51.947249 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:51.946858    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lkc58"]
Sep 17 17:54:51.950353 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:51.950020    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lkc58"]
Sep 17 17:54:51.955562 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:51.955543    2569 scope.go:117] "RemoveContainer" containerID="3e429673497752530aa7151e1c2288da117ce6df193f0ff8554c02641068c854"
Sep 17 17:54:51.955922 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:54:51.955900    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e429673497752530aa7151e1c2288da117ce6df193f0ff8554c02641068c854\": container with ID starting with 3e429673497752530aa7151e1c2288da117ce6df193f0ff8554c02641068c854 not found: ID does not exist" containerID="3e429673497752530aa7151e1c2288da117ce6df193f0ff8554c02641068c854"
Sep 17 17:54:51.956015 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:51.955933    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e429673497752530aa7151e1c2288da117ce6df193f0ff8554c02641068c854"} err="failed to get container status \"3e429673497752530aa7151e1c2288da117ce6df193f0ff8554c02641068c854\": rpc error: code = NotFound desc = could not find container \"3e429673497752530aa7151e1c2288da117ce6df193f0ff8554c02641068c854\": container with ID starting with 3e429673497752530aa7151e1c2288da117ce6df193f0ff8554c02641068c854 not found: ID does not exist"
Sep 17 17:54:51.956015 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:51.955952    2569 scope.go:117] "RemoveContainer" containerID="97d04c1c5d13268063492cbd1e03f3c7a69b9974cfaa8846b40d239be05dbe28"
Sep 17 17:54:51.956443 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:54:51.956395    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97d04c1c5d13268063492cbd1e03f3c7a69b9974cfaa8846b40d239be05dbe28\": container with ID starting with 97d04c1c5d13268063492cbd1e03f3c7a69b9974cfaa8846b40d239be05dbe28 not found: ID does not exist" containerID="97d04c1c5d13268063492cbd1e03f3c7a69b9974cfaa8846b40d239be05dbe28"
Sep 17 17:54:51.956586 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:51.956560    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97d04c1c5d13268063492cbd1e03f3c7a69b9974cfaa8846b40d239be05dbe28"} err="failed to get container status \"97d04c1c5d13268063492cbd1e03f3c7a69b9974cfaa8846b40d239be05dbe28\": rpc error: code = NotFound desc = could not find container \"97d04c1c5d13268063492cbd1e03f3c7a69b9974cfaa8846b40d239be05dbe28\": container with ID starting with 97d04c1c5d13268063492cbd1e03f3c7a69b9974cfaa8846b40d239be05dbe28 not found: ID does not exist"
Sep 17 17:54:51.956685 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:51.956671    2569 scope.go:117] "RemoveContainer" containerID="6eace2fb7e054e727ad9736972d669a83911b49c04efce5be7e85b24318efacc"
Sep 17 17:54:51.957182 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:54:51.957062    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6eace2fb7e054e727ad9736972d669a83911b49c04efce5be7e85b24318efacc\": container with ID starting with 6eace2fb7e054e727ad9736972d669a83911b49c04efce5be7e85b24318efacc not found: ID does not exist" containerID="6eace2fb7e054e727ad9736972d669a83911b49c04efce5be7e85b24318efacc"
Sep 17 17:54:51.957182 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:51.957126    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6eace2fb7e054e727ad9736972d669a83911b49c04efce5be7e85b24318efacc"} err="failed to get container status \"6eace2fb7e054e727ad9736972d669a83911b49c04efce5be7e85b24318efacc\": rpc error: code = NotFound desc = could not find container \"6eace2fb7e054e727ad9736972d669a83911b49c04efce5be7e85b24318efacc\": container with ID starting with 6eace2fb7e054e727ad9736972d669a83911b49c04efce5be7e85b24318efacc not found: ID does not exist"
Sep 17 17:54:52.168646 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:52.168620    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-x8g2s"]
Sep 17 17:54:52.170539 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:54:52.170511    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode16f3122_b1c7_4722_9b7e_63f516e083ea.slice/crio-6552b8a07bc111b144860e8c0eb4c75bfa6bcc6b6d0a7abac28e8b8f59c1f1ad WatchSource:0}: Error finding container 6552b8a07bc111b144860e8c0eb4c75bfa6bcc6b6d0a7abac28e8b8f59c1f1ad: Status 404 returned error can't find the container with id 6552b8a07bc111b144860e8c0eb4c75bfa6bcc6b6d0a7abac28e8b8f59c1f1ad
Sep 17 17:54:52.928732 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:52.928691    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-x8g2s" event={"ID":"e16f3122-b1c7-4722-9b7e-63f516e083ea","Type":"ContainerStarted","Data":"976b01fc194b9bea36e5914684c40ecf4b1037b879c6315f5eb45f31ccec3053"}
Sep 17 17:54:52.929261 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:52.928756    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-x8g2s" event={"ID":"e16f3122-b1c7-4722-9b7e-63f516e083ea","Type":"ContainerStarted","Data":"6552b8a07bc111b144860e8c0eb4c75bfa6bcc6b6d0a7abac28e8b8f59c1f1ad"}
Sep 17 17:54:53.508913 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:53.508876    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23095e57-36d9-44b4-926d-fc7de2158696" path="/var/lib/kubelet/pods/23095e57-36d9-44b4-926d-fc7de2158696/volumes"
Sep 17 17:54:56.946638 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:56.946601    2569 generic.go:356] "Generic (PLEG): container finished" podID="e16f3122-b1c7-4722-9b7e-63f516e083ea" containerID="976b01fc194b9bea36e5914684c40ecf4b1037b879c6315f5eb45f31ccec3053" exitCode=0
Sep 17 17:54:56.946638 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:56.946640    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-x8g2s" event={"ID":"e16f3122-b1c7-4722-9b7e-63f516e083ea","Type":"ContainerDied","Data":"976b01fc194b9bea36e5914684c40ecf4b1037b879c6315f5eb45f31ccec3053"}
Sep 17 17:54:57.952788 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:57.952751    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-x8g2s" event={"ID":"e16f3122-b1c7-4722-9b7e-63f516e083ea","Type":"ContainerStarted","Data":"7a76222e9ce9eac3d273e80d4596e635eeece2a952dbd85ed6829b12a6b1e921"}
Sep 17 17:54:57.953182 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:57.952813    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-x8g2s" event={"ID":"e16f3122-b1c7-4722-9b7e-63f516e083ea","Type":"ContainerStarted","Data":"b99a5264e005ca89d0d5c73d075580c88aec1794580182a6f246f6fb4a7066a1"}
Sep 17 17:54:57.953182 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:57.953107    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-x8g2s"
Sep 17 17:54:57.953281 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:57.953264    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-x8g2s"
Sep 17 17:54:57.954481 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:57.954460    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-x8g2s" podUID="e16f3122-b1c7-4722-9b7e-63f516e083ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 17:54:57.974373 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:57.974320    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-x8g2s" podStartSLOduration=4.055875927 podStartE2EDuration="7.974306471s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.918430544s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:54:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:54:57.973633044 +0000 UTC m=+850.477635670" watchObservedRunningTime="2026-09-17 17:54:57.974306471 +0000 UTC m=+850.478309096"
Sep 17 17:54:58.956748 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:54:58.956710    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-x8g2s" podUID="e16f3122-b1c7-4722-9b7e-63f516e083ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 17:55:03.961676 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:55:03.961648    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-x8g2s"
Sep 17 17:55:03.962149 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:55:03.962121    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-x8g2s" podUID="e16f3122-b1c7-4722-9b7e-63f516e083ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 17:55:13.962993 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:55:13.962947    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-x8g2s" podUID="e16f3122-b1c7-4722-9b7e-63f516e083ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 17:55:23.962314 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:55:23.962274    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-x8g2s" podUID="e16f3122-b1c7-4722-9b7e-63f516e083ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 17:55:33.963072 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:55:33.962983    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-x8g2s" podUID="e16f3122-b1c7-4722-9b7e-63f516e083ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 17:55:43.962801 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:55:43.962755    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-x8g2s" podUID="e16f3122-b1c7-4722-9b7e-63f516e083ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 17:55:49.493989 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:55:49.493964    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-t7pqb_b8d76e28-4655-4d21-98d8-0324e4756aaf/console-operator/2.log"
Sep 17 17:55:49.494569 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:55:49.493964    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-t7pqb_b8d76e28-4655-4d21-98d8-0324e4756aaf/console-operator/2.log"
Sep 17 17:55:49.499927 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:55:49.499902    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k4mqn_6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff/ovn-acl-logging/0.log"
Sep 17 17:55:49.500072 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:55:49.499981    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k4mqn_6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff/ovn-acl-logging/0.log"
Sep 17 17:55:53.962369 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:55:53.962328    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-x8g2s" podUID="e16f3122-b1c7-4722-9b7e-63f516e083ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 17:56:03.963167 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:56:03.963124    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-x8g2s" podUID="e16f3122-b1c7-4722-9b7e-63f516e083ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 17:56:13.962134 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:56:13.962072    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-x8g2s" podUID="e16f3122-b1c7-4722-9b7e-63f516e083ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 17:56:23.963233 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:56:23.963201    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-x8g2s"
Sep 17 17:56:31.582516 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:56:31.582485    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-x8g2s"]
Sep 17 17:56:31.585026 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:56:31.582870    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-x8g2s" podUID="e16f3122-b1c7-4722-9b7e-63f516e083ea" containerName="kserve-container" containerID="cri-o://b99a5264e005ca89d0d5c73d075580c88aec1794580182a6f246f6fb4a7066a1" gracePeriod=30
Sep 17 17:56:31.585026 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:56:31.582902    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-x8g2s" podUID="e16f3122-b1c7-4722-9b7e-63f516e083ea" containerName="kube-rbac-proxy" containerID="cri-o://7a76222e9ce9eac3d273e80d4596e635eeece2a952dbd85ed6829b12a6b1e921" gracePeriod=30
Sep 17 17:56:32.336761 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:56:32.336722    2569 generic.go:356] "Generic (PLEG): container finished" podID="e16f3122-b1c7-4722-9b7e-63f516e083ea" containerID="7a76222e9ce9eac3d273e80d4596e635eeece2a952dbd85ed6829b12a6b1e921" exitCode=2
Sep 17 17:56:32.336932 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:56:32.336799    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-x8g2s" event={"ID":"e16f3122-b1c7-4722-9b7e-63f516e083ea","Type":"ContainerDied","Data":"7a76222e9ce9eac3d273e80d4596e635eeece2a952dbd85ed6829b12a6b1e921"}
Sep 17 17:56:33.957147 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:56:33.957109    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-x8g2s" podUID="e16f3122-b1c7-4722-9b7e-63f516e083ea" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Sep 17 17:56:33.962443 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:56:33.962421    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-x8g2s" podUID="e16f3122-b1c7-4722-9b7e-63f516e083ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 17:56:36.819896 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:56:36.819870    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-x8g2s"
Sep 17 17:56:36.913505 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:56:36.913423    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e16f3122-b1c7-4722-9b7e-63f516e083ea-cabundle-cert\") pod \"e16f3122-b1c7-4722-9b7e-63f516e083ea\" (UID: \"e16f3122-b1c7-4722-9b7e-63f516e083ea\") "
Sep 17 17:56:36.913505 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:56:36.913470    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e16f3122-b1c7-4722-9b7e-63f516e083ea-proxy-tls\") pod \"e16f3122-b1c7-4722-9b7e-63f516e083ea\" (UID: \"e16f3122-b1c7-4722-9b7e-63f516e083ea\") "
Sep 17 17:56:36.913697 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:56:36.913539    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e16f3122-b1c7-4722-9b7e-63f516e083ea-kube-rbac-proxy-sar-config\") pod \"e16f3122-b1c7-4722-9b7e-63f516e083ea\" (UID: \"e16f3122-b1c7-4722-9b7e-63f516e083ea\") "
Sep 17 17:56:36.913697 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:56:36.913579    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fvdvl\" (UniqueName: \"kubernetes.io/projected/e16f3122-b1c7-4722-9b7e-63f516e083ea-kube-api-access-fvdvl\") pod \"e16f3122-b1c7-4722-9b7e-63f516e083ea\" (UID: \"e16f3122-b1c7-4722-9b7e-63f516e083ea\") "
Sep 17 17:56:36.913783 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:56:36.913703    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e16f3122-b1c7-4722-9b7e-63f516e083ea-kserve-provision-location\") pod \"e16f3122-b1c7-4722-9b7e-63f516e083ea\" (UID: \"e16f3122-b1c7-4722-9b7e-63f516e083ea\") "
Sep 17 17:56:36.913852 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:56:36.913830    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e16f3122-b1c7-4722-9b7e-63f516e083ea-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e16f3122-b1c7-4722-9b7e-63f516e083ea" (UID: "e16f3122-b1c7-4722-9b7e-63f516e083ea"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:56:36.914032 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:56:36.914013    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e16f3122-b1c7-4722-9b7e-63f516e083ea-cabundle-cert\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 17:56:36.914032 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:56:36.913900    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e16f3122-b1c7-4722-9b7e-63f516e083ea-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e16f3122-b1c7-4722-9b7e-63f516e083ea" (UID: "e16f3122-b1c7-4722-9b7e-63f516e083ea"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:56:36.914189 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:56:36.914036    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e16f3122-b1c7-4722-9b7e-63f516e083ea-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e16f3122-b1c7-4722-9b7e-63f516e083ea" (UID: "e16f3122-b1c7-4722-9b7e-63f516e083ea"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:56:36.915663 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:56:36.915642    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e16f3122-b1c7-4722-9b7e-63f516e083ea-kube-api-access-fvdvl" (OuterVolumeSpecName: "kube-api-access-fvdvl") pod "e16f3122-b1c7-4722-9b7e-63f516e083ea" (UID: "e16f3122-b1c7-4722-9b7e-63f516e083ea"). InnerVolumeSpecName "kube-api-access-fvdvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:56:36.915738 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:56:36.915670    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e16f3122-b1c7-4722-9b7e-63f516e083ea-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e16f3122-b1c7-4722-9b7e-63f516e083ea" (UID: "e16f3122-b1c7-4722-9b7e-63f516e083ea"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:56:37.014989 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:56:37.014957    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvdvl\" (UniqueName: \"kubernetes.io/projected/e16f3122-b1c7-4722-9b7e-63f516e083ea-kube-api-access-fvdvl\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 17:56:37.014989 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:56:37.014982    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e16f3122-b1c7-4722-9b7e-63f516e083ea-kserve-provision-location\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 17:56:37.014989 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:56:37.014992    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e16f3122-b1c7-4722-9b7e-63f516e083ea-proxy-tls\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 17:56:37.015246 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:56:37.015003    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e16f3122-b1c7-4722-9b7e-63f516e083ea-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 17:56:37.359157 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:56:37.359117    2569 generic.go:356] "Generic (PLEG): container finished" podID="e16f3122-b1c7-4722-9b7e-63f516e083ea" containerID="b99a5264e005ca89d0d5c73d075580c88aec1794580182a6f246f6fb4a7066a1" exitCode=0
Sep 17 17:56:37.359329 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:56:37.359167    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-x8g2s" event={"ID":"e16f3122-b1c7-4722-9b7e-63f516e083ea","Type":"ContainerDied","Data":"b99a5264e005ca89d0d5c73d075580c88aec1794580182a6f246f6fb4a7066a1"}
Sep 17 17:56:37.359329 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:56:37.359171    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-x8g2s"
Sep 17 17:56:37.359329 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:56:37.359225    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-x8g2s" event={"ID":"e16f3122-b1c7-4722-9b7e-63f516e083ea","Type":"ContainerDied","Data":"6552b8a07bc111b144860e8c0eb4c75bfa6bcc6b6d0a7abac28e8b8f59c1f1ad"}
Sep 17 17:56:37.359329 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:56:37.359254    2569 scope.go:117] "RemoveContainer" containerID="7a76222e9ce9eac3d273e80d4596e635eeece2a952dbd85ed6829b12a6b1e921"
Sep 17 17:56:37.369798 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:56:37.369635    2569 scope.go:117] "RemoveContainer" containerID="b99a5264e005ca89d0d5c73d075580c88aec1794580182a6f246f6fb4a7066a1"
Sep 17 17:56:37.377210 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:56:37.377188    2569 scope.go:117] "RemoveContainer" containerID="976b01fc194b9bea36e5914684c40ecf4b1037b879c6315f5eb45f31ccec3053"
Sep 17 17:56:37.384276 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:56:37.384256    2569 scope.go:117] "RemoveContainer" containerID="7a76222e9ce9eac3d273e80d4596e635eeece2a952dbd85ed6829b12a6b1e921"
Sep 17 17:56:37.384550 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:56:37.384532    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a76222e9ce9eac3d273e80d4596e635eeece2a952dbd85ed6829b12a6b1e921\": container with ID starting with 7a76222e9ce9eac3d273e80d4596e635eeece2a952dbd85ed6829b12a6b1e921 not found: ID does not exist" containerID="7a76222e9ce9eac3d273e80d4596e635eeece2a952dbd85ed6829b12a6b1e921"
Sep 17 17:56:37.384615 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:56:37.384557    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a76222e9ce9eac3d273e80d4596e635eeece2a952dbd85ed6829b12a6b1e921"} err="failed to get container status \"7a76222e9ce9eac3d273e80d4596e635eeece2a952dbd85ed6829b12a6b1e921\": rpc error: code = NotFound desc = could not find container \"7a76222e9ce9eac3d273e80d4596e635eeece2a952dbd85ed6829b12a6b1e921\": container with ID starting with 7a76222e9ce9eac3d273e80d4596e635eeece2a952dbd85ed6829b12a6b1e921 not found: ID does not exist"
Sep 17 17:56:37.384615 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:56:37.384572    2569 scope.go:117] "RemoveContainer" containerID="b99a5264e005ca89d0d5c73d075580c88aec1794580182a6f246f6fb4a7066a1"
Sep 17 17:56:37.384788 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:56:37.384774    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b99a5264e005ca89d0d5c73d075580c88aec1794580182a6f246f6fb4a7066a1\": container with ID starting with b99a5264e005ca89d0d5c73d075580c88aec1794580182a6f246f6fb4a7066a1 not found: ID does not exist" containerID="b99a5264e005ca89d0d5c73d075580c88aec1794580182a6f246f6fb4a7066a1"
Sep 17 17:56:37.384984 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:56:37.384791    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b99a5264e005ca89d0d5c73d075580c88aec1794580182a6f246f6fb4a7066a1"} err="failed to get container status \"b99a5264e005ca89d0d5c73d075580c88aec1794580182a6f246f6fb4a7066a1\": rpc error: code = NotFound desc = could not find container \"b99a5264e005ca89d0d5c73d075580c88aec1794580182a6f246f6fb4a7066a1\": container with ID starting with b99a5264e005ca89d0d5c73d075580c88aec1794580182a6f246f6fb4a7066a1 not found: ID does not exist"
Sep 17 17:56:37.384984 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:56:37.384803    2569 scope.go:117] "RemoveContainer" containerID="976b01fc194b9bea36e5914684c40ecf4b1037b879c6315f5eb45f31ccec3053"
Sep 17 17:56:37.385065 ip-10-0-131-14 kubenswrapper[2569]: E0917 17:56:37.385024    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"976b01fc194b9bea36e5914684c40ecf4b1037b879c6315f5eb45f31ccec3053\": container with ID starting with 976b01fc194b9bea36e5914684c40ecf4b1037b879c6315f5eb45f31ccec3053 not found: ID does not exist" containerID="976b01fc194b9bea36e5914684c40ecf4b1037b879c6315f5eb45f31ccec3053"
Sep 17 17:56:37.385065 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:56:37.385048    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"976b01fc194b9bea36e5914684c40ecf4b1037b879c6315f5eb45f31ccec3053"} err="failed to get container status \"976b01fc194b9bea36e5914684c40ecf4b1037b879c6315f5eb45f31ccec3053\": rpc error: code = NotFound desc = could not find container \"976b01fc194b9bea36e5914684c40ecf4b1037b879c6315f5eb45f31ccec3053\": container with ID starting with 976b01fc194b9bea36e5914684c40ecf4b1037b879c6315f5eb45f31ccec3053 not found: ID does not exist"
Sep 17 17:56:37.390411 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:56:37.390390    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-x8g2s"]
Sep 17 17:56:37.394552 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:56:37.394532    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-x8g2s"]
Sep 17 17:56:37.508348 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:56:37.508322    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e16f3122-b1c7-4722-9b7e-63f516e083ea" path="/var/lib/kubelet/pods/e16f3122-b1c7-4722-9b7e-63f516e083ea/volumes"
Sep 17 17:57:45.976202 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:57:45.973421    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-hzwcd"]
Sep 17 17:57:45.976202 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:57:45.974029    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e16f3122-b1c7-4722-9b7e-63f516e083ea" containerName="storage-initializer"
Sep 17 17:57:45.976202 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:57:45.974051    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e16f3122-b1c7-4722-9b7e-63f516e083ea" containerName="storage-initializer"
Sep 17 17:57:45.976202 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:57:45.974087    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23095e57-36d9-44b4-926d-fc7de2158696" containerName="kserve-container"
Sep 17 17:57:45.976202 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:57:45.974096    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="23095e57-36d9-44b4-926d-fc7de2158696" containerName="kserve-container"
Sep 17 17:57:45.976202 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:57:45.974106    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23095e57-36d9-44b4-926d-fc7de2158696" containerName="kube-rbac-proxy"
Sep 17 17:57:45.976202 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:57:45.974115    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="23095e57-36d9-44b4-926d-fc7de2158696" containerName="kube-rbac-proxy"
Sep 17 17:57:45.976202 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:57:45.974138    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e16f3122-b1c7-4722-9b7e-63f516e083ea" containerName="kube-rbac-proxy"
Sep 17 17:57:45.976202 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:57:45.974147    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e16f3122-b1c7-4722-9b7e-63f516e083ea" containerName="kube-rbac-proxy"
Sep 17 17:57:45.976202 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:57:45.974164    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23095e57-36d9-44b4-926d-fc7de2158696" containerName="storage-initializer"
Sep 17 17:57:45.976202 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:57:45.974172    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="23095e57-36d9-44b4-926d-fc7de2158696" containerName="storage-initializer"
Sep 17 17:57:45.976202 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:57:45.974207    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e16f3122-b1c7-4722-9b7e-63f516e083ea" containerName="kserve-container"
Sep 17 17:57:45.976202 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:57:45.974215    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e16f3122-b1c7-4722-9b7e-63f516e083ea" containerName="kserve-container"
Sep 17 17:57:45.976202 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:57:45.974335    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e16f3122-b1c7-4722-9b7e-63f516e083ea" containerName="kserve-container"
Sep 17 17:57:45.976202 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:57:45.974350    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e16f3122-b1c7-4722-9b7e-63f516e083ea" containerName="kube-rbac-proxy"
Sep 17 17:57:45.976202 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:57:45.974365    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="23095e57-36d9-44b4-926d-fc7de2158696" containerName="kube-rbac-proxy"
Sep 17 17:57:45.976202 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:57:45.974382    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="23095e57-36d9-44b4-926d-fc7de2158696" containerName="kserve-container"
Sep 17 17:57:45.978642 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:57:45.978624    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-hzwcd"
Sep 17 17:57:45.989635 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:57:45.989613    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:57:45.990990 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:57:45.990971    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Sep 17 17:57:45.991098 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:57:45.990976    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-kkx8f\""
Sep 17 17:57:45.991244 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:57:45.991229    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 17:57:45.991294 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:57:45.991231    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:57:45.991337 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:57:45.991312    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Sep 17 17:57:45.991959 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:57:45.991945    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:57:46.004244 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:57:46.004219    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-hzwcd"]
Sep 17 17:57:46.066690 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:57:46.066651    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9319d7fb-e576-411f-8853-b1340adcb8bf-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-hzwcd\" (UID: \"9319d7fb-e576-411f-8853-b1340adcb8bf\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-hzwcd"
Sep 17 17:57:46.066869 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:57:46.066697    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9319d7fb-e576-411f-8853-b1340adcb8bf-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-hzwcd\" (UID: \"9319d7fb-e576-411f-8853-b1340adcb8bf\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-hzwcd"
Sep 17 17:57:46.066869 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:57:46.066725    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9fp7\" (UniqueName: \"kubernetes.io/projected/9319d7fb-e576-411f-8853-b1340adcb8bf-kube-api-access-h9fp7\") pod \"isvc-autogluon-v1-predictor-574b5b7495-hzwcd\" (UID: \"9319d7fb-e576-411f-8853-b1340adcb8bf\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-hzwcd"
Sep 17 17:57:46.066869 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:57:46.066755    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9319d7fb-e576-411f-8853-b1340adcb8bf-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-hzwcd\" (UID: \"9319d7fb-e576-411f-8853-b1340adcb8bf\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-hzwcd"
Sep 17 17:57:46.066869 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:57:46.066810    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9319d7fb-e576-411f-8853-b1340adcb8bf-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-hzwcd\" (UID: \"9319d7fb-e576-411f-8853-b1340adcb8bf\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-hzwcd"
Sep 17 17:57:46.167519 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:57:46.167484    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9319d7fb-e576-411f-8853-b1340adcb8bf-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-hzwcd\" (UID: \"9319d7fb-e576-411f-8853-b1340adcb8bf\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-hzwcd"
Sep 17 17:57:46.167519 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:57:46.167518    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9319d7fb-e576-411f-8853-b1340adcb8bf-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-hzwcd\" (UID: \"9319d7fb-e576-411f-8853-b1340adcb8bf\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-hzwcd"
Sep 17 17:57:46.167747 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:57:46.167656    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9319d7fb-e576-411f-8853-b1340adcb8bf-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-hzwcd\" (UID: \"9319d7fb-e576-411f-8853-b1340adcb8bf\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-hzwcd"
Sep 17 17:57:46.167747 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:57:46.167724    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9fp7\" (UniqueName: \"kubernetes.io/projected/9319d7fb-e576-411f-8853-b1340adcb8bf-kube-api-access-h9fp7\") pod \"isvc-autogluon-v1-predictor-574b5b7495-hzwcd\" (UID: \"9319d7fb-e576-411f-8853-b1340adcb8bf\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-hzwcd"
Sep 17 17:57:46.167848 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:57:46.167756    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9319d7fb-e576-411f-8853-b1340adcb8bf-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-hzwcd\" (UID: \"9319d7fb-e576-411f-8853-b1340adcb8bf\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-hzwcd"
Sep 17 17:57:46.167921 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:57:46.167900    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9319d7fb-e576-411f-8853-b1340adcb8bf-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-hzwcd\" (UID: \"9319d7fb-e576-411f-8853-b1340adcb8bf\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-hzwcd"
Sep 17 17:57:46.168329 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:57:46.168306    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9319d7fb-e576-411f-8853-b1340adcb8bf-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-hzwcd\" (UID: \"9319d7fb-e576-411f-8853-b1340adcb8bf\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-hzwcd"
Sep 17 17:57:46.168422 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:57:46.168407    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9319d7fb-e576-411f-8853-b1340adcb8bf-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-hzwcd\" (UID: \"9319d7fb-e576-411f-8853-b1340adcb8bf\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-hzwcd"
Sep 17 17:57:46.170109 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:57:46.170088    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9319d7fb-e576-411f-8853-b1340adcb8bf-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-hzwcd\" (UID: \"9319d7fb-e576-411f-8853-b1340adcb8bf\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-hzwcd"
Sep 17 17:57:46.178194 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:57:46.178171    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9fp7\" (UniqueName: \"kubernetes.io/projected/9319d7fb-e576-411f-8853-b1340adcb8bf-kube-api-access-h9fp7\") pod \"isvc-autogluon-v1-predictor-574b5b7495-hzwcd\" (UID: \"9319d7fb-e576-411f-8853-b1340adcb8bf\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-hzwcd"
Sep 17 17:57:46.288373 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:57:46.288285    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-hzwcd"
Sep 17 17:57:46.567012 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:57:46.566918    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-hzwcd"]
Sep 17 17:57:46.571105 ip-10-0-131-14 kubenswrapper[2569]: W0917 17:57:46.571065    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9319d7fb_e576_411f_8853_b1340adcb8bf.slice/crio-9354ba2cd086a219b804392ed9c9eafd60fd5163263f44da616aeb20ad783fc9 WatchSource:0}: Error finding container 9354ba2cd086a219b804392ed9c9eafd60fd5163263f44da616aeb20ad783fc9: Status 404 returned error can't find the container with id 9354ba2cd086a219b804392ed9c9eafd60fd5163263f44da616aeb20ad783fc9
Sep 17 17:57:46.630402 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:57:46.630376    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-hzwcd" event={"ID":"9319d7fb-e576-411f-8853-b1340adcb8bf","Type":"ContainerStarted","Data":"9354ba2cd086a219b804392ed9c9eafd60fd5163263f44da616aeb20ad783fc9"}
Sep 17 17:57:47.635752 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:57:47.635715    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-hzwcd" event={"ID":"9319d7fb-e576-411f-8853-b1340adcb8bf","Type":"ContainerStarted","Data":"8b90f1be794fef81899b7ae7298384627fea5d7e261e8e633d7bc59fc0c50bb0"}
Sep 17 17:57:52.658325 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:57:52.658292    2569 generic.go:356] "Generic (PLEG): container finished" podID="9319d7fb-e576-411f-8853-b1340adcb8bf" containerID="8b90f1be794fef81899b7ae7298384627fea5d7e261e8e633d7bc59fc0c50bb0" exitCode=0
Sep 17 17:57:52.658740 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:57:52.658376    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-hzwcd" event={"ID":"9319d7fb-e576-411f-8853-b1340adcb8bf","Type":"ContainerDied","Data":"8b90f1be794fef81899b7ae7298384627fea5d7e261e8e633d7bc59fc0c50bb0"}
Sep 17 17:57:52.659623 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:57:52.659607    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:59:56.232051 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:59:56.232019    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-hzwcd" event={"ID":"9319d7fb-e576-411f-8853-b1340adcb8bf","Type":"ContainerStarted","Data":"791c2a7a02829a46aa654d18ce8ae2d5f5339e607642923870108a7bf88d0641"}
Sep 17 17:59:57.238588 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:59:57.238551    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-hzwcd" event={"ID":"9319d7fb-e576-411f-8853-b1340adcb8bf","Type":"ContainerStarted","Data":"d42e672640259fcfd610d0d12c1c0d78f52647c337ba3dc2a2926ccd8b8c1319"}
Sep 17 17:59:57.239035 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:59:57.238770    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-hzwcd"
Sep 17 17:59:57.239035 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:59:57.238840    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-hzwcd"
Sep 17 17:59:57.239955 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:59:57.239930    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-hzwcd" podUID="9319d7fb-e576-411f-8853-b1340adcb8bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 17:59:57.263336 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:59:57.263284    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-hzwcd" podStartSLOduration=3.647032817 podStartE2EDuration="2m12.263268698s" totalImagesPullingTime="2m3.379468323s" totalInitContainerRuntime="5.236767558s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:57:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:59:57.261287803 +0000 UTC m=+1149.765290430" watchObservedRunningTime="2026-09-17 17:59:57.263268698 +0000 UTC m=+1149.767271323"
Sep 17 17:59:58.242747 ip-10-0-131-14 kubenswrapper[2569]: I0917 17:59:58.242710    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-hzwcd" podUID="9319d7fb-e576-411f-8853-b1340adcb8bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 18:00:03.247760 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:00:03.247673    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-hzwcd"
Sep 17 18:00:03.248335 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:00:03.248252    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-hzwcd" podUID="9319d7fb-e576-411f-8853-b1340adcb8bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 18:00:13.248778 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:00:13.248749    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-hzwcd"
Sep 17 18:00:22.162857 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:00:22.162822    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-hzwcd"]
Sep 17 18:00:22.163361 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:00:22.163195    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-hzwcd" podUID="9319d7fb-e576-411f-8853-b1340adcb8bf" containerName="kserve-container" containerID="cri-o://791c2a7a02829a46aa654d18ce8ae2d5f5339e607642923870108a7bf88d0641" gracePeriod=30
Sep 17 18:00:22.163361 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:00:22.163218    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-hzwcd" podUID="9319d7fb-e576-411f-8853-b1340adcb8bf" containerName="kube-rbac-proxy" containerID="cri-o://d42e672640259fcfd610d0d12c1c0d78f52647c337ba3dc2a2926ccd8b8c1319" gracePeriod=30
Sep 17 18:00:22.343489 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:00:22.343458    2569 generic.go:356] "Generic (PLEG): container finished" podID="9319d7fb-e576-411f-8853-b1340adcb8bf" containerID="d42e672640259fcfd610d0d12c1c0d78f52647c337ba3dc2a2926ccd8b8c1319" exitCode=2
Sep 17 18:00:22.343660 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:00:22.343508    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-hzwcd" event={"ID":"9319d7fb-e576-411f-8853-b1340adcb8bf","Type":"ContainerDied","Data":"d42e672640259fcfd610d0d12c1c0d78f52647c337ba3dc2a2926ccd8b8c1319"}
Sep 17 18:00:23.243434 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:00:23.243387    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-hzwcd" podUID="9319d7fb-e576-411f-8853-b1340adcb8bf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Sep 17 18:00:23.248832 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:00:23.248802    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-hzwcd" podUID="9319d7fb-e576-411f-8853-b1340adcb8bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 18:00:23.744533 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:00:23.744505    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-hzwcd"
Sep 17 18:00:23.852432 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:00:23.852348    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9319d7fb-e576-411f-8853-b1340adcb8bf-kube-rbac-proxy-sar-config\") pod \"9319d7fb-e576-411f-8853-b1340adcb8bf\" (UID: \"9319d7fb-e576-411f-8853-b1340adcb8bf\") "
Sep 17 18:00:23.852432 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:00:23.852388    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9319d7fb-e576-411f-8853-b1340adcb8bf-proxy-tls\") pod \"9319d7fb-e576-411f-8853-b1340adcb8bf\" (UID: \"9319d7fb-e576-411f-8853-b1340adcb8bf\") "
Sep 17 18:00:23.852432 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:00:23.852422    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9319d7fb-e576-411f-8853-b1340adcb8bf-cabundle-cert\") pod \"9319d7fb-e576-411f-8853-b1340adcb8bf\" (UID: \"9319d7fb-e576-411f-8853-b1340adcb8bf\") "
Sep 17 18:00:23.852713 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:00:23.852483    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h9fp7\" (UniqueName: \"kubernetes.io/projected/9319d7fb-e576-411f-8853-b1340adcb8bf-kube-api-access-h9fp7\") pod \"9319d7fb-e576-411f-8853-b1340adcb8bf\" (UID: \"9319d7fb-e576-411f-8853-b1340adcb8bf\") "
Sep 17 18:00:23.852713 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:00:23.852506    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9319d7fb-e576-411f-8853-b1340adcb8bf-kserve-provision-location\") pod \"9319d7fb-e576-411f-8853-b1340adcb8bf\" (UID: \"9319d7fb-e576-411f-8853-b1340adcb8bf\") "
Sep 17 18:00:23.852862 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:00:23.852832    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9319d7fb-e576-411f-8853-b1340adcb8bf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9319d7fb-e576-411f-8853-b1340adcb8bf" (UID: "9319d7fb-e576-411f-8853-b1340adcb8bf"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:00:23.852918 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:00:23.852853    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9319d7fb-e576-411f-8853-b1340adcb8bf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9319d7fb-e576-411f-8853-b1340adcb8bf" (UID: "9319d7fb-e576-411f-8853-b1340adcb8bf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:00:23.854607 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:00:23.854579    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9319d7fb-e576-411f-8853-b1340adcb8bf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9319d7fb-e576-411f-8853-b1340adcb8bf" (UID: "9319d7fb-e576-411f-8853-b1340adcb8bf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:00:23.854712 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:00:23.854676    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9319d7fb-e576-411f-8853-b1340adcb8bf-kube-api-access-h9fp7" (OuterVolumeSpecName: "kube-api-access-h9fp7") pod "9319d7fb-e576-411f-8853-b1340adcb8bf" (UID: "9319d7fb-e576-411f-8853-b1340adcb8bf"). InnerVolumeSpecName "kube-api-access-h9fp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:00:23.857282 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:00:23.857261    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9319d7fb-e576-411f-8853-b1340adcb8bf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9319d7fb-e576-411f-8853-b1340adcb8bf" (UID: "9319d7fb-e576-411f-8853-b1340adcb8bf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:00:23.953798 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:00:23.953764    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9fp7\" (UniqueName: \"kubernetes.io/projected/9319d7fb-e576-411f-8853-b1340adcb8bf-kube-api-access-h9fp7\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:00:23.953798 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:00:23.953790    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9319d7fb-e576-411f-8853-b1340adcb8bf-kserve-provision-location\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:00:23.953798 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:00:23.953800    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9319d7fb-e576-411f-8853-b1340adcb8bf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:00:23.953798 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:00:23.953809    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9319d7fb-e576-411f-8853-b1340adcb8bf-proxy-tls\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:00:23.954066 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:00:23.953818    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9319d7fb-e576-411f-8853-b1340adcb8bf-cabundle-cert\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:00:24.354349 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:00:24.354319    2569 generic.go:356] "Generic (PLEG): container finished" podID="9319d7fb-e576-411f-8853-b1340adcb8bf" containerID="791c2a7a02829a46aa654d18ce8ae2d5f5339e607642923870108a7bf88d0641" exitCode=0
Sep 17 18:00:24.354747 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:00:24.354373    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-hzwcd"
Sep 17 18:00:24.354747 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:00:24.354381    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-hzwcd" event={"ID":"9319d7fb-e576-411f-8853-b1340adcb8bf","Type":"ContainerDied","Data":"791c2a7a02829a46aa654d18ce8ae2d5f5339e607642923870108a7bf88d0641"}
Sep 17 18:00:24.354747 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:00:24.354447    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-hzwcd" event={"ID":"9319d7fb-e576-411f-8853-b1340adcb8bf","Type":"ContainerDied","Data":"9354ba2cd086a219b804392ed9c9eafd60fd5163263f44da616aeb20ad783fc9"}
Sep 17 18:00:24.354747 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:00:24.354476    2569 scope.go:117] "RemoveContainer" containerID="d42e672640259fcfd610d0d12c1c0d78f52647c337ba3dc2a2926ccd8b8c1319"
Sep 17 18:00:24.363036 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:00:24.363018    2569 scope.go:117] "RemoveContainer" containerID="791c2a7a02829a46aa654d18ce8ae2d5f5339e607642923870108a7bf88d0641"
Sep 17 18:00:24.370300 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:00:24.370283    2569 scope.go:117] "RemoveContainer" containerID="8b90f1be794fef81899b7ae7298384627fea5d7e261e8e633d7bc59fc0c50bb0"
Sep 17 18:00:24.377209 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:00:24.377189    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-hzwcd"]
Sep 17 18:00:24.377678 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:00:24.377662    2569 scope.go:117] "RemoveContainer" containerID="d42e672640259fcfd610d0d12c1c0d78f52647c337ba3dc2a2926ccd8b8c1319"
Sep 17 18:00:24.377939 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:00:24.377922    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d42e672640259fcfd610d0d12c1c0d78f52647c337ba3dc2a2926ccd8b8c1319\": container with ID starting with d42e672640259fcfd610d0d12c1c0d78f52647c337ba3dc2a2926ccd8b8c1319 not found: ID does not exist" containerID="d42e672640259fcfd610d0d12c1c0d78f52647c337ba3dc2a2926ccd8b8c1319"
Sep 17 18:00:24.378010 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:00:24.377950    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d42e672640259fcfd610d0d12c1c0d78f52647c337ba3dc2a2926ccd8b8c1319"} err="failed to get container status \"d42e672640259fcfd610d0d12c1c0d78f52647c337ba3dc2a2926ccd8b8c1319\": rpc error: code = NotFound desc = could not find container \"d42e672640259fcfd610d0d12c1c0d78f52647c337ba3dc2a2926ccd8b8c1319\": container with ID starting with d42e672640259fcfd610d0d12c1c0d78f52647c337ba3dc2a2926ccd8b8c1319 not found: ID does not exist"
Sep 17 18:00:24.378010 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:00:24.377968    2569 scope.go:117] "RemoveContainer" containerID="791c2a7a02829a46aa654d18ce8ae2d5f5339e607642923870108a7bf88d0641"
Sep 17 18:00:24.378236 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:00:24.378219    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"791c2a7a02829a46aa654d18ce8ae2d5f5339e607642923870108a7bf88d0641\": container with ID starting with 791c2a7a02829a46aa654d18ce8ae2d5f5339e607642923870108a7bf88d0641 not found: ID does not exist" containerID="791c2a7a02829a46aa654d18ce8ae2d5f5339e607642923870108a7bf88d0641"
Sep 17 18:00:24.378283 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:00:24.378243    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"791c2a7a02829a46aa654d18ce8ae2d5f5339e607642923870108a7bf88d0641"} err="failed to get container status \"791c2a7a02829a46aa654d18ce8ae2d5f5339e607642923870108a7bf88d0641\": rpc error: code = NotFound desc = could not find container \"791c2a7a02829a46aa654d18ce8ae2d5f5339e607642923870108a7bf88d0641\": container with ID starting with 791c2a7a02829a46aa654d18ce8ae2d5f5339e607642923870108a7bf88d0641 not found: ID does not exist"
Sep 17 18:00:24.378283 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:00:24.378258    2569 scope.go:117] "RemoveContainer" containerID="8b90f1be794fef81899b7ae7298384627fea5d7e261e8e633d7bc59fc0c50bb0"
Sep 17 18:00:24.378524 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:00:24.378507    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b90f1be794fef81899b7ae7298384627fea5d7e261e8e633d7bc59fc0c50bb0\": container with ID starting with 8b90f1be794fef81899b7ae7298384627fea5d7e261e8e633d7bc59fc0c50bb0 not found: ID does not exist" containerID="8b90f1be794fef81899b7ae7298384627fea5d7e261e8e633d7bc59fc0c50bb0"
Sep 17 18:00:24.378570 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:00:24.378527    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b90f1be794fef81899b7ae7298384627fea5d7e261e8e633d7bc59fc0c50bb0"} err="failed to get container status \"8b90f1be794fef81899b7ae7298384627fea5d7e261e8e633d7bc59fc0c50bb0\": rpc error: code = NotFound desc = could not find container \"8b90f1be794fef81899b7ae7298384627fea5d7e261e8e633d7bc59fc0c50bb0\": container with ID starting with 8b90f1be794fef81899b7ae7298384627fea5d7e261e8e633d7bc59fc0c50bb0 not found: ID does not exist"
Sep 17 18:00:24.382515 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:00:24.382495    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-hzwcd"]
Sep 17 18:00:25.508027 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:00:25.507986    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9319d7fb-e576-411f-8853-b1340adcb8bf" path="/var/lib/kubelet/pods/9319d7fb-e576-411f-8853-b1340adcb8bf/volumes"
Sep 17 18:00:49.521183 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:00:49.521157    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-t7pqb_b8d76e28-4655-4d21-98d8-0324e4756aaf/console-operator/2.log"
Sep 17 18:00:49.522172 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:00:49.522154    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-t7pqb_b8d76e28-4655-4d21-98d8-0324e4756aaf/console-operator/2.log"
Sep 17 18:00:49.526554 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:00:49.526527    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k4mqn_6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff/ovn-acl-logging/0.log"
Sep 17 18:00:49.527706 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:00:49.527684    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k4mqn_6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff/ovn-acl-logging/0.log"
Sep 17 18:01:53.319661 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:01:53.319462    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-z6kzq"]
Sep 17 18:01:53.323614 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:01:53.320029    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9319d7fb-e576-411f-8853-b1340adcb8bf" containerName="storage-initializer"
Sep 17 18:01:53.323614 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:01:53.320046    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="9319d7fb-e576-411f-8853-b1340adcb8bf" containerName="storage-initializer"
Sep 17 18:01:53.323614 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:01:53.320066    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9319d7fb-e576-411f-8853-b1340adcb8bf" containerName="kube-rbac-proxy"
Sep 17 18:01:53.323614 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:01:53.320095    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="9319d7fb-e576-411f-8853-b1340adcb8bf" containerName="kube-rbac-proxy"
Sep 17 18:01:53.323614 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:01:53.320122    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9319d7fb-e576-411f-8853-b1340adcb8bf" containerName="kserve-container"
Sep 17 18:01:53.323614 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:01:53.320131    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="9319d7fb-e576-411f-8853-b1340adcb8bf" containerName="kserve-container"
Sep 17 18:01:53.323614 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:01:53.320186    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="9319d7fb-e576-411f-8853-b1340adcb8bf" containerName="kube-rbac-proxy"
Sep 17 18:01:53.323614 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:01:53.320195    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="9319d7fb-e576-411f-8853-b1340adcb8bf" containerName="kserve-container"
Sep 17 18:01:53.325201 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:01:53.325154    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-z6kzq"
Sep 17 18:01:53.328504 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:01:53.328423    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:01:53.328873 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:01:53.328611    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:01:53.328873 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:01:53.328655    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-kkx8f\""
Sep 17 18:01:53.328873 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:01:53.328714    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:01:53.328873 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:01:53.328761    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:01:53.329139 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:01:53.329001    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-predictor-serving-cert\""
Sep 17 18:01:53.330250 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:01:53.330215    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\""
Sep 17 18:01:53.335253 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:01:53.335192    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-z6kzq"]
Sep 17 18:01:53.481581 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:01:53.481542    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8nx5\" (UniqueName: \"kubernetes.io/projected/62c0acb3-55e9-4b66-8d01-80b77fe0da2f-kube-api-access-h8nx5\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-z6kzq\" (UID: \"62c0acb3-55e9-4b66-8d01-80b77fe0da2f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-z6kzq"
Sep 17 18:01:53.481581 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:01:53.481578    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62c0acb3-55e9-4b66-8d01-80b77fe0da2f-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-z6kzq\" (UID: \"62c0acb3-55e9-4b66-8d01-80b77fe0da2f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-z6kzq"
Sep 17 18:01:53.481801 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:01:53.481613    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62c0acb3-55e9-4b66-8d01-80b77fe0da2f-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-z6kzq\" (UID: \"62c0acb3-55e9-4b66-8d01-80b77fe0da2f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-z6kzq"
Sep 17 18:01:53.481801 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:01:53.481719    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/62c0acb3-55e9-4b66-8d01-80b77fe0da2f-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-z6kzq\" (UID: \"62c0acb3-55e9-4b66-8d01-80b77fe0da2f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-z6kzq"
Sep 17 18:01:53.481801 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:01:53.481780    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62c0acb3-55e9-4b66-8d01-80b77fe0da2f-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-z6kzq\" (UID: \"62c0acb3-55e9-4b66-8d01-80b77fe0da2f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-z6kzq"
Sep 17 18:01:53.582217 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:01:53.582144    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/62c0acb3-55e9-4b66-8d01-80b77fe0da2f-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-z6kzq\" (UID: \"62c0acb3-55e9-4b66-8d01-80b77fe0da2f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-z6kzq"
Sep 17 18:01:53.582217 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:01:53.582191    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62c0acb3-55e9-4b66-8d01-80b77fe0da2f-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-z6kzq\" (UID: \"62c0acb3-55e9-4b66-8d01-80b77fe0da2f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-z6kzq"
Sep 17 18:01:53.582412 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:01:53.582307    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8nx5\" (UniqueName: \"kubernetes.io/projected/62c0acb3-55e9-4b66-8d01-80b77fe0da2f-kube-api-access-h8nx5\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-z6kzq\" (UID: \"62c0acb3-55e9-4b66-8d01-80b77fe0da2f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-z6kzq"
Sep 17 18:01:53.582412 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:01:53.582336    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62c0acb3-55e9-4b66-8d01-80b77fe0da2f-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-z6kzq\" (UID: \"62c0acb3-55e9-4b66-8d01-80b77fe0da2f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-z6kzq"
Sep 17 18:01:53.582412 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:01:53.582369    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62c0acb3-55e9-4b66-8d01-80b77fe0da2f-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-z6kzq\" (UID: \"62c0acb3-55e9-4b66-8d01-80b77fe0da2f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-z6kzq"
Sep 17 18:01:53.582681 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:01:53.582660    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62c0acb3-55e9-4b66-8d01-80b77fe0da2f-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-z6kzq\" (UID: \"62c0acb3-55e9-4b66-8d01-80b77fe0da2f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-z6kzq"
Sep 17 18:01:53.582867 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:01:53.582846    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/62c0acb3-55e9-4b66-8d01-80b77fe0da2f-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-z6kzq\" (UID: \"62c0acb3-55e9-4b66-8d01-80b77fe0da2f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-z6kzq"
Sep 17 18:01:53.582942 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:01:53.582925    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62c0acb3-55e9-4b66-8d01-80b77fe0da2f-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-z6kzq\" (UID: \"62c0acb3-55e9-4b66-8d01-80b77fe0da2f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-z6kzq"
Sep 17 18:01:53.584870 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:01:53.584853    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62c0acb3-55e9-4b66-8d01-80b77fe0da2f-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-z6kzq\" (UID: \"62c0acb3-55e9-4b66-8d01-80b77fe0da2f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-z6kzq"
Sep 17 18:01:53.590824 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:01:53.590802    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8nx5\" (UniqueName: \"kubernetes.io/projected/62c0acb3-55e9-4b66-8d01-80b77fe0da2f-kube-api-access-h8nx5\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-z6kzq\" (UID: \"62c0acb3-55e9-4b66-8d01-80b77fe0da2f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-z6kzq"
Sep 17 18:01:53.645772 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:01:53.645746    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-z6kzq"
Sep 17 18:01:53.934966 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:01:53.934942    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-z6kzq"]
Sep 17 18:01:53.937418 ip-10-0-131-14 kubenswrapper[2569]: W0917 18:01:53.937386    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62c0acb3_55e9_4b66_8d01_80b77fe0da2f.slice/crio-76a8b9f7bd67a0311920270e3857fcab1d8fd0069a08e681da6c50ab65d4d9ca WatchSource:0}: Error finding container 76a8b9f7bd67a0311920270e3857fcab1d8fd0069a08e681da6c50ab65d4d9ca: Status 404 returned error can't find the container with id 76a8b9f7bd67a0311920270e3857fcab1d8fd0069a08e681da6c50ab65d4d9ca
Sep 17 18:01:54.713984 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:01:54.713938    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-z6kzq" event={"ID":"62c0acb3-55e9-4b66-8d01-80b77fe0da2f","Type":"ContainerStarted","Data":"98e3a45db5e76fbfb71bf320f761845150505438af5c564958f56e156607c5a7"}
Sep 17 18:01:54.714404 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:01:54.714014    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-z6kzq" event={"ID":"62c0acb3-55e9-4b66-8d01-80b77fe0da2f","Type":"ContainerStarted","Data":"76a8b9f7bd67a0311920270e3857fcab1d8fd0069a08e681da6c50ab65d4d9ca"}
Sep 17 18:01:58.730921 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:01:58.730888    2569 generic.go:356] "Generic (PLEG): container finished" podID="62c0acb3-55e9-4b66-8d01-80b77fe0da2f" containerID="98e3a45db5e76fbfb71bf320f761845150505438af5c564958f56e156607c5a7" exitCode=0
Sep 17 18:01:58.731400 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:01:58.730945    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-z6kzq" event={"ID":"62c0acb3-55e9-4b66-8d01-80b77fe0da2f","Type":"ContainerDied","Data":"98e3a45db5e76fbfb71bf320f761845150505438af5c564958f56e156607c5a7"}
Sep 17 18:01:59.737478 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:01:59.737446    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-z6kzq" event={"ID":"62c0acb3-55e9-4b66-8d01-80b77fe0da2f","Type":"ContainerStarted","Data":"8a5afcd783b6bd55e311ca9cedd1456b9cc448967b96a97c3282fd8c64453daf"}
Sep 17 18:01:59.737888 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:01:59.737504    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-z6kzq" event={"ID":"62c0acb3-55e9-4b66-8d01-80b77fe0da2f","Type":"ContainerStarted","Data":"abb88eca40f22f466f77522960ef044800e92e886fe635b20012c501ba52292b"}
Sep 17 18:01:59.737888 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:01:59.737781    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-z6kzq"
Sep 17 18:01:59.737888 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:01:59.737845    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-z6kzq"
Sep 17 18:01:59.757823 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:01:59.757770    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-z6kzq" podStartSLOduration=2.996553964 podStartE2EDuration="6.757754098s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.761200134s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:01:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:01:59.757256937 +0000 UTC m=+1272.261259585" watchObservedRunningTime="2026-09-17 18:01:59.757754098 +0000 UTC m=+1272.261756722"
Sep 17 18:02:05.749262 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:02:05.749234    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-z6kzq"
Sep 17 18:02:35.750061 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:02:35.750020    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-z6kzq" podUID="62c0acb3-55e9-4b66-8d01-80b77fe0da2f" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.27:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 18:02:45.750281 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:02:45.750236    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-z6kzq" podUID="62c0acb3-55e9-4b66-8d01-80b77fe0da2f" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.27:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 18:02:55.750714 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:02:55.750668    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-z6kzq" podUID="62c0acb3-55e9-4b66-8d01-80b77fe0da2f" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.27:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 18:03:05.749910 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:05.749874    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-z6kzq" podUID="62c0acb3-55e9-4b66-8d01-80b77fe0da2f" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.27:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 18:03:15.749835 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:15.749795    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-z6kzq" podUID="62c0acb3-55e9-4b66-8d01-80b77fe0da2f" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.27:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 18:03:20.043456 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:20.043417    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-z6kzq"
Sep 17 18:03:23.449091 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:23.449049    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-z6kzq"]
Sep 17 18:03:23.449566 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:23.449304    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-z6kzq" podUID="62c0acb3-55e9-4b66-8d01-80b77fe0da2f" containerName="kserve-container" containerID="cri-o://abb88eca40f22f466f77522960ef044800e92e886fe635b20012c501ba52292b" gracePeriod=30
Sep 17 18:03:23.449566 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:23.449388    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-z6kzq" podUID="62c0acb3-55e9-4b66-8d01-80b77fe0da2f" containerName="kube-rbac-proxy" containerID="cri-o://8a5afcd783b6bd55e311ca9cedd1456b9cc448967b96a97c3282fd8c64453daf" gracePeriod=30
Sep 17 18:03:24.079768 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:24.079738    2569 generic.go:356] "Generic (PLEG): container finished" podID="62c0acb3-55e9-4b66-8d01-80b77fe0da2f" containerID="8a5afcd783b6bd55e311ca9cedd1456b9cc448967b96a97c3282fd8c64453daf" exitCode=2
Sep 17 18:03:24.079940 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:24.079816    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-z6kzq" event={"ID":"62c0acb3-55e9-4b66-8d01-80b77fe0da2f","Type":"ContainerDied","Data":"8a5afcd783b6bd55e311ca9cedd1456b9cc448967b96a97c3282fd8c64453daf"}
Sep 17 18:03:25.743008 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:25.742953    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-z6kzq" podUID="62c0acb3-55e9-4b66-8d01-80b77fe0da2f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Sep 17 18:03:28.789920 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:28.789898    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-z6kzq"
Sep 17 18:03:28.887822 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:28.887793    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62c0acb3-55e9-4b66-8d01-80b77fe0da2f-cabundle-cert\") pod \"62c0acb3-55e9-4b66-8d01-80b77fe0da2f\" (UID: \"62c0acb3-55e9-4b66-8d01-80b77fe0da2f\") "
Sep 17 18:03:28.888005 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:28.887834    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62c0acb3-55e9-4b66-8d01-80b77fe0da2f-proxy-tls\") pod \"62c0acb3-55e9-4b66-8d01-80b77fe0da2f\" (UID: \"62c0acb3-55e9-4b66-8d01-80b77fe0da2f\") "
Sep 17 18:03:28.888005 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:28.887852    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/62c0acb3-55e9-4b66-8d01-80b77fe0da2f-kube-rbac-proxy-sar-config\") pod \"62c0acb3-55e9-4b66-8d01-80b77fe0da2f\" (UID: \"62c0acb3-55e9-4b66-8d01-80b77fe0da2f\") "
Sep 17 18:03:28.888005 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:28.887871    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62c0acb3-55e9-4b66-8d01-80b77fe0da2f-kserve-provision-location\") pod \"62c0acb3-55e9-4b66-8d01-80b77fe0da2f\" (UID: \"62c0acb3-55e9-4b66-8d01-80b77fe0da2f\") "
Sep 17 18:03:28.888005 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:28.887893    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h8nx5\" (UniqueName: \"kubernetes.io/projected/62c0acb3-55e9-4b66-8d01-80b77fe0da2f-kube-api-access-h8nx5\") pod \"62c0acb3-55e9-4b66-8d01-80b77fe0da2f\" (UID: \"62c0acb3-55e9-4b66-8d01-80b77fe0da2f\") "
Sep 17 18:03:28.888254 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:28.888215    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62c0acb3-55e9-4b66-8d01-80b77fe0da2f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "62c0acb3-55e9-4b66-8d01-80b77fe0da2f" (UID: "62c0acb3-55e9-4b66-8d01-80b77fe0da2f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:03:28.888254 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:28.888242    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62c0acb3-55e9-4b66-8d01-80b77fe0da2f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "62c0acb3-55e9-4b66-8d01-80b77fe0da2f" (UID: "62c0acb3-55e9-4b66-8d01-80b77fe0da2f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:03:28.888336 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:28.888261    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62c0acb3-55e9-4b66-8d01-80b77fe0da2f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "62c0acb3-55e9-4b66-8d01-80b77fe0da2f" (UID: "62c0acb3-55e9-4b66-8d01-80b77fe0da2f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:03:28.889919 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:28.889898    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62c0acb3-55e9-4b66-8d01-80b77fe0da2f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "62c0acb3-55e9-4b66-8d01-80b77fe0da2f" (UID: "62c0acb3-55e9-4b66-8d01-80b77fe0da2f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:03:28.890040 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:28.890019    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62c0acb3-55e9-4b66-8d01-80b77fe0da2f-kube-api-access-h8nx5" (OuterVolumeSpecName: "kube-api-access-h8nx5") pod "62c0acb3-55e9-4b66-8d01-80b77fe0da2f" (UID: "62c0acb3-55e9-4b66-8d01-80b77fe0da2f"). InnerVolumeSpecName "kube-api-access-h8nx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:03:28.988415 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:28.988376    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62c0acb3-55e9-4b66-8d01-80b77fe0da2f-cabundle-cert\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:03:28.988415 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:28.988410    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62c0acb3-55e9-4b66-8d01-80b77fe0da2f-proxy-tls\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:03:28.988415 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:28.988419    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/62c0acb3-55e9-4b66-8d01-80b77fe0da2f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:03:28.988415 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:28.988428    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62c0acb3-55e9-4b66-8d01-80b77fe0da2f-kserve-provision-location\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:03:28.988669 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:28.988437    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8nx5\" (UniqueName: \"kubernetes.io/projected/62c0acb3-55e9-4b66-8d01-80b77fe0da2f-kube-api-access-h8nx5\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:03:29.102250 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:29.102221    2569 generic.go:356] "Generic (PLEG): container finished" podID="62c0acb3-55e9-4b66-8d01-80b77fe0da2f" containerID="abb88eca40f22f466f77522960ef044800e92e886fe635b20012c501ba52292b" exitCode=0
Sep 17 18:03:29.102399 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:29.102277    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-z6kzq"
Sep 17 18:03:29.102399 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:29.102276    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-z6kzq" event={"ID":"62c0acb3-55e9-4b66-8d01-80b77fe0da2f","Type":"ContainerDied","Data":"abb88eca40f22f466f77522960ef044800e92e886fe635b20012c501ba52292b"}
Sep 17 18:03:29.102470 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:29.102403    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-z6kzq" event={"ID":"62c0acb3-55e9-4b66-8d01-80b77fe0da2f","Type":"ContainerDied","Data":"76a8b9f7bd67a0311920270e3857fcab1d8fd0069a08e681da6c50ab65d4d9ca"}
Sep 17 18:03:29.102470 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:29.102432    2569 scope.go:117] "RemoveContainer" containerID="8a5afcd783b6bd55e311ca9cedd1456b9cc448967b96a97c3282fd8c64453daf"
Sep 17 18:03:29.110526 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:29.110508    2569 scope.go:117] "RemoveContainer" containerID="abb88eca40f22f466f77522960ef044800e92e886fe635b20012c501ba52292b"
Sep 17 18:03:29.118116 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:29.118093    2569 scope.go:117] "RemoveContainer" containerID="98e3a45db5e76fbfb71bf320f761845150505438af5c564958f56e156607c5a7"
Sep 17 18:03:29.125228 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:29.125176    2569 scope.go:117] "RemoveContainer" containerID="8a5afcd783b6bd55e311ca9cedd1456b9cc448967b96a97c3282fd8c64453daf"
Sep 17 18:03:29.125566 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:03:29.125539    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a5afcd783b6bd55e311ca9cedd1456b9cc448967b96a97c3282fd8c64453daf\": container with ID starting with 8a5afcd783b6bd55e311ca9cedd1456b9cc448967b96a97c3282fd8c64453daf not found: ID does not exist" containerID="8a5afcd783b6bd55e311ca9cedd1456b9cc448967b96a97c3282fd8c64453daf"
Sep 17 18:03:29.125655 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:29.125578    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a5afcd783b6bd55e311ca9cedd1456b9cc448967b96a97c3282fd8c64453daf"} err="failed to get container status \"8a5afcd783b6bd55e311ca9cedd1456b9cc448967b96a97c3282fd8c64453daf\": rpc error: code = NotFound desc = could not find container \"8a5afcd783b6bd55e311ca9cedd1456b9cc448967b96a97c3282fd8c64453daf\": container with ID starting with 8a5afcd783b6bd55e311ca9cedd1456b9cc448967b96a97c3282fd8c64453daf not found: ID does not exist"
Sep 17 18:03:29.125655 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:29.125605    2569 scope.go:117] "RemoveContainer" containerID="abb88eca40f22f466f77522960ef044800e92e886fe635b20012c501ba52292b"
Sep 17 18:03:29.125886 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:03:29.125863    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abb88eca40f22f466f77522960ef044800e92e886fe635b20012c501ba52292b\": container with ID starting with abb88eca40f22f466f77522960ef044800e92e886fe635b20012c501ba52292b not found: ID does not exist" containerID="abb88eca40f22f466f77522960ef044800e92e886fe635b20012c501ba52292b"
Sep 17 18:03:29.125936 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:29.125893    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abb88eca40f22f466f77522960ef044800e92e886fe635b20012c501ba52292b"} err="failed to get container status \"abb88eca40f22f466f77522960ef044800e92e886fe635b20012c501ba52292b\": rpc error: code = NotFound desc = could not find container \"abb88eca40f22f466f77522960ef044800e92e886fe635b20012c501ba52292b\": container with ID starting with abb88eca40f22f466f77522960ef044800e92e886fe635b20012c501ba52292b not found: ID does not exist"
Sep 17 18:03:29.125936 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:29.125910    2569 scope.go:117] "RemoveContainer" containerID="98e3a45db5e76fbfb71bf320f761845150505438af5c564958f56e156607c5a7"
Sep 17 18:03:29.126181 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:03:29.126165    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98e3a45db5e76fbfb71bf320f761845150505438af5c564958f56e156607c5a7\": container with ID starting with 98e3a45db5e76fbfb71bf320f761845150505438af5c564958f56e156607c5a7 not found: ID does not exist" containerID="98e3a45db5e76fbfb71bf320f761845150505438af5c564958f56e156607c5a7"
Sep 17 18:03:29.126236 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:29.126185    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98e3a45db5e76fbfb71bf320f761845150505438af5c564958f56e156607c5a7"} err="failed to get container status \"98e3a45db5e76fbfb71bf320f761845150505438af5c564958f56e156607c5a7\": rpc error: code = NotFound desc = could not find container \"98e3a45db5e76fbfb71bf320f761845150505438af5c564958f56e156607c5a7\": container with ID starting with 98e3a45db5e76fbfb71bf320f761845150505438af5c564958f56e156607c5a7 not found: ID does not exist"
Sep 17 18:03:29.127005 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:29.126987    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-z6kzq"]
Sep 17 18:03:29.131590 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:29.131566    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-z6kzq"]
Sep 17 18:03:29.508777 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:29.508747    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62c0acb3-55e9-4b66-8d01-80b77fe0da2f" path="/var/lib/kubelet/pods/62c0acb3-55e9-4b66-8d01-80b77fe0da2f/volumes"
Sep 17 18:03:46.131132 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:46.131098    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/curl-metrics-1ba573f4"]
Sep 17 18:03:46.131515 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:46.131364    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62c0acb3-55e9-4b66-8d01-80b77fe0da2f" containerName="storage-initializer"
Sep 17 18:03:46.131515 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:46.131375    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="62c0acb3-55e9-4b66-8d01-80b77fe0da2f" containerName="storage-initializer"
Sep 17 18:03:46.131515 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:46.131385    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62c0acb3-55e9-4b66-8d01-80b77fe0da2f" containerName="kube-rbac-proxy"
Sep 17 18:03:46.131515 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:46.131391    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="62c0acb3-55e9-4b66-8d01-80b77fe0da2f" containerName="kube-rbac-proxy"
Sep 17 18:03:46.131515 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:46.131414    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62c0acb3-55e9-4b66-8d01-80b77fe0da2f" containerName="kserve-container"
Sep 17 18:03:46.131515 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:46.131420    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="62c0acb3-55e9-4b66-8d01-80b77fe0da2f" containerName="kserve-container"
Sep 17 18:03:46.131515 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:46.131462    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="62c0acb3-55e9-4b66-8d01-80b77fe0da2f" containerName="kube-rbac-proxy"
Sep 17 18:03:46.131515 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:46.131472    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="62c0acb3-55e9-4b66-8d01-80b77fe0da2f" containerName="kserve-container"
Sep 17 18:03:46.134262 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:46.134246    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-1ba573f4"
Sep 17 18:03:46.137441 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:46.137417    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 18:03:46.137866 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:46.137851    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 18:03:46.138823 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:46.138801    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-dcrjv\""
Sep 17 18:03:46.140532 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:46.140505    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-1ba573f4"]
Sep 17 18:03:46.226591 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:46.226567    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp7hn\" (UniqueName: \"kubernetes.io/projected/15d97d6b-e110-4723-8910-8eae3e60f123-kube-api-access-lp7hn\") pod \"curl-metrics-1ba573f4\" (UID: \"15d97d6b-e110-4723-8910-8eae3e60f123\") " pod="kserve/curl-metrics-1ba573f4"
Sep 17 18:03:46.327830 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:46.327801    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp7hn\" (UniqueName: \"kubernetes.io/projected/15d97d6b-e110-4723-8910-8eae3e60f123-kube-api-access-lp7hn\") pod \"curl-metrics-1ba573f4\" (UID: \"15d97d6b-e110-4723-8910-8eae3e60f123\") " pod="kserve/curl-metrics-1ba573f4"
Sep 17 18:03:46.338994 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:46.338973    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp7hn\" (UniqueName: \"kubernetes.io/projected/15d97d6b-e110-4723-8910-8eae3e60f123-kube-api-access-lp7hn\") pod \"curl-metrics-1ba573f4\" (UID: \"15d97d6b-e110-4723-8910-8eae3e60f123\") " pod="kserve/curl-metrics-1ba573f4"
Sep 17 18:03:46.456923 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:46.456897    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-1ba573f4"
Sep 17 18:03:46.739253 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:46.739221    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-1ba573f4"]
Sep 17 18:03:46.741743 ip-10-0-131-14 kubenswrapper[2569]: W0917 18:03:46.741714    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod15d97d6b_e110_4723_8910_8eae3e60f123.slice/crio-18df7b4147fc42a2352b32573995302e67eba448d4c97795dd47f991089c5a03 WatchSource:0}: Error finding container 18df7b4147fc42a2352b32573995302e67eba448d4c97795dd47f991089c5a03: Status 404 returned error can't find the container with id 18df7b4147fc42a2352b32573995302e67eba448d4c97795dd47f991089c5a03
Sep 17 18:03:46.743727 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:46.743710    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:03:47.172630 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:47.172534    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-1ba573f4" event={"ID":"15d97d6b-e110-4723-8910-8eae3e60f123","Type":"ContainerStarted","Data":"18df7b4147fc42a2352b32573995302e67eba448d4c97795dd47f991089c5a03"}
Sep 17 18:03:48.178626 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:48.178596    2569 generic.go:356] "Generic (PLEG): container finished" podID="15d97d6b-e110-4723-8910-8eae3e60f123" containerID="9f52b17bf11058d44245f92a8ea807ccf1029087db517d70f4c3a4272a2e77bf" exitCode=0
Sep 17 18:03:48.179006 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:48.178686    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-1ba573f4" event={"ID":"15d97d6b-e110-4723-8910-8eae3e60f123","Type":"ContainerDied","Data":"9f52b17bf11058d44245f92a8ea807ccf1029087db517d70f4c3a4272a2e77bf"}
Sep 17 18:03:49.306888 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:49.306861    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-1ba573f4"
Sep 17 18:03:49.386953 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:49.386921    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_curl-metrics-1ba573f4_15d97d6b-e110-4723-8910-8eae3e60f123/curl-metrics-1ba573f4/0.log"
Sep 17 18:03:49.413869 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:49.413841    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/curl-metrics-1ba573f4"]
Sep 17 18:03:49.417391 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:49.417359    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/curl-metrics-1ba573f4"]
Sep 17 18:03:49.452395 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:49.452367    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lp7hn\" (UniqueName: \"kubernetes.io/projected/15d97d6b-e110-4723-8910-8eae3e60f123-kube-api-access-lp7hn\") pod \"15d97d6b-e110-4723-8910-8eae3e60f123\" (UID: \"15d97d6b-e110-4723-8910-8eae3e60f123\") "
Sep 17 18:03:49.454444 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:49.454424    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15d97d6b-e110-4723-8910-8eae3e60f123-kube-api-access-lp7hn" (OuterVolumeSpecName: "kube-api-access-lp7hn") pod "15d97d6b-e110-4723-8910-8eae3e60f123" (UID: "15d97d6b-e110-4723-8910-8eae3e60f123"). InnerVolumeSpecName "kube-api-access-lp7hn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:03:49.509059 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:49.509026    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15d97d6b-e110-4723-8910-8eae3e60f123" path="/var/lib/kubelet/pods/15d97d6b-e110-4723-8910-8eae3e60f123/volumes"
Sep 17 18:03:49.553011 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:49.552984    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp7hn\" (UniqueName: \"kubernetes.io/projected/15d97d6b-e110-4723-8910-8eae3e60f123-kube-api-access-lp7hn\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:03:50.188841 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:50.188811    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-1ba573f4"
Sep 17 18:03:50.189021 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:50.188875    2569 scope.go:117] "RemoveContainer" containerID="9f52b17bf11058d44245f92a8ea807ccf1029087db517d70f4c3a4272a2e77bf"
Sep 17 18:03:54.810571 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:54.810537    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/curl-metrics-972b3e20"]
Sep 17 18:03:54.811167 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:54.810937    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15d97d6b-e110-4723-8910-8eae3e60f123" containerName="curl-metrics-1ba573f4"
Sep 17 18:03:54.811167 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:54.810956    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="15d97d6b-e110-4723-8910-8eae3e60f123" containerName="curl-metrics-1ba573f4"
Sep 17 18:03:54.811167 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:54.811037    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="15d97d6b-e110-4723-8910-8eae3e60f123" containerName="curl-metrics-1ba573f4"
Sep 17 18:03:54.814784 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:54.814765    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-972b3e20"
Sep 17 18:03:54.817724 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:54.817705    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 18:03:54.819100 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:54.819057    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-dcrjv\""
Sep 17 18:03:54.819210 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:54.819062    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 18:03:54.821031 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:54.821011    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-972b3e20"]
Sep 17 18:03:54.897069 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:54.897041    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2rvd\" (UniqueName: \"kubernetes.io/projected/b77434ee-d703-4678-890a-95edb3b1c555-kube-api-access-n2rvd\") pod \"curl-metrics-972b3e20\" (UID: \"b77434ee-d703-4678-890a-95edb3b1c555\") " pod="kserve/curl-metrics-972b3e20"
Sep 17 18:03:54.997931 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:54.997890    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2rvd\" (UniqueName: \"kubernetes.io/projected/b77434ee-d703-4678-890a-95edb3b1c555-kube-api-access-n2rvd\") pod \"curl-metrics-972b3e20\" (UID: \"b77434ee-d703-4678-890a-95edb3b1c555\") " pod="kserve/curl-metrics-972b3e20"
Sep 17 18:03:55.008039 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:55.008015    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2rvd\" (UniqueName: \"kubernetes.io/projected/b77434ee-d703-4678-890a-95edb3b1c555-kube-api-access-n2rvd\") pod \"curl-metrics-972b3e20\" (UID: \"b77434ee-d703-4678-890a-95edb3b1c555\") " pod="kserve/curl-metrics-972b3e20"
Sep 17 18:03:55.134384 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:55.134302    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-972b3e20"
Sep 17 18:03:55.405467 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:55.405157    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-972b3e20"]
Sep 17 18:03:55.410423 ip-10-0-131-14 kubenswrapper[2569]: W0917 18:03:55.410397    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb77434ee_d703_4678_890a_95edb3b1c555.slice/crio-84a8889976db9c24d261ac0319bf2879a84656f97c91797994805d5ca2cb431c WatchSource:0}: Error finding container 84a8889976db9c24d261ac0319bf2879a84656f97c91797994805d5ca2cb431c: Status 404 returned error can't find the container with id 84a8889976db9c24d261ac0319bf2879a84656f97c91797994805d5ca2cb431c
Sep 17 18:03:56.213904 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:56.213872    2569 generic.go:356] "Generic (PLEG): container finished" podID="b77434ee-d703-4678-890a-95edb3b1c555" containerID="6f5551a7942bad5784f8522baece2b731c0024e1b9080203e0fcce00cd4d21d3" exitCode=0
Sep 17 18:03:56.214299 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:56.213927    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-972b3e20" event={"ID":"b77434ee-d703-4678-890a-95edb3b1c555","Type":"ContainerDied","Data":"6f5551a7942bad5784f8522baece2b731c0024e1b9080203e0fcce00cd4d21d3"}
Sep 17 18:03:56.214299 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:56.213955    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-972b3e20" event={"ID":"b77434ee-d703-4678-890a-95edb3b1c555","Type":"ContainerStarted","Data":"84a8889976db9c24d261ac0319bf2879a84656f97c91797994805d5ca2cb431c"}
Sep 17 18:03:57.343343 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:57.343321    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-972b3e20"
Sep 17 18:03:57.369153 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:57.369132    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_curl-metrics-972b3e20_b77434ee-d703-4678-890a-95edb3b1c555/curl-metrics-972b3e20/0.log"
Sep 17 18:03:57.396289 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:57.396262    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/curl-metrics-972b3e20"]
Sep 17 18:03:57.399476 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:57.399456    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/curl-metrics-972b3e20"]
Sep 17 18:03:57.419578 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:57.419555    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n2rvd\" (UniqueName: \"kubernetes.io/projected/b77434ee-d703-4678-890a-95edb3b1c555-kube-api-access-n2rvd\") pod \"b77434ee-d703-4678-890a-95edb3b1c555\" (UID: \"b77434ee-d703-4678-890a-95edb3b1c555\") "
Sep 17 18:03:57.421536 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:57.421515    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b77434ee-d703-4678-890a-95edb3b1c555-kube-api-access-n2rvd" (OuterVolumeSpecName: "kube-api-access-n2rvd") pod "b77434ee-d703-4678-890a-95edb3b1c555" (UID: "b77434ee-d703-4678-890a-95edb3b1c555"). InnerVolumeSpecName "kube-api-access-n2rvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:03:57.508107 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:57.508026    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b77434ee-d703-4678-890a-95edb3b1c555" path="/var/lib/kubelet/pods/b77434ee-d703-4678-890a-95edb3b1c555/volumes"
Sep 17 18:03:57.520178 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:57.520158    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2rvd\" (UniqueName: \"kubernetes.io/projected/b77434ee-d703-4678-890a-95edb3b1c555-kube-api-access-n2rvd\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:03:58.227802 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:58.227774    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-972b3e20"
Sep 17 18:03:58.227981 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:03:58.227825    2569 scope.go:117] "RemoveContainer" containerID="6f5551a7942bad5784f8522baece2b731c0024e1b9080203e0fcce00cd4d21d3"
Sep 17 18:05:38.118090 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:05:38.118058    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-4r8q4"]
Sep 17 18:05:38.118522 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:05:38.118351    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b77434ee-d703-4678-890a-95edb3b1c555" containerName="curl-metrics-972b3e20"
Sep 17 18:05:38.118522 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:05:38.118362    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b77434ee-d703-4678-890a-95edb3b1c555" containerName="curl-metrics-972b3e20"
Sep 17 18:05:38.118522 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:05:38.118413    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b77434ee-d703-4678-890a-95edb3b1c555" containerName="curl-metrics-972b3e20"
Sep 17 18:05:38.121481 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:05:38.121465    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-4r8q4"
Sep 17 18:05:38.124400 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:05:38.124372    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-predictor-serving-cert\""
Sep 17 18:05:38.124547 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:05:38.124466    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-kkx8f\""
Sep 17 18:05:38.124621 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:05:38.124594    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 17 18:05:38.125159 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:05:38.125142    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:05:38.125228 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:05:38.125148    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:05:38.125774 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:05:38.125756    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:05:38.125853 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:05:38.125758    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:05:38.134420 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:05:38.134396    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-4r8q4"]
Sep 17 18:05:38.253400 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:05:38.253368    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/36dd2595-a01d-4fe6-a0e4-bacafa311d73-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-4r8q4\" (UID: \"36dd2595-a01d-4fe6-a0e4-bacafa311d73\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-4r8q4"
Sep 17 18:05:38.253541 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:05:38.253413    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/36dd2595-a01d-4fe6-a0e4-bacafa311d73-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-4r8q4\" (UID: \"36dd2595-a01d-4fe6-a0e4-bacafa311d73\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-4r8q4"
Sep 17 18:05:38.253541 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:05:38.253449    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8cgh\" (UniqueName: \"kubernetes.io/projected/36dd2595-a01d-4fe6-a0e4-bacafa311d73-kube-api-access-p8cgh\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-4r8q4\" (UID: \"36dd2595-a01d-4fe6-a0e4-bacafa311d73\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-4r8q4"
Sep 17 18:05:38.253541 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:05:38.253477    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/36dd2595-a01d-4fe6-a0e4-bacafa311d73-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-4r8q4\" (UID: \"36dd2595-a01d-4fe6-a0e4-bacafa311d73\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-4r8q4"
Sep 17 18:05:38.253541 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:05:38.253514    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/36dd2595-a01d-4fe6-a0e4-bacafa311d73-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-4r8q4\" (UID: \"36dd2595-a01d-4fe6-a0e4-bacafa311d73\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-4r8q4"
Sep 17 18:05:38.354205 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:05:38.354170    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/36dd2595-a01d-4fe6-a0e4-bacafa311d73-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-4r8q4\" (UID: \"36dd2595-a01d-4fe6-a0e4-bacafa311d73\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-4r8q4"
Sep 17 18:05:38.354359 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:05:38.354214    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/36dd2595-a01d-4fe6-a0e4-bacafa311d73-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-4r8q4\" (UID: \"36dd2595-a01d-4fe6-a0e4-bacafa311d73\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-4r8q4"
Sep 17 18:05:38.354359 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:05:38.354250    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8cgh\" (UniqueName: \"kubernetes.io/projected/36dd2595-a01d-4fe6-a0e4-bacafa311d73-kube-api-access-p8cgh\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-4r8q4\" (UID: \"36dd2595-a01d-4fe6-a0e4-bacafa311d73\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-4r8q4"
Sep 17 18:05:38.354359 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:05:38.354274    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/36dd2595-a01d-4fe6-a0e4-bacafa311d73-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-4r8q4\" (UID: \"36dd2595-a01d-4fe6-a0e4-bacafa311d73\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-4r8q4"
Sep 17 18:05:38.354359 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:05:38.354307    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/36dd2595-a01d-4fe6-a0e4-bacafa311d73-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-4r8q4\" (UID: \"36dd2595-a01d-4fe6-a0e4-bacafa311d73\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-4r8q4"
Sep 17 18:05:38.354576 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:05:38.354443    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-serving-cert: secret "sklearn-v2-mlserver-predictor-serving-cert" not found
Sep 17 18:05:38.354576 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:05:38.354523    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36dd2595-a01d-4fe6-a0e4-bacafa311d73-proxy-tls podName:36dd2595-a01d-4fe6-a0e4-bacafa311d73 nodeName:}" failed. No retries permitted until 2026-09-17 18:05:38.85450125 +0000 UTC m=+1491.358503861 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/36dd2595-a01d-4fe6-a0e4-bacafa311d73-proxy-tls") pod "sklearn-v2-mlserver-predictor-7f568cdc6b-4r8q4" (UID: "36dd2595-a01d-4fe6-a0e4-bacafa311d73") : secret "sklearn-v2-mlserver-predictor-serving-cert" not found
Sep 17 18:05:38.354694 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:05:38.354585    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/36dd2595-a01d-4fe6-a0e4-bacafa311d73-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-4r8q4\" (UID: \"36dd2595-a01d-4fe6-a0e4-bacafa311d73\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-4r8q4"
Sep 17 18:05:38.354926 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:05:38.354905    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/36dd2595-a01d-4fe6-a0e4-bacafa311d73-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-4r8q4\" (UID: \"36dd2595-a01d-4fe6-a0e4-bacafa311d73\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-4r8q4"
Sep 17 18:05:38.354994 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:05:38.354939    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/36dd2595-a01d-4fe6-a0e4-bacafa311d73-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-4r8q4\" (UID: \"36dd2595-a01d-4fe6-a0e4-bacafa311d73\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-4r8q4"
Sep 17 18:05:38.372645 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:05:38.372590    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8cgh\" (UniqueName: \"kubernetes.io/projected/36dd2595-a01d-4fe6-a0e4-bacafa311d73-kube-api-access-p8cgh\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-4r8q4\" (UID: \"36dd2595-a01d-4fe6-a0e4-bacafa311d73\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-4r8q4"
Sep 17 18:05:38.858631 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:05:38.858596    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/36dd2595-a01d-4fe6-a0e4-bacafa311d73-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-4r8q4\" (UID: \"36dd2595-a01d-4fe6-a0e4-bacafa311d73\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-4r8q4"
Sep 17 18:05:38.860823 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:05:38.860803    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/36dd2595-a01d-4fe6-a0e4-bacafa311d73-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-4r8q4\" (UID: \"36dd2595-a01d-4fe6-a0e4-bacafa311d73\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-4r8q4"
Sep 17 18:05:39.031473 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:05:39.031441    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-4r8q4"
Sep 17 18:05:39.316905 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:05:39.316883    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-4r8q4"]
Sep 17 18:05:39.319276 ip-10-0-131-14 kubenswrapper[2569]: W0917 18:05:39.319248    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36dd2595_a01d_4fe6_a0e4_bacafa311d73.slice/crio-71b8c9c58970c7b27f75f18254aca7e491fe1296d6e57da40fdb5db1d948f08e WatchSource:0}: Error finding container 71b8c9c58970c7b27f75f18254aca7e491fe1296d6e57da40fdb5db1d948f08e: Status 404 returned error can't find the container with id 71b8c9c58970c7b27f75f18254aca7e491fe1296d6e57da40fdb5db1d948f08e
Sep 17 18:05:39.607117 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:05:39.607017    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-4r8q4" event={"ID":"36dd2595-a01d-4fe6-a0e4-bacafa311d73","Type":"ContainerStarted","Data":"cb8475b80d143be124482d2f34a0666896c1d4afaa811c5a5a5cdc5c88bd9a3f"}
Sep 17 18:05:39.607117 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:05:39.607110    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-4r8q4" event={"ID":"36dd2595-a01d-4fe6-a0e4-bacafa311d73","Type":"ContainerStarted","Data":"71b8c9c58970c7b27f75f18254aca7e491fe1296d6e57da40fdb5db1d948f08e"}
Sep 17 18:05:43.623302 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:05:43.623271    2569 generic.go:356] "Generic (PLEG): container finished" podID="36dd2595-a01d-4fe6-a0e4-bacafa311d73" containerID="cb8475b80d143be124482d2f34a0666896c1d4afaa811c5a5a5cdc5c88bd9a3f" exitCode=0
Sep 17 18:05:43.623643 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:05:43.623343    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-4r8q4" event={"ID":"36dd2595-a01d-4fe6-a0e4-bacafa311d73","Type":"ContainerDied","Data":"cb8475b80d143be124482d2f34a0666896c1d4afaa811c5a5a5cdc5c88bd9a3f"}
Sep 17 18:05:49.550435 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:05:49.550404    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-t7pqb_b8d76e28-4655-4d21-98d8-0324e4756aaf/console-operator/2.log"
Sep 17 18:05:49.552797 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:05:49.552769    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-t7pqb_b8d76e28-4655-4d21-98d8-0324e4756aaf/console-operator/2.log"
Sep 17 18:05:49.556829 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:05:49.556810    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k4mqn_6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff/ovn-acl-logging/0.log"
Sep 17 18:05:49.559491 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:05:49.559470    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k4mqn_6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff/ovn-acl-logging/0.log"
Sep 17 18:07:59.247778 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:07:59.247736    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-4r8q4" event={"ID":"36dd2595-a01d-4fe6-a0e4-bacafa311d73","Type":"ContainerStarted","Data":"c06f505d529ee38b6fee601b1ba36915f0487e06ef0e20aa6373949a98d36dff"}
Sep 17 18:07:59.248297 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:07:59.247810    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-4r8q4" event={"ID":"36dd2595-a01d-4fe6-a0e4-bacafa311d73","Type":"ContainerStarted","Data":"195970df91a0862e3d434593a06a50d6f268ddedafe778438e39cd6420a3bf18"}
Sep 17 18:07:59.248297 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:07:59.247853    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-4r8q4"
Sep 17 18:07:59.271231 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:07:59.271161    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-4r8q4" podStartSLOduration=2.5597109749999998 podStartE2EDuration="2m21.271145992s" totalImagesPullingTime="2m14.905371455s" totalInitContainerRuntime="3.806063562s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:05:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:07:59.269600273 +0000 UTC m=+1631.773602911" watchObservedRunningTime="2026-09-17 18:07:59.271145992 +0000 UTC m=+1631.775148617"
Sep 17 18:08:00.251647 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:08:00.251614    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-4r8q4"
Sep 17 18:08:06.260649 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:08:06.260619    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-4r8q4"
Sep 17 18:08:36.341046 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:08:36.340995    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-4r8q4" podUID="36dd2595-a01d-4fe6-a0e4-bacafa311d73" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 17 18:08:46.264017 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:08:46.263985    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-4r8q4"
Sep 17 18:08:48.223622 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:08:48.223589    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-4r8q4"]
Sep 17 18:08:48.223987 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:08:48.223860    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-4r8q4" podUID="36dd2595-a01d-4fe6-a0e4-bacafa311d73" containerName="kserve-container" containerID="cri-o://195970df91a0862e3d434593a06a50d6f268ddedafe778438e39cd6420a3bf18" gracePeriod=30
Sep 17 18:08:48.223987 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:08:48.223904    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-4r8q4" podUID="36dd2595-a01d-4fe6-a0e4-bacafa311d73" containerName="kube-rbac-proxy" containerID="cri-o://c06f505d529ee38b6fee601b1ba36915f0487e06ef0e20aa6373949a98d36dff" gracePeriod=30
Sep 17 18:08:48.442164 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:08:48.442130    2569 generic.go:356] "Generic (PLEG): container finished" podID="36dd2595-a01d-4fe6-a0e4-bacafa311d73" containerID="c06f505d529ee38b6fee601b1ba36915f0487e06ef0e20aa6373949a98d36dff" exitCode=2
Sep 17 18:08:48.442332 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:08:48.442208    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-4r8q4" event={"ID":"36dd2595-a01d-4fe6-a0e4-bacafa311d73","Type":"ContainerDied","Data":"c06f505d529ee38b6fee601b1ba36915f0487e06ef0e20aa6373949a98d36dff"}
Sep 17 18:08:51.256056 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:08:51.256016    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-4r8q4" podUID="36dd2595-a01d-4fe6-a0e4-bacafa311d73" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Sep 17 18:08:56.067880 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:08:56.067854    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-4r8q4"
Sep 17 18:08:56.130341 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:08:56.130249    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/36dd2595-a01d-4fe6-a0e4-bacafa311d73-kserve-provision-location\") pod \"36dd2595-a01d-4fe6-a0e4-bacafa311d73\" (UID: \"36dd2595-a01d-4fe6-a0e4-bacafa311d73\") "
Sep 17 18:08:56.130341 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:08:56.130290    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p8cgh\" (UniqueName: \"kubernetes.io/projected/36dd2595-a01d-4fe6-a0e4-bacafa311d73-kube-api-access-p8cgh\") pod \"36dd2595-a01d-4fe6-a0e4-bacafa311d73\" (UID: \"36dd2595-a01d-4fe6-a0e4-bacafa311d73\") "
Sep 17 18:08:56.130341 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:08:56.130342    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/36dd2595-a01d-4fe6-a0e4-bacafa311d73-cabundle-cert\") pod \"36dd2595-a01d-4fe6-a0e4-bacafa311d73\" (UID: \"36dd2595-a01d-4fe6-a0e4-bacafa311d73\") "
Sep 17 18:08:56.130607 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:08:56.130374    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/36dd2595-a01d-4fe6-a0e4-bacafa311d73-kube-rbac-proxy-sar-config\") pod \"36dd2595-a01d-4fe6-a0e4-bacafa311d73\" (UID: \"36dd2595-a01d-4fe6-a0e4-bacafa311d73\") "
Sep 17 18:08:56.130607 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:08:56.130394    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/36dd2595-a01d-4fe6-a0e4-bacafa311d73-proxy-tls\") pod \"36dd2595-a01d-4fe6-a0e4-bacafa311d73\" (UID: \"36dd2595-a01d-4fe6-a0e4-bacafa311d73\") "
Sep 17 18:08:56.130715 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:08:56.130692    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36dd2595-a01d-4fe6-a0e4-bacafa311d73-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "36dd2595-a01d-4fe6-a0e4-bacafa311d73" (UID: "36dd2595-a01d-4fe6-a0e4-bacafa311d73"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:08:56.130799 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:08:56.130774    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36dd2595-a01d-4fe6-a0e4-bacafa311d73-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "36dd2595-a01d-4fe6-a0e4-bacafa311d73" (UID: "36dd2595-a01d-4fe6-a0e4-bacafa311d73"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:08:56.130851 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:08:56.130782    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36dd2595-a01d-4fe6-a0e4-bacafa311d73-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "36dd2595-a01d-4fe6-a0e4-bacafa311d73" (UID: "36dd2595-a01d-4fe6-a0e4-bacafa311d73"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:08:56.132445 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:08:56.132419    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36dd2595-a01d-4fe6-a0e4-bacafa311d73-kube-api-access-p8cgh" (OuterVolumeSpecName: "kube-api-access-p8cgh") pod "36dd2595-a01d-4fe6-a0e4-bacafa311d73" (UID: "36dd2595-a01d-4fe6-a0e4-bacafa311d73"). InnerVolumeSpecName "kube-api-access-p8cgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:08:56.132544 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:08:56.132524    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36dd2595-a01d-4fe6-a0e4-bacafa311d73-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "36dd2595-a01d-4fe6-a0e4-bacafa311d73" (UID: "36dd2595-a01d-4fe6-a0e4-bacafa311d73"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:08:56.231304 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:08:56.231267    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/36dd2595-a01d-4fe6-a0e4-bacafa311d73-kserve-provision-location\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:08:56.231304 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:08:56.231296    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8cgh\" (UniqueName: \"kubernetes.io/projected/36dd2595-a01d-4fe6-a0e4-bacafa311d73-kube-api-access-p8cgh\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:08:56.231304 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:08:56.231308    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/36dd2595-a01d-4fe6-a0e4-bacafa311d73-cabundle-cert\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:08:56.231304 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:08:56.231317    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/36dd2595-a01d-4fe6-a0e4-bacafa311d73-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:08:56.231586 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:08:56.231325    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/36dd2595-a01d-4fe6-a0e4-bacafa311d73-proxy-tls\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:08:56.478900 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:08:56.478868    2569 generic.go:356] "Generic (PLEG): container finished" podID="36dd2595-a01d-4fe6-a0e4-bacafa311d73" containerID="195970df91a0862e3d434593a06a50d6f268ddedafe778438e39cd6420a3bf18" exitCode=0
Sep 17 18:08:56.479135 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:08:56.478924    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-4r8q4" event={"ID":"36dd2595-a01d-4fe6-a0e4-bacafa311d73","Type":"ContainerDied","Data":"195970df91a0862e3d434593a06a50d6f268ddedafe778438e39cd6420a3bf18"}
Sep 17 18:08:56.479135 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:08:56.478937    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-4r8q4"
Sep 17 18:08:56.479135 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:08:56.478975    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-4r8q4" event={"ID":"36dd2595-a01d-4fe6-a0e4-bacafa311d73","Type":"ContainerDied","Data":"71b8c9c58970c7b27f75f18254aca7e491fe1296d6e57da40fdb5db1d948f08e"}
Sep 17 18:08:56.479135 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:08:56.479002    2569 scope.go:117] "RemoveContainer" containerID="c06f505d529ee38b6fee601b1ba36915f0487e06ef0e20aa6373949a98d36dff"
Sep 17 18:08:56.488269 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:08:56.488243    2569 scope.go:117] "RemoveContainer" containerID="195970df91a0862e3d434593a06a50d6f268ddedafe778438e39cd6420a3bf18"
Sep 17 18:08:56.496000 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:08:56.495983    2569 scope.go:117] "RemoveContainer" containerID="cb8475b80d143be124482d2f34a0666896c1d4afaa811c5a5a5cdc5c88bd9a3f"
Sep 17 18:08:56.502825 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:08:56.502800    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-4r8q4"]
Sep 17 18:08:56.503477 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:08:56.503464    2569 scope.go:117] "RemoveContainer" containerID="c06f505d529ee38b6fee601b1ba36915f0487e06ef0e20aa6373949a98d36dff"
Sep 17 18:08:56.503762 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:08:56.503744    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c06f505d529ee38b6fee601b1ba36915f0487e06ef0e20aa6373949a98d36dff\": container with ID starting with c06f505d529ee38b6fee601b1ba36915f0487e06ef0e20aa6373949a98d36dff not found: ID does not exist" containerID="c06f505d529ee38b6fee601b1ba36915f0487e06ef0e20aa6373949a98d36dff"
Sep 17 18:08:56.503807 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:08:56.503771    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c06f505d529ee38b6fee601b1ba36915f0487e06ef0e20aa6373949a98d36dff"} err="failed to get container status \"c06f505d529ee38b6fee601b1ba36915f0487e06ef0e20aa6373949a98d36dff\": rpc error: code = NotFound desc = could not find container \"c06f505d529ee38b6fee601b1ba36915f0487e06ef0e20aa6373949a98d36dff\": container with ID starting with c06f505d529ee38b6fee601b1ba36915f0487e06ef0e20aa6373949a98d36dff not found: ID does not exist"
Sep 17 18:08:56.503807 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:08:56.503789    2569 scope.go:117] "RemoveContainer" containerID="195970df91a0862e3d434593a06a50d6f268ddedafe778438e39cd6420a3bf18"
Sep 17 18:08:56.504029 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:08:56.504011    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"195970df91a0862e3d434593a06a50d6f268ddedafe778438e39cd6420a3bf18\": container with ID starting with 195970df91a0862e3d434593a06a50d6f268ddedafe778438e39cd6420a3bf18 not found: ID does not exist" containerID="195970df91a0862e3d434593a06a50d6f268ddedafe778438e39cd6420a3bf18"
Sep 17 18:08:56.504112 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:08:56.504037    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"195970df91a0862e3d434593a06a50d6f268ddedafe778438e39cd6420a3bf18"} err="failed to get container status \"195970df91a0862e3d434593a06a50d6f268ddedafe778438e39cd6420a3bf18\": rpc error: code = NotFound desc = could not find container \"195970df91a0862e3d434593a06a50d6f268ddedafe778438e39cd6420a3bf18\": container with ID starting with 195970df91a0862e3d434593a06a50d6f268ddedafe778438e39cd6420a3bf18 not found: ID does not exist"
Sep 17 18:08:56.504112 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:08:56.504058    2569 scope.go:117] "RemoveContainer" containerID="cb8475b80d143be124482d2f34a0666896c1d4afaa811c5a5a5cdc5c88bd9a3f"
Sep 17 18:08:56.504331 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:08:56.504304    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb8475b80d143be124482d2f34a0666896c1d4afaa811c5a5a5cdc5c88bd9a3f\": container with ID starting with cb8475b80d143be124482d2f34a0666896c1d4afaa811c5a5a5cdc5c88bd9a3f not found: ID does not exist" containerID="cb8475b80d143be124482d2f34a0666896c1d4afaa811c5a5a5cdc5c88bd9a3f"
Sep 17 18:08:56.504331 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:08:56.504318    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb8475b80d143be124482d2f34a0666896c1d4afaa811c5a5a5cdc5c88bd9a3f"} err="failed to get container status \"cb8475b80d143be124482d2f34a0666896c1d4afaa811c5a5a5cdc5c88bd9a3f\": rpc error: code = NotFound desc = could not find container \"cb8475b80d143be124482d2f34a0666896c1d4afaa811c5a5a5cdc5c88bd9a3f\": container with ID starting with cb8475b80d143be124482d2f34a0666896c1d4afaa811c5a5a5cdc5c88bd9a3f not found: ID does not exist"
Sep 17 18:08:56.507205 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:08:56.507183    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-4r8q4"]
Sep 17 18:08:57.508012 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:08:57.507980    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36dd2595-a01d-4fe6-a0e4-bacafa311d73" path="/var/lib/kubelet/pods/36dd2595-a01d-4fe6-a0e4-bacafa311d73/volumes"
Sep 17 18:09:30.493524 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:09:30.493490    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-khfr8"]
Sep 17 18:09:30.494017 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:09:30.493772    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36dd2595-a01d-4fe6-a0e4-bacafa311d73" containerName="kserve-container"
Sep 17 18:09:30.494017 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:09:30.493791    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="36dd2595-a01d-4fe6-a0e4-bacafa311d73" containerName="kserve-container"
Sep 17 18:09:30.494017 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:09:30.493807    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36dd2595-a01d-4fe6-a0e4-bacafa311d73" containerName="storage-initializer"
Sep 17 18:09:30.494017 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:09:30.493813    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="36dd2595-a01d-4fe6-a0e4-bacafa311d73" containerName="storage-initializer"
Sep 17 18:09:30.494017 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:09:30.493819    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36dd2595-a01d-4fe6-a0e4-bacafa311d73" containerName="kube-rbac-proxy"
Sep 17 18:09:30.494017 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:09:30.493825    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="36dd2595-a01d-4fe6-a0e4-bacafa311d73" containerName="kube-rbac-proxy"
Sep 17 18:09:30.494017 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:09:30.493879    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="36dd2595-a01d-4fe6-a0e4-bacafa311d73" containerName="kube-rbac-proxy"
Sep 17 18:09:30.494017 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:09:30.493885    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="36dd2595-a01d-4fe6-a0e4-bacafa311d73" containerName="kserve-container"
Sep 17 18:09:30.497085 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:09:30.497060    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-khfr8"
Sep 17 18:09:30.500239 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:09:30.500214    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:09:30.500380 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:09:30.500214    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:09:30.500380 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:09:30.500220    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-kkx8f\""
Sep 17 18:09:30.500380 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:09:30.500220    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:09:30.501179 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:09:30.501163    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-predictor-serving-cert\""
Sep 17 18:09:30.501179 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:09:30.501171    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 17 18:09:30.501299 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:09:30.501265    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:09:30.508600 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:09:30.508579    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-khfr8"]
Sep 17 18:09:30.523867 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:09:30.523837    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ff23b0f-f993-4384-90e4-83aacac37842-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-khfr8\" (UID: \"0ff23b0f-f993-4384-90e4-83aacac37842\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-khfr8"
Sep 17 18:09:30.524015 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:09:30.523877    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g47wt\" (UniqueName: \"kubernetes.io/projected/0ff23b0f-f993-4384-90e4-83aacac37842-kube-api-access-g47wt\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-khfr8\" (UID: \"0ff23b0f-f993-4384-90e4-83aacac37842\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-khfr8"
Sep 17 18:09:30.524015 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:09:30.523920    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0ff23b0f-f993-4384-90e4-83aacac37842-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-khfr8\" (UID: \"0ff23b0f-f993-4384-90e4-83aacac37842\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-khfr8"
Sep 17 18:09:30.524015 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:09:30.523962    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0ff23b0f-f993-4384-90e4-83aacac37842-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-khfr8\" (UID: \"0ff23b0f-f993-4384-90e4-83aacac37842\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-khfr8"
Sep 17 18:09:30.524174 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:09:30.524015    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ff23b0f-f993-4384-90e4-83aacac37842-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-khfr8\" (UID: \"0ff23b0f-f993-4384-90e4-83aacac37842\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-khfr8"
Sep 17 18:09:30.624700 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:09:30.624669    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ff23b0f-f993-4384-90e4-83aacac37842-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-khfr8\" (UID: \"0ff23b0f-f993-4384-90e4-83aacac37842\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-khfr8"
Sep 17 18:09:30.624914 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:09:30.624710    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g47wt\" (UniqueName: \"kubernetes.io/projected/0ff23b0f-f993-4384-90e4-83aacac37842-kube-api-access-g47wt\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-khfr8\" (UID: \"0ff23b0f-f993-4384-90e4-83aacac37842\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-khfr8"
Sep 17 18:09:30.624914 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:09:30.624740    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0ff23b0f-f993-4384-90e4-83aacac37842-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-khfr8\" (UID: \"0ff23b0f-f993-4384-90e4-83aacac37842\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-khfr8"
Sep 17 18:09:30.624914 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:09:30.624776    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0ff23b0f-f993-4384-90e4-83aacac37842-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-khfr8\" (UID: \"0ff23b0f-f993-4384-90e4-83aacac37842\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-khfr8"
Sep 17 18:09:30.624914 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:09:30.624795    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ff23b0f-f993-4384-90e4-83aacac37842-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-khfr8\" (UID: \"0ff23b0f-f993-4384-90e4-83aacac37842\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-khfr8"
Sep 17 18:09:30.625200 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:09:30.625179    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ff23b0f-f993-4384-90e4-83aacac37842-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-khfr8\" (UID: \"0ff23b0f-f993-4384-90e4-83aacac37842\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-khfr8"
Sep 17 18:09:30.625485 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:09:30.625462    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0ff23b0f-f993-4384-90e4-83aacac37842-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-khfr8\" (UID: \"0ff23b0f-f993-4384-90e4-83aacac37842\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-khfr8"
Sep 17 18:09:30.625532 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:09:30.625512    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0ff23b0f-f993-4384-90e4-83aacac37842-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-khfr8\" (UID: \"0ff23b0f-f993-4384-90e4-83aacac37842\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-khfr8"
Sep 17 18:09:30.627369 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:09:30.627353    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ff23b0f-f993-4384-90e4-83aacac37842-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-khfr8\" (UID: \"0ff23b0f-f993-4384-90e4-83aacac37842\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-khfr8"
Sep 17 18:09:30.634092 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:09:30.634056    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g47wt\" (UniqueName: \"kubernetes.io/projected/0ff23b0f-f993-4384-90e4-83aacac37842-kube-api-access-g47wt\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-khfr8\" (UID: \"0ff23b0f-f993-4384-90e4-83aacac37842\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-khfr8"
Sep 17 18:09:30.807664 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:09:30.807569    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-khfr8"
Sep 17 18:09:31.083684 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:09:31.083637    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-khfr8"]
Sep 17 18:09:31.086350 ip-10-0-131-14 kubenswrapper[2569]: W0917 18:09:31.086320    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ff23b0f_f993_4384_90e4_83aacac37842.slice/crio-3839da6c3bba0711177f51fabd65342381739a23e5b41933cf3b3192896e9db1 WatchSource:0}: Error finding container 3839da6c3bba0711177f51fabd65342381739a23e5b41933cf3b3192896e9db1: Status 404 returned error can't find the container with id 3839da6c3bba0711177f51fabd65342381739a23e5b41933cf3b3192896e9db1
Sep 17 18:09:31.621867 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:09:31.621824    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-khfr8" event={"ID":"0ff23b0f-f993-4384-90e4-83aacac37842","Type":"ContainerStarted","Data":"04a07259aca7383565df97851a266374c8d1669a351b1dba7a95c6c189308e7f"}
Sep 17 18:09:31.622326 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:09:31.621901    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-khfr8" event={"ID":"0ff23b0f-f993-4384-90e4-83aacac37842","Type":"ContainerStarted","Data":"3839da6c3bba0711177f51fabd65342381739a23e5b41933cf3b3192896e9db1"}
Sep 17 18:09:35.639182 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:09:35.639143    2569 generic.go:356] "Generic (PLEG): container finished" podID="0ff23b0f-f993-4384-90e4-83aacac37842" containerID="04a07259aca7383565df97851a266374c8d1669a351b1dba7a95c6c189308e7f" exitCode=0
Sep 17 18:09:35.639623 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:09:35.639217    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-khfr8" event={"ID":"0ff23b0f-f993-4384-90e4-83aacac37842","Type":"ContainerDied","Data":"04a07259aca7383565df97851a266374c8d1669a351b1dba7a95c6c189308e7f"}
Sep 17 18:09:36.645260 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:09:36.645227    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-khfr8" event={"ID":"0ff23b0f-f993-4384-90e4-83aacac37842","Type":"ContainerStarted","Data":"9ee0ee38ef6a83def04f73dfb0a648a8b37432b500eada790dd27e22527e7574"}
Sep 17 18:09:36.645720 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:09:36.645294    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-khfr8" event={"ID":"0ff23b0f-f993-4384-90e4-83aacac37842","Type":"ContainerStarted","Data":"f0df7a93822dc36d169386c30b8a00fe37aa96e8a89c5a37e1098c662fe402d2"}
Sep 17 18:09:36.645720 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:09:36.645483    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-khfr8"
Sep 17 18:09:36.691718 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:09:36.691669    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-khfr8" podStartSLOduration=2.657277939 podStartE2EDuration="6.691652411s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.034374472s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:09:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:09:36.686098697 +0000 UTC m=+1729.190101322" watchObservedRunningTime="2026-09-17 18:09:36.691652411 +0000 UTC m=+1729.195655036"
Sep 17 18:09:37.649137 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:09:37.649101    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-khfr8"
Sep 17 18:09:43.657487 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:09:43.657453    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-khfr8"
Sep 17 18:10:13.741479 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:10:13.741434    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-khfr8" podUID="0ff23b0f-f993-4384-90e4-83aacac37842" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 17 18:10:23.660820 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:10:23.660789    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-khfr8"
Sep 17 18:10:30.622942 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:10:30.622911    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-khfr8"]
Sep 17 18:10:30.623424 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:10:30.623143    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-khfr8" podUID="0ff23b0f-f993-4384-90e4-83aacac37842" containerName="kserve-container" containerID="cri-o://f0df7a93822dc36d169386c30b8a00fe37aa96e8a89c5a37e1098c662fe402d2" gracePeriod=30
Sep 17 18:10:30.623424 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:10:30.623211    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-khfr8" podUID="0ff23b0f-f993-4384-90e4-83aacac37842" containerName="kube-rbac-proxy" containerID="cri-o://9ee0ee38ef6a83def04f73dfb0a648a8b37432b500eada790dd27e22527e7574" gracePeriod=30
Sep 17 18:10:30.861127 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:10:30.861071    2569 generic.go:356] "Generic (PLEG): container finished" podID="0ff23b0f-f993-4384-90e4-83aacac37842" containerID="9ee0ee38ef6a83def04f73dfb0a648a8b37432b500eada790dd27e22527e7574" exitCode=2
Sep 17 18:10:30.861296 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:10:30.861145    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-khfr8" event={"ID":"0ff23b0f-f993-4384-90e4-83aacac37842","Type":"ContainerDied","Data":"9ee0ee38ef6a83def04f73dfb0a648a8b37432b500eada790dd27e22527e7574"}
Sep 17 18:10:33.653317 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:10:33.653228    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-khfr8" podUID="0ff23b0f-f993-4384-90e4-83aacac37842" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Sep 17 18:10:38.162514 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:10:38.162491    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-khfr8"
Sep 17 18:10:38.195821 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:10:38.195790    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g47wt\" (UniqueName: \"kubernetes.io/projected/0ff23b0f-f993-4384-90e4-83aacac37842-kube-api-access-g47wt\") pod \"0ff23b0f-f993-4384-90e4-83aacac37842\" (UID: \"0ff23b0f-f993-4384-90e4-83aacac37842\") "
Sep 17 18:10:38.195821 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:10:38.195823    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0ff23b0f-f993-4384-90e4-83aacac37842-kube-rbac-proxy-sar-config\") pod \"0ff23b0f-f993-4384-90e4-83aacac37842\" (UID: \"0ff23b0f-f993-4384-90e4-83aacac37842\") "
Sep 17 18:10:38.196071 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:10:38.195867    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0ff23b0f-f993-4384-90e4-83aacac37842-cabundle-cert\") pod \"0ff23b0f-f993-4384-90e4-83aacac37842\" (UID: \"0ff23b0f-f993-4384-90e4-83aacac37842\") "
Sep 17 18:10:38.196071 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:10:38.196025    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ff23b0f-f993-4384-90e4-83aacac37842-kserve-provision-location\") pod \"0ff23b0f-f993-4384-90e4-83aacac37842\" (UID: \"0ff23b0f-f993-4384-90e4-83aacac37842\") "
Sep 17 18:10:38.196071 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:10:38.196054    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ff23b0f-f993-4384-90e4-83aacac37842-proxy-tls\") pod \"0ff23b0f-f993-4384-90e4-83aacac37842\" (UID: \"0ff23b0f-f993-4384-90e4-83aacac37842\") "
Sep 17 18:10:38.196330 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:10:38.196305    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ff23b0f-f993-4384-90e4-83aacac37842-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0ff23b0f-f993-4384-90e4-83aacac37842" (UID: "0ff23b0f-f993-4384-90e4-83aacac37842"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:10:38.196330 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:10:38.196312    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff23b0f-f993-4384-90e4-83aacac37842-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0ff23b0f-f993-4384-90e4-83aacac37842" (UID: "0ff23b0f-f993-4384-90e4-83aacac37842"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:10:38.196456 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:10:38.196316    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ff23b0f-f993-4384-90e4-83aacac37842-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0ff23b0f-f993-4384-90e4-83aacac37842" (UID: "0ff23b0f-f993-4384-90e4-83aacac37842"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:10:38.197909 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:10:38.197886    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ff23b0f-f993-4384-90e4-83aacac37842-kube-api-access-g47wt" (OuterVolumeSpecName: "kube-api-access-g47wt") pod "0ff23b0f-f993-4384-90e4-83aacac37842" (UID: "0ff23b0f-f993-4384-90e4-83aacac37842"). InnerVolumeSpecName "kube-api-access-g47wt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:10:38.198062 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:10:38.198037    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ff23b0f-f993-4384-90e4-83aacac37842-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0ff23b0f-f993-4384-90e4-83aacac37842" (UID: "0ff23b0f-f993-4384-90e4-83aacac37842"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:10:38.297380 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:10:38.297333    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g47wt\" (UniqueName: \"kubernetes.io/projected/0ff23b0f-f993-4384-90e4-83aacac37842-kube-api-access-g47wt\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:10:38.297380 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:10:38.297374    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0ff23b0f-f993-4384-90e4-83aacac37842-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:10:38.297380 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:10:38.297385    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0ff23b0f-f993-4384-90e4-83aacac37842-cabundle-cert\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:10:38.297380 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:10:38.297394    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ff23b0f-f993-4384-90e4-83aacac37842-kserve-provision-location\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:10:38.297662 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:10:38.297403    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ff23b0f-f993-4384-90e4-83aacac37842-proxy-tls\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:10:38.893479 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:10:38.893447    2569 generic.go:356] "Generic (PLEG): container finished" podID="0ff23b0f-f993-4384-90e4-83aacac37842" containerID="f0df7a93822dc36d169386c30b8a00fe37aa96e8a89c5a37e1098c662fe402d2" exitCode=0
Sep 17 18:10:38.893675 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:10:38.893519    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-khfr8"
Sep 17 18:10:38.893675 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:10:38.893532    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-khfr8" event={"ID":"0ff23b0f-f993-4384-90e4-83aacac37842","Type":"ContainerDied","Data":"f0df7a93822dc36d169386c30b8a00fe37aa96e8a89c5a37e1098c662fe402d2"}
Sep 17 18:10:38.893675 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:10:38.893592    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-khfr8" event={"ID":"0ff23b0f-f993-4384-90e4-83aacac37842","Type":"ContainerDied","Data":"3839da6c3bba0711177f51fabd65342381739a23e5b41933cf3b3192896e9db1"}
Sep 17 18:10:38.893675 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:10:38.893618    2569 scope.go:117] "RemoveContainer" containerID="9ee0ee38ef6a83def04f73dfb0a648a8b37432b500eada790dd27e22527e7574"
Sep 17 18:10:38.902396 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:10:38.902379    2569 scope.go:117] "RemoveContainer" containerID="f0df7a93822dc36d169386c30b8a00fe37aa96e8a89c5a37e1098c662fe402d2"
Sep 17 18:10:38.909561 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:10:38.909545    2569 scope.go:117] "RemoveContainer" containerID="04a07259aca7383565df97851a266374c8d1669a351b1dba7a95c6c189308e7f"
Sep 17 18:10:38.916362 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:10:38.916339    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-khfr8"]
Sep 17 18:10:38.917616 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:10:38.917600    2569 scope.go:117] "RemoveContainer" containerID="9ee0ee38ef6a83def04f73dfb0a648a8b37432b500eada790dd27e22527e7574"
Sep 17 18:10:38.917991 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:10:38.917928    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ee0ee38ef6a83def04f73dfb0a648a8b37432b500eada790dd27e22527e7574\": container with ID starting with 9ee0ee38ef6a83def04f73dfb0a648a8b37432b500eada790dd27e22527e7574 not found: ID does not exist" containerID="9ee0ee38ef6a83def04f73dfb0a648a8b37432b500eada790dd27e22527e7574"
Sep 17 18:10:38.918131 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:10:38.918001    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ee0ee38ef6a83def04f73dfb0a648a8b37432b500eada790dd27e22527e7574"} err="failed to get container status \"9ee0ee38ef6a83def04f73dfb0a648a8b37432b500eada790dd27e22527e7574\": rpc error: code = NotFound desc = could not find container \"9ee0ee38ef6a83def04f73dfb0a648a8b37432b500eada790dd27e22527e7574\": container with ID starting with 9ee0ee38ef6a83def04f73dfb0a648a8b37432b500eada790dd27e22527e7574 not found: ID does not exist"
Sep 17 18:10:38.918131 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:10:38.918027    2569 scope.go:117] "RemoveContainer" containerID="f0df7a93822dc36d169386c30b8a00fe37aa96e8a89c5a37e1098c662fe402d2"
Sep 17 18:10:38.918388 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:10:38.918361    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0df7a93822dc36d169386c30b8a00fe37aa96e8a89c5a37e1098c662fe402d2\": container with ID starting with f0df7a93822dc36d169386c30b8a00fe37aa96e8a89c5a37e1098c662fe402d2 not found: ID does not exist" containerID="f0df7a93822dc36d169386c30b8a00fe37aa96e8a89c5a37e1098c662fe402d2"
Sep 17 18:10:38.918477 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:10:38.918397    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0df7a93822dc36d169386c30b8a00fe37aa96e8a89c5a37e1098c662fe402d2"} err="failed to get container status \"f0df7a93822dc36d169386c30b8a00fe37aa96e8a89c5a37e1098c662fe402d2\": rpc error: code = NotFound desc = could not find container \"f0df7a93822dc36d169386c30b8a00fe37aa96e8a89c5a37e1098c662fe402d2\": container with ID starting with f0df7a93822dc36d169386c30b8a00fe37aa96e8a89c5a37e1098c662fe402d2 not found: ID does not exist"
Sep 17 18:10:38.918477 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:10:38.918419    2569 scope.go:117] "RemoveContainer" containerID="04a07259aca7383565df97851a266374c8d1669a351b1dba7a95c6c189308e7f"
Sep 17 18:10:38.918775 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:10:38.918753    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04a07259aca7383565df97851a266374c8d1669a351b1dba7a95c6c189308e7f\": container with ID starting with 04a07259aca7383565df97851a266374c8d1669a351b1dba7a95c6c189308e7f not found: ID does not exist" containerID="04a07259aca7383565df97851a266374c8d1669a351b1dba7a95c6c189308e7f"
Sep 17 18:10:38.918896 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:10:38.918872    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04a07259aca7383565df97851a266374c8d1669a351b1dba7a95c6c189308e7f"} err="failed to get container status \"04a07259aca7383565df97851a266374c8d1669a351b1dba7a95c6c189308e7f\": rpc error: code = NotFound desc = could not find container \"04a07259aca7383565df97851a266374c8d1669a351b1dba7a95c6c189308e7f\": container with ID starting with 04a07259aca7383565df97851a266374c8d1669a351b1dba7a95c6c189308e7f not found: ID does not exist"
Sep 17 18:10:38.920305 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:10:38.920283    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-khfr8"]
Sep 17 18:10:39.508657 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:10:39.508627    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ff23b0f-f993-4384-90e4-83aacac37842" path="/var/lib/kubelet/pods/0ff23b0f-f993-4384-90e4-83aacac37842/volumes"
Sep 17 18:10:49.577145 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:10:49.577117    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-t7pqb_b8d76e28-4655-4d21-98d8-0324e4756aaf/console-operator/2.log"
Sep 17 18:10:49.580064 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:10:49.580043    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-t7pqb_b8d76e28-4655-4d21-98d8-0324e4756aaf/console-operator/2.log"
Sep 17 18:10:49.582886 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:10:49.582866    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k4mqn_6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff/ovn-acl-logging/0.log"
Sep 17 18:10:49.585475 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:10:49.585458    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k4mqn_6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff/ovn-acl-logging/0.log"
Sep 17 18:11:10.241838 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:11:10.241801    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-z2l4q"]
Sep 17 18:11:10.242487 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:11:10.242262    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ff23b0f-f993-4384-90e4-83aacac37842" containerName="storage-initializer"
Sep 17 18:11:10.242487 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:11:10.242280    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ff23b0f-f993-4384-90e4-83aacac37842" containerName="storage-initializer"
Sep 17 18:11:10.242487 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:11:10.242298    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ff23b0f-f993-4384-90e4-83aacac37842" containerName="kube-rbac-proxy"
Sep 17 18:11:10.242487 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:11:10.242306    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ff23b0f-f993-4384-90e4-83aacac37842" containerName="kube-rbac-proxy"
Sep 17 18:11:10.242487 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:11:10.242326    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ff23b0f-f993-4384-90e4-83aacac37842" containerName="kserve-container"
Sep 17 18:11:10.242487 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:11:10.242334    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ff23b0f-f993-4384-90e4-83aacac37842" containerName="kserve-container"
Sep 17 18:11:10.242487 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:11:10.242412    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ff23b0f-f993-4384-90e4-83aacac37842" containerName="kube-rbac-proxy"
Sep 17 18:11:10.242487 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:11:10.242430    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ff23b0f-f993-4384-90e4-83aacac37842" containerName="kserve-container"
Sep 17 18:11:10.245700 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:11:10.245680    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-z2l4q"
Sep 17 18:11:10.248428 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:11:10.248402    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 17 18:11:10.248538 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:11:10.248463    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:11:10.248656 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:11:10.248619    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:11:10.248656 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:11:10.248640    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:11:10.248864 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:11:10.248751    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 17 18:11:10.249744 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:11:10.249725    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:11:10.250405 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:11:10.250259    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-kkx8f\""
Sep 17 18:11:10.260032 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:11:10.260006    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-z2l4q"]
Sep 17 18:11:10.271142 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:11:10.271119    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26d26802-41f4-4d17-8856-9073e4391d66-kserve-provision-location\") pod \"isvc-canary-create-predictor-5f845bd94d-z2l4q\" (UID: \"26d26802-41f4-4d17-8856-9073e4391d66\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-z2l4q"
Sep 17 18:11:10.271283 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:11:10.271152    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26d26802-41f4-4d17-8856-9073e4391d66-cabundle-cert\") pod \"isvc-canary-create-predictor-5f845bd94d-z2l4q\" (UID: \"26d26802-41f4-4d17-8856-9073e4391d66\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-z2l4q"
Sep 17 18:11:10.271283 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:11:10.271178    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26d26802-41f4-4d17-8856-9073e4391d66-proxy-tls\") pod \"isvc-canary-create-predictor-5f845bd94d-z2l4q\" (UID: \"26d26802-41f4-4d17-8856-9073e4391d66\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-z2l4q"
Sep 17 18:11:10.271283 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:11:10.271274    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srtcb\" (UniqueName: \"kubernetes.io/projected/26d26802-41f4-4d17-8856-9073e4391d66-kube-api-access-srtcb\") pod \"isvc-canary-create-predictor-5f845bd94d-z2l4q\" (UID: \"26d26802-41f4-4d17-8856-9073e4391d66\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-z2l4q"
Sep 17 18:11:10.271459 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:11:10.271301    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26d26802-41f4-4d17-8856-9073e4391d66-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5f845bd94d-z2l4q\" (UID: \"26d26802-41f4-4d17-8856-9073e4391d66\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-z2l4q"
Sep 17 18:11:10.371904 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:11:10.371869    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26d26802-41f4-4d17-8856-9073e4391d66-kserve-provision-location\") pod \"isvc-canary-create-predictor-5f845bd94d-z2l4q\" (UID: \"26d26802-41f4-4d17-8856-9073e4391d66\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-z2l4q"
Sep 17 18:11:10.372120 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:11:10.371918    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26d26802-41f4-4d17-8856-9073e4391d66-cabundle-cert\") pod \"isvc-canary-create-predictor-5f845bd94d-z2l4q\" (UID: \"26d26802-41f4-4d17-8856-9073e4391d66\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-z2l4q"
Sep 17 18:11:10.372120 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:11:10.371939    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26d26802-41f4-4d17-8856-9073e4391d66-proxy-tls\") pod \"isvc-canary-create-predictor-5f845bd94d-z2l4q\" (UID: \"26d26802-41f4-4d17-8856-9073e4391d66\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-z2l4q"
Sep 17 18:11:10.372120 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:11:10.371976    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srtcb\" (UniqueName: \"kubernetes.io/projected/26d26802-41f4-4d17-8856-9073e4391d66-kube-api-access-srtcb\") pod \"isvc-canary-create-predictor-5f845bd94d-z2l4q\" (UID: \"26d26802-41f4-4d17-8856-9073e4391d66\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-z2l4q"
Sep 17 18:11:10.372120 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:11:10.372001    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26d26802-41f4-4d17-8856-9073e4391d66-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5f845bd94d-z2l4q\" (UID: \"26d26802-41f4-4d17-8856-9073e4391d66\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-z2l4q"
Sep 17 18:11:10.372374 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:11:10.372164    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-predictor-serving-cert: secret "isvc-canary-create-predictor-serving-cert" not found
Sep 17 18:11:10.372374 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:11:10.372249    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26d26802-41f4-4d17-8856-9073e4391d66-proxy-tls podName:26d26802-41f4-4d17-8856-9073e4391d66 nodeName:}" failed. No retries permitted until 2026-09-17 18:11:10.872225166 +0000 UTC m=+1823.376227785 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/26d26802-41f4-4d17-8856-9073e4391d66-proxy-tls") pod "isvc-canary-create-predictor-5f845bd94d-z2l4q" (UID: "26d26802-41f4-4d17-8856-9073e4391d66") : secret "isvc-canary-create-predictor-serving-cert" not found
Sep 17 18:11:10.372483 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:11:10.372415    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26d26802-41f4-4d17-8856-9073e4391d66-kserve-provision-location\") pod \"isvc-canary-create-predictor-5f845bd94d-z2l4q\" (UID: \"26d26802-41f4-4d17-8856-9073e4391d66\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-z2l4q"
Sep 17 18:11:10.372709 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:11:10.372691    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26d26802-41f4-4d17-8856-9073e4391d66-cabundle-cert\") pod \"isvc-canary-create-predictor-5f845bd94d-z2l4q\" (UID: \"26d26802-41f4-4d17-8856-9073e4391d66\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-z2l4q"
Sep 17 18:11:10.372709 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:11:10.372701    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26d26802-41f4-4d17-8856-9073e4391d66-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5f845bd94d-z2l4q\" (UID: \"26d26802-41f4-4d17-8856-9073e4391d66\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-z2l4q"
Sep 17 18:11:10.385251 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:11:10.385199    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srtcb\" (UniqueName: \"kubernetes.io/projected/26d26802-41f4-4d17-8856-9073e4391d66-kube-api-access-srtcb\") pod \"isvc-canary-create-predictor-5f845bd94d-z2l4q\" (UID: \"26d26802-41f4-4d17-8856-9073e4391d66\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-z2l4q"
Sep 17 18:11:10.875643 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:11:10.875599    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26d26802-41f4-4d17-8856-9073e4391d66-proxy-tls\") pod \"isvc-canary-create-predictor-5f845bd94d-z2l4q\" (UID: \"26d26802-41f4-4d17-8856-9073e4391d66\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-z2l4q"
Sep 17 18:11:10.878247 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:11:10.878224    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26d26802-41f4-4d17-8856-9073e4391d66-proxy-tls\") pod \"isvc-canary-create-predictor-5f845bd94d-z2l4q\" (UID: \"26d26802-41f4-4d17-8856-9073e4391d66\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-z2l4q"
Sep 17 18:11:11.161819 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:11:11.161716    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-z2l4q"
Sep 17 18:11:11.436560 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:11:11.436534    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-z2l4q"]
Sep 17 18:11:11.439055 ip-10-0-131-14 kubenswrapper[2569]: W0917 18:11:11.439027    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26d26802_41f4_4d17_8856_9073e4391d66.slice/crio-edaa785fef6397b133218e51dc947f1d5bd10e974ac49eb789049b78012c5240 WatchSource:0}: Error finding container edaa785fef6397b133218e51dc947f1d5bd10e974ac49eb789049b78012c5240: Status 404 returned error can't find the container with id edaa785fef6397b133218e51dc947f1d5bd10e974ac49eb789049b78012c5240
Sep 17 18:11:12.030195 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:11:12.030161    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-z2l4q" event={"ID":"26d26802-41f4-4d17-8856-9073e4391d66","Type":"ContainerStarted","Data":"90ac0f3a60944b1435c364fd0531ae5b38a6e5b40678aa2e16ce7ecfec1e6dec"}
Sep 17 18:11:12.030384 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:11:12.030219    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-z2l4q" event={"ID":"26d26802-41f4-4d17-8856-9073e4391d66","Type":"ContainerStarted","Data":"edaa785fef6397b133218e51dc947f1d5bd10e974ac49eb789049b78012c5240"}
Sep 17 18:11:16.048420 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:11:16.048380    2569 generic.go:356] "Generic (PLEG): container finished" podID="26d26802-41f4-4d17-8856-9073e4391d66" containerID="90ac0f3a60944b1435c364fd0531ae5b38a6e5b40678aa2e16ce7ecfec1e6dec" exitCode=0
Sep 17 18:11:16.048420 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:11:16.048424    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-z2l4q" event={"ID":"26d26802-41f4-4d17-8856-9073e4391d66","Type":"ContainerDied","Data":"90ac0f3a60944b1435c364fd0531ae5b38a6e5b40678aa2e16ce7ecfec1e6dec"}
Sep 17 18:11:16.049652 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:11:16.049634    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:11:31.123093 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:11:31.123048    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-z2l4q" event={"ID":"26d26802-41f4-4d17-8856-9073e4391d66","Type":"ContainerStarted","Data":"1a456126daf234ca647af372adad6a26b5b356400eb2ce88ba68b212a729ddae"}
Sep 17 18:11:31.123476 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:11:31.123152    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-z2l4q" event={"ID":"26d26802-41f4-4d17-8856-9073e4391d66","Type":"ContainerStarted","Data":"bc786068de9ef22c49417635ffd47166d3c7f3936cd43c7531eb9280d485394f"}
Sep 17 18:11:31.123476 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:11:31.123466    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-z2l4q"
Sep 17 18:11:31.123614 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:11:31.123597    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-z2l4q"
Sep 17 18:11:31.124993 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:11:31.124966    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-z2l4q" podUID="26d26802-41f4-4d17-8856-9073e4391d66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 18:11:31.145326 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:11:31.145276    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-z2l4q" podStartSLOduration=2.639031797 podStartE2EDuration="21.14526286s" totalImagesPullingTime="14.441080583s" totalInitContainerRuntime="4.06515048s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:11:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:11:31.144000662 +0000 UTC m=+1843.648003304" watchObservedRunningTime="2026-09-17 18:11:31.14526286 +0000 UTC m=+1843.649265484"
Sep 17 18:11:32.127812 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:11:32.127774    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-z2l4q" podUID="26d26802-41f4-4d17-8856-9073e4391d66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 18:11:37.132207 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:11:37.132173    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-z2l4q"
Sep 17 18:11:37.132700 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:11:37.132654    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-z2l4q" podUID="26d26802-41f4-4d17-8856-9073e4391d66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 18:11:47.133434 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:11:47.133394    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-z2l4q" podUID="26d26802-41f4-4d17-8856-9073e4391d66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 18:11:57.132974 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:11:57.132926    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-z2l4q" podUID="26d26802-41f4-4d17-8856-9073e4391d66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 18:12:07.133236 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:12:07.133194    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-z2l4q" podUID="26d26802-41f4-4d17-8856-9073e4391d66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 18:12:17.133136 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:12:17.133066    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-z2l4q" podUID="26d26802-41f4-4d17-8856-9073e4391d66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 18:12:27.133423 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:12:27.133381    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-z2l4q" podUID="26d26802-41f4-4d17-8856-9073e4391d66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 18:12:33.797455 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:12:33.797413    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-z2l4q"]
Sep 17 18:12:33.798015 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:12:33.797669    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-z2l4q" podUID="26d26802-41f4-4d17-8856-9073e4391d66" containerName="kserve-container" containerID="cri-o://bc786068de9ef22c49417635ffd47166d3c7f3936cd43c7531eb9280d485394f" gracePeriod=30
Sep 17 18:12:33.798015 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:12:33.797723    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-z2l4q" podUID="26d26802-41f4-4d17-8856-9073e4391d66" containerName="kube-rbac-proxy" containerID="cri-o://1a456126daf234ca647af372adad6a26b5b356400eb2ce88ba68b212a729ddae" gracePeriod=30
Sep 17 18:12:33.798015 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:12:33.797921    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-z2l4q" podUID="26d26802-41f4-4d17-8856-9073e4391d66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 18:12:34.384766 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:12:34.384733    2569 generic.go:356] "Generic (PLEG): container finished" podID="26d26802-41f4-4d17-8856-9073e4391d66" containerID="1a456126daf234ca647af372adad6a26b5b356400eb2ce88ba68b212a729ddae" exitCode=2
Sep 17 18:12:34.384955 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:12:34.384798    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-z2l4q" event={"ID":"26d26802-41f4-4d17-8856-9073e4391d66","Type":"ContainerDied","Data":"1a456126daf234ca647af372adad6a26b5b356400eb2ce88ba68b212a729ddae"}
Sep 17 18:12:37.128384 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:12:37.128345    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-z2l4q" podUID="26d26802-41f4-4d17-8856-9073e4391d66" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Sep 17 18:12:42.128738 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:12:42.128690    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-z2l4q" podUID="26d26802-41f4-4d17-8856-9073e4391d66" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Sep 17 18:12:43.798884 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:12:43.798838    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-z2l4q" podUID="26d26802-41f4-4d17-8856-9073e4391d66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 18:12:45.433853 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:12:45.433817    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-26fvh"]
Sep 17 18:12:45.437213 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:12:45.437185    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-26fvh"
Sep 17 18:12:45.439941 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:12:45.439917    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 17 18:12:45.440070 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:12:45.439959    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 17 18:12:45.449745 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:12:45.449720    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-26fvh"]
Sep 17 18:12:45.561161 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:12:45.561128    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b74dee3-f906-43ae-8a01-9bd07dd3b779-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-567686cc76-26fvh\" (UID: \"7b74dee3-f906-43ae-8a01-9bd07dd3b779\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-26fvh"
Sep 17 18:12:45.561161 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:12:45.561162    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4ts6\" (UniqueName: \"kubernetes.io/projected/7b74dee3-f906-43ae-8a01-9bd07dd3b779-kube-api-access-p4ts6\") pod \"isvc-canary-promote-predictor-567686cc76-26fvh\" (UID: \"7b74dee3-f906-43ae-8a01-9bd07dd3b779\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-26fvh"
Sep 17 18:12:45.561419 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:12:45.561204    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b74dee3-f906-43ae-8a01-9bd07dd3b779-kserve-provision-location\") pod \"isvc-canary-promote-predictor-567686cc76-26fvh\" (UID: \"7b74dee3-f906-43ae-8a01-9bd07dd3b779\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-26fvh"
Sep 17 18:12:45.561419 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:12:45.561311    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b74dee3-f906-43ae-8a01-9bd07dd3b779-proxy-tls\") pod \"isvc-canary-promote-predictor-567686cc76-26fvh\" (UID: \"7b74dee3-f906-43ae-8a01-9bd07dd3b779\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-26fvh"
Sep 17 18:12:45.561419 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:12:45.561348    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b74dee3-f906-43ae-8a01-9bd07dd3b779-cabundle-cert\") pod \"isvc-canary-promote-predictor-567686cc76-26fvh\" (UID: \"7b74dee3-f906-43ae-8a01-9bd07dd3b779\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-26fvh"
Sep 17 18:12:45.662048 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:12:45.662018    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b74dee3-f906-43ae-8a01-9bd07dd3b779-proxy-tls\") pod \"isvc-canary-promote-predictor-567686cc76-26fvh\" (UID: \"7b74dee3-f906-43ae-8a01-9bd07dd3b779\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-26fvh"
Sep 17 18:12:45.662048 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:12:45.662052    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b74dee3-f906-43ae-8a01-9bd07dd3b779-cabundle-cert\") pod \"isvc-canary-promote-predictor-567686cc76-26fvh\" (UID: \"7b74dee3-f906-43ae-8a01-9bd07dd3b779\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-26fvh"
Sep 17 18:12:45.662315 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:12:45.662089    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b74dee3-f906-43ae-8a01-9bd07dd3b779-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-567686cc76-26fvh\" (UID: \"7b74dee3-f906-43ae-8a01-9bd07dd3b779\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-26fvh"
Sep 17 18:12:45.662315 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:12:45.662109    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4ts6\" (UniqueName: \"kubernetes.io/projected/7b74dee3-f906-43ae-8a01-9bd07dd3b779-kube-api-access-p4ts6\") pod \"isvc-canary-promote-predictor-567686cc76-26fvh\" (UID: \"7b74dee3-f906-43ae-8a01-9bd07dd3b779\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-26fvh"
Sep 17 18:12:45.662315 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:12:45.662160    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b74dee3-f906-43ae-8a01-9bd07dd3b779-kserve-provision-location\") pod \"isvc-canary-promote-predictor-567686cc76-26fvh\" (UID: \"7b74dee3-f906-43ae-8a01-9bd07dd3b779\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-26fvh"
Sep 17 18:12:45.662315 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:12:45.662203    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-predictor-serving-cert: secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 17 18:12:45.662315 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:12:45.662286    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b74dee3-f906-43ae-8a01-9bd07dd3b779-proxy-tls podName:7b74dee3-f906-43ae-8a01-9bd07dd3b779 nodeName:}" failed. No retries permitted until 2026-09-17 18:12:46.162266615 +0000 UTC m=+1918.666269234 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7b74dee3-f906-43ae-8a01-9bd07dd3b779-proxy-tls") pod "isvc-canary-promote-predictor-567686cc76-26fvh" (UID: "7b74dee3-f906-43ae-8a01-9bd07dd3b779") : secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 17 18:12:45.662653 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:12:45.662630    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b74dee3-f906-43ae-8a01-9bd07dd3b779-kserve-provision-location\") pod \"isvc-canary-promote-predictor-567686cc76-26fvh\" (UID: \"7b74dee3-f906-43ae-8a01-9bd07dd3b779\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-26fvh"
Sep 17 18:12:45.662882 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:12:45.662852    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b74dee3-f906-43ae-8a01-9bd07dd3b779-cabundle-cert\") pod \"isvc-canary-promote-predictor-567686cc76-26fvh\" (UID: \"7b74dee3-f906-43ae-8a01-9bd07dd3b779\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-26fvh"
Sep 17 18:12:45.663015 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:12:45.662861    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b74dee3-f906-43ae-8a01-9bd07dd3b779-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-567686cc76-26fvh\" (UID: \"7b74dee3-f906-43ae-8a01-9bd07dd3b779\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-26fvh"
Sep 17 18:12:45.671291 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:12:45.671268    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4ts6\" (UniqueName: \"kubernetes.io/projected/7b74dee3-f906-43ae-8a01-9bd07dd3b779-kube-api-access-p4ts6\") pod \"isvc-canary-promote-predictor-567686cc76-26fvh\" (UID: \"7b74dee3-f906-43ae-8a01-9bd07dd3b779\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-26fvh"
Sep 17 18:12:46.166849 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:12:46.166808    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b74dee3-f906-43ae-8a01-9bd07dd3b779-proxy-tls\") pod \"isvc-canary-promote-predictor-567686cc76-26fvh\" (UID: \"7b74dee3-f906-43ae-8a01-9bd07dd3b779\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-26fvh"
Sep 17 18:12:46.169202 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:12:46.169184    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b74dee3-f906-43ae-8a01-9bd07dd3b779-proxy-tls\") pod \"isvc-canary-promote-predictor-567686cc76-26fvh\" (UID: \"7b74dee3-f906-43ae-8a01-9bd07dd3b779\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-26fvh"
Sep 17 18:12:46.346860 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:12:46.346808    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-26fvh"
Sep 17 18:12:46.620359 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:12:46.620330    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-26fvh"]
Sep 17 18:12:46.622009 ip-10-0-131-14 kubenswrapper[2569]: W0917 18:12:46.621982    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b74dee3_f906_43ae_8a01_9bd07dd3b779.slice/crio-39796a2f245a4be20b586e87c8d81808363526e6d54cdbd94efd212ce7334256 WatchSource:0}: Error finding container 39796a2f245a4be20b586e87c8d81808363526e6d54cdbd94efd212ce7334256: Status 404 returned error can't find the container with id 39796a2f245a4be20b586e87c8d81808363526e6d54cdbd94efd212ce7334256
Sep 17 18:12:47.128750 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:12:47.128708    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-z2l4q" podUID="26d26802-41f4-4d17-8856-9073e4391d66" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Sep 17 18:12:47.128937 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:12:47.128851    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-z2l4q"
Sep 17 18:12:47.437916 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:12:47.437825    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-26fvh" event={"ID":"7b74dee3-f906-43ae-8a01-9bd07dd3b779","Type":"ContainerStarted","Data":"1ffad1d2f0921bfdb9a5ba123c03eb7f1ffb7d9b78eb5f9176c9032b63d5b4ec"}
Sep 17 18:12:47.437916 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:12:47.437879    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-26fvh" event={"ID":"7b74dee3-f906-43ae-8a01-9bd07dd3b779","Type":"ContainerStarted","Data":"39796a2f245a4be20b586e87c8d81808363526e6d54cdbd94efd212ce7334256"}
Sep 17 18:12:51.460800 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:12:51.460759    2569 generic.go:356] "Generic (PLEG): container finished" podID="7b74dee3-f906-43ae-8a01-9bd07dd3b779" containerID="1ffad1d2f0921bfdb9a5ba123c03eb7f1ffb7d9b78eb5f9176c9032b63d5b4ec" exitCode=0
Sep 17 18:12:51.461207 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:12:51.460833    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-26fvh" event={"ID":"7b74dee3-f906-43ae-8a01-9bd07dd3b779","Type":"ContainerDied","Data":"1ffad1d2f0921bfdb9a5ba123c03eb7f1ffb7d9b78eb5f9176c9032b63d5b4ec"}
Sep 17 18:12:52.128403 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:12:52.128353    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-z2l4q" podUID="26d26802-41f4-4d17-8856-9073e4391d66" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Sep 17 18:12:52.466881 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:12:52.466849    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-26fvh" event={"ID":"7b74dee3-f906-43ae-8a01-9bd07dd3b779","Type":"ContainerStarted","Data":"42992624a2767da332e681668ba7ce771ab9a93aa2fe386968f3210a2bcdd5f9"}
Sep 17 18:12:52.467286 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:12:52.466917    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-26fvh" event={"ID":"7b74dee3-f906-43ae-8a01-9bd07dd3b779","Type":"ContainerStarted","Data":"56bd4897bae298551ff97620b5a2ba4ac2a553d71e155a5550d3ec2f27c900e9"}
Sep 17 18:12:52.467286 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:12:52.467229    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-26fvh"
Sep 17 18:12:52.490054 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:12:52.490007    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-26fvh" podStartSLOduration=3.666686342 podStartE2EDuration="7.48999273s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.823306388s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:12:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:12:52.488953296 +0000 UTC m=+1924.992955922" watchObservedRunningTime="2026-09-17 18:12:52.48999273 +0000 UTC m=+1924.993995356"
Sep 17 18:12:53.471069 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:12:53.471042    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-26fvh"
Sep 17 18:12:53.472336 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:12:53.472308    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-26fvh" podUID="7b74dee3-f906-43ae-8a01-9bd07dd3b779" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 18:12:53.799786 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:12:53.799706    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-z2l4q"
Sep 17 18:12:54.475230 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:12:54.475193    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-26fvh" podUID="7b74dee3-f906-43ae-8a01-9bd07dd3b779" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 18:12:57.128401 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:12:57.128362    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-z2l4q" podUID="26d26802-41f4-4d17-8856-9073e4391d66" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Sep 17 18:12:59.480171 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:12:59.480140    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-26fvh"
Sep 17 18:12:59.480666 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:12:59.480630    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-26fvh" podUID="7b74dee3-f906-43ae-8a01-9bd07dd3b779" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 18:13:02.128431 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:13:02.128393    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-z2l4q" podUID="26d26802-41f4-4d17-8856-9073e4391d66" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Sep 17 18:13:04.441506 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:13:04.441482    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-z2l4q"
Sep 17 18:13:04.519693 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:13:04.519661    2569 generic.go:356] "Generic (PLEG): container finished" podID="26d26802-41f4-4d17-8856-9073e4391d66" containerID="bc786068de9ef22c49417635ffd47166d3c7f3936cd43c7531eb9280d485394f" exitCode=137
Sep 17 18:13:04.519893 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:13:04.519734    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-z2l4q"
Sep 17 18:13:04.519893 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:13:04.519732    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-z2l4q" event={"ID":"26d26802-41f4-4d17-8856-9073e4391d66","Type":"ContainerDied","Data":"bc786068de9ef22c49417635ffd47166d3c7f3936cd43c7531eb9280d485394f"}
Sep 17 18:13:04.519893 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:13:04.519827    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-z2l4q" event={"ID":"26d26802-41f4-4d17-8856-9073e4391d66","Type":"ContainerDied","Data":"edaa785fef6397b133218e51dc947f1d5bd10e974ac49eb789049b78012c5240"}
Sep 17 18:13:04.519893 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:13:04.519862    2569 scope.go:117] "RemoveContainer" containerID="1a456126daf234ca647af372adad6a26b5b356400eb2ce88ba68b212a729ddae"
Sep 17 18:13:04.529962 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:13:04.529944    2569 scope.go:117] "RemoveContainer" containerID="bc786068de9ef22c49417635ffd47166d3c7f3936cd43c7531eb9280d485394f"
Sep 17 18:13:04.533329 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:13:04.533310    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26d26802-41f4-4d17-8856-9073e4391d66-cabundle-cert\") pod \"26d26802-41f4-4d17-8856-9073e4391d66\" (UID: \"26d26802-41f4-4d17-8856-9073e4391d66\") "
Sep 17 18:13:04.533453 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:13:04.533345    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26d26802-41f4-4d17-8856-9073e4391d66-proxy-tls\") pod \"26d26802-41f4-4d17-8856-9073e4391d66\" (UID: \"26d26802-41f4-4d17-8856-9073e4391d66\") "
Sep 17 18:13:04.533453 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:13:04.533383    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26d26802-41f4-4d17-8856-9073e4391d66-kserve-provision-location\") pod \"26d26802-41f4-4d17-8856-9073e4391d66\" (UID: \"26d26802-41f4-4d17-8856-9073e4391d66\") "
Sep 17 18:13:04.533453 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:13:04.533414    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26d26802-41f4-4d17-8856-9073e4391d66-kube-rbac-proxy-sar-config\") pod \"26d26802-41f4-4d17-8856-9073e4391d66\" (UID: \"26d26802-41f4-4d17-8856-9073e4391d66\") "
Sep 17 18:13:04.533453 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:13:04.533449    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-srtcb\" (UniqueName: \"kubernetes.io/projected/26d26802-41f4-4d17-8856-9073e4391d66-kube-api-access-srtcb\") pod \"26d26802-41f4-4d17-8856-9073e4391d66\" (UID: \"26d26802-41f4-4d17-8856-9073e4391d66\") "
Sep 17 18:13:04.534021 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:13:04.533985    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26d26802-41f4-4d17-8856-9073e4391d66-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "26d26802-41f4-4d17-8856-9073e4391d66" (UID: "26d26802-41f4-4d17-8856-9073e4391d66"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:13:04.534147 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:13:04.534024    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26d26802-41f4-4d17-8856-9073e4391d66-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "26d26802-41f4-4d17-8856-9073e4391d66" (UID: "26d26802-41f4-4d17-8856-9073e4391d66"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:13:04.534147 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:13:04.534034    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26d26802-41f4-4d17-8856-9073e4391d66-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "26d26802-41f4-4d17-8856-9073e4391d66" (UID: "26d26802-41f4-4d17-8856-9073e4391d66"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:13:04.535700 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:13:04.535675    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26d26802-41f4-4d17-8856-9073e4391d66-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "26d26802-41f4-4d17-8856-9073e4391d66" (UID: "26d26802-41f4-4d17-8856-9073e4391d66"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:13:04.536010 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:13:04.535983    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26d26802-41f4-4d17-8856-9073e4391d66-kube-api-access-srtcb" (OuterVolumeSpecName: "kube-api-access-srtcb") pod "26d26802-41f4-4d17-8856-9073e4391d66" (UID: "26d26802-41f4-4d17-8856-9073e4391d66"). InnerVolumeSpecName "kube-api-access-srtcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:13:04.538099 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:13:04.538059    2569 scope.go:117] "RemoveContainer" containerID="90ac0f3a60944b1435c364fd0531ae5b38a6e5b40678aa2e16ce7ecfec1e6dec"
Sep 17 18:13:04.553331 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:13:04.553311    2569 scope.go:117] "RemoveContainer" containerID="1a456126daf234ca647af372adad6a26b5b356400eb2ce88ba68b212a729ddae"
Sep 17 18:13:04.553593 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:13:04.553576    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a456126daf234ca647af372adad6a26b5b356400eb2ce88ba68b212a729ddae\": container with ID starting with 1a456126daf234ca647af372adad6a26b5b356400eb2ce88ba68b212a729ddae not found: ID does not exist" containerID="1a456126daf234ca647af372adad6a26b5b356400eb2ce88ba68b212a729ddae"
Sep 17 18:13:04.553641 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:13:04.553600    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a456126daf234ca647af372adad6a26b5b356400eb2ce88ba68b212a729ddae"} err="failed to get container status \"1a456126daf234ca647af372adad6a26b5b356400eb2ce88ba68b212a729ddae\": rpc error: code = NotFound desc = could not find container \"1a456126daf234ca647af372adad6a26b5b356400eb2ce88ba68b212a729ddae\": container with ID starting with 1a456126daf234ca647af372adad6a26b5b356400eb2ce88ba68b212a729ddae not found: ID does not exist"
Sep 17 18:13:04.553641 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:13:04.553618    2569 scope.go:117] "RemoveContainer" containerID="bc786068de9ef22c49417635ffd47166d3c7f3936cd43c7531eb9280d485394f"
Sep 17 18:13:04.553913 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:13:04.553894    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc786068de9ef22c49417635ffd47166d3c7f3936cd43c7531eb9280d485394f\": container with ID starting with bc786068de9ef22c49417635ffd47166d3c7f3936cd43c7531eb9280d485394f not found: ID does not exist" containerID="bc786068de9ef22c49417635ffd47166d3c7f3936cd43c7531eb9280d485394f"
Sep 17 18:13:04.553965 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:13:04.553919    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc786068de9ef22c49417635ffd47166d3c7f3936cd43c7531eb9280d485394f"} err="failed to get container status \"bc786068de9ef22c49417635ffd47166d3c7f3936cd43c7531eb9280d485394f\": rpc error: code = NotFound desc = could not find container \"bc786068de9ef22c49417635ffd47166d3c7f3936cd43c7531eb9280d485394f\": container with ID starting with bc786068de9ef22c49417635ffd47166d3c7f3936cd43c7531eb9280d485394f not found: ID does not exist"
Sep 17 18:13:04.553965 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:13:04.553934    2569 scope.go:117] "RemoveContainer" containerID="90ac0f3a60944b1435c364fd0531ae5b38a6e5b40678aa2e16ce7ecfec1e6dec"
Sep 17 18:13:04.554217 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:13:04.554177    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90ac0f3a60944b1435c364fd0531ae5b38a6e5b40678aa2e16ce7ecfec1e6dec\": container with ID starting with 90ac0f3a60944b1435c364fd0531ae5b38a6e5b40678aa2e16ce7ecfec1e6dec not found: ID does not exist" containerID="90ac0f3a60944b1435c364fd0531ae5b38a6e5b40678aa2e16ce7ecfec1e6dec"
Sep 17 18:13:04.554298 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:13:04.554226    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90ac0f3a60944b1435c364fd0531ae5b38a6e5b40678aa2e16ce7ecfec1e6dec"} err="failed to get container status \"90ac0f3a60944b1435c364fd0531ae5b38a6e5b40678aa2e16ce7ecfec1e6dec\": rpc error: code = NotFound desc = could not find container \"90ac0f3a60944b1435c364fd0531ae5b38a6e5b40678aa2e16ce7ecfec1e6dec\": container with ID starting with 90ac0f3a60944b1435c364fd0531ae5b38a6e5b40678aa2e16ce7ecfec1e6dec not found: ID does not exist"
Sep 17 18:13:04.634513 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:13:04.634428    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26d26802-41f4-4d17-8856-9073e4391d66-cabundle-cert\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:13:04.634513 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:13:04.634452    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26d26802-41f4-4d17-8856-9073e4391d66-proxy-tls\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:13:04.634513 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:13:04.634461    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26d26802-41f4-4d17-8856-9073e4391d66-kserve-provision-location\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:13:04.634513 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:13:04.634472    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26d26802-41f4-4d17-8856-9073e4391d66-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:13:04.634513 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:13:04.634481    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srtcb\" (UniqueName: \"kubernetes.io/projected/26d26802-41f4-4d17-8856-9073e4391d66-kube-api-access-srtcb\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:13:04.842697 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:13:04.842660    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-z2l4q"]
Sep 17 18:13:04.846506 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:13:04.846481    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-z2l4q"]
Sep 17 18:13:05.507878 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:13:05.507836    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26d26802-41f4-4d17-8856-9073e4391d66" path="/var/lib/kubelet/pods/26d26802-41f4-4d17-8856-9073e4391d66/volumes"
Sep 17 18:13:09.481434 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:13:09.481394    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-26fvh" podUID="7b74dee3-f906-43ae-8a01-9bd07dd3b779" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 18:13:19.480584 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:13:19.480548    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-26fvh" podUID="7b74dee3-f906-43ae-8a01-9bd07dd3b779" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 18:13:29.481048 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:13:29.481004    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-26fvh" podUID="7b74dee3-f906-43ae-8a01-9bd07dd3b779" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 18:13:39.481213 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:13:39.481175    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-26fvh" podUID="7b74dee3-f906-43ae-8a01-9bd07dd3b779" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 18:13:49.481384 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:13:49.481346    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-26fvh" podUID="7b74dee3-f906-43ae-8a01-9bd07dd3b779" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 18:13:59.481063 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:13:59.481019    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-26fvh" podUID="7b74dee3-f906-43ae-8a01-9bd07dd3b779" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 18:14:09.272333 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:09.272291    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-26fvh"]
Sep 17 18:14:09.272832 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:09.272550    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-26fvh" podUID="7b74dee3-f906-43ae-8a01-9bd07dd3b779" containerName="kserve-container" containerID="cri-o://56bd4897bae298551ff97620b5a2ba4ac2a553d71e155a5550d3ec2f27c900e9" gracePeriod=30
Sep 17 18:14:09.272832 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:09.272657    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-26fvh" podUID="7b74dee3-f906-43ae-8a01-9bd07dd3b779" containerName="kube-rbac-proxy" containerID="cri-o://42992624a2767da332e681668ba7ce771ab9a93aa2fe386968f3210a2bcdd5f9" gracePeriod=30
Sep 17 18:14:09.273404 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:09.273252    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-26fvh"
Sep 17 18:14:09.476314 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:09.476274    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-26fvh" podUID="7b74dee3-f906-43ae-8a01-9bd07dd3b779" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Sep 17 18:14:09.788391 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:09.788358    2569 generic.go:356] "Generic (PLEG): container finished" podID="7b74dee3-f906-43ae-8a01-9bd07dd3b779" containerID="42992624a2767da332e681668ba7ce771ab9a93aa2fe386968f3210a2bcdd5f9" exitCode=2
Sep 17 18:14:09.788570 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:09.788431    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-26fvh" event={"ID":"7b74dee3-f906-43ae-8a01-9bd07dd3b779","Type":"ContainerDied","Data":"42992624a2767da332e681668ba7ce771ab9a93aa2fe386968f3210a2bcdd5f9"}
Sep 17 18:14:14.127157 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:14.127132    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-26fvh"
Sep 17 18:14:14.199057 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:14.199034    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b74dee3-f906-43ae-8a01-9bd07dd3b779-cabundle-cert\") pod \"7b74dee3-f906-43ae-8a01-9bd07dd3b779\" (UID: \"7b74dee3-f906-43ae-8a01-9bd07dd3b779\") "
Sep 17 18:14:14.199226 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:14.199067    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b74dee3-f906-43ae-8a01-9bd07dd3b779-kserve-provision-location\") pod \"7b74dee3-f906-43ae-8a01-9bd07dd3b779\" (UID: \"7b74dee3-f906-43ae-8a01-9bd07dd3b779\") "
Sep 17 18:14:14.199226 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:14.199148    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b74dee3-f906-43ae-8a01-9bd07dd3b779-proxy-tls\") pod \"7b74dee3-f906-43ae-8a01-9bd07dd3b779\" (UID: \"7b74dee3-f906-43ae-8a01-9bd07dd3b779\") "
Sep 17 18:14:14.199226 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:14.199177    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p4ts6\" (UniqueName: \"kubernetes.io/projected/7b74dee3-f906-43ae-8a01-9bd07dd3b779-kube-api-access-p4ts6\") pod \"7b74dee3-f906-43ae-8a01-9bd07dd3b779\" (UID: \"7b74dee3-f906-43ae-8a01-9bd07dd3b779\") "
Sep 17 18:14:14.199226 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:14.199200    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b74dee3-f906-43ae-8a01-9bd07dd3b779-kube-rbac-proxy-sar-config\") pod \"7b74dee3-f906-43ae-8a01-9bd07dd3b779\" (UID: \"7b74dee3-f906-43ae-8a01-9bd07dd3b779\") "
Sep 17 18:14:14.199508 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:14.199414    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b74dee3-f906-43ae-8a01-9bd07dd3b779-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7b74dee3-f906-43ae-8a01-9bd07dd3b779" (UID: "7b74dee3-f906-43ae-8a01-9bd07dd3b779"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:14:14.199508 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:14.199472    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b74dee3-f906-43ae-8a01-9bd07dd3b779-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7b74dee3-f906-43ae-8a01-9bd07dd3b779" (UID: "7b74dee3-f906-43ae-8a01-9bd07dd3b779"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:14:14.199642 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:14.199607    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b74dee3-f906-43ae-8a01-9bd07dd3b779-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7b74dee3-f906-43ae-8a01-9bd07dd3b779" (UID: "7b74dee3-f906-43ae-8a01-9bd07dd3b779"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:14:14.201294 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:14.201270    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b74dee3-f906-43ae-8a01-9bd07dd3b779-kube-api-access-p4ts6" (OuterVolumeSpecName: "kube-api-access-p4ts6") pod "7b74dee3-f906-43ae-8a01-9bd07dd3b779" (UID: "7b74dee3-f906-43ae-8a01-9bd07dd3b779"). InnerVolumeSpecName "kube-api-access-p4ts6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:14:14.201361 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:14.201294    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b74dee3-f906-43ae-8a01-9bd07dd3b779-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7b74dee3-f906-43ae-8a01-9bd07dd3b779" (UID: "7b74dee3-f906-43ae-8a01-9bd07dd3b779"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:14:14.299822 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:14.299790    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b74dee3-f906-43ae-8a01-9bd07dd3b779-cabundle-cert\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:14:14.299822 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:14.299821    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b74dee3-f906-43ae-8a01-9bd07dd3b779-kserve-provision-location\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:14:14.300011 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:14.299832    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b74dee3-f906-43ae-8a01-9bd07dd3b779-proxy-tls\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:14:14.300011 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:14.299842    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4ts6\" (UniqueName: \"kubernetes.io/projected/7b74dee3-f906-43ae-8a01-9bd07dd3b779-kube-api-access-p4ts6\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:14:14.300011 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:14.299850    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b74dee3-f906-43ae-8a01-9bd07dd3b779-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:14:14.811942 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:14.811909    2569 generic.go:356] "Generic (PLEG): container finished" podID="7b74dee3-f906-43ae-8a01-9bd07dd3b779" containerID="56bd4897bae298551ff97620b5a2ba4ac2a553d71e155a5550d3ec2f27c900e9" exitCode=0
Sep 17 18:14:14.812142 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:14.811966    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-26fvh"
Sep 17 18:14:14.812142 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:14.811991    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-26fvh" event={"ID":"7b74dee3-f906-43ae-8a01-9bd07dd3b779","Type":"ContainerDied","Data":"56bd4897bae298551ff97620b5a2ba4ac2a553d71e155a5550d3ec2f27c900e9"}
Sep 17 18:14:14.812142 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:14.812058    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-26fvh" event={"ID":"7b74dee3-f906-43ae-8a01-9bd07dd3b779","Type":"ContainerDied","Data":"39796a2f245a4be20b586e87c8d81808363526e6d54cdbd94efd212ce7334256"}
Sep 17 18:14:14.812142 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:14.812107    2569 scope.go:117] "RemoveContainer" containerID="42992624a2767da332e681668ba7ce771ab9a93aa2fe386968f3210a2bcdd5f9"
Sep 17 18:14:14.820632 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:14.820521    2569 scope.go:117] "RemoveContainer" containerID="56bd4897bae298551ff97620b5a2ba4ac2a553d71e155a5550d3ec2f27c900e9"
Sep 17 18:14:14.827759 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:14.827741    2569 scope.go:117] "RemoveContainer" containerID="1ffad1d2f0921bfdb9a5ba123c03eb7f1ffb7d9b78eb5f9176c9032b63d5b4ec"
Sep 17 18:14:14.833612 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:14.833589    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-26fvh"]
Sep 17 18:14:14.834907 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:14.834886    2569 scope.go:117] "RemoveContainer" containerID="42992624a2767da332e681668ba7ce771ab9a93aa2fe386968f3210a2bcdd5f9"
Sep 17 18:14:14.835177 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:14:14.835157    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42992624a2767da332e681668ba7ce771ab9a93aa2fe386968f3210a2bcdd5f9\": container with ID starting with 42992624a2767da332e681668ba7ce771ab9a93aa2fe386968f3210a2bcdd5f9 not found: ID does not exist" containerID="42992624a2767da332e681668ba7ce771ab9a93aa2fe386968f3210a2bcdd5f9"
Sep 17 18:14:14.835239 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:14.835185    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42992624a2767da332e681668ba7ce771ab9a93aa2fe386968f3210a2bcdd5f9"} err="failed to get container status \"42992624a2767da332e681668ba7ce771ab9a93aa2fe386968f3210a2bcdd5f9\": rpc error: code = NotFound desc = could not find container \"42992624a2767da332e681668ba7ce771ab9a93aa2fe386968f3210a2bcdd5f9\": container with ID starting with 42992624a2767da332e681668ba7ce771ab9a93aa2fe386968f3210a2bcdd5f9 not found: ID does not exist"
Sep 17 18:14:14.835239 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:14.835201    2569 scope.go:117] "RemoveContainer" containerID="56bd4897bae298551ff97620b5a2ba4ac2a553d71e155a5550d3ec2f27c900e9"
Sep 17 18:14:14.835410 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:14:14.835395    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56bd4897bae298551ff97620b5a2ba4ac2a553d71e155a5550d3ec2f27c900e9\": container with ID starting with 56bd4897bae298551ff97620b5a2ba4ac2a553d71e155a5550d3ec2f27c900e9 not found: ID does not exist" containerID="56bd4897bae298551ff97620b5a2ba4ac2a553d71e155a5550d3ec2f27c900e9"
Sep 17 18:14:14.835453 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:14.835413    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56bd4897bae298551ff97620b5a2ba4ac2a553d71e155a5550d3ec2f27c900e9"} err="failed to get container status \"56bd4897bae298551ff97620b5a2ba4ac2a553d71e155a5550d3ec2f27c900e9\": rpc error: code = NotFound desc = could not find container \"56bd4897bae298551ff97620b5a2ba4ac2a553d71e155a5550d3ec2f27c900e9\": container with ID starting with 56bd4897bae298551ff97620b5a2ba4ac2a553d71e155a5550d3ec2f27c900e9 not found: ID does not exist"
Sep 17 18:14:14.835453 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:14.835424    2569 scope.go:117] "RemoveContainer" containerID="1ffad1d2f0921bfdb9a5ba123c03eb7f1ffb7d9b78eb5f9176c9032b63d5b4ec"
Sep 17 18:14:14.835647 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:14:14.835633    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ffad1d2f0921bfdb9a5ba123c03eb7f1ffb7d9b78eb5f9176c9032b63d5b4ec\": container with ID starting with 1ffad1d2f0921bfdb9a5ba123c03eb7f1ffb7d9b78eb5f9176c9032b63d5b4ec not found: ID does not exist" containerID="1ffad1d2f0921bfdb9a5ba123c03eb7f1ffb7d9b78eb5f9176c9032b63d5b4ec"
Sep 17 18:14:14.835739 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:14.835648    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ffad1d2f0921bfdb9a5ba123c03eb7f1ffb7d9b78eb5f9176c9032b63d5b4ec"} err="failed to get container status \"1ffad1d2f0921bfdb9a5ba123c03eb7f1ffb7d9b78eb5f9176c9032b63d5b4ec\": rpc error: code = NotFound desc = could not find container \"1ffad1d2f0921bfdb9a5ba123c03eb7f1ffb7d9b78eb5f9176c9032b63d5b4ec\": container with ID starting with 1ffad1d2f0921bfdb9a5ba123c03eb7f1ffb7d9b78eb5f9176c9032b63d5b4ec not found: ID does not exist"
Sep 17 18:14:14.839783 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:14.839762    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-26fvh"]
Sep 17 18:14:15.480210 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:15.480175    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x6dsj"]
Sep 17 18:14:15.480672 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:15.480571    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b74dee3-f906-43ae-8a01-9bd07dd3b779" containerName="storage-initializer"
Sep 17 18:14:15.480672 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:15.480590    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b74dee3-f906-43ae-8a01-9bd07dd3b779" containerName="storage-initializer"
Sep 17 18:14:15.480672 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:15.480606    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b74dee3-f906-43ae-8a01-9bd07dd3b779" containerName="kube-rbac-proxy"
Sep 17 18:14:15.480672 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:15.480613    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b74dee3-f906-43ae-8a01-9bd07dd3b779" containerName="kube-rbac-proxy"
Sep 17 18:14:15.480672 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:15.480625    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26d26802-41f4-4d17-8856-9073e4391d66" containerName="storage-initializer"
Sep 17 18:14:15.480672 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:15.480633    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="26d26802-41f4-4d17-8856-9073e4391d66" containerName="storage-initializer"
Sep 17 18:14:15.480672 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:15.480643    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26d26802-41f4-4d17-8856-9073e4391d66" containerName="kserve-container"
Sep 17 18:14:15.480672 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:15.480650    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="26d26802-41f4-4d17-8856-9073e4391d66" containerName="kserve-container"
Sep 17 18:14:15.480672 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:15.480659    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b74dee3-f906-43ae-8a01-9bd07dd3b779" containerName="kserve-container"
Sep 17 18:14:15.480672 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:15.480667    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b74dee3-f906-43ae-8a01-9bd07dd3b779" containerName="kserve-container"
Sep 17 18:14:15.481202 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:15.480689    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26d26802-41f4-4d17-8856-9073e4391d66" containerName="kube-rbac-proxy"
Sep 17 18:14:15.481202 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:15.480696    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="26d26802-41f4-4d17-8856-9073e4391d66" containerName="kube-rbac-proxy"
Sep 17 18:14:15.481202 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:15.480778    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="26d26802-41f4-4d17-8856-9073e4391d66" containerName="kserve-container"
Sep 17 18:14:15.481202 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:15.480790    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="26d26802-41f4-4d17-8856-9073e4391d66" containerName="kube-rbac-proxy"
Sep 17 18:14:15.481202 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:15.480804    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b74dee3-f906-43ae-8a01-9bd07dd3b779" containerName="kserve-container"
Sep 17 18:14:15.481202 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:15.480814    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b74dee3-f906-43ae-8a01-9bd07dd3b779" containerName="kube-rbac-proxy"
Sep 17 18:14:15.483965 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:15.483943    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x6dsj"
Sep 17 18:14:15.486711 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:15.486691    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 17 18:14:15.486837 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:15.486809    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:14:15.487238 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:15.487220    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:14:15.488088 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:15.488056    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 17 18:14:15.488088 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:15.488070    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-kkx8f\""
Sep 17 18:14:15.488224 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:15.488099    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:14:15.488224 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:15.488056    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:14:15.495481 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:15.495461    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x6dsj"]
Sep 17 18:14:15.508349 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:15.508329    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b74dee3-f906-43ae-8a01-9bd07dd3b779" path="/var/lib/kubelet/pods/7b74dee3-f906-43ae-8a01-9bd07dd3b779/volumes"
Sep 17 18:14:15.508800 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:15.508784    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ktnm\" (UniqueName: \"kubernetes.io/projected/04bf96d9-6455-43e8-bc33-59eec7b8c8d1-kube-api-access-4ktnm\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-x6dsj\" (UID: \"04bf96d9-6455-43e8-bc33-59eec7b8c8d1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x6dsj"
Sep 17 18:14:15.508865 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:15.508848    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/04bf96d9-6455-43e8-bc33-59eec7b8c8d1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-x6dsj\" (UID: \"04bf96d9-6455-43e8-bc33-59eec7b8c8d1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x6dsj"
Sep 17 18:14:15.508913 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:15.508876    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04bf96d9-6455-43e8-bc33-59eec7b8c8d1-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-x6dsj\" (UID: \"04bf96d9-6455-43e8-bc33-59eec7b8c8d1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x6dsj"
Sep 17 18:14:15.508913 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:15.508893    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/04bf96d9-6455-43e8-bc33-59eec7b8c8d1-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-x6dsj\" (UID: \"04bf96d9-6455-43e8-bc33-59eec7b8c8d1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x6dsj"
Sep 17 18:14:15.509007 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:15.508990    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/04bf96d9-6455-43e8-bc33-59eec7b8c8d1-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-x6dsj\" (UID: \"04bf96d9-6455-43e8-bc33-59eec7b8c8d1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x6dsj"
Sep 17 18:14:15.609575 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:15.609540    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/04bf96d9-6455-43e8-bc33-59eec7b8c8d1-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-x6dsj\" (UID: \"04bf96d9-6455-43e8-bc33-59eec7b8c8d1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x6dsj"
Sep 17 18:14:15.609575 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:15.609581    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ktnm\" (UniqueName: \"kubernetes.io/projected/04bf96d9-6455-43e8-bc33-59eec7b8c8d1-kube-api-access-4ktnm\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-x6dsj\" (UID: \"04bf96d9-6455-43e8-bc33-59eec7b8c8d1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x6dsj"
Sep 17 18:14:15.609787 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:15.609654    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/04bf96d9-6455-43e8-bc33-59eec7b8c8d1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-x6dsj\" (UID: \"04bf96d9-6455-43e8-bc33-59eec7b8c8d1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x6dsj"
Sep 17 18:14:15.609787 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:15.609684    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04bf96d9-6455-43e8-bc33-59eec7b8c8d1-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-x6dsj\" (UID: \"04bf96d9-6455-43e8-bc33-59eec7b8c8d1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x6dsj"
Sep 17 18:14:15.609787 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:15.609706    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/04bf96d9-6455-43e8-bc33-59eec7b8c8d1-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-x6dsj\" (UID: \"04bf96d9-6455-43e8-bc33-59eec7b8c8d1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x6dsj"
Sep 17 18:14:15.610026 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:15.610001    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/04bf96d9-6455-43e8-bc33-59eec7b8c8d1-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-x6dsj\" (UID: \"04bf96d9-6455-43e8-bc33-59eec7b8c8d1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x6dsj"
Sep 17 18:14:15.610373 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:15.610352    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/04bf96d9-6455-43e8-bc33-59eec7b8c8d1-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-x6dsj\" (UID: \"04bf96d9-6455-43e8-bc33-59eec7b8c8d1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x6dsj"
Sep 17 18:14:15.610469 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:15.610369    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/04bf96d9-6455-43e8-bc33-59eec7b8c8d1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-x6dsj\" (UID: \"04bf96d9-6455-43e8-bc33-59eec7b8c8d1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x6dsj"
Sep 17 18:14:15.612181 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:15.612162    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04bf96d9-6455-43e8-bc33-59eec7b8c8d1-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-x6dsj\" (UID: \"04bf96d9-6455-43e8-bc33-59eec7b8c8d1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x6dsj"
Sep 17 18:14:15.623066 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:15.623038    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ktnm\" (UniqueName: \"kubernetes.io/projected/04bf96d9-6455-43e8-bc33-59eec7b8c8d1-kube-api-access-4ktnm\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-x6dsj\" (UID: \"04bf96d9-6455-43e8-bc33-59eec7b8c8d1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x6dsj"
Sep 17 18:14:15.795423 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:15.795339    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x6dsj"
Sep 17 18:14:16.068220 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:16.068196    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x6dsj"]
Sep 17 18:14:16.070272 ip-10-0-131-14 kubenswrapper[2569]: W0917 18:14:16.070243    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04bf96d9_6455_43e8_bc33_59eec7b8c8d1.slice/crio-cdeb30b68c6cb6cec668781d4635cd238028cf3678ca7a74e0dff8d80403054d WatchSource:0}: Error finding container cdeb30b68c6cb6cec668781d4635cd238028cf3678ca7a74e0dff8d80403054d: Status 404 returned error can't find the container with id cdeb30b68c6cb6cec668781d4635cd238028cf3678ca7a74e0dff8d80403054d
Sep 17 18:14:16.824151 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:16.824116    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x6dsj" event={"ID":"04bf96d9-6455-43e8-bc33-59eec7b8c8d1","Type":"ContainerStarted","Data":"acc1448baea4460475e9755ee63b3fd04b93aec83c6332c605545944e5c3c9c9"}
Sep 17 18:14:16.824645 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:16.824194    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x6dsj" event={"ID":"04bf96d9-6455-43e8-bc33-59eec7b8c8d1","Type":"ContainerStarted","Data":"cdeb30b68c6cb6cec668781d4635cd238028cf3678ca7a74e0dff8d80403054d"}
Sep 17 18:14:20.842459 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:20.842422    2569 generic.go:356] "Generic (PLEG): container finished" podID="04bf96d9-6455-43e8-bc33-59eec7b8c8d1" containerID="acc1448baea4460475e9755ee63b3fd04b93aec83c6332c605545944e5c3c9c9" exitCode=0
Sep 17 18:14:20.842848 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:20.842494    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x6dsj" event={"ID":"04bf96d9-6455-43e8-bc33-59eec7b8c8d1","Type":"ContainerDied","Data":"acc1448baea4460475e9755ee63b3fd04b93aec83c6332c605545944e5c3c9c9"}
Sep 17 18:14:21.849429 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:21.849394    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x6dsj" event={"ID":"04bf96d9-6455-43e8-bc33-59eec7b8c8d1","Type":"ContainerStarted","Data":"72bf8cff7d30a696cb45acf864cdee237e03e0ef0e46760d483dce1bbab92b70"}
Sep 17 18:14:21.849859 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:21.849457    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x6dsj" event={"ID":"04bf96d9-6455-43e8-bc33-59eec7b8c8d1","Type":"ContainerStarted","Data":"52b4110754ea6e204bb2582dd5d0fd649977f8c79c8f3200ee9ebcdc7ee9aa70"}
Sep 17 18:14:21.849859 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:21.849721    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x6dsj"
Sep 17 18:14:21.849938 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:21.849866    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x6dsj"
Sep 17 18:14:21.851266 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:21.851237    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x6dsj" podUID="04bf96d9-6455-43e8-bc33-59eec7b8c8d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 18:14:21.870777 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:21.870735    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x6dsj" podStartSLOduration=2.873222551 podStartE2EDuration="6.870722238s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.997499687s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:14:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:14:21.869251666 +0000 UTC m=+2014.373254301" watchObservedRunningTime="2026-09-17 18:14:21.870722238 +0000 UTC m=+2014.374724863"
Sep 17 18:14:22.853577 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:22.853543    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x6dsj" podUID="04bf96d9-6455-43e8-bc33-59eec7b8c8d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 18:14:27.858631 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:27.858603    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x6dsj"
Sep 17 18:14:27.859116 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:27.859093    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x6dsj" podUID="04bf96d9-6455-43e8-bc33-59eec7b8c8d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 18:14:37.859567 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:37.859526    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x6dsj" podUID="04bf96d9-6455-43e8-bc33-59eec7b8c8d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 18:14:47.859992 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:47.859950    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x6dsj" podUID="04bf96d9-6455-43e8-bc33-59eec7b8c8d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 18:14:57.859531 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:14:57.859490    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x6dsj" podUID="04bf96d9-6455-43e8-bc33-59eec7b8c8d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 18:15:07.859836 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:07.859795    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x6dsj" podUID="04bf96d9-6455-43e8-bc33-59eec7b8c8d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 18:15:17.859321 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:17.859283    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x6dsj" podUID="04bf96d9-6455-43e8-bc33-59eec7b8c8d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 18:15:27.859931 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:27.859887    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x6dsj" podUID="04bf96d9-6455-43e8-bc33-59eec7b8c8d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 18:15:37.859784 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:37.859754    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x6dsj"
Sep 17 18:15:45.621177 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:45.621140    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x6dsj"]
Sep 17 18:15:45.621678 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:45.621398    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x6dsj" podUID="04bf96d9-6455-43e8-bc33-59eec7b8c8d1" containerName="kserve-container" containerID="cri-o://52b4110754ea6e204bb2582dd5d0fd649977f8c79c8f3200ee9ebcdc7ee9aa70" gracePeriod=30
Sep 17 18:15:45.621678 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:45.621440    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x6dsj" podUID="04bf96d9-6455-43e8-bc33-59eec7b8c8d1" containerName="kube-rbac-proxy" containerID="cri-o://72bf8cff7d30a696cb45acf864cdee237e03e0ef0e46760d483dce1bbab92b70" gracePeriod=30
Sep 17 18:15:46.183883 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:46.183852    2569 generic.go:356] "Generic (PLEG): container finished" podID="04bf96d9-6455-43e8-bc33-59eec7b8c8d1" containerID="72bf8cff7d30a696cb45acf864cdee237e03e0ef0e46760d483dce1bbab92b70" exitCode=2
Sep 17 18:15:46.184053 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:46.183897    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x6dsj" event={"ID":"04bf96d9-6455-43e8-bc33-59eec7b8c8d1","Type":"ContainerDied","Data":"72bf8cff7d30a696cb45acf864cdee237e03e0ef0e46760d483dce1bbab92b70"}
Sep 17 18:15:47.854783 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:47.854680    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x6dsj" podUID="04bf96d9-6455-43e8-bc33-59eec7b8c8d1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Sep 17 18:15:47.859540 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:47.859464    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x6dsj" podUID="04bf96d9-6455-43e8-bc33-59eec7b8c8d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 18:15:49.603565 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:49.603534    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-t7pqb_b8d76e28-4655-4d21-98d8-0324e4756aaf/console-operator/2.log"
Sep 17 18:15:49.606326 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:49.606306    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-t7pqb_b8d76e28-4655-4d21-98d8-0324e4756aaf/console-operator/2.log"
Sep 17 18:15:49.609291 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:49.609272    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k4mqn_6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff/ovn-acl-logging/0.log"
Sep 17 18:15:49.612552 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:49.612533    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k4mqn_6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff/ovn-acl-logging/0.log"
Sep 17 18:15:50.463039 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:50.463016    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x6dsj"
Sep 17 18:15:50.602781 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:50.602681    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4ktnm\" (UniqueName: \"kubernetes.io/projected/04bf96d9-6455-43e8-bc33-59eec7b8c8d1-kube-api-access-4ktnm\") pod \"04bf96d9-6455-43e8-bc33-59eec7b8c8d1\" (UID: \"04bf96d9-6455-43e8-bc33-59eec7b8c8d1\") "
Sep 17 18:15:50.602781 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:50.602752    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04bf96d9-6455-43e8-bc33-59eec7b8c8d1-proxy-tls\") pod \"04bf96d9-6455-43e8-bc33-59eec7b8c8d1\" (UID: \"04bf96d9-6455-43e8-bc33-59eec7b8c8d1\") "
Sep 17 18:15:50.602781 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:50.602780    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/04bf96d9-6455-43e8-bc33-59eec7b8c8d1-kube-rbac-proxy-sar-config\") pod \"04bf96d9-6455-43e8-bc33-59eec7b8c8d1\" (UID: \"04bf96d9-6455-43e8-bc33-59eec7b8c8d1\") "
Sep 17 18:15:50.603114 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:50.602827    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/04bf96d9-6455-43e8-bc33-59eec7b8c8d1-kserve-provision-location\") pod \"04bf96d9-6455-43e8-bc33-59eec7b8c8d1\" (UID: \"04bf96d9-6455-43e8-bc33-59eec7b8c8d1\") "
Sep 17 18:15:50.603114 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:50.602855    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/04bf96d9-6455-43e8-bc33-59eec7b8c8d1-cabundle-cert\") pod \"04bf96d9-6455-43e8-bc33-59eec7b8c8d1\" (UID: \"04bf96d9-6455-43e8-bc33-59eec7b8c8d1\") "
Sep 17 18:15:50.603239 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:50.603149    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04bf96d9-6455-43e8-bc33-59eec7b8c8d1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "04bf96d9-6455-43e8-bc33-59eec7b8c8d1" (UID: "04bf96d9-6455-43e8-bc33-59eec7b8c8d1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:15:50.603239 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:50.603197    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04bf96d9-6455-43e8-bc33-59eec7b8c8d1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "04bf96d9-6455-43e8-bc33-59eec7b8c8d1" (UID: "04bf96d9-6455-43e8-bc33-59eec7b8c8d1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:15:50.603311 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:50.603240    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04bf96d9-6455-43e8-bc33-59eec7b8c8d1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "04bf96d9-6455-43e8-bc33-59eec7b8c8d1" (UID: "04bf96d9-6455-43e8-bc33-59eec7b8c8d1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:15:50.604935 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:50.604912    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04bf96d9-6455-43e8-bc33-59eec7b8c8d1-kube-api-access-4ktnm" (OuterVolumeSpecName: "kube-api-access-4ktnm") pod "04bf96d9-6455-43e8-bc33-59eec7b8c8d1" (UID: "04bf96d9-6455-43e8-bc33-59eec7b8c8d1"). InnerVolumeSpecName "kube-api-access-4ktnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:15:50.605246 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:50.604944    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04bf96d9-6455-43e8-bc33-59eec7b8c8d1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "04bf96d9-6455-43e8-bc33-59eec7b8c8d1" (UID: "04bf96d9-6455-43e8-bc33-59eec7b8c8d1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:15:50.704438 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:50.704398    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ktnm\" (UniqueName: \"kubernetes.io/projected/04bf96d9-6455-43e8-bc33-59eec7b8c8d1-kube-api-access-4ktnm\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:15:50.704438 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:50.704428    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04bf96d9-6455-43e8-bc33-59eec7b8c8d1-proxy-tls\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:15:50.704438 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:50.704440    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/04bf96d9-6455-43e8-bc33-59eec7b8c8d1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:15:50.704682 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:50.704449    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/04bf96d9-6455-43e8-bc33-59eec7b8c8d1-kserve-provision-location\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:15:50.704682 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:50.704458    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/04bf96d9-6455-43e8-bc33-59eec7b8c8d1-cabundle-cert\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:15:50.849517 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:50.849485    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-lczcv"]
Sep 17 18:15:50.849780 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:50.849770    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04bf96d9-6455-43e8-bc33-59eec7b8c8d1" containerName="storage-initializer"
Sep 17 18:15:50.849846 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:50.849783    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="04bf96d9-6455-43e8-bc33-59eec7b8c8d1" containerName="storage-initializer"
Sep 17 18:15:50.849846 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:50.849808    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04bf96d9-6455-43e8-bc33-59eec7b8c8d1" containerName="kserve-container"
Sep 17 18:15:50.849846 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:50.849816    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="04bf96d9-6455-43e8-bc33-59eec7b8c8d1" containerName="kserve-container"
Sep 17 18:15:50.849846 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:50.849832    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04bf96d9-6455-43e8-bc33-59eec7b8c8d1" containerName="kube-rbac-proxy"
Sep 17 18:15:50.849846 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:50.849838    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="04bf96d9-6455-43e8-bc33-59eec7b8c8d1" containerName="kube-rbac-proxy"
Sep 17 18:15:50.850001 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:50.849883    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="04bf96d9-6455-43e8-bc33-59eec7b8c8d1" containerName="kube-rbac-proxy"
Sep 17 18:15:50.850001 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:50.849892    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="04bf96d9-6455-43e8-bc33-59eec7b8c8d1" containerName="kserve-container"
Sep 17 18:15:50.852988 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:50.852947    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-lczcv"
Sep 17 18:15:50.855511 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:50.855495    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Sep 17 18:15:50.855593 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:50.855576    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 17 18:15:50.862726 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:50.862701    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-lczcv"]
Sep 17 18:15:50.965905 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:50.965870    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-m5687"]
Sep 17 18:15:50.969249 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:50.969230    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-m5687"
Sep 17 18:15:50.971668 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:50.971648    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 17 18:15:50.982369 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:50.982343    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-m5687"]
Sep 17 18:15:51.007544 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:51.007515    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrmgv\" (UniqueName: \"kubernetes.io/projected/bd2e9cc4-f6aa-4f25-b347-f7a2689b5831-kube-api-access-vrmgv\") pod \"isvc-canary-rollback-v2-predictor-78968789f7-lczcv\" (UID: \"bd2e9cc4-f6aa-4f25-b347-f7a2689b5831\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-lczcv"
Sep 17 18:15:51.007667 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:51.007554    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd2e9cc4-f6aa-4f25-b347-f7a2689b5831-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-78968789f7-lczcv\" (UID: \"bd2e9cc4-f6aa-4f25-b347-f7a2689b5831\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-lczcv"
Sep 17 18:15:51.007667 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:51.007575    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd2e9cc4-f6aa-4f25-b347-f7a2689b5831-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-78968789f7-lczcv\" (UID: \"bd2e9cc4-f6aa-4f25-b347-f7a2689b5831\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-lczcv"
Sep 17 18:15:51.007667 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:51.007648    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd2e9cc4-f6aa-4f25-b347-f7a2689b5831-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-78968789f7-lczcv\" (UID: \"bd2e9cc4-f6aa-4f25-b347-f7a2689b5831\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-lczcv"
Sep 17 18:15:51.007800 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:51.007683    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd2e9cc4-f6aa-4f25-b347-f7a2689b5831-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-78968789f7-lczcv\" (UID: \"bd2e9cc4-f6aa-4f25-b347-f7a2689b5831\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-lczcv"
Sep 17 18:15:51.108419 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:51.108330    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c51911a9-04b0-41e4-917f-b1fdd27600d8-proxy-tls\") pod \"isvc-canary-rollback-predictor-76cbc5464-m5687\" (UID: \"c51911a9-04b0-41e4-917f-b1fdd27600d8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-m5687"
Sep 17 18:15:51.108419 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:51.108368    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c51911a9-04b0-41e4-917f-b1fdd27600d8-cabundle-cert\") pod \"isvc-canary-rollback-predictor-76cbc5464-m5687\" (UID: \"c51911a9-04b0-41e4-917f-b1fdd27600d8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-m5687"
Sep 17 18:15:51.108651 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:51.108419    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrmgv\" (UniqueName: \"kubernetes.io/projected/bd2e9cc4-f6aa-4f25-b347-f7a2689b5831-kube-api-access-vrmgv\") pod \"isvc-canary-rollback-v2-predictor-78968789f7-lczcv\" (UID: \"bd2e9cc4-f6aa-4f25-b347-f7a2689b5831\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-lczcv"
Sep 17 18:15:51.108651 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:51.108456    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd2e9cc4-f6aa-4f25-b347-f7a2689b5831-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-78968789f7-lczcv\" (UID: \"bd2e9cc4-f6aa-4f25-b347-f7a2689b5831\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-lczcv"
Sep 17 18:15:51.108651 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:51.108484    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd2e9cc4-f6aa-4f25-b347-f7a2689b5831-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-78968789f7-lczcv\" (UID: \"bd2e9cc4-f6aa-4f25-b347-f7a2689b5831\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-lczcv"
Sep 17 18:15:51.108828 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:51.108645    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zvnk\" (UniqueName: \"kubernetes.io/projected/c51911a9-04b0-41e4-917f-b1fdd27600d8-kube-api-access-8zvnk\") pod \"isvc-canary-rollback-predictor-76cbc5464-m5687\" (UID: \"c51911a9-04b0-41e4-917f-b1fdd27600d8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-m5687"
Sep 17 18:15:51.108828 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:51.108681    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c51911a9-04b0-41e4-917f-b1fdd27600d8-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-76cbc5464-m5687\" (UID: \"c51911a9-04b0-41e4-917f-b1fdd27600d8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-m5687"
Sep 17 18:15:51.108828 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:51.108712    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd2e9cc4-f6aa-4f25-b347-f7a2689b5831-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-78968789f7-lczcv\" (UID: \"bd2e9cc4-f6aa-4f25-b347-f7a2689b5831\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-lczcv"
Sep 17 18:15:51.108828 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:51.108757    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c51911a9-04b0-41e4-917f-b1fdd27600d8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-76cbc5464-m5687\" (UID: \"c51911a9-04b0-41e4-917f-b1fdd27600d8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-m5687"
Sep 17 18:15:51.108828 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:51.108812    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd2e9cc4-f6aa-4f25-b347-f7a2689b5831-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-78968789f7-lczcv\" (UID: \"bd2e9cc4-f6aa-4f25-b347-f7a2689b5831\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-lczcv"
Sep 17 18:15:51.109029 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:51.108878    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd2e9cc4-f6aa-4f25-b347-f7a2689b5831-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-78968789f7-lczcv\" (UID: \"bd2e9cc4-f6aa-4f25-b347-f7a2689b5831\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-lczcv"
Sep 17 18:15:51.109029 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:15:51.108929    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-serving-cert: secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Sep 17 18:15:51.109029 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:15:51.108982    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd2e9cc4-f6aa-4f25-b347-f7a2689b5831-proxy-tls podName:bd2e9cc4-f6aa-4f25-b347-f7a2689b5831 nodeName:}" failed. No retries permitted until 2026-09-17 18:15:51.608969046 +0000 UTC m=+2104.112971653 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/bd2e9cc4-f6aa-4f25-b347-f7a2689b5831-proxy-tls") pod "isvc-canary-rollback-v2-predictor-78968789f7-lczcv" (UID: "bd2e9cc4-f6aa-4f25-b347-f7a2689b5831") : secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Sep 17 18:15:51.109235 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:51.109218    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd2e9cc4-f6aa-4f25-b347-f7a2689b5831-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-78968789f7-lczcv\" (UID: \"bd2e9cc4-f6aa-4f25-b347-f7a2689b5831\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-lczcv"
Sep 17 18:15:51.109368 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:51.109352    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd2e9cc4-f6aa-4f25-b347-f7a2689b5831-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-78968789f7-lczcv\" (UID: \"bd2e9cc4-f6aa-4f25-b347-f7a2689b5831\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-lczcv"
Sep 17 18:15:51.119316 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:51.119295    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrmgv\" (UniqueName: \"kubernetes.io/projected/bd2e9cc4-f6aa-4f25-b347-f7a2689b5831-kube-api-access-vrmgv\") pod \"isvc-canary-rollback-v2-predictor-78968789f7-lczcv\" (UID: \"bd2e9cc4-f6aa-4f25-b347-f7a2689b5831\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-lczcv"
Sep 17 18:15:51.205089 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:51.205052    2569 generic.go:356] "Generic (PLEG): container finished" podID="04bf96d9-6455-43e8-bc33-59eec7b8c8d1" containerID="52b4110754ea6e204bb2582dd5d0fd649977f8c79c8f3200ee9ebcdc7ee9aa70" exitCode=0
Sep 17 18:15:51.205270 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:51.205130    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x6dsj"
Sep 17 18:15:51.205270 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:51.205146    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x6dsj" event={"ID":"04bf96d9-6455-43e8-bc33-59eec7b8c8d1","Type":"ContainerDied","Data":"52b4110754ea6e204bb2582dd5d0fd649977f8c79c8f3200ee9ebcdc7ee9aa70"}
Sep 17 18:15:51.205270 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:51.205213    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x6dsj" event={"ID":"04bf96d9-6455-43e8-bc33-59eec7b8c8d1","Type":"ContainerDied","Data":"cdeb30b68c6cb6cec668781d4635cd238028cf3678ca7a74e0dff8d80403054d"}
Sep 17 18:15:51.205270 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:51.205239    2569 scope.go:117] "RemoveContainer" containerID="72bf8cff7d30a696cb45acf864cdee237e03e0ef0e46760d483dce1bbab92b70"
Sep 17 18:15:51.209632 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:51.209607    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zvnk\" (UniqueName: \"kubernetes.io/projected/c51911a9-04b0-41e4-917f-b1fdd27600d8-kube-api-access-8zvnk\") pod \"isvc-canary-rollback-predictor-76cbc5464-m5687\" (UID: \"c51911a9-04b0-41e4-917f-b1fdd27600d8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-m5687"
Sep 17 18:15:51.209779 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:51.209646    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c51911a9-04b0-41e4-917f-b1fdd27600d8-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-76cbc5464-m5687\" (UID: \"c51911a9-04b0-41e4-917f-b1fdd27600d8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-m5687"
Sep 17 18:15:51.209865 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:51.209773    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c51911a9-04b0-41e4-917f-b1fdd27600d8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-76cbc5464-m5687\" (UID: \"c51911a9-04b0-41e4-917f-b1fdd27600d8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-m5687"
Sep 17 18:15:51.209928 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:51.209886    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c51911a9-04b0-41e4-917f-b1fdd27600d8-proxy-tls\") pod \"isvc-canary-rollback-predictor-76cbc5464-m5687\" (UID: \"c51911a9-04b0-41e4-917f-b1fdd27600d8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-m5687"
Sep 17 18:15:51.209986 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:51.209954    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c51911a9-04b0-41e4-917f-b1fdd27600d8-cabundle-cert\") pod \"isvc-canary-rollback-predictor-76cbc5464-m5687\" (UID: \"c51911a9-04b0-41e4-917f-b1fdd27600d8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-m5687"
Sep 17 18:15:51.210045 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:51.210023    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c51911a9-04b0-41e4-917f-b1fdd27600d8-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-76cbc5464-m5687\" (UID: \"c51911a9-04b0-41e4-917f-b1fdd27600d8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-m5687"
Sep 17 18:15:51.215682 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:51.210979    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c51911a9-04b0-41e4-917f-b1fdd27600d8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-76cbc5464-m5687\" (UID: \"c51911a9-04b0-41e4-917f-b1fdd27600d8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-m5687"
Sep 17 18:15:51.215682 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:51.211665    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c51911a9-04b0-41e4-917f-b1fdd27600d8-cabundle-cert\") pod \"isvc-canary-rollback-predictor-76cbc5464-m5687\" (UID: \"c51911a9-04b0-41e4-917f-b1fdd27600d8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-m5687"
Sep 17 18:15:51.215682 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:51.215334    2569 scope.go:117] "RemoveContainer" containerID="52b4110754ea6e204bb2582dd5d0fd649977f8c79c8f3200ee9ebcdc7ee9aa70"
Sep 17 18:15:51.218403 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:51.218378    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zvnk\" (UniqueName: \"kubernetes.io/projected/c51911a9-04b0-41e4-917f-b1fdd27600d8-kube-api-access-8zvnk\") pod \"isvc-canary-rollback-predictor-76cbc5464-m5687\" (UID: \"c51911a9-04b0-41e4-917f-b1fdd27600d8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-m5687"
Sep 17 18:15:51.219744 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:51.219724    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c51911a9-04b0-41e4-917f-b1fdd27600d8-proxy-tls\") pod \"isvc-canary-rollback-predictor-76cbc5464-m5687\" (UID: \"c51911a9-04b0-41e4-917f-b1fdd27600d8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-m5687"
Sep 17 18:15:51.224338 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:51.224318    2569 scope.go:117] "RemoveContainer" containerID="acc1448baea4460475e9755ee63b3fd04b93aec83c6332c605545944e5c3c9c9"
Sep 17 18:15:51.233166 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:51.233140    2569 scope.go:117] "RemoveContainer" containerID="72bf8cff7d30a696cb45acf864cdee237e03e0ef0e46760d483dce1bbab92b70"
Sep 17 18:15:51.233599 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:15:51.233579    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72bf8cff7d30a696cb45acf864cdee237e03e0ef0e46760d483dce1bbab92b70\": container with ID starting with 72bf8cff7d30a696cb45acf864cdee237e03e0ef0e46760d483dce1bbab92b70 not found: ID does not exist" containerID="72bf8cff7d30a696cb45acf864cdee237e03e0ef0e46760d483dce1bbab92b70"
Sep 17 18:15:51.233704 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:51.233609    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72bf8cff7d30a696cb45acf864cdee237e03e0ef0e46760d483dce1bbab92b70"} err="failed to get container status \"72bf8cff7d30a696cb45acf864cdee237e03e0ef0e46760d483dce1bbab92b70\": rpc error: code = NotFound desc = could not find container \"72bf8cff7d30a696cb45acf864cdee237e03e0ef0e46760d483dce1bbab92b70\": container with ID starting with 72bf8cff7d30a696cb45acf864cdee237e03e0ef0e46760d483dce1bbab92b70 not found: ID does not exist"
Sep 17 18:15:51.233704 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:51.233631    2569 scope.go:117] "RemoveContainer" containerID="52b4110754ea6e204bb2582dd5d0fd649977f8c79c8f3200ee9ebcdc7ee9aa70"
Sep 17 18:15:51.233914 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:15:51.233897    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52b4110754ea6e204bb2582dd5d0fd649977f8c79c8f3200ee9ebcdc7ee9aa70\": container with ID starting with 52b4110754ea6e204bb2582dd5d0fd649977f8c79c8f3200ee9ebcdc7ee9aa70 not found: ID does not exist" containerID="52b4110754ea6e204bb2582dd5d0fd649977f8c79c8f3200ee9ebcdc7ee9aa70"
Sep 17 18:15:51.233988 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:51.233923    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52b4110754ea6e204bb2582dd5d0fd649977f8c79c8f3200ee9ebcdc7ee9aa70"} err="failed to get container status \"52b4110754ea6e204bb2582dd5d0fd649977f8c79c8f3200ee9ebcdc7ee9aa70\": rpc error: code = NotFound desc = could not find container \"52b4110754ea6e204bb2582dd5d0fd649977f8c79c8f3200ee9ebcdc7ee9aa70\": container with ID starting with 52b4110754ea6e204bb2582dd5d0fd649977f8c79c8f3200ee9ebcdc7ee9aa70 not found: ID does not exist"
Sep 17 18:15:51.233988 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:51.233976    2569 scope.go:117] "RemoveContainer" containerID="acc1448baea4460475e9755ee63b3fd04b93aec83c6332c605545944e5c3c9c9"
Sep 17 18:15:51.234783 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:15:51.234759    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acc1448baea4460475e9755ee63b3fd04b93aec83c6332c605545944e5c3c9c9\": container with ID starting with acc1448baea4460475e9755ee63b3fd04b93aec83c6332c605545944e5c3c9c9 not found: ID does not exist" containerID="acc1448baea4460475e9755ee63b3fd04b93aec83c6332c605545944e5c3c9c9"
Sep 17 18:15:51.234900 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:51.234791    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acc1448baea4460475e9755ee63b3fd04b93aec83c6332c605545944e5c3c9c9"} err="failed to get container status \"acc1448baea4460475e9755ee63b3fd04b93aec83c6332c605545944e5c3c9c9\": rpc error: code = NotFound desc = could not find container \"acc1448baea4460475e9755ee63b3fd04b93aec83c6332c605545944e5c3c9c9\": container with ID starting with acc1448baea4460475e9755ee63b3fd04b93aec83c6332c605545944e5c3c9c9 not found: ID does not exist"
Sep 17 18:15:51.235805 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:51.235791    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x6dsj"]
Sep 17 18:15:51.237168 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:51.237149    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-x6dsj"]
Sep 17 18:15:51.279120 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:51.279090    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-m5687"
Sep 17 18:15:51.508463 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:51.508435    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04bf96d9-6455-43e8-bc33-59eec7b8c8d1" path="/var/lib/kubelet/pods/04bf96d9-6455-43e8-bc33-59eec7b8c8d1/volumes"
Sep 17 18:15:51.549440 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:51.549410    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-m5687"]
Sep 17 18:15:51.552155 ip-10-0-131-14 kubenswrapper[2569]: W0917 18:15:51.552129    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc51911a9_04b0_41e4_917f_b1fdd27600d8.slice/crio-767ee53c4675db908053d3fb493fbe8899e83b2acc36ab191ef20e2707612067 WatchSource:0}: Error finding container 767ee53c4675db908053d3fb493fbe8899e83b2acc36ab191ef20e2707612067: Status 404 returned error can't find the container with id 767ee53c4675db908053d3fb493fbe8899e83b2acc36ab191ef20e2707612067
Sep 17 18:15:51.614510 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:51.614479    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd2e9cc4-f6aa-4f25-b347-f7a2689b5831-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-78968789f7-lczcv\" (UID: \"bd2e9cc4-f6aa-4f25-b347-f7a2689b5831\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-lczcv"
Sep 17 18:15:51.616740 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:51.616717    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd2e9cc4-f6aa-4f25-b347-f7a2689b5831-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-78968789f7-lczcv\" (UID: \"bd2e9cc4-f6aa-4f25-b347-f7a2689b5831\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-lczcv"
Sep 17 18:15:51.763672 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:51.763584    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-lczcv"
Sep 17 18:15:52.036191 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:52.036166    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-lczcv"]
Sep 17 18:15:52.037599 ip-10-0-131-14 kubenswrapper[2569]: W0917 18:15:52.037576    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd2e9cc4_f6aa_4f25_b347_f7a2689b5831.slice/crio-c58aab55d645b5671e17d7f0a0b85dd22ef38580efdb8d8a18f265a91dfbe640 WatchSource:0}: Error finding container c58aab55d645b5671e17d7f0a0b85dd22ef38580efdb8d8a18f265a91dfbe640: Status 404 returned error can't find the container with id c58aab55d645b5671e17d7f0a0b85dd22ef38580efdb8d8a18f265a91dfbe640
Sep 17 18:15:52.212430 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:52.212383    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-m5687" event={"ID":"c51911a9-04b0-41e4-917f-b1fdd27600d8","Type":"ContainerStarted","Data":"c08c115f19b87bd6bf76d5906b48ec5e55cabe3da96cfdbae67d3308398ecdea"}
Sep 17 18:15:52.212623 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:52.212454    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-m5687" event={"ID":"c51911a9-04b0-41e4-917f-b1fdd27600d8","Type":"ContainerStarted","Data":"767ee53c4675db908053d3fb493fbe8899e83b2acc36ab191ef20e2707612067"}
Sep 17 18:15:52.214211 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:52.214182    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-lczcv" event={"ID":"bd2e9cc4-f6aa-4f25-b347-f7a2689b5831","Type":"ContainerStarted","Data":"0f321e9c429cbf5612a89f0453e55fa80403c4ebebc58f697e3e3068f407b448"}
Sep 17 18:15:52.214377 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:52.214236    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-lczcv" event={"ID":"bd2e9cc4-f6aa-4f25-b347-f7a2689b5831","Type":"ContainerStarted","Data":"c58aab55d645b5671e17d7f0a0b85dd22ef38580efdb8d8a18f265a91dfbe640"}
Sep 17 18:15:56.233320 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:56.233285    2569 generic.go:356] "Generic (PLEG): container finished" podID="c51911a9-04b0-41e4-917f-b1fdd27600d8" containerID="c08c115f19b87bd6bf76d5906b48ec5e55cabe3da96cfdbae67d3308398ecdea" exitCode=0
Sep 17 18:15:56.233768 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:56.233362    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-m5687" event={"ID":"c51911a9-04b0-41e4-917f-b1fdd27600d8","Type":"ContainerDied","Data":"c08c115f19b87bd6bf76d5906b48ec5e55cabe3da96cfdbae67d3308398ecdea"}
Sep 17 18:15:56.234973 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:56.234950    2569 generic.go:356] "Generic (PLEG): container finished" podID="bd2e9cc4-f6aa-4f25-b347-f7a2689b5831" containerID="0f321e9c429cbf5612a89f0453e55fa80403c4ebebc58f697e3e3068f407b448" exitCode=0
Sep 17 18:15:56.235118 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:56.235020    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-lczcv" event={"ID":"bd2e9cc4-f6aa-4f25-b347-f7a2689b5831","Type":"ContainerDied","Data":"0f321e9c429cbf5612a89f0453e55fa80403c4ebebc58f697e3e3068f407b448"}
Sep 17 18:15:57.241460 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:57.241424    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-lczcv" event={"ID":"bd2e9cc4-f6aa-4f25-b347-f7a2689b5831","Type":"ContainerStarted","Data":"52b8c131fa8270126b980078d08ec6a4cfad70685d09c084b27484f1701a077a"}
Sep 17 18:15:57.241961 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:57.241497    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-lczcv" event={"ID":"bd2e9cc4-f6aa-4f25-b347-f7a2689b5831","Type":"ContainerStarted","Data":"5b8aae732f53361e1de09a477c16640d01d4deb7408766516a5e2ebc9c68d387"}
Sep 17 18:15:57.241961 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:57.241827    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-lczcv"
Sep 17 18:15:57.241961 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:57.241933    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-lczcv"
Sep 17 18:15:57.243435 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:57.243402    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-lczcv" podUID="bd2e9cc4-f6aa-4f25-b347-f7a2689b5831" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 18:15:57.243911 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:57.243891    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-m5687" event={"ID":"c51911a9-04b0-41e4-917f-b1fdd27600d8","Type":"ContainerStarted","Data":"b9e5829dc6263406a071643054328e5f8311f15ca24f916f3f1199acb1fd290a"}
Sep 17 18:15:57.243969 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:57.243944    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-m5687" event={"ID":"c51911a9-04b0-41e4-917f-b1fdd27600d8","Type":"ContainerStarted","Data":"ff242569b5986f4f301f3c8c6d8615e28ee61abbdf199a403d89705310c13f3a"}
Sep 17 18:15:57.244241 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:57.244224    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-m5687"
Sep 17 18:15:57.262604 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:57.262552    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-lczcv" podStartSLOduration=3.482230784 podStartE2EDuration="7.262538363s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.780307579s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:15:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:15:57.261702214 +0000 UTC m=+2109.765704840" watchObservedRunningTime="2026-09-17 18:15:57.262538363 +0000 UTC m=+2109.766540988"
Sep 17 18:15:57.280487 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:57.280445    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-m5687" podStartSLOduration=3.482728231 podStartE2EDuration="7.280432182s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.797703951s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:15:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:15:57.27954152 +0000 UTC m=+2109.783544147" watchObservedRunningTime="2026-09-17 18:15:57.280432182 +0000 UTC m=+2109.784434806"
Sep 17 18:15:58.249177 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:58.249139    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-lczcv" podUID="bd2e9cc4-f6aa-4f25-b347-f7a2689b5831" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 18:15:58.249595 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:58.249195    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-m5687"
Sep 17 18:15:58.250389 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:58.250366    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-m5687" podUID="c51911a9-04b0-41e4-917f-b1fdd27600d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 18:15:59.253481 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:15:59.253445    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-m5687" podUID="c51911a9-04b0-41e4-917f-b1fdd27600d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 18:16:03.254804 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:16:03.254772    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-lczcv"
Sep 17 18:16:03.255265 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:16:03.255239    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-lczcv" podUID="bd2e9cc4-f6aa-4f25-b347-f7a2689b5831" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 18:16:04.258478 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:16:04.258450    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-m5687"
Sep 17 18:16:04.258990 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:16:04.258964    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-m5687" podUID="c51911a9-04b0-41e4-917f-b1fdd27600d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 18:16:13.255700 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:16:13.255660    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-lczcv" podUID="bd2e9cc4-f6aa-4f25-b347-f7a2689b5831" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 18:16:14.259819 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:16:14.259783    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-m5687" podUID="c51911a9-04b0-41e4-917f-b1fdd27600d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 18:16:23.255330 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:16:23.255288    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-lczcv" podUID="bd2e9cc4-f6aa-4f25-b347-f7a2689b5831" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 18:16:24.258923 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:16:24.258884    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-m5687" podUID="c51911a9-04b0-41e4-917f-b1fdd27600d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 18:16:33.255772 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:16:33.255689    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-lczcv" podUID="bd2e9cc4-f6aa-4f25-b347-f7a2689b5831" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 18:16:34.258932 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:16:34.258889    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-m5687" podUID="c51911a9-04b0-41e4-917f-b1fdd27600d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 18:16:43.255396 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:16:43.255348    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-lczcv" podUID="bd2e9cc4-f6aa-4f25-b347-f7a2689b5831" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 18:16:44.258895 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:16:44.258856    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-m5687" podUID="c51911a9-04b0-41e4-917f-b1fdd27600d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 18:16:53.255313 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:16:53.255268    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-lczcv" podUID="bd2e9cc4-f6aa-4f25-b347-f7a2689b5831" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 18:16:54.259494 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:16:54.259456    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-m5687" podUID="c51911a9-04b0-41e4-917f-b1fdd27600d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 18:17:03.255372 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:17:03.255332    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-lczcv" podUID="bd2e9cc4-f6aa-4f25-b347-f7a2689b5831" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 18:17:04.259757 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:17:04.259718    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-m5687" podUID="c51911a9-04b0-41e4-917f-b1fdd27600d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 18:17:05.504327 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:17:05.504280    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-lczcv" podUID="bd2e9cc4-f6aa-4f25-b347-f7a2689b5831" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 18:17:14.260221 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:17:14.260192    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-m5687"
Sep 17 18:17:15.508816 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:17:15.508789    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-lczcv"
Sep 17 18:17:20.876580 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:17:20.876548    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-lczcv"]
Sep 17 18:17:20.877187 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:17:20.876783    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-lczcv" podUID="bd2e9cc4-f6aa-4f25-b347-f7a2689b5831" containerName="kserve-container" containerID="cri-o://5b8aae732f53361e1de09a477c16640d01d4deb7408766516a5e2ebc9c68d387" gracePeriod=30
Sep 17 18:17:20.877187 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:17:20.876839    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-lczcv" podUID="bd2e9cc4-f6aa-4f25-b347-f7a2689b5831" containerName="kube-rbac-proxy" containerID="cri-o://52b8c131fa8270126b980078d08ec6a4cfad70685d09c084b27484f1701a077a" gracePeriod=30
Sep 17 18:17:21.594680 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:17:21.594646    2569 generic.go:356] "Generic (PLEG): container finished" podID="bd2e9cc4-f6aa-4f25-b347-f7a2689b5831" containerID="52b8c131fa8270126b980078d08ec6a4cfad70685d09c084b27484f1701a077a" exitCode=2
Sep 17 18:17:21.594848 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:17:21.594712    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-lczcv" event={"ID":"bd2e9cc4-f6aa-4f25-b347-f7a2689b5831","Type":"ContainerDied","Data":"52b8c131fa8270126b980078d08ec6a4cfad70685d09c084b27484f1701a077a"}
Sep 17 18:17:23.249782 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:17:23.249741    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-lczcv" podUID="bd2e9cc4-f6aa-4f25-b347-f7a2689b5831" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Sep 17 18:17:25.421436 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:17:25.421414    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-lczcv"
Sep 17 18:17:25.513392 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:17:25.513320    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd2e9cc4-f6aa-4f25-b347-f7a2689b5831-kserve-provision-location\") pod \"bd2e9cc4-f6aa-4f25-b347-f7a2689b5831\" (UID: \"bd2e9cc4-f6aa-4f25-b347-f7a2689b5831\") "
Sep 17 18:17:25.513392 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:17:25.513383    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd2e9cc4-f6aa-4f25-b347-f7a2689b5831-kube-rbac-proxy-sar-config\") pod \"bd2e9cc4-f6aa-4f25-b347-f7a2689b5831\" (UID: \"bd2e9cc4-f6aa-4f25-b347-f7a2689b5831\") "
Sep 17 18:17:25.513614 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:17:25.513416    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd2e9cc4-f6aa-4f25-b347-f7a2689b5831-proxy-tls\") pod \"bd2e9cc4-f6aa-4f25-b347-f7a2689b5831\" (UID: \"bd2e9cc4-f6aa-4f25-b347-f7a2689b5831\") "
Sep 17 18:17:25.513614 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:17:25.513452    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd2e9cc4-f6aa-4f25-b347-f7a2689b5831-cabundle-cert\") pod \"bd2e9cc4-f6aa-4f25-b347-f7a2689b5831\" (UID: \"bd2e9cc4-f6aa-4f25-b347-f7a2689b5831\") "
Sep 17 18:17:25.513614 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:17:25.513475    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrmgv\" (UniqueName: \"kubernetes.io/projected/bd2e9cc4-f6aa-4f25-b347-f7a2689b5831-kube-api-access-vrmgv\") pod \"bd2e9cc4-f6aa-4f25-b347-f7a2689b5831\" (UID: \"bd2e9cc4-f6aa-4f25-b347-f7a2689b5831\") "
Sep 17 18:17:25.513759 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:17:25.513699    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd2e9cc4-f6aa-4f25-b347-f7a2689b5831-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bd2e9cc4-f6aa-4f25-b347-f7a2689b5831" (UID: "bd2e9cc4-f6aa-4f25-b347-f7a2689b5831"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:17:25.513759 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:17:25.513728    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd2e9cc4-f6aa-4f25-b347-f7a2689b5831-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bd2e9cc4-f6aa-4f25-b347-f7a2689b5831" (UID: "bd2e9cc4-f6aa-4f25-b347-f7a2689b5831"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:17:25.513853 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:17:25.513809    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd2e9cc4-f6aa-4f25-b347-f7a2689b5831-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bd2e9cc4-f6aa-4f25-b347-f7a2689b5831" (UID: "bd2e9cc4-f6aa-4f25-b347-f7a2689b5831"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:17:25.515497 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:17:25.515476    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd2e9cc4-f6aa-4f25-b347-f7a2689b5831-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bd2e9cc4-f6aa-4f25-b347-f7a2689b5831" (UID: "bd2e9cc4-f6aa-4f25-b347-f7a2689b5831"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:17:25.515627 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:17:25.515591    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd2e9cc4-f6aa-4f25-b347-f7a2689b5831-kube-api-access-vrmgv" (OuterVolumeSpecName: "kube-api-access-vrmgv") pod "bd2e9cc4-f6aa-4f25-b347-f7a2689b5831" (UID: "bd2e9cc4-f6aa-4f25-b347-f7a2689b5831"). InnerVolumeSpecName "kube-api-access-vrmgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:17:25.612764 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:17:25.612733    2569 generic.go:356] "Generic (PLEG): container finished" podID="bd2e9cc4-f6aa-4f25-b347-f7a2689b5831" containerID="5b8aae732f53361e1de09a477c16640d01d4deb7408766516a5e2ebc9c68d387" exitCode=0
Sep 17 18:17:25.612913 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:17:25.612792    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-lczcv"
Sep 17 18:17:25.612913 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:17:25.612815    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-lczcv" event={"ID":"bd2e9cc4-f6aa-4f25-b347-f7a2689b5831","Type":"ContainerDied","Data":"5b8aae732f53361e1de09a477c16640d01d4deb7408766516a5e2ebc9c68d387"}
Sep 17 18:17:25.612913 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:17:25.612881    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-lczcv" event={"ID":"bd2e9cc4-f6aa-4f25-b347-f7a2689b5831","Type":"ContainerDied","Data":"c58aab55d645b5671e17d7f0a0b85dd22ef38580efdb8d8a18f265a91dfbe640"}
Sep 17 18:17:25.612913 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:17:25.612909    2569 scope.go:117] "RemoveContainer" containerID="52b8c131fa8270126b980078d08ec6a4cfad70685d09c084b27484f1701a077a"
Sep 17 18:17:25.615974 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:17:25.614743    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd2e9cc4-f6aa-4f25-b347-f7a2689b5831-proxy-tls\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:17:25.615974 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:17:25.614772    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd2e9cc4-f6aa-4f25-b347-f7a2689b5831-cabundle-cert\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:17:25.615974 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:17:25.614792    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrmgv\" (UniqueName: \"kubernetes.io/projected/bd2e9cc4-f6aa-4f25-b347-f7a2689b5831-kube-api-access-vrmgv\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:17:25.615974 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:17:25.614805    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd2e9cc4-f6aa-4f25-b347-f7a2689b5831-kserve-provision-location\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:17:25.615974 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:17:25.614820    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd2e9cc4-f6aa-4f25-b347-f7a2689b5831-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:17:25.628283 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:17:25.628251    2569 scope.go:117] "RemoveContainer" containerID="5b8aae732f53361e1de09a477c16640d01d4deb7408766516a5e2ebc9c68d387"
Sep 17 18:17:25.637837 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:17:25.637793    2569 scope.go:117] "RemoveContainer" containerID="0f321e9c429cbf5612a89f0453e55fa80403c4ebebc58f697e3e3068f407b448"
Sep 17 18:17:25.638414 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:17:25.638390    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-lczcv"]
Sep 17 18:17:25.644612 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:17:25.644591    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-78968789f7-lczcv"]
Sep 17 18:17:25.646019 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:17:25.645999    2569 scope.go:117] "RemoveContainer" containerID="52b8c131fa8270126b980078d08ec6a4cfad70685d09c084b27484f1701a077a"
Sep 17 18:17:25.646308 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:17:25.646288    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52b8c131fa8270126b980078d08ec6a4cfad70685d09c084b27484f1701a077a\": container with ID starting with 52b8c131fa8270126b980078d08ec6a4cfad70685d09c084b27484f1701a077a not found: ID does not exist" containerID="52b8c131fa8270126b980078d08ec6a4cfad70685d09c084b27484f1701a077a"
Sep 17 18:17:25.646361 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:17:25.646315    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52b8c131fa8270126b980078d08ec6a4cfad70685d09c084b27484f1701a077a"} err="failed to get container status \"52b8c131fa8270126b980078d08ec6a4cfad70685d09c084b27484f1701a077a\": rpc error: code = NotFound desc = could not find container \"52b8c131fa8270126b980078d08ec6a4cfad70685d09c084b27484f1701a077a\": container with ID starting with 52b8c131fa8270126b980078d08ec6a4cfad70685d09c084b27484f1701a077a not found: ID does not exist"
Sep 17 18:17:25.646361 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:17:25.646332    2569 scope.go:117] "RemoveContainer" containerID="5b8aae732f53361e1de09a477c16640d01d4deb7408766516a5e2ebc9c68d387"
Sep 17 18:17:25.646563 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:17:25.646545    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b8aae732f53361e1de09a477c16640d01d4deb7408766516a5e2ebc9c68d387\": container with ID starting with 5b8aae732f53361e1de09a477c16640d01d4deb7408766516a5e2ebc9c68d387 not found: ID does not exist" containerID="5b8aae732f53361e1de09a477c16640d01d4deb7408766516a5e2ebc9c68d387"
Sep 17 18:17:25.646608 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:17:25.646575    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b8aae732f53361e1de09a477c16640d01d4deb7408766516a5e2ebc9c68d387"} err="failed to get container status \"5b8aae732f53361e1de09a477c16640d01d4deb7408766516a5e2ebc9c68d387\": rpc error: code = NotFound desc = could not find container \"5b8aae732f53361e1de09a477c16640d01d4deb7408766516a5e2ebc9c68d387\": container with ID starting with 5b8aae732f53361e1de09a477c16640d01d4deb7408766516a5e2ebc9c68d387 not found: ID does not exist"
Sep 17 18:17:25.646608 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:17:25.646590    2569 scope.go:117] "RemoveContainer" containerID="0f321e9c429cbf5612a89f0453e55fa80403c4ebebc58f697e3e3068f407b448"
Sep 17 18:17:25.646804 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:17:25.646791    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f321e9c429cbf5612a89f0453e55fa80403c4ebebc58f697e3e3068f407b448\": container with ID starting with 0f321e9c429cbf5612a89f0453e55fa80403c4ebebc58f697e3e3068f407b448 not found: ID does not exist" containerID="0f321e9c429cbf5612a89f0453e55fa80403c4ebebc58f697e3e3068f407b448"
Sep 17 18:17:25.646846 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:17:25.646806    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f321e9c429cbf5612a89f0453e55fa80403c4ebebc58f697e3e3068f407b448"} err="failed to get container status \"0f321e9c429cbf5612a89f0453e55fa80403c4ebebc58f697e3e3068f407b448\": rpc error: code = NotFound desc = could not find container \"0f321e9c429cbf5612a89f0453e55fa80403c4ebebc58f697e3e3068f407b448\": container with ID starting with 0f321e9c429cbf5612a89f0453e55fa80403c4ebebc58f697e3e3068f407b448 not found: ID does not exist"
Sep 17 18:17:27.508226 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:17:27.508195    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd2e9cc4-f6aa-4f25-b347-f7a2689b5831" path="/var/lib/kubelet/pods/bd2e9cc4-f6aa-4f25-b347-f7a2689b5831/volumes"
Sep 17 18:18:51.022357 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:51.022327    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-m5687"]
Sep 17 18:18:51.022855 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:51.022592    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-m5687" podUID="c51911a9-04b0-41e4-917f-b1fdd27600d8" containerName="kserve-container" containerID="cri-o://ff242569b5986f4f301f3c8c6d8615e28ee61abbdf199a403d89705310c13f3a" gracePeriod=30
Sep 17 18:18:51.022855 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:51.022637    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-m5687" podUID="c51911a9-04b0-41e4-917f-b1fdd27600d8" containerName="kube-rbac-proxy" containerID="cri-o://b9e5829dc6263406a071643054328e5f8311f15ca24f916f3f1199acb1fd290a" gracePeriod=30
Sep 17 18:18:51.964695 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:51.964658    2569 generic.go:356] "Generic (PLEG): container finished" podID="c51911a9-04b0-41e4-917f-b1fdd27600d8" containerID="b9e5829dc6263406a071643054328e5f8311f15ca24f916f3f1199acb1fd290a" exitCode=2
Sep 17 18:18:51.964869 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:51.964731    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-m5687" event={"ID":"c51911a9-04b0-41e4-917f-b1fdd27600d8","Type":"ContainerDied","Data":"b9e5829dc6263406a071643054328e5f8311f15ca24f916f3f1199acb1fd290a"}
Sep 17 18:18:54.254613 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:54.254575    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-m5687" podUID="c51911a9-04b0-41e4-917f-b1fdd27600d8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Sep 17 18:18:54.258897 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:54.258872    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-m5687" podUID="c51911a9-04b0-41e4-917f-b1fdd27600d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 18:18:55.463298 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:55.463275    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-m5687"
Sep 17 18:18:55.534999 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:55.534934    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c51911a9-04b0-41e4-917f-b1fdd27600d8-kube-rbac-proxy-sar-config\") pod \"c51911a9-04b0-41e4-917f-b1fdd27600d8\" (UID: \"c51911a9-04b0-41e4-917f-b1fdd27600d8\") "
Sep 17 18:18:55.535140 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:55.535016    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c51911a9-04b0-41e4-917f-b1fdd27600d8-kserve-provision-location\") pod \"c51911a9-04b0-41e4-917f-b1fdd27600d8\" (UID: \"c51911a9-04b0-41e4-917f-b1fdd27600d8\") "
Sep 17 18:18:55.535140 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:55.535102    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c51911a9-04b0-41e4-917f-b1fdd27600d8-proxy-tls\") pod \"c51911a9-04b0-41e4-917f-b1fdd27600d8\" (UID: \"c51911a9-04b0-41e4-917f-b1fdd27600d8\") "
Sep 17 18:18:55.535140 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:55.535132    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c51911a9-04b0-41e4-917f-b1fdd27600d8-cabundle-cert\") pod \"c51911a9-04b0-41e4-917f-b1fdd27600d8\" (UID: \"c51911a9-04b0-41e4-917f-b1fdd27600d8\") "
Sep 17 18:18:55.535308 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:55.535155    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8zvnk\" (UniqueName: \"kubernetes.io/projected/c51911a9-04b0-41e4-917f-b1fdd27600d8-kube-api-access-8zvnk\") pod \"c51911a9-04b0-41e4-917f-b1fdd27600d8\" (UID: \"c51911a9-04b0-41e4-917f-b1fdd27600d8\") "
Sep 17 18:18:55.535367 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:55.535332    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c51911a9-04b0-41e4-917f-b1fdd27600d8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c51911a9-04b0-41e4-917f-b1fdd27600d8" (UID: "c51911a9-04b0-41e4-917f-b1fdd27600d8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:18:55.535597 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:55.535349    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c51911a9-04b0-41e4-917f-b1fdd27600d8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c51911a9-04b0-41e4-917f-b1fdd27600d8" (UID: "c51911a9-04b0-41e4-917f-b1fdd27600d8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:18:55.535597 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:55.535487    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c51911a9-04b0-41e4-917f-b1fdd27600d8-kserve-provision-location\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:18:55.535597 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:55.535500    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c51911a9-04b0-41e4-917f-b1fdd27600d8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:18:55.535597 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:55.535549    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c51911a9-04b0-41e4-917f-b1fdd27600d8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c51911a9-04b0-41e4-917f-b1fdd27600d8" (UID: "c51911a9-04b0-41e4-917f-b1fdd27600d8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:18:55.537240 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:55.537214    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c51911a9-04b0-41e4-917f-b1fdd27600d8-kube-api-access-8zvnk" (OuterVolumeSpecName: "kube-api-access-8zvnk") pod "c51911a9-04b0-41e4-917f-b1fdd27600d8" (UID: "c51911a9-04b0-41e4-917f-b1fdd27600d8"). InnerVolumeSpecName "kube-api-access-8zvnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:18:55.537313 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:55.537239    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c51911a9-04b0-41e4-917f-b1fdd27600d8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c51911a9-04b0-41e4-917f-b1fdd27600d8" (UID: "c51911a9-04b0-41e4-917f-b1fdd27600d8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:18:55.636755 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:55.636732    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c51911a9-04b0-41e4-917f-b1fdd27600d8-cabundle-cert\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:18:55.636755 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:55.636754    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zvnk\" (UniqueName: \"kubernetes.io/projected/c51911a9-04b0-41e4-917f-b1fdd27600d8-kube-api-access-8zvnk\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:18:55.636911 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:55.636765    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c51911a9-04b0-41e4-917f-b1fdd27600d8-proxy-tls\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:18:55.982919 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:55.982884    2569 generic.go:356] "Generic (PLEG): container finished" podID="c51911a9-04b0-41e4-917f-b1fdd27600d8" containerID="ff242569b5986f4f301f3c8c6d8615e28ee61abbdf199a403d89705310c13f3a" exitCode=0
Sep 17 18:18:55.983112 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:55.982924    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-m5687" event={"ID":"c51911a9-04b0-41e4-917f-b1fdd27600d8","Type":"ContainerDied","Data":"ff242569b5986f4f301f3c8c6d8615e28ee61abbdf199a403d89705310c13f3a"}
Sep 17 18:18:55.983112 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:55.982935    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-m5687"
Sep 17 18:18:55.983112 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:55.982982    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-m5687" event={"ID":"c51911a9-04b0-41e4-917f-b1fdd27600d8","Type":"ContainerDied","Data":"767ee53c4675db908053d3fb493fbe8899e83b2acc36ab191ef20e2707612067"}
Sep 17 18:18:55.983112 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:55.983017    2569 scope.go:117] "RemoveContainer" containerID="b9e5829dc6263406a071643054328e5f8311f15ca24f916f3f1199acb1fd290a"
Sep 17 18:18:55.991891 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:55.991870    2569 scope.go:117] "RemoveContainer" containerID="ff242569b5986f4f301f3c8c6d8615e28ee61abbdf199a403d89705310c13f3a"
Sep 17 18:18:55.999012 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:55.998995    2569 scope.go:117] "RemoveContainer" containerID="c08c115f19b87bd6bf76d5906b48ec5e55cabe3da96cfdbae67d3308398ecdea"
Sep 17 18:18:56.005500 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.005480    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-m5687"]
Sep 17 18:18:56.006117 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.006104    2569 scope.go:117] "RemoveContainer" containerID="b9e5829dc6263406a071643054328e5f8311f15ca24f916f3f1199acb1fd290a"
Sep 17 18:18:56.006362 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:18:56.006346    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9e5829dc6263406a071643054328e5f8311f15ca24f916f3f1199acb1fd290a\": container with ID starting with b9e5829dc6263406a071643054328e5f8311f15ca24f916f3f1199acb1fd290a not found: ID does not exist" containerID="b9e5829dc6263406a071643054328e5f8311f15ca24f916f3f1199acb1fd290a"
Sep 17 18:18:56.006406 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.006369    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9e5829dc6263406a071643054328e5f8311f15ca24f916f3f1199acb1fd290a"} err="failed to get container status \"b9e5829dc6263406a071643054328e5f8311f15ca24f916f3f1199acb1fd290a\": rpc error: code = NotFound desc = could not find container \"b9e5829dc6263406a071643054328e5f8311f15ca24f916f3f1199acb1fd290a\": container with ID starting with b9e5829dc6263406a071643054328e5f8311f15ca24f916f3f1199acb1fd290a not found: ID does not exist"
Sep 17 18:18:56.006406 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.006385    2569 scope.go:117] "RemoveContainer" containerID="ff242569b5986f4f301f3c8c6d8615e28ee61abbdf199a403d89705310c13f3a"
Sep 17 18:18:56.006635 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:18:56.006617    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff242569b5986f4f301f3c8c6d8615e28ee61abbdf199a403d89705310c13f3a\": container with ID starting with ff242569b5986f4f301f3c8c6d8615e28ee61abbdf199a403d89705310c13f3a not found: ID does not exist" containerID="ff242569b5986f4f301f3c8c6d8615e28ee61abbdf199a403d89705310c13f3a"
Sep 17 18:18:56.006681 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.006641    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff242569b5986f4f301f3c8c6d8615e28ee61abbdf199a403d89705310c13f3a"} err="failed to get container status \"ff242569b5986f4f301f3c8c6d8615e28ee61abbdf199a403d89705310c13f3a\": rpc error: code = NotFound desc = could not find container \"ff242569b5986f4f301f3c8c6d8615e28ee61abbdf199a403d89705310c13f3a\": container with ID starting with ff242569b5986f4f301f3c8c6d8615e28ee61abbdf199a403d89705310c13f3a not found: ID does not exist"
Sep 17 18:18:56.006681 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.006657    2569 scope.go:117] "RemoveContainer" containerID="c08c115f19b87bd6bf76d5906b48ec5e55cabe3da96cfdbae67d3308398ecdea"
Sep 17 18:18:56.006887 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:18:56.006870    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c08c115f19b87bd6bf76d5906b48ec5e55cabe3da96cfdbae67d3308398ecdea\": container with ID starting with c08c115f19b87bd6bf76d5906b48ec5e55cabe3da96cfdbae67d3308398ecdea not found: ID does not exist" containerID="c08c115f19b87bd6bf76d5906b48ec5e55cabe3da96cfdbae67d3308398ecdea"
Sep 17 18:18:56.006941 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.006896    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c08c115f19b87bd6bf76d5906b48ec5e55cabe3da96cfdbae67d3308398ecdea"} err="failed to get container status \"c08c115f19b87bd6bf76d5906b48ec5e55cabe3da96cfdbae67d3308398ecdea\": rpc error: code = NotFound desc = could not find container \"c08c115f19b87bd6bf76d5906b48ec5e55cabe3da96cfdbae67d3308398ecdea\": container with ID starting with c08c115f19b87bd6bf76d5906b48ec5e55cabe3da96cfdbae67d3308398ecdea not found: ID does not exist"
Sep 17 18:18:56.010841 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.010822    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-m5687"]
Sep 17 18:18:56.191271 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.191243    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-sfv8x"]
Sep 17 18:18:56.191540 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.191527    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd2e9cc4-f6aa-4f25-b347-f7a2689b5831" containerName="storage-initializer"
Sep 17 18:18:56.191540 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.191540    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd2e9cc4-f6aa-4f25-b347-f7a2689b5831" containerName="storage-initializer"
Sep 17 18:18:56.191621 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.191548    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c51911a9-04b0-41e4-917f-b1fdd27600d8" containerName="kserve-container"
Sep 17 18:18:56.191621 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.191554    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c51911a9-04b0-41e4-917f-b1fdd27600d8" containerName="kserve-container"
Sep 17 18:18:56.191621 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.191564    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c51911a9-04b0-41e4-917f-b1fdd27600d8" containerName="kube-rbac-proxy"
Sep 17 18:18:56.191621 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.191569    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c51911a9-04b0-41e4-917f-b1fdd27600d8" containerName="kube-rbac-proxy"
Sep 17 18:18:56.191621 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.191579    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c51911a9-04b0-41e4-917f-b1fdd27600d8" containerName="storage-initializer"
Sep 17 18:18:56.191621 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.191584    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c51911a9-04b0-41e4-917f-b1fdd27600d8" containerName="storage-initializer"
Sep 17 18:18:56.191621 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.191593    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd2e9cc4-f6aa-4f25-b347-f7a2689b5831" containerName="kube-rbac-proxy"
Sep 17 18:18:56.191621 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.191599    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd2e9cc4-f6aa-4f25-b347-f7a2689b5831" containerName="kube-rbac-proxy"
Sep 17 18:18:56.191621 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.191608    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd2e9cc4-f6aa-4f25-b347-f7a2689b5831" containerName="kserve-container"
Sep 17 18:18:56.191621 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.191613    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd2e9cc4-f6aa-4f25-b347-f7a2689b5831" containerName="kserve-container"
Sep 17 18:18:56.191958 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.191654    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c51911a9-04b0-41e4-917f-b1fdd27600d8" containerName="kube-rbac-proxy"
Sep 17 18:18:56.191958 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.191662    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c51911a9-04b0-41e4-917f-b1fdd27600d8" containerName="kserve-container"
Sep 17 18:18:56.191958 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.191671    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd2e9cc4-f6aa-4f25-b347-f7a2689b5831" containerName="kube-rbac-proxy"
Sep 17 18:18:56.191958 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.191677    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd2e9cc4-f6aa-4f25-b347-f7a2689b5831" containerName="kserve-container"
Sep 17 18:18:56.195999 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.195982    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-sfv8x"
Sep 17 18:18:56.198650 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.198626    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Sep 17 18:18:56.198856 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.198839    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-kkx8f\""
Sep 17 18:18:56.198901 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.198873    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:18:56.199118 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.199100    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:18:56.199182 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.199127    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 17 18:18:56.199237 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.199130    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:18:56.199237 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.199199    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:18:56.205599 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.205575    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-sfv8x"]
Sep 17 18:18:56.231171 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.231144    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-lh5cv"]
Sep 17 18:18:56.234447 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.234405    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-lh5cv"
Sep 17 18:18:56.236968 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.236948    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 17 18:18:56.242016 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.241998    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxp9h\" (UniqueName: \"kubernetes.io/projected/87fa04bd-2308-41bc-8818-07e1c6b51295-kube-api-access-gxp9h\") pod \"isvc-canary-stop-v2-predictor-6cbcf9f68f-sfv8x\" (UID: \"87fa04bd-2308-41bc-8818-07e1c6b51295\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-sfv8x"
Sep 17 18:18:56.242139 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.242044    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/87fa04bd-2308-41bc-8818-07e1c6b51295-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-6cbcf9f68f-sfv8x\" (UID: \"87fa04bd-2308-41bc-8818-07e1c6b51295\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-sfv8x"
Sep 17 18:18:56.242139 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.242070    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87fa04bd-2308-41bc-8818-07e1c6b51295-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-6cbcf9f68f-sfv8x\" (UID: \"87fa04bd-2308-41bc-8818-07e1c6b51295\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-sfv8x"
Sep 17 18:18:56.242139 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.242126    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/87fa04bd-2308-41bc-8818-07e1c6b51295-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-6cbcf9f68f-sfv8x\" (UID: \"87fa04bd-2308-41bc-8818-07e1c6b51295\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-sfv8x"
Sep 17 18:18:56.242272 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.242193    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87fa04bd-2308-41bc-8818-07e1c6b51295-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-6cbcf9f68f-sfv8x\" (UID: \"87fa04bd-2308-41bc-8818-07e1c6b51295\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-sfv8x"
Sep 17 18:18:56.244981 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.244959    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-lh5cv"]
Sep 17 18:18:56.343534 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.343504    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87fa04bd-2308-41bc-8818-07e1c6b51295-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-6cbcf9f68f-sfv8x\" (UID: \"87fa04bd-2308-41bc-8818-07e1c6b51295\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-sfv8x"
Sep 17 18:18:56.343674 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.343545    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/20456cd0-3433-42d5-bf03-8c69516c0774-proxy-tls\") pod \"isvc-canary-stop-predictor-646f5bdbcb-lh5cv\" (UID: \"20456cd0-3433-42d5-bf03-8c69516c0774\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-lh5cv"
Sep 17 18:18:56.343674 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.343583    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20456cd0-3433-42d5-bf03-8c69516c0774-kserve-provision-location\") pod \"isvc-canary-stop-predictor-646f5bdbcb-lh5cv\" (UID: \"20456cd0-3433-42d5-bf03-8c69516c0774\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-lh5cv"
Sep 17 18:18:56.343674 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.343615    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxp9h\" (UniqueName: \"kubernetes.io/projected/87fa04bd-2308-41bc-8818-07e1c6b51295-kube-api-access-gxp9h\") pod \"isvc-canary-stop-v2-predictor-6cbcf9f68f-sfv8x\" (UID: \"87fa04bd-2308-41bc-8818-07e1c6b51295\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-sfv8x"
Sep 17 18:18:56.343674 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.343648    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/20456cd0-3433-42d5-bf03-8c69516c0774-cabundle-cert\") pod \"isvc-canary-stop-predictor-646f5bdbcb-lh5cv\" (UID: \"20456cd0-3433-42d5-bf03-8c69516c0774\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-lh5cv"
Sep 17 18:18:56.343867 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.343808    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/87fa04bd-2308-41bc-8818-07e1c6b51295-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-6cbcf9f68f-sfv8x\" (UID: \"87fa04bd-2308-41bc-8818-07e1c6b51295\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-sfv8x"
Sep 17 18:18:56.343867 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.343842    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87fa04bd-2308-41bc-8818-07e1c6b51295-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-6cbcf9f68f-sfv8x\" (UID: \"87fa04bd-2308-41bc-8818-07e1c6b51295\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-sfv8x"
Sep 17 18:18:56.343978 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.343882    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrwcs\" (UniqueName: \"kubernetes.io/projected/20456cd0-3433-42d5-bf03-8c69516c0774-kube-api-access-mrwcs\") pod \"isvc-canary-stop-predictor-646f5bdbcb-lh5cv\" (UID: \"20456cd0-3433-42d5-bf03-8c69516c0774\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-lh5cv"
Sep 17 18:18:56.343978 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.343924    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/87fa04bd-2308-41bc-8818-07e1c6b51295-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-6cbcf9f68f-sfv8x\" (UID: \"87fa04bd-2308-41bc-8818-07e1c6b51295\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-sfv8x"
Sep 17 18:18:56.343978 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.343958    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/20456cd0-3433-42d5-bf03-8c69516c0774-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-646f5bdbcb-lh5cv\" (UID: \"20456cd0-3433-42d5-bf03-8c69516c0774\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-lh5cv"
Sep 17 18:18:56.344329 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.344306    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87fa04bd-2308-41bc-8818-07e1c6b51295-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-6cbcf9f68f-sfv8x\" (UID: \"87fa04bd-2308-41bc-8818-07e1c6b51295\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-sfv8x"
Sep 17 18:18:56.344560 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.344541    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/87fa04bd-2308-41bc-8818-07e1c6b51295-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-6cbcf9f68f-sfv8x\" (UID: \"87fa04bd-2308-41bc-8818-07e1c6b51295\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-sfv8x"
Sep 17 18:18:56.344646 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.344571    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/87fa04bd-2308-41bc-8818-07e1c6b51295-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-6cbcf9f68f-sfv8x\" (UID: \"87fa04bd-2308-41bc-8818-07e1c6b51295\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-sfv8x"
Sep 17 18:18:56.346061 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.346042    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87fa04bd-2308-41bc-8818-07e1c6b51295-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-6cbcf9f68f-sfv8x\" (UID: \"87fa04bd-2308-41bc-8818-07e1c6b51295\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-sfv8x"
Sep 17 18:18:56.351593 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.351574    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxp9h\" (UniqueName: \"kubernetes.io/projected/87fa04bd-2308-41bc-8818-07e1c6b51295-kube-api-access-gxp9h\") pod \"isvc-canary-stop-v2-predictor-6cbcf9f68f-sfv8x\" (UID: \"87fa04bd-2308-41bc-8818-07e1c6b51295\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-sfv8x"
Sep 17 18:18:56.445199 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.445159    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/20456cd0-3433-42d5-bf03-8c69516c0774-proxy-tls\") pod \"isvc-canary-stop-predictor-646f5bdbcb-lh5cv\" (UID: \"20456cd0-3433-42d5-bf03-8c69516c0774\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-lh5cv"
Sep 17 18:18:56.445346 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.445218    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20456cd0-3433-42d5-bf03-8c69516c0774-kserve-provision-location\") pod \"isvc-canary-stop-predictor-646f5bdbcb-lh5cv\" (UID: \"20456cd0-3433-42d5-bf03-8c69516c0774\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-lh5cv"
Sep 17 18:18:56.445346 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.445262    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/20456cd0-3433-42d5-bf03-8c69516c0774-cabundle-cert\") pod \"isvc-canary-stop-predictor-646f5bdbcb-lh5cv\" (UID: \"20456cd0-3433-42d5-bf03-8c69516c0774\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-lh5cv"
Sep 17 18:18:56.445346 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.445306    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrwcs\" (UniqueName: \"kubernetes.io/projected/20456cd0-3433-42d5-bf03-8c69516c0774-kube-api-access-mrwcs\") pod \"isvc-canary-stop-predictor-646f5bdbcb-lh5cv\" (UID: \"20456cd0-3433-42d5-bf03-8c69516c0774\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-lh5cv"
Sep 17 18:18:56.445346 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:18:56.445312    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-predictor-serving-cert: secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 17 18:18:56.445556 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.445346    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/20456cd0-3433-42d5-bf03-8c69516c0774-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-646f5bdbcb-lh5cv\" (UID: \"20456cd0-3433-42d5-bf03-8c69516c0774\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-lh5cv"
Sep 17 18:18:56.445556 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:18:56.445368    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20456cd0-3433-42d5-bf03-8c69516c0774-proxy-tls podName:20456cd0-3433-42d5-bf03-8c69516c0774 nodeName:}" failed. No retries permitted until 2026-09-17 18:18:56.945353274 +0000 UTC m=+2289.449355876 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/20456cd0-3433-42d5-bf03-8c69516c0774-proxy-tls") pod "isvc-canary-stop-predictor-646f5bdbcb-lh5cv" (UID: "20456cd0-3433-42d5-bf03-8c69516c0774") : secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 17 18:18:56.445701 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.445640    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20456cd0-3433-42d5-bf03-8c69516c0774-kserve-provision-location\") pod \"isvc-canary-stop-predictor-646f5bdbcb-lh5cv\" (UID: \"20456cd0-3433-42d5-bf03-8c69516c0774\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-lh5cv"
Sep 17 18:18:56.445923 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.445903    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/20456cd0-3433-42d5-bf03-8c69516c0774-cabundle-cert\") pod \"isvc-canary-stop-predictor-646f5bdbcb-lh5cv\" (UID: \"20456cd0-3433-42d5-bf03-8c69516c0774\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-lh5cv"
Sep 17 18:18:56.446016 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.445994    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/20456cd0-3433-42d5-bf03-8c69516c0774-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-646f5bdbcb-lh5cv\" (UID: \"20456cd0-3433-42d5-bf03-8c69516c0774\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-lh5cv"
Sep 17 18:18:56.455837 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.455806    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrwcs\" (UniqueName: \"kubernetes.io/projected/20456cd0-3433-42d5-bf03-8c69516c0774-kube-api-access-mrwcs\") pod \"isvc-canary-stop-predictor-646f5bdbcb-lh5cv\" (UID: \"20456cd0-3433-42d5-bf03-8c69516c0774\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-lh5cv"
Sep 17 18:18:56.506014 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.505961    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-sfv8x"
Sep 17 18:18:56.776263 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.776238    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-sfv8x"]
Sep 17 18:18:56.778310 ip-10-0-131-14 kubenswrapper[2569]: W0917 18:18:56.778288    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87fa04bd_2308_41bc_8818_07e1c6b51295.slice/crio-0a2788e18b52b7e83bcc40d18426f91e8220f0b5b612cf138a4879afcb88ee6c WatchSource:0}: Error finding container 0a2788e18b52b7e83bcc40d18426f91e8220f0b5b612cf138a4879afcb88ee6c: Status 404 returned error can't find the container with id 0a2788e18b52b7e83bcc40d18426f91e8220f0b5b612cf138a4879afcb88ee6c
Sep 17 18:18:56.949457 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.949423    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/20456cd0-3433-42d5-bf03-8c69516c0774-proxy-tls\") pod \"isvc-canary-stop-predictor-646f5bdbcb-lh5cv\" (UID: \"20456cd0-3433-42d5-bf03-8c69516c0774\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-lh5cv"
Sep 17 18:18:56.951741 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.951717    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/20456cd0-3433-42d5-bf03-8c69516c0774-proxy-tls\") pod \"isvc-canary-stop-predictor-646f5bdbcb-lh5cv\" (UID: \"20456cd0-3433-42d5-bf03-8c69516c0774\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-lh5cv"
Sep 17 18:18:56.988583 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.988553    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-sfv8x" event={"ID":"87fa04bd-2308-41bc-8818-07e1c6b51295","Type":"ContainerStarted","Data":"b98aba38fc4712ab7aeb5a41d9c93985fd6365cd5318c6aa013ea1c72bceda7a"}
Sep 17 18:18:56.988717 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:56.988621    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-sfv8x" event={"ID":"87fa04bd-2308-41bc-8818-07e1c6b51295","Type":"ContainerStarted","Data":"0a2788e18b52b7e83bcc40d18426f91e8220f0b5b612cf138a4879afcb88ee6c"}
Sep 17 18:18:57.143945 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:57.143864    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-lh5cv"
Sep 17 18:18:57.416941 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:57.416865    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-lh5cv"]
Sep 17 18:18:57.420630 ip-10-0-131-14 kubenswrapper[2569]: W0917 18:18:57.420597    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20456cd0_3433_42d5_bf03_8c69516c0774.slice/crio-908979bc646be1ce5236045bb15aabfdf12c0c2aa995370c1d55d0beaf74491f WatchSource:0}: Error finding container 908979bc646be1ce5236045bb15aabfdf12c0c2aa995370c1d55d0beaf74491f: Status 404 returned error can't find the container with id 908979bc646be1ce5236045bb15aabfdf12c0c2aa995370c1d55d0beaf74491f
Sep 17 18:18:57.508729 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:57.508703    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c51911a9-04b0-41e4-917f-b1fdd27600d8" path="/var/lib/kubelet/pods/c51911a9-04b0-41e4-917f-b1fdd27600d8/volumes"
Sep 17 18:18:57.995488 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:57.995449    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-lh5cv" event={"ID":"20456cd0-3433-42d5-bf03-8c69516c0774","Type":"ContainerStarted","Data":"eaed17ccd88d0a4511729bcb11ecfe227e08e4794de4687d8523a6abb0acbbeb"}
Sep 17 18:18:57.995696 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:18:57.995511    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-lh5cv" event={"ID":"20456cd0-3433-42d5-bf03-8c69516c0774","Type":"ContainerStarted","Data":"908979bc646be1ce5236045bb15aabfdf12c0c2aa995370c1d55d0beaf74491f"}
Sep 17 18:19:01.008899 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:19:01.008865    2569 generic.go:356] "Generic (PLEG): container finished" podID="87fa04bd-2308-41bc-8818-07e1c6b51295" containerID="b98aba38fc4712ab7aeb5a41d9c93985fd6365cd5318c6aa013ea1c72bceda7a" exitCode=0
Sep 17 18:19:01.009290 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:19:01.008947    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-sfv8x" event={"ID":"87fa04bd-2308-41bc-8818-07e1c6b51295","Type":"ContainerDied","Data":"b98aba38fc4712ab7aeb5a41d9c93985fd6365cd5318c6aa013ea1c72bceda7a"}
Sep 17 18:19:02.014261 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:19:02.014230    2569 generic.go:356] "Generic (PLEG): container finished" podID="20456cd0-3433-42d5-bf03-8c69516c0774" containerID="eaed17ccd88d0a4511729bcb11ecfe227e08e4794de4687d8523a6abb0acbbeb" exitCode=0
Sep 17 18:19:02.014711 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:19:02.014312    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-lh5cv" event={"ID":"20456cd0-3433-42d5-bf03-8c69516c0774","Type":"ContainerDied","Data":"eaed17ccd88d0a4511729bcb11ecfe227e08e4794de4687d8523a6abb0acbbeb"}
Sep 17 18:19:02.016663 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:19:02.016640    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-sfv8x" event={"ID":"87fa04bd-2308-41bc-8818-07e1c6b51295","Type":"ContainerStarted","Data":"82e54b02948a4048dc1c69e68489fd19df719c6d83fd0404a5d216a333189529"}
Sep 17 18:19:02.016796 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:19:02.016696    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-sfv8x" event={"ID":"87fa04bd-2308-41bc-8818-07e1c6b51295","Type":"ContainerStarted","Data":"fd499b5d3e80ca664ed8ee7ec8956f81ce82c7a7d39e2802bee75394590fe0d3"}
Sep 17 18:19:02.016990 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:19:02.016919    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-sfv8x"
Sep 17 18:19:02.054690 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:19:02.054646    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-sfv8x" podStartSLOduration=2.197131875 podStartE2EDuration="6.054628575s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.8574967s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:18:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:19:02.053234031 +0000 UTC m=+2294.557236656" watchObservedRunningTime="2026-09-17 18:19:02.054628575 +0000 UTC m=+2294.558631202"
Sep 17 18:19:03.023316 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:19:03.023279    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-lh5cv" event={"ID":"20456cd0-3433-42d5-bf03-8c69516c0774","Type":"ContainerStarted","Data":"3e6bc48a71420810dc8b4bd394c08ab864c8447008778cdcd9ab1d1813d48988"}
Sep 17 18:19:03.023698 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:19:03.023353    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-lh5cv" event={"ID":"20456cd0-3433-42d5-bf03-8c69516c0774","Type":"ContainerStarted","Data":"535d74f12a86e6a47eeae80c328d79bdde951c7a824f84a2883714f9883e532b"}
Sep 17 18:19:03.023698 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:19:03.023589    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-sfv8x"
Sep 17 18:19:03.023861 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:19:03.023842    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-lh5cv"
Sep 17 18:19:03.024028 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:19:03.024003    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-lh5cv"
Sep 17 18:19:03.024863 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:19:03.024842    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-sfv8x" podUID="87fa04bd-2308-41bc-8818-07e1c6b51295" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 18:19:03.024918 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:19:03.024884    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-lh5cv" podUID="20456cd0-3433-42d5-bf03-8c69516c0774" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 18:19:03.045056 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:19:03.045004    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-lh5cv" podStartSLOduration=3.220816541 podStartE2EDuration="7.044992199s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.824175658s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:18:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:19:03.043302361 +0000 UTC m=+2295.547304986" watchObservedRunningTime="2026-09-17 18:19:03.044992199 +0000 UTC m=+2295.548994823"
Sep 17 18:19:04.027838 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:19:04.027789    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-lh5cv" podUID="20456cd0-3433-42d5-bf03-8c69516c0774" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 18:19:04.028231 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:19:04.027930    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-sfv8x" podUID="87fa04bd-2308-41bc-8818-07e1c6b51295" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 18:19:09.032146 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:19:09.032117    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-lh5cv"
Sep 17 18:19:09.032511 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:19:09.032206    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-sfv8x"
Sep 17 18:19:09.032627 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:19:09.032598    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-lh5cv" podUID="20456cd0-3433-42d5-bf03-8c69516c0774" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 18:19:09.032705 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:19:09.032681    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-sfv8x" podUID="87fa04bd-2308-41bc-8818-07e1c6b51295" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 18:19:19.032651 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:19:19.032609    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-sfv8x" podUID="87fa04bd-2308-41bc-8818-07e1c6b51295" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 18:19:19.033040 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:19:19.032610    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-lh5cv" podUID="20456cd0-3433-42d5-bf03-8c69516c0774" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 18:19:20.102583 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:19:20.102556    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qjps6"]
Sep 17 18:19:20.106262 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:19:20.106242    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qjps6"
Sep 17 18:19:20.109111 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:19:20.109069    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-predictor-serving-cert\""
Sep 17 18:19:20.109249 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:19:20.109096    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 17 18:19:20.116270 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:19:20.116245    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qjps6"]
Sep 17 18:19:20.245595 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:19:20.245560    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1765ef9b-2007-42e1-b88f-f5cef196384a-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qjps6\" (UID: \"1765ef9b-2007-42e1-b88f-f5cef196384a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qjps6"
Sep 17 18:19:20.245595 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:19:20.245602    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1765ef9b-2007-42e1-b88f-f5cef196384a-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qjps6\" (UID: \"1765ef9b-2007-42e1-b88f-f5cef196384a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qjps6"
Sep 17 18:19:20.245861 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:19:20.245619    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1765ef9b-2007-42e1-b88f-f5cef196384a-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qjps6\" (UID: \"1765ef9b-2007-42e1-b88f-f5cef196384a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qjps6"
Sep 17 18:19:20.245861 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:19:20.245768    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khccj\" (UniqueName: \"kubernetes.io/projected/1765ef9b-2007-42e1-b88f-f5cef196384a-kube-api-access-khccj\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qjps6\" (UID: \"1765ef9b-2007-42e1-b88f-f5cef196384a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qjps6"
Sep 17 18:19:20.245861 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:19:20.245813    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1765ef9b-2007-42e1-b88f-f5cef196384a-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qjps6\" (UID: \"1765ef9b-2007-42e1-b88f-f5cef196384a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qjps6"
Sep 17 18:19:20.346591 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:19:20.346552    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khccj\" (UniqueName: \"kubernetes.io/projected/1765ef9b-2007-42e1-b88f-f5cef196384a-kube-api-access-khccj\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qjps6\" (UID: \"1765ef9b-2007-42e1-b88f-f5cef196384a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qjps6"
Sep 17 18:19:20.346759 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:19:20.346605    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1765ef9b-2007-42e1-b88f-f5cef196384a-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qjps6\" (UID: \"1765ef9b-2007-42e1-b88f-f5cef196384a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qjps6"
Sep 17 18:19:20.346759 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:19:20.346638    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1765ef9b-2007-42e1-b88f-f5cef196384a-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qjps6\" (UID: \"1765ef9b-2007-42e1-b88f-f5cef196384a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qjps6"
Sep 17 18:19:20.346759 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:19:20.346683    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1765ef9b-2007-42e1-b88f-f5cef196384a-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qjps6\" (UID: \"1765ef9b-2007-42e1-b88f-f5cef196384a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qjps6"
Sep 17 18:19:20.346759 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:19:20.346711    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1765ef9b-2007-42e1-b88f-f5cef196384a-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qjps6\" (UID: \"1765ef9b-2007-42e1-b88f-f5cef196384a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qjps6"
Sep 17 18:19:20.346986 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:19:20.346815    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-serving-cert: secret "isvc-xgboost-v2-mlserver-predictor-serving-cert" not found
Sep 17 18:19:20.346986 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:19:20.346882    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1765ef9b-2007-42e1-b88f-f5cef196384a-proxy-tls podName:1765ef9b-2007-42e1-b88f-f5cef196384a nodeName:}" failed. No retries permitted until 2026-09-17 18:19:20.846862577 +0000 UTC m=+2313.350865191 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1765ef9b-2007-42e1-b88f-f5cef196384a-proxy-tls") pod "isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qjps6" (UID: "1765ef9b-2007-42e1-b88f-f5cef196384a") : secret "isvc-xgboost-v2-mlserver-predictor-serving-cert" not found
Sep 17 18:19:20.347393 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:19:20.347357    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1765ef9b-2007-42e1-b88f-f5cef196384a-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qjps6\" (UID: \"1765ef9b-2007-42e1-b88f-f5cef196384a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qjps6"
Sep 17 18:19:20.347519 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:19:20.347460    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1765ef9b-2007-42e1-b88f-f5cef196384a-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qjps6\" (UID: \"1765ef9b-2007-42e1-b88f-f5cef196384a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qjps6"
Sep 17 18:19:20.347573 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:19:20.347540    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1765ef9b-2007-42e1-b88f-f5cef196384a-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qjps6\" (UID: \"1765ef9b-2007-42e1-b88f-f5cef196384a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qjps6"
Sep 17 18:19:20.358732 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:19:20.358672    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khccj\" (UniqueName: \"kubernetes.io/projected/1765ef9b-2007-42e1-b88f-f5cef196384a-kube-api-access-khccj\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qjps6\" (UID: \"1765ef9b-2007-42e1-b88f-f5cef196384a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qjps6"
Sep 17 18:19:20.850396 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:19:20.850363    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1765ef9b-2007-42e1-b88f-f5cef196384a-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qjps6\" (UID: \"1765ef9b-2007-42e1-b88f-f5cef196384a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qjps6"
Sep 17 18:19:20.852803 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:19:20.852770    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1765ef9b-2007-42e1-b88f-f5cef196384a-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qjps6\" (UID: \"1765ef9b-2007-42e1-b88f-f5cef196384a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qjps6"
Sep 17 18:19:21.016793 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:19:21.016757    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qjps6"
Sep 17 18:19:21.291590 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:19:21.291567    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qjps6"]
Sep 17 18:19:21.293397 ip-10-0-131-14 kubenswrapper[2569]: W0917 18:19:21.293372    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1765ef9b_2007_42e1_b88f_f5cef196384a.slice/crio-aa56cfa87a0bddb08b3b5b53b7cdc75c1a7a08522c391e865a24b0247578da32 WatchSource:0}: Error finding container aa56cfa87a0bddb08b3b5b53b7cdc75c1a7a08522c391e865a24b0247578da32: Status 404 returned error can't find the container with id aa56cfa87a0bddb08b3b5b53b7cdc75c1a7a08522c391e865a24b0247578da32
Sep 17 18:19:22.106584 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:19:22.106547    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qjps6" event={"ID":"1765ef9b-2007-42e1-b88f-f5cef196384a","Type":"ContainerStarted","Data":"0a212f20018a97b5ad7469ec9fbdd069bf49937ff2a0c87054c3e265bed1af9e"}
Sep 17 18:19:22.106779 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:19:22.106614    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qjps6" event={"ID":"1765ef9b-2007-42e1-b88f-f5cef196384a","Type":"ContainerStarted","Data":"aa56cfa87a0bddb08b3b5b53b7cdc75c1a7a08522c391e865a24b0247578da32"}
Sep 17 18:19:26.124940 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:19:26.124906    2569 generic.go:356] "Generic (PLEG): container finished" podID="1765ef9b-2007-42e1-b88f-f5cef196384a" containerID="0a212f20018a97b5ad7469ec9fbdd069bf49937ff2a0c87054c3e265bed1af9e" exitCode=0
Sep 17 18:19:26.125344 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:19:26.124986    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qjps6" event={"ID":"1765ef9b-2007-42e1-b88f-f5cef196384a","Type":"ContainerDied","Data":"0a212f20018a97b5ad7469ec9fbdd069bf49937ff2a0c87054c3e265bed1af9e"}
Sep 17 18:19:27.131399 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:19:27.131367    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qjps6" event={"ID":"1765ef9b-2007-42e1-b88f-f5cef196384a","Type":"ContainerStarted","Data":"8feb8305c077292cc90f9422029c54597016b015f6f1f3809643ec175166e430"}
Sep 17 18:19:27.131912 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:19:27.131426    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qjps6" event={"ID":"1765ef9b-2007-42e1-b88f-f5cef196384a","Type":"ContainerStarted","Data":"a2d2652f88cbf8cdda82e643cd9e08d0802c231b26a6521de9fb284f221283d5"}
Sep 17 18:19:27.131912 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:19:27.131636    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qjps6"
Sep 17 18:19:27.153571 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:19:27.153515    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qjps6" podStartSLOduration=3.072293363 podStartE2EDuration="7.153498233s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.08120487s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:19:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:19:27.151882141 +0000 UTC m=+2319.655884766" watchObservedRunningTime="2026-09-17 18:19:27.153498233 +0000 UTC m=+2319.657500858"
Sep 17 18:19:28.136085 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:19:28.136054    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qjps6"
Sep 17 18:19:29.032688 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:19:29.032649    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-sfv8x" podUID="87fa04bd-2308-41bc-8818-07e1c6b51295" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 18:19:29.032866 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:19:29.032642    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-lh5cv" podUID="20456cd0-3433-42d5-bf03-8c69516c0774" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 18:19:34.146716 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:19:34.146631    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qjps6"
Sep 17 18:19:39.032819 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:19:39.032779    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-lh5cv" podUID="20456cd0-3433-42d5-bf03-8c69516c0774" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 18:19:39.033239 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:19:39.032779    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-sfv8x" podUID="87fa04bd-2308-41bc-8818-07e1c6b51295" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 18:19:49.032638 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:19:49.032595    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-sfv8x" podUID="87fa04bd-2308-41bc-8818-07e1c6b51295" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 18:19:49.033241 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:19:49.032595    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-lh5cv" podUID="20456cd0-3433-42d5-bf03-8c69516c0774" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 18:19:59.032844 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:19:59.032804    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-sfv8x" podUID="87fa04bd-2308-41bc-8818-07e1c6b51295" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 18:19:59.033333 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:19:59.032813    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-lh5cv" podUID="20456cd0-3433-42d5-bf03-8c69516c0774" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 18:20:04.151175 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:04.151147    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qjps6"
Sep 17 18:20:09.033192 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:09.033154    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-lh5cv" podUID="20456cd0-3433-42d5-bf03-8c69516c0774" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 18:20:09.033643 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:09.033161    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-sfv8x" podUID="87fa04bd-2308-41bc-8818-07e1c6b51295" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 18:20:10.141370 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:10.141333    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qjps6"]
Sep 17 18:20:10.141768 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:10.141643    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qjps6" podUID="1765ef9b-2007-42e1-b88f-f5cef196384a" containerName="kserve-container" containerID="cri-o://a2d2652f88cbf8cdda82e643cd9e08d0802c231b26a6521de9fb284f221283d5" gracePeriod=30
Sep 17 18:20:10.141840 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:10.141807    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qjps6" podUID="1765ef9b-2007-42e1-b88f-f5cef196384a" containerName="kube-rbac-proxy" containerID="cri-o://8feb8305c077292cc90f9422029c54597016b015f6f1f3809643ec175166e430" gracePeriod=30
Sep 17 18:20:10.314320 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:10.314289    2569 generic.go:356] "Generic (PLEG): container finished" podID="1765ef9b-2007-42e1-b88f-f5cef196384a" containerID="8feb8305c077292cc90f9422029c54597016b015f6f1f3809643ec175166e430" exitCode=2
Sep 17 18:20:10.314505 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:10.314379    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qjps6" event={"ID":"1765ef9b-2007-42e1-b88f-f5cef196384a","Type":"ContainerDied","Data":"8feb8305c077292cc90f9422029c54597016b015f6f1f3809643ec175166e430"}
Sep 17 18:20:14.140419 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:14.140383    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qjps6" podUID="1765ef9b-2007-42e1-b88f-f5cef196384a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.39:8643/healthz\": dial tcp 10.133.0.39:8643: connect: connection refused"
Sep 17 18:20:14.147895 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:14.147857    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qjps6" podUID="1765ef9b-2007-42e1-b88f-f5cef196384a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.39:8080/v2/models/isvc-xgboost-v2-mlserver/ready\": dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 18:20:15.340732 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:15.340701    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-5gw5z"]
Sep 17 18:20:15.344367 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:15.344343    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-5gw5z"
Sep 17 18:20:15.347007 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:15.346979    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-predictor-serving-cert\""
Sep 17 18:20:15.347007 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:15.346999    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 17 18:20:15.355523 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:15.355501    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-5gw5z"]
Sep 17 18:20:15.373345 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:15.373322    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81cd0cbc-9d19-4b19-8db9-28edbded0d19-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-5gw5z\" (UID: \"81cd0cbc-9d19-4b19-8db9-28edbded0d19\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-5gw5z"
Sep 17 18:20:15.373457 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:15.373359    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81cd0cbc-9d19-4b19-8db9-28edbded0d19-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-5gw5z\" (UID: \"81cd0cbc-9d19-4b19-8db9-28edbded0d19\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-5gw5z"
Sep 17 18:20:15.373457 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:15.373388    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81cd0cbc-9d19-4b19-8db9-28edbded0d19-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-5gw5z\" (UID: \"81cd0cbc-9d19-4b19-8db9-28edbded0d19\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-5gw5z"
Sep 17 18:20:15.373543 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:15.373486    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fthj\" (UniqueName: \"kubernetes.io/projected/81cd0cbc-9d19-4b19-8db9-28edbded0d19-kube-api-access-6fthj\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-5gw5z\" (UID: \"81cd0cbc-9d19-4b19-8db9-28edbded0d19\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-5gw5z"
Sep 17 18:20:15.373543 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:15.373513    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81cd0cbc-9d19-4b19-8db9-28edbded0d19-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-5gw5z\" (UID: \"81cd0cbc-9d19-4b19-8db9-28edbded0d19\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-5gw5z"
Sep 17 18:20:15.474383 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:15.474351    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fthj\" (UniqueName: \"kubernetes.io/projected/81cd0cbc-9d19-4b19-8db9-28edbded0d19-kube-api-access-6fthj\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-5gw5z\" (UID: \"81cd0cbc-9d19-4b19-8db9-28edbded0d19\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-5gw5z"
Sep 17 18:20:15.474383 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:15.474384    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81cd0cbc-9d19-4b19-8db9-28edbded0d19-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-5gw5z\" (UID: \"81cd0cbc-9d19-4b19-8db9-28edbded0d19\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-5gw5z"
Sep 17 18:20:15.474575 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:15.474406    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81cd0cbc-9d19-4b19-8db9-28edbded0d19-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-5gw5z\" (UID: \"81cd0cbc-9d19-4b19-8db9-28edbded0d19\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-5gw5z"
Sep 17 18:20:15.474575 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:15.474510    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81cd0cbc-9d19-4b19-8db9-28edbded0d19-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-5gw5z\" (UID: \"81cd0cbc-9d19-4b19-8db9-28edbded0d19\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-5gw5z"
Sep 17 18:20:15.474683 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:15.474575    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81cd0cbc-9d19-4b19-8db9-28edbded0d19-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-5gw5z\" (UID: \"81cd0cbc-9d19-4b19-8db9-28edbded0d19\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-5gw5z"
Sep 17 18:20:15.474891 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:15.474872    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81cd0cbc-9d19-4b19-8db9-28edbded0d19-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-5gw5z\" (UID: \"81cd0cbc-9d19-4b19-8db9-28edbded0d19\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-5gw5z"
Sep 17 18:20:15.475213 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:15.475197    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81cd0cbc-9d19-4b19-8db9-28edbded0d19-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-5gw5z\" (UID: \"81cd0cbc-9d19-4b19-8db9-28edbded0d19\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-5gw5z"
Sep 17 18:20:15.475263 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:15.475243    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81cd0cbc-9d19-4b19-8db9-28edbded0d19-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-5gw5z\" (UID: \"81cd0cbc-9d19-4b19-8db9-28edbded0d19\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-5gw5z"
Sep 17 18:20:15.476780 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:15.476764    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81cd0cbc-9d19-4b19-8db9-28edbded0d19-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-5gw5z\" (UID: \"81cd0cbc-9d19-4b19-8db9-28edbded0d19\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-5gw5z"
Sep 17 18:20:15.482662 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:15.482635    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fthj\" (UniqueName: \"kubernetes.io/projected/81cd0cbc-9d19-4b19-8db9-28edbded0d19-kube-api-access-6fthj\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-5gw5z\" (UID: \"81cd0cbc-9d19-4b19-8db9-28edbded0d19\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-5gw5z"
Sep 17 18:20:15.508423 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:15.508406    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-lh5cv"
Sep 17 18:20:15.655283 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:15.655194    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-5gw5z"
Sep 17 18:20:15.930367 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:15.930342    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-5gw5z"]
Sep 17 18:20:15.932138 ip-10-0-131-14 kubenswrapper[2569]: W0917 18:20:15.932113    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81cd0cbc_9d19_4b19_8db9_28edbded0d19.slice/crio-827fd0d1be7ad4d3c3e3a2881aa540b3dc6d38b11c46c112c5b15b38228c2ecd WatchSource:0}: Error finding container 827fd0d1be7ad4d3c3e3a2881aa540b3dc6d38b11c46c112c5b15b38228c2ecd: Status 404 returned error can't find the container with id 827fd0d1be7ad4d3c3e3a2881aa540b3dc6d38b11c46c112c5b15b38228c2ecd
Sep 17 18:20:16.343048 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:16.343016    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-5gw5z" event={"ID":"81cd0cbc-9d19-4b19-8db9-28edbded0d19","Type":"ContainerStarted","Data":"be2ea5d767a4fa951b4ffce7a78c38dd0bd290342d8ff7c83454f63c5113f901"}
Sep 17 18:20:16.343480 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:16.343145    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-5gw5z" event={"ID":"81cd0cbc-9d19-4b19-8db9-28edbded0d19","Type":"ContainerStarted","Data":"827fd0d1be7ad4d3c3e3a2881aa540b3dc6d38b11c46c112c5b15b38228c2ecd"}
Sep 17 18:20:16.345586 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:16.345557    2569 generic.go:356] "Generic (PLEG): container finished" podID="1765ef9b-2007-42e1-b88f-f5cef196384a" containerID="a2d2652f88cbf8cdda82e643cd9e08d0802c231b26a6521de9fb284f221283d5" exitCode=0
Sep 17 18:20:16.345709 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:16.345620    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qjps6" event={"ID":"1765ef9b-2007-42e1-b88f-f5cef196384a","Type":"ContainerDied","Data":"a2d2652f88cbf8cdda82e643cd9e08d0802c231b26a6521de9fb284f221283d5"}
Sep 17 18:20:16.372733 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:16.372715    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qjps6"
Sep 17 18:20:16.482627 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:16.482584    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-khccj\" (UniqueName: \"kubernetes.io/projected/1765ef9b-2007-42e1-b88f-f5cef196384a-kube-api-access-khccj\") pod \"1765ef9b-2007-42e1-b88f-f5cef196384a\" (UID: \"1765ef9b-2007-42e1-b88f-f5cef196384a\") "
Sep 17 18:20:16.483506 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:16.482659    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1765ef9b-2007-42e1-b88f-f5cef196384a-kube-rbac-proxy-sar-config\") pod \"1765ef9b-2007-42e1-b88f-f5cef196384a\" (UID: \"1765ef9b-2007-42e1-b88f-f5cef196384a\") "
Sep 17 18:20:16.483506 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:16.482690    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1765ef9b-2007-42e1-b88f-f5cef196384a-cabundle-cert\") pod \"1765ef9b-2007-42e1-b88f-f5cef196384a\" (UID: \"1765ef9b-2007-42e1-b88f-f5cef196384a\") "
Sep 17 18:20:16.483506 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:16.482808    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1765ef9b-2007-42e1-b88f-f5cef196384a-proxy-tls\") pod \"1765ef9b-2007-42e1-b88f-f5cef196384a\" (UID: \"1765ef9b-2007-42e1-b88f-f5cef196384a\") "
Sep 17 18:20:16.483506 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:16.482874    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1765ef9b-2007-42e1-b88f-f5cef196384a-kserve-provision-location\") pod \"1765ef9b-2007-42e1-b88f-f5cef196384a\" (UID: \"1765ef9b-2007-42e1-b88f-f5cef196384a\") "
Sep 17 18:20:16.483506 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:16.483167    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1765ef9b-2007-42e1-b88f-f5cef196384a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1765ef9b-2007-42e1-b88f-f5cef196384a" (UID: "1765ef9b-2007-42e1-b88f-f5cef196384a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:20:16.483506 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:16.483194    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1765ef9b-2007-42e1-b88f-f5cef196384a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1765ef9b-2007-42e1-b88f-f5cef196384a" (UID: "1765ef9b-2007-42e1-b88f-f5cef196384a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:20:16.483506 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:16.483367    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1765ef9b-2007-42e1-b88f-f5cef196384a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1765ef9b-2007-42e1-b88f-f5cef196384a" (UID: "1765ef9b-2007-42e1-b88f-f5cef196384a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:20:16.485266 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:16.485229    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1765ef9b-2007-42e1-b88f-f5cef196384a-kube-api-access-khccj" (OuterVolumeSpecName: "kube-api-access-khccj") pod "1765ef9b-2007-42e1-b88f-f5cef196384a" (UID: "1765ef9b-2007-42e1-b88f-f5cef196384a"). InnerVolumeSpecName "kube-api-access-khccj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:20:16.485367 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:16.485261    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1765ef9b-2007-42e1-b88f-f5cef196384a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1765ef9b-2007-42e1-b88f-f5cef196384a" (UID: "1765ef9b-2007-42e1-b88f-f5cef196384a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:20:16.583683 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:16.583606    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1765ef9b-2007-42e1-b88f-f5cef196384a-proxy-tls\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:20:16.583683 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:16.583635    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1765ef9b-2007-42e1-b88f-f5cef196384a-kserve-provision-location\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:20:16.583683 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:16.583648    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khccj\" (UniqueName: \"kubernetes.io/projected/1765ef9b-2007-42e1-b88f-f5cef196384a-kube-api-access-khccj\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:20:16.583683 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:16.583657    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1765ef9b-2007-42e1-b88f-f5cef196384a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:20:16.583683 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:16.583666    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1765ef9b-2007-42e1-b88f-f5cef196384a-cabundle-cert\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:20:17.353004 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:17.352978    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qjps6"
Sep 17 18:20:17.353004 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:17.352993    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qjps6" event={"ID":"1765ef9b-2007-42e1-b88f-f5cef196384a","Type":"ContainerDied","Data":"aa56cfa87a0bddb08b3b5b53b7cdc75c1a7a08522c391e865a24b0247578da32"}
Sep 17 18:20:17.353580 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:17.353063    2569 scope.go:117] "RemoveContainer" containerID="8feb8305c077292cc90f9422029c54597016b015f6f1f3809643ec175166e430"
Sep 17 18:20:17.362065 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:17.362046    2569 scope.go:117] "RemoveContainer" containerID="a2d2652f88cbf8cdda82e643cd9e08d0802c231b26a6521de9fb284f221283d5"
Sep 17 18:20:17.369659 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:17.369643    2569 scope.go:117] "RemoveContainer" containerID="0a212f20018a97b5ad7469ec9fbdd069bf49937ff2a0c87054c3e265bed1af9e"
Sep 17 18:20:17.375589 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:17.375569    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qjps6"]
Sep 17 18:20:17.379415 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:17.379395    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qjps6"]
Sep 17 18:20:17.508215 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:17.508187    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1765ef9b-2007-42e1-b88f-f5cef196384a" path="/var/lib/kubelet/pods/1765ef9b-2007-42e1-b88f-f5cef196384a/volumes"
Sep 17 18:20:19.033831 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:19.033803    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-sfv8x"
Sep 17 18:20:19.149876 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:19.149844    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-lh5cv"]
Sep 17 18:20:19.150110 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:19.150053    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-lh5cv" podUID="20456cd0-3433-42d5-bf03-8c69516c0774" containerName="kserve-container" containerID="cri-o://535d74f12a86e6a47eeae80c328d79bdde951c7a824f84a2883714f9883e532b" gracePeriod=30
Sep 17 18:20:19.150226 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:19.150106    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-lh5cv" podUID="20456cd0-3433-42d5-bf03-8c69516c0774" containerName="kube-rbac-proxy" containerID="cri-o://3e6bc48a71420810dc8b4bd394c08ab864c8447008778cdcd9ab1d1813d48988" gracePeriod=30
Sep 17 18:20:19.365046 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:19.364975    2569 generic.go:356] "Generic (PLEG): container finished" podID="20456cd0-3433-42d5-bf03-8c69516c0774" containerID="3e6bc48a71420810dc8b4bd394c08ab864c8447008778cdcd9ab1d1813d48988" exitCode=2
Sep 17 18:20:19.365177 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:19.365045    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-lh5cv" event={"ID":"20456cd0-3433-42d5-bf03-8c69516c0774","Type":"ContainerDied","Data":"3e6bc48a71420810dc8b4bd394c08ab864c8447008778cdcd9ab1d1813d48988"}
Sep 17 18:20:20.374505 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:20.374472    2569 generic.go:356] "Generic (PLEG): container finished" podID="81cd0cbc-9d19-4b19-8db9-28edbded0d19" containerID="be2ea5d767a4fa951b4ffce7a78c38dd0bd290342d8ff7c83454f63c5113f901" exitCode=0
Sep 17 18:20:20.374889 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:20.374515    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-5gw5z" event={"ID":"81cd0cbc-9d19-4b19-8db9-28edbded0d19","Type":"ContainerDied","Data":"be2ea5d767a4fa951b4ffce7a78c38dd0bd290342d8ff7c83454f63c5113f901"}
Sep 17 18:20:21.380587 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:21.380558    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-5gw5z" event={"ID":"81cd0cbc-9d19-4b19-8db9-28edbded0d19","Type":"ContainerStarted","Data":"543caa400c7efc111371f97c99d4f445e87c47086ff9c3532bd6ddc5aaa33cd3"}
Sep 17 18:20:21.380986 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:21.380615    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-5gw5z" event={"ID":"81cd0cbc-9d19-4b19-8db9-28edbded0d19","Type":"ContainerStarted","Data":"b4e3ba0b72ea0e3106f68423cfddb5e3701780e5724a1f56fd2e2a6c45763bf8"}
Sep 17 18:20:21.380986 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:21.380799    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-5gw5z"
Sep 17 18:20:21.402941 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:21.402891    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-5gw5z" podStartSLOduration=2.1313262 podStartE2EDuration="6.402879295s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.271553095s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:20:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:20:21.401680379 +0000 UTC m=+2373.905683018" watchObservedRunningTime="2026-09-17 18:20:21.402879295 +0000 UTC m=+2373.906881919"
Sep 17 18:20:22.385096 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:22.385057    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-5gw5z"
Sep 17 18:20:23.391007 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:23.390980    2569 generic.go:356] "Generic (PLEG): container finished" podID="20456cd0-3433-42d5-bf03-8c69516c0774" containerID="535d74f12a86e6a47eeae80c328d79bdde951c7a824f84a2883714f9883e532b" exitCode=0
Sep 17 18:20:23.391347 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:23.391047    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-lh5cv" event={"ID":"20456cd0-3433-42d5-bf03-8c69516c0774","Type":"ContainerDied","Data":"535d74f12a86e6a47eeae80c328d79bdde951c7a824f84a2883714f9883e532b"}
Sep 17 18:20:23.391347 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:23.391128    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-lh5cv" event={"ID":"20456cd0-3433-42d5-bf03-8c69516c0774","Type":"ContainerDied","Data":"908979bc646be1ce5236045bb15aabfdf12c0c2aa995370c1d55d0beaf74491f"}
Sep 17 18:20:23.391347 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:23.391160    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="908979bc646be1ce5236045bb15aabfdf12c0c2aa995370c1d55d0beaf74491f"
Sep 17 18:20:23.395883 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:23.395866    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-lh5cv"
Sep 17 18:20:23.439644 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:23.439619    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/20456cd0-3433-42d5-bf03-8c69516c0774-kube-rbac-proxy-sar-config\") pod \"20456cd0-3433-42d5-bf03-8c69516c0774\" (UID: \"20456cd0-3433-42d5-bf03-8c69516c0774\") "
Sep 17 18:20:23.439779 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:23.439671    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/20456cd0-3433-42d5-bf03-8c69516c0774-proxy-tls\") pod \"20456cd0-3433-42d5-bf03-8c69516c0774\" (UID: \"20456cd0-3433-42d5-bf03-8c69516c0774\") "
Sep 17 18:20:23.439779 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:23.439692    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/20456cd0-3433-42d5-bf03-8c69516c0774-cabundle-cert\") pod \"20456cd0-3433-42d5-bf03-8c69516c0774\" (UID: \"20456cd0-3433-42d5-bf03-8c69516c0774\") "
Sep 17 18:20:23.439779 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:23.439739    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mrwcs\" (UniqueName: \"kubernetes.io/projected/20456cd0-3433-42d5-bf03-8c69516c0774-kube-api-access-mrwcs\") pod \"20456cd0-3433-42d5-bf03-8c69516c0774\" (UID: \"20456cd0-3433-42d5-bf03-8c69516c0774\") "
Sep 17 18:20:23.439779 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:23.439767    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20456cd0-3433-42d5-bf03-8c69516c0774-kserve-provision-location\") pod \"20456cd0-3433-42d5-bf03-8c69516c0774\" (UID: \"20456cd0-3433-42d5-bf03-8c69516c0774\") "
Sep 17 18:20:23.440355 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:23.440125    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20456cd0-3433-42d5-bf03-8c69516c0774-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "20456cd0-3433-42d5-bf03-8c69516c0774" (UID: "20456cd0-3433-42d5-bf03-8c69516c0774"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:20:23.440467 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:23.440389    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20456cd0-3433-42d5-bf03-8c69516c0774-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "20456cd0-3433-42d5-bf03-8c69516c0774" (UID: "20456cd0-3433-42d5-bf03-8c69516c0774"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:20:23.440535 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:23.440500    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20456cd0-3433-42d5-bf03-8c69516c0774-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "20456cd0-3433-42d5-bf03-8c69516c0774" (UID: "20456cd0-3433-42d5-bf03-8c69516c0774"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:20:23.441994 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:23.441973    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20456cd0-3433-42d5-bf03-8c69516c0774-kube-api-access-mrwcs" (OuterVolumeSpecName: "kube-api-access-mrwcs") pod "20456cd0-3433-42d5-bf03-8c69516c0774" (UID: "20456cd0-3433-42d5-bf03-8c69516c0774"). InnerVolumeSpecName "kube-api-access-mrwcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:20:23.442301 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:23.442283    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20456cd0-3433-42d5-bf03-8c69516c0774-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "20456cd0-3433-42d5-bf03-8c69516c0774" (UID: "20456cd0-3433-42d5-bf03-8c69516c0774"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:20:23.540700 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:23.540675    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/20456cd0-3433-42d5-bf03-8c69516c0774-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:20:23.540700 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:23.540699    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/20456cd0-3433-42d5-bf03-8c69516c0774-proxy-tls\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:20:23.540839 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:23.540710    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/20456cd0-3433-42d5-bf03-8c69516c0774-cabundle-cert\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:20:23.540839 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:23.540719    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrwcs\" (UniqueName: \"kubernetes.io/projected/20456cd0-3433-42d5-bf03-8c69516c0774-kube-api-access-mrwcs\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:20:23.540839 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:23.540726    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20456cd0-3433-42d5-bf03-8c69516c0774-kserve-provision-location\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:20:24.395359 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:24.395325    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-lh5cv"
Sep 17 18:20:24.414420 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:24.414393    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-lh5cv"]
Sep 17 18:20:24.418005 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:24.417981    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-lh5cv"]
Sep 17 18:20:25.508207 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:25.508174    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20456cd0-3433-42d5-bf03-8c69516c0774" path="/var/lib/kubelet/pods/20456cd0-3433-42d5-bf03-8c69516c0774/volumes"
Sep 17 18:20:26.190761 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:26.190721    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-sfv8x"]
Sep 17 18:20:26.191046 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:26.191020    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-sfv8x" podUID="87fa04bd-2308-41bc-8818-07e1c6b51295" containerName="kserve-container" containerID="cri-o://fd499b5d3e80ca664ed8ee7ec8956f81ce82c7a7d39e2802bee75394590fe0d3" gracePeriod=30
Sep 17 18:20:26.191181 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:26.191047    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-sfv8x" podUID="87fa04bd-2308-41bc-8818-07e1c6b51295" containerName="kube-rbac-proxy" containerID="cri-o://82e54b02948a4048dc1c69e68489fd19df719c6d83fd0404a5d216a333189529" gracePeriod=30
Sep 17 18:20:26.405849 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:26.405819    2569 generic.go:356] "Generic (PLEG): container finished" podID="87fa04bd-2308-41bc-8818-07e1c6b51295" containerID="82e54b02948a4048dc1c69e68489fd19df719c6d83fd0404a5d216a333189529" exitCode=2
Sep 17 18:20:26.405975 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:26.405854    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-sfv8x" event={"ID":"87fa04bd-2308-41bc-8818-07e1c6b51295","Type":"ContainerDied","Data":"82e54b02948a4048dc1c69e68489fd19df719c6d83fd0404a5d216a333189529"}
Sep 17 18:20:28.396321 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:28.396295    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-5gw5z"
Sep 17 18:20:29.028391 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:29.028353    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-sfv8x" podUID="87fa04bd-2308-41bc-8818-07e1c6b51295" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Sep 17 18:20:29.032609 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:29.032580    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-sfv8x" podUID="87fa04bd-2308-41bc-8818-07e1c6b51295" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 18:20:30.425232 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:30.425205    2569 generic.go:356] "Generic (PLEG): container finished" podID="87fa04bd-2308-41bc-8818-07e1c6b51295" containerID="fd499b5d3e80ca664ed8ee7ec8956f81ce82c7a7d39e2802bee75394590fe0d3" exitCode=0
Sep 17 18:20:30.425557 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:30.425279    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-sfv8x" event={"ID":"87fa04bd-2308-41bc-8818-07e1c6b51295","Type":"ContainerDied","Data":"fd499b5d3e80ca664ed8ee7ec8956f81ce82c7a7d39e2802bee75394590fe0d3"}
Sep 17 18:20:30.425557 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:30.425345    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-sfv8x" event={"ID":"87fa04bd-2308-41bc-8818-07e1c6b51295","Type":"ContainerDied","Data":"0a2788e18b52b7e83bcc40d18426f91e8220f0b5b612cf138a4879afcb88ee6c"}
Sep 17 18:20:30.425557 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:30.425372    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-sfv8x"
Sep 17 18:20:30.425557 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:30.425375    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a2788e18b52b7e83bcc40d18426f91e8220f0b5b612cf138a4879afcb88ee6c"
Sep 17 18:20:30.497456 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:30.497398    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/87fa04bd-2308-41bc-8818-07e1c6b51295-cabundle-cert\") pod \"87fa04bd-2308-41bc-8818-07e1c6b51295\" (UID: \"87fa04bd-2308-41bc-8818-07e1c6b51295\") "
Sep 17 18:20:30.497569 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:30.497463    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87fa04bd-2308-41bc-8818-07e1c6b51295-proxy-tls\") pod \"87fa04bd-2308-41bc-8818-07e1c6b51295\" (UID: \"87fa04bd-2308-41bc-8818-07e1c6b51295\") "
Sep 17 18:20:30.497569 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:30.497512    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87fa04bd-2308-41bc-8818-07e1c6b51295-kserve-provision-location\") pod \"87fa04bd-2308-41bc-8818-07e1c6b51295\" (UID: \"87fa04bd-2308-41bc-8818-07e1c6b51295\") "
Sep 17 18:20:30.497569 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:30.497533    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gxp9h\" (UniqueName: \"kubernetes.io/projected/87fa04bd-2308-41bc-8818-07e1c6b51295-kube-api-access-gxp9h\") pod \"87fa04bd-2308-41bc-8818-07e1c6b51295\" (UID: \"87fa04bd-2308-41bc-8818-07e1c6b51295\") "
Sep 17 18:20:30.497569 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:30.497564    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/87fa04bd-2308-41bc-8818-07e1c6b51295-kube-rbac-proxy-sar-config\") pod \"87fa04bd-2308-41bc-8818-07e1c6b51295\" (UID: \"87fa04bd-2308-41bc-8818-07e1c6b51295\") "
Sep 17 18:20:30.497823 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:30.497793    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87fa04bd-2308-41bc-8818-07e1c6b51295-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "87fa04bd-2308-41bc-8818-07e1c6b51295" (UID: "87fa04bd-2308-41bc-8818-07e1c6b51295"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:20:30.497823 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:30.497813    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87fa04bd-2308-41bc-8818-07e1c6b51295-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "87fa04bd-2308-41bc-8818-07e1c6b51295" (UID: "87fa04bd-2308-41bc-8818-07e1c6b51295"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:20:30.498012 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:30.497989    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87fa04bd-2308-41bc-8818-07e1c6b51295-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "87fa04bd-2308-41bc-8818-07e1c6b51295" (UID: "87fa04bd-2308-41bc-8818-07e1c6b51295"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:20:30.499478 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:30.499460    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87fa04bd-2308-41bc-8818-07e1c6b51295-kube-api-access-gxp9h" (OuterVolumeSpecName: "kube-api-access-gxp9h") pod "87fa04bd-2308-41bc-8818-07e1c6b51295" (UID: "87fa04bd-2308-41bc-8818-07e1c6b51295"). InnerVolumeSpecName "kube-api-access-gxp9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:20:30.499580 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:30.499564    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87fa04bd-2308-41bc-8818-07e1c6b51295-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "87fa04bd-2308-41bc-8818-07e1c6b51295" (UID: "87fa04bd-2308-41bc-8818-07e1c6b51295"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:20:30.598600 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:30.598577    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87fa04bd-2308-41bc-8818-07e1c6b51295-kserve-provision-location\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:20:30.598703 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:30.598603    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxp9h\" (UniqueName: \"kubernetes.io/projected/87fa04bd-2308-41bc-8818-07e1c6b51295-kube-api-access-gxp9h\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:20:30.598703 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:30.598614    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/87fa04bd-2308-41bc-8818-07e1c6b51295-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:20:30.598703 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:30.598624    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/87fa04bd-2308-41bc-8818-07e1c6b51295-cabundle-cert\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:20:30.598703 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:30.598632    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87fa04bd-2308-41bc-8818-07e1c6b51295-proxy-tls\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:20:31.429073 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:31.429042    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-sfv8x"
Sep 17 18:20:31.449352 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:31.449316    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-sfv8x"]
Sep 17 18:20:31.453823 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:31.453801    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-sfv8x"]
Sep 17 18:20:31.508160 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:31.508139    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87fa04bd-2308-41bc-8818-07e1c6b51295" path="/var/lib/kubelet/pods/87fa04bd-2308-41bc-8818-07e1c6b51295/volumes"
Sep 17 18:20:36.501669 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:36.501635    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6255j"]
Sep 17 18:20:36.502161 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:36.502103    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20456cd0-3433-42d5-bf03-8c69516c0774" containerName="storage-initializer"
Sep 17 18:20:36.502161 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:36.502121    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="20456cd0-3433-42d5-bf03-8c69516c0774" containerName="storage-initializer"
Sep 17 18:20:36.502161 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:36.502134    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20456cd0-3433-42d5-bf03-8c69516c0774" containerName="kserve-container"
Sep 17 18:20:36.502161 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:36.502142    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="20456cd0-3433-42d5-bf03-8c69516c0774" containerName="kserve-container"
Sep 17 18:20:36.502161 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:36.502156    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87fa04bd-2308-41bc-8818-07e1c6b51295" containerName="storage-initializer"
Sep 17 18:20:36.502161 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:36.502164    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="87fa04bd-2308-41bc-8818-07e1c6b51295" containerName="storage-initializer"
Sep 17 18:20:36.502503 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:36.502179    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1765ef9b-2007-42e1-b88f-f5cef196384a" containerName="kserve-container"
Sep 17 18:20:36.502503 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:36.502187    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1765ef9b-2007-42e1-b88f-f5cef196384a" containerName="kserve-container"
Sep 17 18:20:36.502503 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:36.502204    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87fa04bd-2308-41bc-8818-07e1c6b51295" containerName="kserve-container"
Sep 17 18:20:36.502503 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:36.502211    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="87fa04bd-2308-41bc-8818-07e1c6b51295" containerName="kserve-container"
Sep 17 18:20:36.502503 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:36.502221    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87fa04bd-2308-41bc-8818-07e1c6b51295" containerName="kube-rbac-proxy"
Sep 17 18:20:36.502503 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:36.502229    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="87fa04bd-2308-41bc-8818-07e1c6b51295" containerName="kube-rbac-proxy"
Sep 17 18:20:36.502503 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:36.502240    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1765ef9b-2007-42e1-b88f-f5cef196384a" containerName="storage-initializer"
Sep 17 18:20:36.502503 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:36.502248    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1765ef9b-2007-42e1-b88f-f5cef196384a" containerName="storage-initializer"
Sep 17 18:20:36.502503 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:36.502259    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20456cd0-3433-42d5-bf03-8c69516c0774" containerName="kube-rbac-proxy"
Sep 17 18:20:36.502503 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:36.502267    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="20456cd0-3433-42d5-bf03-8c69516c0774" containerName="kube-rbac-proxy"
Sep 17 18:20:36.502503 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:36.502276    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1765ef9b-2007-42e1-b88f-f5cef196384a" containerName="kube-rbac-proxy"
Sep 17 18:20:36.502503 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:36.502285    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1765ef9b-2007-42e1-b88f-f5cef196384a" containerName="kube-rbac-proxy"
Sep 17 18:20:36.502503 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:36.502368    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="87fa04bd-2308-41bc-8818-07e1c6b51295" containerName="kube-rbac-proxy"
Sep 17 18:20:36.502503 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:36.502381    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="1765ef9b-2007-42e1-b88f-f5cef196384a" containerName="kserve-container"
Sep 17 18:20:36.502503 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:36.502395    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="20456cd0-3433-42d5-bf03-8c69516c0774" containerName="kube-rbac-proxy"
Sep 17 18:20:36.502503 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:36.502405    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="20456cd0-3433-42d5-bf03-8c69516c0774" containerName="kserve-container"
Sep 17 18:20:36.502503 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:36.502414    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="87fa04bd-2308-41bc-8818-07e1c6b51295" containerName="kserve-container"
Sep 17 18:20:36.502503 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:36.502423    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="1765ef9b-2007-42e1-b88f-f5cef196384a" containerName="kube-rbac-proxy"
Sep 17 18:20:36.507104 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:36.507062    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6255j"
Sep 17 18:20:36.510138 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:36.510119    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-kube-rbac-proxy-sar-config\""
Sep 17 18:20:36.510219 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:36.510124    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-predictor-serving-cert\""
Sep 17 18:20:36.515837 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:36.515816    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6255j"]
Sep 17 18:20:36.638089 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:36.638050    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9adc9c04-3250-428d-bcda-bfc8ed97077a-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-6255j\" (UID: \"9adc9c04-3250-428d-bcda-bfc8ed97077a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6255j"
Sep 17 18:20:36.638183 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:36.638114    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9adc9c04-3250-428d-bcda-bfc8ed97077a-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-6255j\" (UID: \"9adc9c04-3250-428d-bcda-bfc8ed97077a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6255j"
Sep 17 18:20:36.638183 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:36.638146    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9adc9c04-3250-428d-bcda-bfc8ed97077a-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-6255j\" (UID: \"9adc9c04-3250-428d-bcda-bfc8ed97077a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6255j"
Sep 17 18:20:36.638257 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:36.638199    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9adc9c04-3250-428d-bcda-bfc8ed97077a-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-6255j\" (UID: \"9adc9c04-3250-428d-bcda-bfc8ed97077a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6255j"
Sep 17 18:20:36.638257 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:36.638218    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbrx6\" (UniqueName: \"kubernetes.io/projected/9adc9c04-3250-428d-bcda-bfc8ed97077a-kube-api-access-nbrx6\") pod \"isvc-lightgbm-predictor-54f85ddf88-6255j\" (UID: \"9adc9c04-3250-428d-bcda-bfc8ed97077a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6255j"
Sep 17 18:20:36.739418 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:36.739390    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9adc9c04-3250-428d-bcda-bfc8ed97077a-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-6255j\" (UID: \"9adc9c04-3250-428d-bcda-bfc8ed97077a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6255j"
Sep 17 18:20:36.739520 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:36.739431    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbrx6\" (UniqueName: \"kubernetes.io/projected/9adc9c04-3250-428d-bcda-bfc8ed97077a-kube-api-access-nbrx6\") pod \"isvc-lightgbm-predictor-54f85ddf88-6255j\" (UID: \"9adc9c04-3250-428d-bcda-bfc8ed97077a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6255j"
Sep 17 18:20:36.739520 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:36.739474    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9adc9c04-3250-428d-bcda-bfc8ed97077a-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-6255j\" (UID: \"9adc9c04-3250-428d-bcda-bfc8ed97077a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6255j"
Sep 17 18:20:36.739520 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:36.739515    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9adc9c04-3250-428d-bcda-bfc8ed97077a-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-6255j\" (UID: \"9adc9c04-3250-428d-bcda-bfc8ed97077a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6255j"
Sep 17 18:20:36.739681 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:36.739552    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9adc9c04-3250-428d-bcda-bfc8ed97077a-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-6255j\" (UID: \"9adc9c04-3250-428d-bcda-bfc8ed97077a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6255j"
Sep 17 18:20:36.739943 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:36.739916    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9adc9c04-3250-428d-bcda-bfc8ed97077a-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-6255j\" (UID: \"9adc9c04-3250-428d-bcda-bfc8ed97077a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6255j"
Sep 17 18:20:36.740349 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:36.740329    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9adc9c04-3250-428d-bcda-bfc8ed97077a-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-6255j\" (UID: \"9adc9c04-3250-428d-bcda-bfc8ed97077a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6255j"
Sep 17 18:20:36.740431 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:36.740371    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9adc9c04-3250-428d-bcda-bfc8ed97077a-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-6255j\" (UID: \"9adc9c04-3250-428d-bcda-bfc8ed97077a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6255j"
Sep 17 18:20:36.741874 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:36.741851    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9adc9c04-3250-428d-bcda-bfc8ed97077a-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-6255j\" (UID: \"9adc9c04-3250-428d-bcda-bfc8ed97077a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6255j"
Sep 17 18:20:36.748831 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:36.748812    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbrx6\" (UniqueName: \"kubernetes.io/projected/9adc9c04-3250-428d-bcda-bfc8ed97077a-kube-api-access-nbrx6\") pod \"isvc-lightgbm-predictor-54f85ddf88-6255j\" (UID: \"9adc9c04-3250-428d-bcda-bfc8ed97077a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6255j"
Sep 17 18:20:36.817232 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:36.817176    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6255j"
Sep 17 18:20:37.091602 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:37.091578    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6255j"]
Sep 17 18:20:37.093428 ip-10-0-131-14 kubenswrapper[2569]: W0917 18:20:37.093402    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9adc9c04_3250_428d_bcda_bfc8ed97077a.slice/crio-f9e999dabdd13d1534126cb09e835e2bffb37c8691f710bd8be15dd07f671681 WatchSource:0}: Error finding container f9e999dabdd13d1534126cb09e835e2bffb37c8691f710bd8be15dd07f671681: Status 404 returned error can't find the container with id f9e999dabdd13d1534126cb09e835e2bffb37c8691f710bd8be15dd07f671681
Sep 17 18:20:37.462474 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:37.462440    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6255j" event={"ID":"9adc9c04-3250-428d-bcda-bfc8ed97077a","Type":"ContainerStarted","Data":"2ff8911cea5a9352db30dbc7dc25bdf160afe2a690252abf2f9792e8739d937e"}
Sep 17 18:20:37.462655 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:37.462539    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6255j" event={"ID":"9adc9c04-3250-428d-bcda-bfc8ed97077a","Type":"ContainerStarted","Data":"f9e999dabdd13d1534126cb09e835e2bffb37c8691f710bd8be15dd07f671681"}
Sep 17 18:20:41.481226 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:41.481158    2569 generic.go:356] "Generic (PLEG): container finished" podID="9adc9c04-3250-428d-bcda-bfc8ed97077a" containerID="2ff8911cea5a9352db30dbc7dc25bdf160afe2a690252abf2f9792e8739d937e" exitCode=0
Sep 17 18:20:41.481226 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:41.481200    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6255j" event={"ID":"9adc9c04-3250-428d-bcda-bfc8ed97077a","Type":"ContainerDied","Data":"2ff8911cea5a9352db30dbc7dc25bdf160afe2a690252abf2f9792e8739d937e"}
Sep 17 18:20:41.482371 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:41.482355    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:20:48.516232 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:48.516198    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6255j" event={"ID":"9adc9c04-3250-428d-bcda-bfc8ed97077a","Type":"ContainerStarted","Data":"aae77b1553711f8f406c7c1bae812c33b0b6e6a68ca1d6d4d4eec484177a2ad1"}
Sep 17 18:20:48.516699 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:48.516270    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6255j" event={"ID":"9adc9c04-3250-428d-bcda-bfc8ed97077a","Type":"ContainerStarted","Data":"269d1c3eaebd415b2f55ba415c5b33164f54fdd855b1cb9202c7a7f071e8bd58"}
Sep 17 18:20:48.516699 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:48.516482    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6255j"
Sep 17 18:20:48.537357 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:48.537313    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6255j" podStartSLOduration=2.065613099 podStartE2EDuration="12.537299555s" totalImagesPullingTime="6.416225821s" totalInitContainerRuntime="4.055460635s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:20:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:20:48.535801951 +0000 UTC m=+2401.039804577" watchObservedRunningTime="2026-09-17 18:20:48.537299555 +0000 UTC m=+2401.041302180"
Sep 17 18:20:49.520658 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:49.520630    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6255j"
Sep 17 18:20:49.521999 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:49.521973    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6255j" podUID="9adc9c04-3250-428d-bcda-bfc8ed97077a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 18:20:49.636875 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:49.636853    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-t7pqb_b8d76e28-4655-4d21-98d8-0324e4756aaf/console-operator/2.log"
Sep 17 18:20:49.639634 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:49.639609    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-t7pqb_b8d76e28-4655-4d21-98d8-0324e4756aaf/console-operator/2.log"
Sep 17 18:20:49.643478 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:49.643461    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k4mqn_6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff/ovn-acl-logging/0.log"
Sep 17 18:20:49.646181 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:49.646164    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k4mqn_6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff/ovn-acl-logging/0.log"
Sep 17 18:20:50.524641 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:50.524606    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6255j" podUID="9adc9c04-3250-428d-bcda-bfc8ed97077a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 18:20:55.529089 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:55.529041    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6255j"
Sep 17 18:20:55.529631 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:55.529597    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6255j" podUID="9adc9c04-3250-428d-bcda-bfc8ed97077a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 18:20:58.400271 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:20:58.400243    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-5gw5z"
Sep 17 18:21:05.409531 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:05.409493    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-5gw5z"]
Sep 17 18:21:05.410019 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:05.409851    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-5gw5z" podUID="81cd0cbc-9d19-4b19-8db9-28edbded0d19" containerName="kserve-container" containerID="cri-o://b4e3ba0b72ea0e3106f68423cfddb5e3701780e5724a1f56fd2e2a6c45763bf8" gracePeriod=30
Sep 17 18:21:05.410019 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:05.409867    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-5gw5z" podUID="81cd0cbc-9d19-4b19-8db9-28edbded0d19" containerName="kube-rbac-proxy" containerID="cri-o://543caa400c7efc111371f97c99d4f445e87c47086ff9c3532bd6ddc5aaa33cd3" gracePeriod=30
Sep 17 18:21:05.529589 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:05.529551    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6255j" podUID="9adc9c04-3250-428d-bcda-bfc8ed97077a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 18:21:05.592645 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:05.592619    2569 generic.go:356] "Generic (PLEG): container finished" podID="81cd0cbc-9d19-4b19-8db9-28edbded0d19" containerID="543caa400c7efc111371f97c99d4f445e87c47086ff9c3532bd6ddc5aaa33cd3" exitCode=2
Sep 17 18:21:05.592771 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:05.592693    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-5gw5z" event={"ID":"81cd0cbc-9d19-4b19-8db9-28edbded0d19","Type":"ContainerDied","Data":"543caa400c7efc111371f97c99d4f445e87c47086ff9c3532bd6ddc5aaa33cd3"}
Sep 17 18:21:08.392452 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:08.392411    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-5gw5z" podUID="81cd0cbc-9d19-4b19-8db9-28edbded0d19" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
Sep 17 18:21:08.396791 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:08.396762    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-5gw5z" podUID="81cd0cbc-9d19-4b19-8db9-28edbded0d19" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.40:8080/v2/models/xgboost-v2-mlserver/ready\": dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 18:21:10.555967 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:10.555946    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-5gw5z"
Sep 17 18:21:10.575210 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:10.575184    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gslgl"]
Sep 17 18:21:10.575514 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:10.575498    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81cd0cbc-9d19-4b19-8db9-28edbded0d19" containerName="kserve-container"
Sep 17 18:21:10.575590 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:10.575518    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="81cd0cbc-9d19-4b19-8db9-28edbded0d19" containerName="kserve-container"
Sep 17 18:21:10.575590 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:10.575547    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81cd0cbc-9d19-4b19-8db9-28edbded0d19" containerName="storage-initializer"
Sep 17 18:21:10.575590 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:10.575558    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="81cd0cbc-9d19-4b19-8db9-28edbded0d19" containerName="storage-initializer"
Sep 17 18:21:10.575590 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:10.575570    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81cd0cbc-9d19-4b19-8db9-28edbded0d19" containerName="kube-rbac-proxy"
Sep 17 18:21:10.575590 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:10.575579    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="81cd0cbc-9d19-4b19-8db9-28edbded0d19" containerName="kube-rbac-proxy"
Sep 17 18:21:10.575859 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:10.575670    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="81cd0cbc-9d19-4b19-8db9-28edbded0d19" containerName="kube-rbac-proxy"
Sep 17 18:21:10.575859 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:10.575682    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="81cd0cbc-9d19-4b19-8db9-28edbded0d19" containerName="kserve-container"
Sep 17 18:21:10.578643 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:10.578625    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gslgl"
Sep 17 18:21:10.581259 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:10.581233    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-predictor-serving-cert\""
Sep 17 18:21:10.581363 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:10.581287    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\""
Sep 17 18:21:10.588051 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:10.588028    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gslgl"]
Sep 17 18:21:10.619038 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:10.618973    2569 generic.go:356] "Generic (PLEG): container finished" podID="81cd0cbc-9d19-4b19-8db9-28edbded0d19" containerID="b4e3ba0b72ea0e3106f68423cfddb5e3701780e5724a1f56fd2e2a6c45763bf8" exitCode=0
Sep 17 18:21:10.619038 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:10.619006    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-5gw5z" event={"ID":"81cd0cbc-9d19-4b19-8db9-28edbded0d19","Type":"ContainerDied","Data":"b4e3ba0b72ea0e3106f68423cfddb5e3701780e5724a1f56fd2e2a6c45763bf8"}
Sep 17 18:21:10.619038 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:10.619033    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-5gw5z"
Sep 17 18:21:10.619308 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:10.619128    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-5gw5z" event={"ID":"81cd0cbc-9d19-4b19-8db9-28edbded0d19","Type":"ContainerDied","Data":"827fd0d1be7ad4d3c3e3a2881aa540b3dc6d38b11c46c112c5b15b38228c2ecd"}
Sep 17 18:21:10.619308 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:10.619171    2569 scope.go:117] "RemoveContainer" containerID="543caa400c7efc111371f97c99d4f445e87c47086ff9c3532bd6ddc5aaa33cd3"
Sep 17 18:21:10.627122 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:10.627107    2569 scope.go:117] "RemoveContainer" containerID="b4e3ba0b72ea0e3106f68423cfddb5e3701780e5724a1f56fd2e2a6c45763bf8"
Sep 17 18:21:10.633999 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:10.633984    2569 scope.go:117] "RemoveContainer" containerID="be2ea5d767a4fa951b4ffce7a78c38dd0bd290342d8ff7c83454f63c5113f901"
Sep 17 18:21:10.640668 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:10.640651    2569 scope.go:117] "RemoveContainer" containerID="543caa400c7efc111371f97c99d4f445e87c47086ff9c3532bd6ddc5aaa33cd3"
Sep 17 18:21:10.640938 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:21:10.640920    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"543caa400c7efc111371f97c99d4f445e87c47086ff9c3532bd6ddc5aaa33cd3\": container with ID starting with 543caa400c7efc111371f97c99d4f445e87c47086ff9c3532bd6ddc5aaa33cd3 not found: ID does not exist" containerID="543caa400c7efc111371f97c99d4f445e87c47086ff9c3532bd6ddc5aaa33cd3"
Sep 17 18:21:10.641010 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:10.640950    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"543caa400c7efc111371f97c99d4f445e87c47086ff9c3532bd6ddc5aaa33cd3"} err="failed to get container status \"543caa400c7efc111371f97c99d4f445e87c47086ff9c3532bd6ddc5aaa33cd3\": rpc error: code = NotFound desc = could not find container \"543caa400c7efc111371f97c99d4f445e87c47086ff9c3532bd6ddc5aaa33cd3\": container with ID starting with 543caa400c7efc111371f97c99d4f445e87c47086ff9c3532bd6ddc5aaa33cd3 not found: ID does not exist"
Sep 17 18:21:10.641010 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:10.640973    2569 scope.go:117] "RemoveContainer" containerID="b4e3ba0b72ea0e3106f68423cfddb5e3701780e5724a1f56fd2e2a6c45763bf8"
Sep 17 18:21:10.641250 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:21:10.641234    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4e3ba0b72ea0e3106f68423cfddb5e3701780e5724a1f56fd2e2a6c45763bf8\": container with ID starting with b4e3ba0b72ea0e3106f68423cfddb5e3701780e5724a1f56fd2e2a6c45763bf8 not found: ID does not exist" containerID="b4e3ba0b72ea0e3106f68423cfddb5e3701780e5724a1f56fd2e2a6c45763bf8"
Sep 17 18:21:10.641325 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:10.641257    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4e3ba0b72ea0e3106f68423cfddb5e3701780e5724a1f56fd2e2a6c45763bf8"} err="failed to get container status \"b4e3ba0b72ea0e3106f68423cfddb5e3701780e5724a1f56fd2e2a6c45763bf8\": rpc error: code = NotFound desc = could not find container \"b4e3ba0b72ea0e3106f68423cfddb5e3701780e5724a1f56fd2e2a6c45763bf8\": container with ID starting with b4e3ba0b72ea0e3106f68423cfddb5e3701780e5724a1f56fd2e2a6c45763bf8 not found: ID does not exist"
Sep 17 18:21:10.641325 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:10.641276    2569 scope.go:117] "RemoveContainer" containerID="be2ea5d767a4fa951b4ffce7a78c38dd0bd290342d8ff7c83454f63c5113f901"
Sep 17 18:21:10.641499 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:21:10.641484    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be2ea5d767a4fa951b4ffce7a78c38dd0bd290342d8ff7c83454f63c5113f901\": container with ID starting with be2ea5d767a4fa951b4ffce7a78c38dd0bd290342d8ff7c83454f63c5113f901 not found: ID does not exist" containerID="be2ea5d767a4fa951b4ffce7a78c38dd0bd290342d8ff7c83454f63c5113f901"
Sep 17 18:21:10.641546 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:10.641504    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be2ea5d767a4fa951b4ffce7a78c38dd0bd290342d8ff7c83454f63c5113f901"} err="failed to get container status \"be2ea5d767a4fa951b4ffce7a78c38dd0bd290342d8ff7c83454f63c5113f901\": rpc error: code = NotFound desc = could not find container \"be2ea5d767a4fa951b4ffce7a78c38dd0bd290342d8ff7c83454f63c5113f901\": container with ID starting with be2ea5d767a4fa951b4ffce7a78c38dd0bd290342d8ff7c83454f63c5113f901 not found: ID does not exist"
Sep 17 18:21:10.698259 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:10.698235    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81cd0cbc-9d19-4b19-8db9-28edbded0d19-kube-rbac-proxy-sar-config\") pod \"81cd0cbc-9d19-4b19-8db9-28edbded0d19\" (UID: \"81cd0cbc-9d19-4b19-8db9-28edbded0d19\") "
Sep 17 18:21:10.698355 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:10.698318    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6fthj\" (UniqueName: \"kubernetes.io/projected/81cd0cbc-9d19-4b19-8db9-28edbded0d19-kube-api-access-6fthj\") pod \"81cd0cbc-9d19-4b19-8db9-28edbded0d19\" (UID: \"81cd0cbc-9d19-4b19-8db9-28edbded0d19\") "
Sep 17 18:21:10.698422 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:10.698364    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81cd0cbc-9d19-4b19-8db9-28edbded0d19-kserve-provision-location\") pod \"81cd0cbc-9d19-4b19-8db9-28edbded0d19\" (UID: \"81cd0cbc-9d19-4b19-8db9-28edbded0d19\") "
Sep 17 18:21:10.698422 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:10.698392    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81cd0cbc-9d19-4b19-8db9-28edbded0d19-cabundle-cert\") pod \"81cd0cbc-9d19-4b19-8db9-28edbded0d19\" (UID: \"81cd0cbc-9d19-4b19-8db9-28edbded0d19\") "
Sep 17 18:21:10.698527 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:10.698460    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81cd0cbc-9d19-4b19-8db9-28edbded0d19-proxy-tls\") pod \"81cd0cbc-9d19-4b19-8db9-28edbded0d19\" (UID: \"81cd0cbc-9d19-4b19-8db9-28edbded0d19\") "
Sep 17 18:21:10.698579 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:10.698548    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81cd0cbc-9d19-4b19-8db9-28edbded0d19-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "81cd0cbc-9d19-4b19-8db9-28edbded0d19" (UID: "81cd0cbc-9d19-4b19-8db9-28edbded0d19"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:21:10.698637 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:10.698580    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z5hj\" (UniqueName: \"kubernetes.io/projected/8dd1fe07-7ba4-484d-9beb-79c5751cd739-kube-api-access-9z5hj\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-gslgl\" (UID: \"8dd1fe07-7ba4-484d-9beb-79c5751cd739\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gslgl"
Sep 17 18:21:10.698637 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:10.698615    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8dd1fe07-7ba4-484d-9beb-79c5751cd739-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-gslgl\" (UID: \"8dd1fe07-7ba4-484d-9beb-79c5751cd739\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gslgl"
Sep 17 18:21:10.698753 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:10.698717    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8dd1fe07-7ba4-484d-9beb-79c5751cd739-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-gslgl\" (UID: \"8dd1fe07-7ba4-484d-9beb-79c5751cd739\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gslgl"
Sep 17 18:21:10.698753 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:10.698711    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81cd0cbc-9d19-4b19-8db9-28edbded0d19-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "81cd0cbc-9d19-4b19-8db9-28edbded0d19" (UID: "81cd0cbc-9d19-4b19-8db9-28edbded0d19"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:21:10.698753 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:10.698734    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81cd0cbc-9d19-4b19-8db9-28edbded0d19-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "81cd0cbc-9d19-4b19-8db9-28edbded0d19" (UID: "81cd0cbc-9d19-4b19-8db9-28edbded0d19"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:21:10.698915 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:10.698813    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8dd1fe07-7ba4-484d-9beb-79c5751cd739-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-gslgl\" (UID: \"8dd1fe07-7ba4-484d-9beb-79c5751cd739\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gslgl"
Sep 17 18:21:10.698915 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:10.698853    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8dd1fe07-7ba4-484d-9beb-79c5751cd739-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-gslgl\" (UID: \"8dd1fe07-7ba4-484d-9beb-79c5751cd739\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gslgl"
Sep 17 18:21:10.698915 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:10.698894    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81cd0cbc-9d19-4b19-8db9-28edbded0d19-kserve-provision-location\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:21:10.698915 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:10.698903    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81cd0cbc-9d19-4b19-8db9-28edbded0d19-cabundle-cert\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:21:10.698915 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:10.698912    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81cd0cbc-9d19-4b19-8db9-28edbded0d19-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:21:10.700355 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:10.700335    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81cd0cbc-9d19-4b19-8db9-28edbded0d19-kube-api-access-6fthj" (OuterVolumeSpecName: "kube-api-access-6fthj") pod "81cd0cbc-9d19-4b19-8db9-28edbded0d19" (UID: "81cd0cbc-9d19-4b19-8db9-28edbded0d19"). InnerVolumeSpecName "kube-api-access-6fthj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:21:10.700489 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:10.700470    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81cd0cbc-9d19-4b19-8db9-28edbded0d19-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "81cd0cbc-9d19-4b19-8db9-28edbded0d19" (UID: "81cd0cbc-9d19-4b19-8db9-28edbded0d19"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:21:10.799899 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:10.799874    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8dd1fe07-7ba4-484d-9beb-79c5751cd739-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-gslgl\" (UID: \"8dd1fe07-7ba4-484d-9beb-79c5751cd739\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gslgl"
Sep 17 18:21:10.799987 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:10.799910    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8dd1fe07-7ba4-484d-9beb-79c5751cd739-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-gslgl\" (UID: \"8dd1fe07-7ba4-484d-9beb-79c5751cd739\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gslgl"
Sep 17 18:21:10.799987 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:10.799933    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z5hj\" (UniqueName: \"kubernetes.io/projected/8dd1fe07-7ba4-484d-9beb-79c5751cd739-kube-api-access-9z5hj\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-gslgl\" (UID: \"8dd1fe07-7ba4-484d-9beb-79c5751cd739\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gslgl"
Sep 17 18:21:10.799987 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:10.799951    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8dd1fe07-7ba4-484d-9beb-79c5751cd739-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-gslgl\" (UID: \"8dd1fe07-7ba4-484d-9beb-79c5751cd739\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gslgl"
Sep 17 18:21:10.800132 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:10.799987    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8dd1fe07-7ba4-484d-9beb-79c5751cd739-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-gslgl\" (UID: \"8dd1fe07-7ba4-484d-9beb-79c5751cd739\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gslgl"
Sep 17 18:21:10.800132 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:10.800028    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81cd0cbc-9d19-4b19-8db9-28edbded0d19-proxy-tls\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:21:10.800132 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:10.800046    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fthj\" (UniqueName: \"kubernetes.io/projected/81cd0cbc-9d19-4b19-8db9-28edbded0d19-kube-api-access-6fthj\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:21:10.800437 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:10.800413    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8dd1fe07-7ba4-484d-9beb-79c5751cd739-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-gslgl\" (UID: \"8dd1fe07-7ba4-484d-9beb-79c5751cd739\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gslgl"
Sep 17 18:21:10.800696 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:10.800673    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8dd1fe07-7ba4-484d-9beb-79c5751cd739-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-gslgl\" (UID: \"8dd1fe07-7ba4-484d-9beb-79c5751cd739\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gslgl"
Sep 17 18:21:10.800774 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:10.800673    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8dd1fe07-7ba4-484d-9beb-79c5751cd739-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-gslgl\" (UID: \"8dd1fe07-7ba4-484d-9beb-79c5751cd739\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gslgl"
Sep 17 18:21:10.802180 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:10.802162    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8dd1fe07-7ba4-484d-9beb-79c5751cd739-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-gslgl\" (UID: \"8dd1fe07-7ba4-484d-9beb-79c5751cd739\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gslgl"
Sep 17 18:21:10.807716 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:10.807694    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z5hj\" (UniqueName: \"kubernetes.io/projected/8dd1fe07-7ba4-484d-9beb-79c5751cd739-kube-api-access-9z5hj\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-gslgl\" (UID: \"8dd1fe07-7ba4-484d-9beb-79c5751cd739\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gslgl"
Sep 17 18:21:10.889479 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:10.889432    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gslgl"
Sep 17 18:21:10.944720 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:10.944655    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-5gw5z"]
Sep 17 18:21:10.946993 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:10.946968    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-5gw5z"]
Sep 17 18:21:11.161794 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:11.161755    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gslgl"]
Sep 17 18:21:11.162987 ip-10-0-131-14 kubenswrapper[2569]: W0917 18:21:11.162962    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dd1fe07_7ba4_484d_9beb_79c5751cd739.slice/crio-67ac4a26757c5b436f459a3f24e4d1514ccc9c721d06754f5415ff901279447b WatchSource:0}: Error finding container 67ac4a26757c5b436f459a3f24e4d1514ccc9c721d06754f5415ff901279447b: Status 404 returned error can't find the container with id 67ac4a26757c5b436f459a3f24e4d1514ccc9c721d06754f5415ff901279447b
Sep 17 18:21:11.509415 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:11.509384    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81cd0cbc-9d19-4b19-8db9-28edbded0d19" path="/var/lib/kubelet/pods/81cd0cbc-9d19-4b19-8db9-28edbded0d19/volumes"
Sep 17 18:21:11.627275 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:11.627237    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gslgl" event={"ID":"8dd1fe07-7ba4-484d-9beb-79c5751cd739","Type":"ContainerStarted","Data":"286e2db5f6452ac05da7b8a08dcc4bed9d3b5bf09de1c2d2a06a7d13c17ed2d0"}
Sep 17 18:21:11.627614 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:11.627299    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gslgl" event={"ID":"8dd1fe07-7ba4-484d-9beb-79c5751cd739","Type":"ContainerStarted","Data":"67ac4a26757c5b436f459a3f24e4d1514ccc9c721d06754f5415ff901279447b"}
Sep 17 18:21:15.529717 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:15.529681    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6255j" podUID="9adc9c04-3250-428d-bcda-bfc8ed97077a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 18:21:15.646484 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:15.646458    2569 generic.go:356] "Generic (PLEG): container finished" podID="8dd1fe07-7ba4-484d-9beb-79c5751cd739" containerID="286e2db5f6452ac05da7b8a08dcc4bed9d3b5bf09de1c2d2a06a7d13c17ed2d0" exitCode=0
Sep 17 18:21:15.646624 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:15.646531    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gslgl" event={"ID":"8dd1fe07-7ba4-484d-9beb-79c5751cd739","Type":"ContainerDied","Data":"286e2db5f6452ac05da7b8a08dcc4bed9d3b5bf09de1c2d2a06a7d13c17ed2d0"}
Sep 17 18:21:25.529894 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:25.529855    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6255j" podUID="9adc9c04-3250-428d-bcda-bfc8ed97077a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 18:21:35.529745 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:35.529708    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6255j" podUID="9adc9c04-3250-428d-bcda-bfc8ed97077a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 18:21:35.748047 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:35.748013    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gslgl" event={"ID":"8dd1fe07-7ba4-484d-9beb-79c5751cd739","Type":"ContainerStarted","Data":"0757b97bc4ded2cecefbf8611218d8297af442129fbc35c2933352ed76017ed8"}
Sep 17 18:21:35.748215 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:35.748120    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gslgl" event={"ID":"8dd1fe07-7ba4-484d-9beb-79c5751cd739","Type":"ContainerStarted","Data":"3d6b21cb76bbd9ed499cb82fefcc0d57edffe2b2a2c786069ad326c79ab6f72e"}
Sep 17 18:21:35.748367 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:35.748349    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gslgl"
Sep 17 18:21:35.767520 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:35.767461    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gslgl" podStartSLOduration=2.79350175 podStartE2EDuration="25.767443162s" totalImagesPullingTime="19.106597366s" totalInitContainerRuntime="3.867344046s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:21:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:21:35.765881965 +0000 UTC m=+2448.269884589" watchObservedRunningTime="2026-09-17 18:21:35.767443162 +0000 UTC m=+2448.271445788"
Sep 17 18:21:36.752863 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:36.752825    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gslgl"
Sep 17 18:21:36.754161 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:36.754123    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gslgl" podUID="8dd1fe07-7ba4-484d-9beb-79c5751cd739" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 17 18:21:37.757744 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:37.757689    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gslgl" podUID="8dd1fe07-7ba4-484d-9beb-79c5751cd739" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 17 18:21:42.763290 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:42.763263    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gslgl"
Sep 17 18:21:42.764015 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:42.763961    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gslgl" podUID="8dd1fe07-7ba4-484d-9beb-79c5751cd739" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 17 18:21:45.529924 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:45.529882    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6255j" podUID="9adc9c04-3250-428d-bcda-bfc8ed97077a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 18:21:52.763803 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:52.763748    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gslgl" podUID="8dd1fe07-7ba4-484d-9beb-79c5751cd739" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 17 18:21:55.530226 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:21:55.530180    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6255j" podUID="9adc9c04-3250-428d-bcda-bfc8ed97077a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 18:22:02.764477 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:02.764443    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gslgl" podUID="8dd1fe07-7ba4-484d-9beb-79c5751cd739" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 17 18:22:05.530459 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:05.530420    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6255j" podUID="9adc9c04-3250-428d-bcda-bfc8ed97077a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 18:22:12.764314 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:12.764276    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gslgl" podUID="8dd1fe07-7ba4-484d-9beb-79c5751cd739" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 17 18:22:15.530386 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:15.530346    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6255j" podUID="9adc9c04-3250-428d-bcda-bfc8ed97077a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 18:22:16.503093 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:16.503040    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6255j" podUID="9adc9c04-3250-428d-bcda-bfc8ed97077a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 18:22:22.764546 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:22.764505    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gslgl" podUID="8dd1fe07-7ba4-484d-9beb-79c5751cd739" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 17 18:22:26.503270 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:26.503234    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6255j"
Sep 17 18:22:32.764320 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:32.764239    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gslgl" podUID="8dd1fe07-7ba4-484d-9beb-79c5751cd739" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 17 18:22:37.160819 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:37.160766    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6255j"]
Sep 17 18:22:37.161487 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:37.161193    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6255j" podUID="9adc9c04-3250-428d-bcda-bfc8ed97077a" containerName="kserve-container" containerID="cri-o://269d1c3eaebd415b2f55ba415c5b33164f54fdd855b1cb9202c7a7f071e8bd58" gracePeriod=30
Sep 17 18:22:37.161487 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:37.161216    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6255j" podUID="9adc9c04-3250-428d-bcda-bfc8ed97077a" containerName="kube-rbac-proxy" containerID="cri-o://aae77b1553711f8f406c7c1bae812c33b0b6e6a68ca1d6d4d4eec484177a2ad1" gracePeriod=30
Sep 17 18:22:38.014014 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:38.013982    2569 generic.go:356] "Generic (PLEG): container finished" podID="9adc9c04-3250-428d-bcda-bfc8ed97077a" containerID="aae77b1553711f8f406c7c1bae812c33b0b6e6a68ca1d6d4d4eec484177a2ad1" exitCode=2
Sep 17 18:22:38.014198 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:38.014020    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6255j" event={"ID":"9adc9c04-3250-428d-bcda-bfc8ed97077a","Type":"ContainerDied","Data":"aae77b1553711f8f406c7c1bae812c33b0b6e6a68ca1d6d4d4eec484177a2ad1"}
Sep 17 18:22:40.525425 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:40.525384    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6255j" podUID="9adc9c04-3250-428d-bcda-bfc8ed97077a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.41:8643/healthz\": dial tcp 10.133.0.41:8643: connect: connection refused"
Sep 17 18:22:42.004721 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:42.004699    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6255j"
Sep 17 18:22:42.032171 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:42.032141    2569 generic.go:356] "Generic (PLEG): container finished" podID="9adc9c04-3250-428d-bcda-bfc8ed97077a" containerID="269d1c3eaebd415b2f55ba415c5b33164f54fdd855b1cb9202c7a7f071e8bd58" exitCode=0
Sep 17 18:22:42.032301 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:42.032195    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6255j"
Sep 17 18:22:42.032301 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:42.032225    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6255j" event={"ID":"9adc9c04-3250-428d-bcda-bfc8ed97077a","Type":"ContainerDied","Data":"269d1c3eaebd415b2f55ba415c5b33164f54fdd855b1cb9202c7a7f071e8bd58"}
Sep 17 18:22:42.032301 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:42.032295    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6255j" event={"ID":"9adc9c04-3250-428d-bcda-bfc8ed97077a","Type":"ContainerDied","Data":"f9e999dabdd13d1534126cb09e835e2bffb37c8691f710bd8be15dd07f671681"}
Sep 17 18:22:42.032481 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:42.032336    2569 scope.go:117] "RemoveContainer" containerID="aae77b1553711f8f406c7c1bae812c33b0b6e6a68ca1d6d4d4eec484177a2ad1"
Sep 17 18:22:42.040185 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:42.040169    2569 scope.go:117] "RemoveContainer" containerID="269d1c3eaebd415b2f55ba415c5b33164f54fdd855b1cb9202c7a7f071e8bd58"
Sep 17 18:22:42.047226 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:42.047207    2569 scope.go:117] "RemoveContainer" containerID="2ff8911cea5a9352db30dbc7dc25bdf160afe2a690252abf2f9792e8739d937e"
Sep 17 18:22:42.055615 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:42.055597    2569 scope.go:117] "RemoveContainer" containerID="aae77b1553711f8f406c7c1bae812c33b0b6e6a68ca1d6d4d4eec484177a2ad1"
Sep 17 18:22:42.055885 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:22:42.055866    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aae77b1553711f8f406c7c1bae812c33b0b6e6a68ca1d6d4d4eec484177a2ad1\": container with ID starting with aae77b1553711f8f406c7c1bae812c33b0b6e6a68ca1d6d4d4eec484177a2ad1 not found: ID does not exist" containerID="aae77b1553711f8f406c7c1bae812c33b0b6e6a68ca1d6d4d4eec484177a2ad1"
Sep 17 18:22:42.055956 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:42.055892    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aae77b1553711f8f406c7c1bae812c33b0b6e6a68ca1d6d4d4eec484177a2ad1"} err="failed to get container status \"aae77b1553711f8f406c7c1bae812c33b0b6e6a68ca1d6d4d4eec484177a2ad1\": rpc error: code = NotFound desc = could not find container \"aae77b1553711f8f406c7c1bae812c33b0b6e6a68ca1d6d4d4eec484177a2ad1\": container with ID starting with aae77b1553711f8f406c7c1bae812c33b0b6e6a68ca1d6d4d4eec484177a2ad1 not found: ID does not exist"
Sep 17 18:22:42.055956 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:42.055907    2569 scope.go:117] "RemoveContainer" containerID="269d1c3eaebd415b2f55ba415c5b33164f54fdd855b1cb9202c7a7f071e8bd58"
Sep 17 18:22:42.056206 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:22:42.056190    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"269d1c3eaebd415b2f55ba415c5b33164f54fdd855b1cb9202c7a7f071e8bd58\": container with ID starting with 269d1c3eaebd415b2f55ba415c5b33164f54fdd855b1cb9202c7a7f071e8bd58 not found: ID does not exist" containerID="269d1c3eaebd415b2f55ba415c5b33164f54fdd855b1cb9202c7a7f071e8bd58"
Sep 17 18:22:42.056252 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:42.056209    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"269d1c3eaebd415b2f55ba415c5b33164f54fdd855b1cb9202c7a7f071e8bd58"} err="failed to get container status \"269d1c3eaebd415b2f55ba415c5b33164f54fdd855b1cb9202c7a7f071e8bd58\": rpc error: code = NotFound desc = could not find container \"269d1c3eaebd415b2f55ba415c5b33164f54fdd855b1cb9202c7a7f071e8bd58\": container with ID starting with 269d1c3eaebd415b2f55ba415c5b33164f54fdd855b1cb9202c7a7f071e8bd58 not found: ID does not exist"
Sep 17 18:22:42.056252 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:42.056221    2569 scope.go:117] "RemoveContainer" containerID="2ff8911cea5a9352db30dbc7dc25bdf160afe2a690252abf2f9792e8739d937e"
Sep 17 18:22:42.056452 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:22:42.056438    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ff8911cea5a9352db30dbc7dc25bdf160afe2a690252abf2f9792e8739d937e\": container with ID starting with 2ff8911cea5a9352db30dbc7dc25bdf160afe2a690252abf2f9792e8739d937e not found: ID does not exist" containerID="2ff8911cea5a9352db30dbc7dc25bdf160afe2a690252abf2f9792e8739d937e"
Sep 17 18:22:42.056496 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:42.056457    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ff8911cea5a9352db30dbc7dc25bdf160afe2a690252abf2f9792e8739d937e"} err="failed to get container status \"2ff8911cea5a9352db30dbc7dc25bdf160afe2a690252abf2f9792e8739d937e\": rpc error: code = NotFound desc = could not find container \"2ff8911cea5a9352db30dbc7dc25bdf160afe2a690252abf2f9792e8739d937e\": container with ID starting with 2ff8911cea5a9352db30dbc7dc25bdf160afe2a690252abf2f9792e8739d937e not found: ID does not exist"
Sep 17 18:22:42.162542 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:42.162483    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9adc9c04-3250-428d-bcda-bfc8ed97077a-kube-rbac-proxy-sar-config\") pod \"9adc9c04-3250-428d-bcda-bfc8ed97077a\" (UID: \"9adc9c04-3250-428d-bcda-bfc8ed97077a\") "
Sep 17 18:22:42.162542 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:42.162526    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9adc9c04-3250-428d-bcda-bfc8ed97077a-cabundle-cert\") pod \"9adc9c04-3250-428d-bcda-bfc8ed97077a\" (UID: \"9adc9c04-3250-428d-bcda-bfc8ed97077a\") "
Sep 17 18:22:42.162696 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:42.162568    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9adc9c04-3250-428d-bcda-bfc8ed97077a-kserve-provision-location\") pod \"9adc9c04-3250-428d-bcda-bfc8ed97077a\" (UID: \"9adc9c04-3250-428d-bcda-bfc8ed97077a\") "
Sep 17 18:22:42.162696 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:42.162594    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9adc9c04-3250-428d-bcda-bfc8ed97077a-proxy-tls\") pod \"9adc9c04-3250-428d-bcda-bfc8ed97077a\" (UID: \"9adc9c04-3250-428d-bcda-bfc8ed97077a\") "
Sep 17 18:22:42.162696 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:42.162654    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nbrx6\" (UniqueName: \"kubernetes.io/projected/9adc9c04-3250-428d-bcda-bfc8ed97077a-kube-api-access-nbrx6\") pod \"9adc9c04-3250-428d-bcda-bfc8ed97077a\" (UID: \"9adc9c04-3250-428d-bcda-bfc8ed97077a\") "
Sep 17 18:22:42.162944 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:42.162920    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9adc9c04-3250-428d-bcda-bfc8ed97077a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9adc9c04-3250-428d-bcda-bfc8ed97077a" (UID: "9adc9c04-3250-428d-bcda-bfc8ed97077a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:22:42.162990 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:42.162924    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9adc9c04-3250-428d-bcda-bfc8ed97077a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9adc9c04-3250-428d-bcda-bfc8ed97077a" (UID: "9adc9c04-3250-428d-bcda-bfc8ed97077a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:22:42.163265 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:42.163237    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9adc9c04-3250-428d-bcda-bfc8ed97077a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9adc9c04-3250-428d-bcda-bfc8ed97077a" (UID: "9adc9c04-3250-428d-bcda-bfc8ed97077a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:22:42.164606 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:42.164583    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9adc9c04-3250-428d-bcda-bfc8ed97077a-kube-api-access-nbrx6" (OuterVolumeSpecName: "kube-api-access-nbrx6") pod "9adc9c04-3250-428d-bcda-bfc8ed97077a" (UID: "9adc9c04-3250-428d-bcda-bfc8ed97077a"). InnerVolumeSpecName "kube-api-access-nbrx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:22:42.164724 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:42.164701    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9adc9c04-3250-428d-bcda-bfc8ed97077a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9adc9c04-3250-428d-bcda-bfc8ed97077a" (UID: "9adc9c04-3250-428d-bcda-bfc8ed97077a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:22:42.264250 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:42.264222    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9adc9c04-3250-428d-bcda-bfc8ed97077a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:22:42.264250 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:42.264247    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9adc9c04-3250-428d-bcda-bfc8ed97077a-cabundle-cert\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:22:42.264399 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:42.264256    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9adc9c04-3250-428d-bcda-bfc8ed97077a-kserve-provision-location\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:22:42.264399 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:42.264278    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9adc9c04-3250-428d-bcda-bfc8ed97077a-proxy-tls\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:22:42.264399 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:42.264289    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbrx6\" (UniqueName: \"kubernetes.io/projected/9adc9c04-3250-428d-bcda-bfc8ed97077a-kube-api-access-nbrx6\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:22:42.354036 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:42.354009    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6255j"]
Sep 17 18:22:42.357405 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:42.357383    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6255j"]
Sep 17 18:22:42.369982 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:42.369957    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qx456"]
Sep 17 18:22:42.370259 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:42.370247    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9adc9c04-3250-428d-bcda-bfc8ed97077a" containerName="kserve-container"
Sep 17 18:22:42.370310 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:42.370260    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="9adc9c04-3250-428d-bcda-bfc8ed97077a" containerName="kserve-container"
Sep 17 18:22:42.370310 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:42.370269    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9adc9c04-3250-428d-bcda-bfc8ed97077a" containerName="kube-rbac-proxy"
Sep 17 18:22:42.370310 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:42.370275    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="9adc9c04-3250-428d-bcda-bfc8ed97077a" containerName="kube-rbac-proxy"
Sep 17 18:22:42.370310 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:42.370284    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9adc9c04-3250-428d-bcda-bfc8ed97077a" containerName="storage-initializer"
Sep 17 18:22:42.370310 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:42.370290    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="9adc9c04-3250-428d-bcda-bfc8ed97077a" containerName="storage-initializer"
Sep 17 18:22:42.370466 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:42.370344    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="9adc9c04-3250-428d-bcda-bfc8ed97077a" containerName="kserve-container"
Sep 17 18:22:42.370466 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:42.370354    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="9adc9c04-3250-428d-bcda-bfc8ed97077a" containerName="kube-rbac-proxy"
Sep 17 18:22:42.374697 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:42.374682    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qx456"
Sep 17 18:22:42.377217 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:42.377200    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-predictor-serving-cert\""
Sep 17 18:22:42.377334 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:42.377261    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\""
Sep 17 18:22:42.385314 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:42.385296    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qx456"]
Sep 17 18:22:42.465876 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:42.465854    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2pcb\" (UniqueName: \"kubernetes.io/projected/71c71769-a0f6-4c7d-8bf7-a41c4da6b022-kube-api-access-v2pcb\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-qx456\" (UID: \"71c71769-a0f6-4c7d-8bf7-a41c4da6b022\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qx456"
Sep 17 18:22:42.465958 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:42.465894    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71c71769-a0f6-4c7d-8bf7-a41c4da6b022-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-qx456\" (UID: \"71c71769-a0f6-4c7d-8bf7-a41c4da6b022\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qx456"
Sep 17 18:22:42.465958 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:42.465918    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71c71769-a0f6-4c7d-8bf7-a41c4da6b022-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-qx456\" (UID: \"71c71769-a0f6-4c7d-8bf7-a41c4da6b022\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qx456"
Sep 17 18:22:42.466037 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:42.465998    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71c71769-a0f6-4c7d-8bf7-a41c4da6b022-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-qx456\" (UID: \"71c71769-a0f6-4c7d-8bf7-a41c4da6b022\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qx456"
Sep 17 18:22:42.466097 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:42.466038    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71c71769-a0f6-4c7d-8bf7-a41c4da6b022-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-qx456\" (UID: \"71c71769-a0f6-4c7d-8bf7-a41c4da6b022\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qx456"
Sep 17 18:22:42.567273 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:42.567248    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71c71769-a0f6-4c7d-8bf7-a41c4da6b022-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-qx456\" (UID: \"71c71769-a0f6-4c7d-8bf7-a41c4da6b022\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qx456"
Sep 17 18:22:42.567377 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:42.567296    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71c71769-a0f6-4c7d-8bf7-a41c4da6b022-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-qx456\" (UID: \"71c71769-a0f6-4c7d-8bf7-a41c4da6b022\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qx456"
Sep 17 18:22:42.567377 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:42.567336    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71c71769-a0f6-4c7d-8bf7-a41c4da6b022-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-qx456\" (UID: \"71c71769-a0f6-4c7d-8bf7-a41c4da6b022\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qx456"
Sep 17 18:22:42.567509 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:42.567392    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2pcb\" (UniqueName: \"kubernetes.io/projected/71c71769-a0f6-4c7d-8bf7-a41c4da6b022-kube-api-access-v2pcb\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-qx456\" (UID: \"71c71769-a0f6-4c7d-8bf7-a41c4da6b022\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qx456"
Sep 17 18:22:42.567509 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:42.567442    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71c71769-a0f6-4c7d-8bf7-a41c4da6b022-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-qx456\" (UID: \"71c71769-a0f6-4c7d-8bf7-a41c4da6b022\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qx456"
Sep 17 18:22:42.567656 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:42.567632    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71c71769-a0f6-4c7d-8bf7-a41c4da6b022-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-qx456\" (UID: \"71c71769-a0f6-4c7d-8bf7-a41c4da6b022\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qx456"
Sep 17 18:22:42.567959 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:42.567935    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71c71769-a0f6-4c7d-8bf7-a41c4da6b022-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-qx456\" (UID: \"71c71769-a0f6-4c7d-8bf7-a41c4da6b022\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qx456"
Sep 17 18:22:42.568034 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:42.567990    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71c71769-a0f6-4c7d-8bf7-a41c4da6b022-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-qx456\" (UID: \"71c71769-a0f6-4c7d-8bf7-a41c4da6b022\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qx456"
Sep 17 18:22:42.569959 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:42.569939    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71c71769-a0f6-4c7d-8bf7-a41c4da6b022-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-qx456\" (UID: \"71c71769-a0f6-4c7d-8bf7-a41c4da6b022\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qx456"
Sep 17 18:22:42.576502 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:42.576483    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2pcb\" (UniqueName: \"kubernetes.io/projected/71c71769-a0f6-4c7d-8bf7-a41c4da6b022-kube-api-access-v2pcb\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-qx456\" (UID: \"71c71769-a0f6-4c7d-8bf7-a41c4da6b022\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qx456"
Sep 17 18:22:42.685509 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:42.685469    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qx456"
Sep 17 18:22:42.764552 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:42.764531    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gslgl"
Sep 17 18:22:42.960711 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:42.960687    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qx456"]
Sep 17 18:22:42.962337 ip-10-0-131-14 kubenswrapper[2569]: W0917 18:22:42.962312    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71c71769_a0f6_4c7d_8bf7_a41c4da6b022.slice/crio-1da8ae2d2b43f2c3a5cbf86cd7cd8a5ee1aa35eb691398bd02b509dbdadc469d WatchSource:0}: Error finding container 1da8ae2d2b43f2c3a5cbf86cd7cd8a5ee1aa35eb691398bd02b509dbdadc469d: Status 404 returned error can't find the container with id 1da8ae2d2b43f2c3a5cbf86cd7cd8a5ee1aa35eb691398bd02b509dbdadc469d
Sep 17 18:22:43.038205 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:43.038175    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qx456" event={"ID":"71c71769-a0f6-4c7d-8bf7-a41c4da6b022","Type":"ContainerStarted","Data":"cf2fb74c35c05b14b097536b1e0483af83838bf13ad1f29378d9c856f7c7179f"}
Sep 17 18:22:43.038588 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:43.038233    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qx456" event={"ID":"71c71769-a0f6-4c7d-8bf7-a41c4da6b022","Type":"ContainerStarted","Data":"1da8ae2d2b43f2c3a5cbf86cd7cd8a5ee1aa35eb691398bd02b509dbdadc469d"}
Sep 17 18:22:43.508088 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:43.508040    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9adc9c04-3250-428d-bcda-bfc8ed97077a" path="/var/lib/kubelet/pods/9adc9c04-3250-428d-bcda-bfc8ed97077a/volumes"
Sep 17 18:22:47.061916 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:47.061844    2569 generic.go:356] "Generic (PLEG): container finished" podID="71c71769-a0f6-4c7d-8bf7-a41c4da6b022" containerID="cf2fb74c35c05b14b097536b1e0483af83838bf13ad1f29378d9c856f7c7179f" exitCode=0
Sep 17 18:22:47.062284 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:47.061919    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qx456" event={"ID":"71c71769-a0f6-4c7d-8bf7-a41c4da6b022","Type":"ContainerDied","Data":"cf2fb74c35c05b14b097536b1e0483af83838bf13ad1f29378d9c856f7c7179f"}
Sep 17 18:22:48.070114 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:48.070068    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qx456" event={"ID":"71c71769-a0f6-4c7d-8bf7-a41c4da6b022","Type":"ContainerStarted","Data":"73e0302b95e04d02b910dea2c9dfe158c37d6be187e5df45d536b4d6665d707c"}
Sep 17 18:22:48.070507 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:48.070144    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qx456" event={"ID":"71c71769-a0f6-4c7d-8bf7-a41c4da6b022","Type":"ContainerStarted","Data":"6861e9c5dccfb4dadd20b291bcd846e1db2252f795e70b70eed3dcbeef4b8e4a"}
Sep 17 18:22:48.070507 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:48.070450    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qx456"
Sep 17 18:22:48.070599 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:48.070580    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qx456"
Sep 17 18:22:48.071995 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:48.071966    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qx456" podUID="71c71769-a0f6-4c7d-8bf7-a41c4da6b022" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 18:22:48.090513 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:48.090473    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qx456" podStartSLOduration=2.333685932 podStartE2EDuration="6.090460006s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.756774074s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:22:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:22:48.089688196 +0000 UTC m=+2520.593690832" watchObservedRunningTime="2026-09-17 18:22:48.090460006 +0000 UTC m=+2520.594462631"
Sep 17 18:22:49.074652 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:49.074611    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qx456" podUID="71c71769-a0f6-4c7d-8bf7-a41c4da6b022" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 18:22:51.053788 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:51.053756    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gslgl"]
Sep 17 18:22:51.054229 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:51.054051    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gslgl" podUID="8dd1fe07-7ba4-484d-9beb-79c5751cd739" containerName="kserve-container" containerID="cri-o://3d6b21cb76bbd9ed499cb82fefcc0d57edffe2b2a2c786069ad326c79ab6f72e" gracePeriod=30
Sep 17 18:22:51.054229 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:51.054111    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gslgl" podUID="8dd1fe07-7ba4-484d-9beb-79c5751cd739" containerName="kube-rbac-proxy" containerID="cri-o://0757b97bc4ded2cecefbf8611218d8297af442129fbc35c2933352ed76017ed8" gracePeriod=30
Sep 17 18:22:52.088314 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:52.088278    2569 generic.go:356] "Generic (PLEG): container finished" podID="8dd1fe07-7ba4-484d-9beb-79c5751cd739" containerID="0757b97bc4ded2cecefbf8611218d8297af442129fbc35c2933352ed76017ed8" exitCode=2
Sep 17 18:22:52.088676 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:52.088350    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gslgl" event={"ID":"8dd1fe07-7ba4-484d-9beb-79c5751cd739","Type":"ContainerDied","Data":"0757b97bc4ded2cecefbf8611218d8297af442129fbc35c2933352ed76017ed8"}
Sep 17 18:22:52.758333 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:52.758296    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gslgl" podUID="8dd1fe07-7ba4-484d-9beb-79c5751cd739" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.42:8643/healthz\": dial tcp 10.133.0.42:8643: connect: connection refused"
Sep 17 18:22:52.763681 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:52.763656    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gslgl" podUID="8dd1fe07-7ba4-484d-9beb-79c5751cd739" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 17 18:22:54.079241 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:54.079216    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qx456"
Sep 17 18:22:54.079790 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:54.079765    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qx456" podUID="71c71769-a0f6-4c7d-8bf7-a41c4da6b022" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 18:22:54.507441 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:54.507420    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gslgl"
Sep 17 18:22:54.656158 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:54.656099    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8dd1fe07-7ba4-484d-9beb-79c5751cd739-kube-rbac-proxy-sar-config\") pod \"8dd1fe07-7ba4-484d-9beb-79c5751cd739\" (UID: \"8dd1fe07-7ba4-484d-9beb-79c5751cd739\") "
Sep 17 18:22:54.656158 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:54.656133    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9z5hj\" (UniqueName: \"kubernetes.io/projected/8dd1fe07-7ba4-484d-9beb-79c5751cd739-kube-api-access-9z5hj\") pod \"8dd1fe07-7ba4-484d-9beb-79c5751cd739\" (UID: \"8dd1fe07-7ba4-484d-9beb-79c5751cd739\") "
Sep 17 18:22:54.656341 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:54.656173    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8dd1fe07-7ba4-484d-9beb-79c5751cd739-proxy-tls\") pod \"8dd1fe07-7ba4-484d-9beb-79c5751cd739\" (UID: \"8dd1fe07-7ba4-484d-9beb-79c5751cd739\") "
Sep 17 18:22:54.656341 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:54.656216    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8dd1fe07-7ba4-484d-9beb-79c5751cd739-kserve-provision-location\") pod \"8dd1fe07-7ba4-484d-9beb-79c5751cd739\" (UID: \"8dd1fe07-7ba4-484d-9beb-79c5751cd739\") "
Sep 17 18:22:54.656341 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:54.656267    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8dd1fe07-7ba4-484d-9beb-79c5751cd739-cabundle-cert\") pod \"8dd1fe07-7ba4-484d-9beb-79c5751cd739\" (UID: \"8dd1fe07-7ba4-484d-9beb-79c5751cd739\") "
Sep 17 18:22:54.656551 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:54.656532    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dd1fe07-7ba4-484d-9beb-79c5751cd739-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8dd1fe07-7ba4-484d-9beb-79c5751cd739" (UID: "8dd1fe07-7ba4-484d-9beb-79c5751cd739"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:22:54.656605 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:54.656547    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dd1fe07-7ba4-484d-9beb-79c5751cd739-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8dd1fe07-7ba4-484d-9beb-79c5751cd739" (UID: "8dd1fe07-7ba4-484d-9beb-79c5751cd739"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:22:54.656643 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:54.656603    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dd1fe07-7ba4-484d-9beb-79c5751cd739-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8dd1fe07-7ba4-484d-9beb-79c5751cd739" (UID: "8dd1fe07-7ba4-484d-9beb-79c5751cd739"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:22:54.657974 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:54.657955    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dd1fe07-7ba4-484d-9beb-79c5751cd739-kube-api-access-9z5hj" (OuterVolumeSpecName: "kube-api-access-9z5hj") pod "8dd1fe07-7ba4-484d-9beb-79c5751cd739" (UID: "8dd1fe07-7ba4-484d-9beb-79c5751cd739"). InnerVolumeSpecName "kube-api-access-9z5hj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:22:54.658072 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:54.658051    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dd1fe07-7ba4-484d-9beb-79c5751cd739-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8dd1fe07-7ba4-484d-9beb-79c5751cd739" (UID: "8dd1fe07-7ba4-484d-9beb-79c5751cd739"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:22:54.757364 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:54.757341    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8dd1fe07-7ba4-484d-9beb-79c5751cd739-proxy-tls\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:22:54.757364 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:54.757361    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8dd1fe07-7ba4-484d-9beb-79c5751cd739-kserve-provision-location\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:22:54.757499 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:54.757370    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8dd1fe07-7ba4-484d-9beb-79c5751cd739-cabundle-cert\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:22:54.757499 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:54.757378    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8dd1fe07-7ba4-484d-9beb-79c5751cd739-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:22:54.757499 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:54.757388    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9z5hj\" (UniqueName: \"kubernetes.io/projected/8dd1fe07-7ba4-484d-9beb-79c5751cd739-kube-api-access-9z5hj\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:22:55.102906 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:55.102877    2569 generic.go:356] "Generic (PLEG): container finished" podID="8dd1fe07-7ba4-484d-9beb-79c5751cd739" containerID="3d6b21cb76bbd9ed499cb82fefcc0d57edffe2b2a2c786069ad326c79ab6f72e" exitCode=0
Sep 17 18:22:55.103244 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:55.102930    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gslgl" event={"ID":"8dd1fe07-7ba4-484d-9beb-79c5751cd739","Type":"ContainerDied","Data":"3d6b21cb76bbd9ed499cb82fefcc0d57edffe2b2a2c786069ad326c79ab6f72e"}
Sep 17 18:22:55.103244 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:55.102976    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gslgl" event={"ID":"8dd1fe07-7ba4-484d-9beb-79c5751cd739","Type":"ContainerDied","Data":"67ac4a26757c5b436f459a3f24e4d1514ccc9c721d06754f5415ff901279447b"}
Sep 17 18:22:55.103244 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:55.102930    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gslgl"
Sep 17 18:22:55.103244 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:55.103002    2569 scope.go:117] "RemoveContainer" containerID="0757b97bc4ded2cecefbf8611218d8297af442129fbc35c2933352ed76017ed8"
Sep 17 18:22:55.111253 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:55.111236    2569 scope.go:117] "RemoveContainer" containerID="3d6b21cb76bbd9ed499cb82fefcc0d57edffe2b2a2c786069ad326c79ab6f72e"
Sep 17 18:22:55.118524 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:55.118508    2569 scope.go:117] "RemoveContainer" containerID="286e2db5f6452ac05da7b8a08dcc4bed9d3b5bf09de1c2d2a06a7d13c17ed2d0"
Sep 17 18:22:55.123952 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:55.123927    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gslgl"]
Sep 17 18:22:55.125729 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:55.125714    2569 scope.go:117] "RemoveContainer" containerID="0757b97bc4ded2cecefbf8611218d8297af442129fbc35c2933352ed76017ed8"
Sep 17 18:22:55.126012 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:22:55.125993    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0757b97bc4ded2cecefbf8611218d8297af442129fbc35c2933352ed76017ed8\": container with ID starting with 0757b97bc4ded2cecefbf8611218d8297af442129fbc35c2933352ed76017ed8 not found: ID does not exist" containerID="0757b97bc4ded2cecefbf8611218d8297af442129fbc35c2933352ed76017ed8"
Sep 17 18:22:55.126125 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:55.126024    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0757b97bc4ded2cecefbf8611218d8297af442129fbc35c2933352ed76017ed8"} err="failed to get container status \"0757b97bc4ded2cecefbf8611218d8297af442129fbc35c2933352ed76017ed8\": rpc error: code = NotFound desc = could not find container \"0757b97bc4ded2cecefbf8611218d8297af442129fbc35c2933352ed76017ed8\": container with ID starting with 0757b97bc4ded2cecefbf8611218d8297af442129fbc35c2933352ed76017ed8 not found: ID does not exist"
Sep 17 18:22:55.126125 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:55.126051    2569 scope.go:117] "RemoveContainer" containerID="3d6b21cb76bbd9ed499cb82fefcc0d57edffe2b2a2c786069ad326c79ab6f72e"
Sep 17 18:22:55.126328 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:22:55.126311    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d6b21cb76bbd9ed499cb82fefcc0d57edffe2b2a2c786069ad326c79ab6f72e\": container with ID starting with 3d6b21cb76bbd9ed499cb82fefcc0d57edffe2b2a2c786069ad326c79ab6f72e not found: ID does not exist" containerID="3d6b21cb76bbd9ed499cb82fefcc0d57edffe2b2a2c786069ad326c79ab6f72e"
Sep 17 18:22:55.126370 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:55.126332    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d6b21cb76bbd9ed499cb82fefcc0d57edffe2b2a2c786069ad326c79ab6f72e"} err="failed to get container status \"3d6b21cb76bbd9ed499cb82fefcc0d57edffe2b2a2c786069ad326c79ab6f72e\": rpc error: code = NotFound desc = could not find container \"3d6b21cb76bbd9ed499cb82fefcc0d57edffe2b2a2c786069ad326c79ab6f72e\": container with ID starting with 3d6b21cb76bbd9ed499cb82fefcc0d57edffe2b2a2c786069ad326c79ab6f72e not found: ID does not exist"
Sep 17 18:22:55.126370 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:55.126347    2569 scope.go:117] "RemoveContainer" containerID="286e2db5f6452ac05da7b8a08dcc4bed9d3b5bf09de1c2d2a06a7d13c17ed2d0"
Sep 17 18:22:55.126628 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:22:55.126610    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"286e2db5f6452ac05da7b8a08dcc4bed9d3b5bf09de1c2d2a06a7d13c17ed2d0\": container with ID starting with 286e2db5f6452ac05da7b8a08dcc4bed9d3b5bf09de1c2d2a06a7d13c17ed2d0 not found: ID does not exist" containerID="286e2db5f6452ac05da7b8a08dcc4bed9d3b5bf09de1c2d2a06a7d13c17ed2d0"
Sep 17 18:22:55.126686 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:55.126632    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"286e2db5f6452ac05da7b8a08dcc4bed9d3b5bf09de1c2d2a06a7d13c17ed2d0"} err="failed to get container status \"286e2db5f6452ac05da7b8a08dcc4bed9d3b5bf09de1c2d2a06a7d13c17ed2d0\": rpc error: code = NotFound desc = could not find container \"286e2db5f6452ac05da7b8a08dcc4bed9d3b5bf09de1c2d2a06a7d13c17ed2d0\": container with ID starting with 286e2db5f6452ac05da7b8a08dcc4bed9d3b5bf09de1c2d2a06a7d13c17ed2d0 not found: ID does not exist"
Sep 17 18:22:55.129601 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:55.129578    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-gslgl"]
Sep 17 18:22:55.507702 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:55.507675    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8dd1fe07-7ba4-484d-9beb-79c5751cd739" path="/var/lib/kubelet/pods/8dd1fe07-7ba4-484d-9beb-79c5751cd739/volumes"
Sep 17 18:22:56.237876 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:56.237843    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zj8jr"]
Sep 17 18:22:56.238270 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:56.238141    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8dd1fe07-7ba4-484d-9beb-79c5751cd739" containerName="kserve-container"
Sep 17 18:22:56.238270 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:56.238153    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8dd1fe07-7ba4-484d-9beb-79c5751cd739" containerName="kserve-container"
Sep 17 18:22:56.238270 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:56.238176    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8dd1fe07-7ba4-484d-9beb-79c5751cd739" containerName="storage-initializer"
Sep 17 18:22:56.238270 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:56.238181    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8dd1fe07-7ba4-484d-9beb-79c5751cd739" containerName="storage-initializer"
Sep 17 18:22:56.238270 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:56.238189    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8dd1fe07-7ba4-484d-9beb-79c5751cd739" containerName="kube-rbac-proxy"
Sep 17 18:22:56.238270 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:56.238194    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8dd1fe07-7ba4-484d-9beb-79c5751cd739" containerName="kube-rbac-proxy"
Sep 17 18:22:56.238270 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:56.238237    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="8dd1fe07-7ba4-484d-9beb-79c5751cd739" containerName="kserve-container"
Sep 17 18:22:56.238270 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:56.238246    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="8dd1fe07-7ba4-484d-9beb-79c5751cd739" containerName="kube-rbac-proxy"
Sep 17 18:22:56.242801 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:56.242785    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zj8jr"
Sep 17 18:22:56.245230 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:56.245210    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-predictor-serving-cert\""
Sep 17 18:22:56.245340 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:56.245283    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 17 18:22:56.254724 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:56.254702    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zj8jr"]
Sep 17 18:22:56.369903 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:56.369878    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ebd5bf44-b1be-43d3-bf96-25df4b31ddea-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-zj8jr\" (UID: \"ebd5bf44-b1be-43d3-bf96-25df4b31ddea\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zj8jr"
Sep 17 18:22:56.370006 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:56.369909    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ebd5bf44-b1be-43d3-bf96-25df4b31ddea-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-zj8jr\" (UID: \"ebd5bf44-b1be-43d3-bf96-25df4b31ddea\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zj8jr"
Sep 17 18:22:56.370006 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:56.369933    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebd5bf44-b1be-43d3-bf96-25df4b31ddea-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-zj8jr\" (UID: \"ebd5bf44-b1be-43d3-bf96-25df4b31ddea\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zj8jr"
Sep 17 18:22:56.370006 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:56.369953    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb4zz\" (UniqueName: \"kubernetes.io/projected/ebd5bf44-b1be-43d3-bf96-25df4b31ddea-kube-api-access-wb4zz\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-zj8jr\" (UID: \"ebd5bf44-b1be-43d3-bf96-25df4b31ddea\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zj8jr"
Sep 17 18:22:56.370149 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:56.370009    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ebd5bf44-b1be-43d3-bf96-25df4b31ddea-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-zj8jr\" (UID: \"ebd5bf44-b1be-43d3-bf96-25df4b31ddea\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zj8jr"
Sep 17 18:22:56.470966 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:56.470947    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ebd5bf44-b1be-43d3-bf96-25df4b31ddea-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-zj8jr\" (UID: \"ebd5bf44-b1be-43d3-bf96-25df4b31ddea\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zj8jr"
Sep 17 18:22:56.471054 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:56.470996    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ebd5bf44-b1be-43d3-bf96-25df4b31ddea-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-zj8jr\" (UID: \"ebd5bf44-b1be-43d3-bf96-25df4b31ddea\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zj8jr"
Sep 17 18:22:56.471054 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:56.471016    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ebd5bf44-b1be-43d3-bf96-25df4b31ddea-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-zj8jr\" (UID: \"ebd5bf44-b1be-43d3-bf96-25df4b31ddea\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zj8jr"
Sep 17 18:22:56.471054 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:56.471043    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebd5bf44-b1be-43d3-bf96-25df4b31ddea-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-zj8jr\" (UID: \"ebd5bf44-b1be-43d3-bf96-25df4b31ddea\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zj8jr"
Sep 17 18:22:56.471228 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:56.471063    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb4zz\" (UniqueName: \"kubernetes.io/projected/ebd5bf44-b1be-43d3-bf96-25df4b31ddea-kube-api-access-wb4zz\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-zj8jr\" (UID: \"ebd5bf44-b1be-43d3-bf96-25df4b31ddea\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zj8jr"
Sep 17 18:22:56.471489 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:56.471467    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ebd5bf44-b1be-43d3-bf96-25df4b31ddea-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-zj8jr\" (UID: \"ebd5bf44-b1be-43d3-bf96-25df4b31ddea\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zj8jr"
Sep 17 18:22:56.471758 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:56.471737    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ebd5bf44-b1be-43d3-bf96-25df4b31ddea-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-zj8jr\" (UID: \"ebd5bf44-b1be-43d3-bf96-25df4b31ddea\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zj8jr"
Sep 17 18:22:56.471807 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:56.471737    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ebd5bf44-b1be-43d3-bf96-25df4b31ddea-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-zj8jr\" (UID: \"ebd5bf44-b1be-43d3-bf96-25df4b31ddea\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zj8jr"
Sep 17 18:22:56.473543 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:56.473523    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebd5bf44-b1be-43d3-bf96-25df4b31ddea-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-zj8jr\" (UID: \"ebd5bf44-b1be-43d3-bf96-25df4b31ddea\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zj8jr"
Sep 17 18:22:56.479224 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:56.479202    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb4zz\" (UniqueName: \"kubernetes.io/projected/ebd5bf44-b1be-43d3-bf96-25df4b31ddea-kube-api-access-wb4zz\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-zj8jr\" (UID: \"ebd5bf44-b1be-43d3-bf96-25df4b31ddea\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zj8jr"
Sep 17 18:22:56.555594 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:56.555544    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zj8jr"
Sep 17 18:22:56.824412 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:56.824388    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zj8jr"]
Sep 17 18:22:56.826103 ip-10-0-131-14 kubenswrapper[2569]: W0917 18:22:56.826059    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebd5bf44_b1be_43d3_bf96_25df4b31ddea.slice/crio-91a4d1b908ed0a981c1798c75112948a22d75ded2c254b72a887baca964cc599 WatchSource:0}: Error finding container 91a4d1b908ed0a981c1798c75112948a22d75ded2c254b72a887baca964cc599: Status 404 returned error can't find the container with id 91a4d1b908ed0a981c1798c75112948a22d75ded2c254b72a887baca964cc599
Sep 17 18:22:57.114197 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:57.114115    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zj8jr" event={"ID":"ebd5bf44-b1be-43d3-bf96-25df4b31ddea","Type":"ContainerStarted","Data":"9ea2a60f443053cbd4f339b8dd980120be204d49c34e1f847a5e754979a7e622"}
Sep 17 18:22:57.114197 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:22:57.114181    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zj8jr" event={"ID":"ebd5bf44-b1be-43d3-bf96-25df4b31ddea","Type":"ContainerStarted","Data":"91a4d1b908ed0a981c1798c75112948a22d75ded2c254b72a887baca964cc599"}
Sep 17 18:23:01.134868 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:23:01.134790    2569 generic.go:356] "Generic (PLEG): container finished" podID="ebd5bf44-b1be-43d3-bf96-25df4b31ddea" containerID="9ea2a60f443053cbd4f339b8dd980120be204d49c34e1f847a5e754979a7e622" exitCode=0
Sep 17 18:23:01.134868 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:23:01.134858    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zj8jr" event={"ID":"ebd5bf44-b1be-43d3-bf96-25df4b31ddea","Type":"ContainerDied","Data":"9ea2a60f443053cbd4f339b8dd980120be204d49c34e1f847a5e754979a7e622"}
Sep 17 18:23:02.142453 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:23:02.142416    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zj8jr" event={"ID":"ebd5bf44-b1be-43d3-bf96-25df4b31ddea","Type":"ContainerStarted","Data":"73e3ee87cb1671fe3cb15b512fab521f7993c96104402979b0b7b09809dfcc42"}
Sep 17 18:23:02.142882 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:23:02.142493    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zj8jr" event={"ID":"ebd5bf44-b1be-43d3-bf96-25df4b31ddea","Type":"ContainerStarted","Data":"7a70af6c74e0fd99c2724bec3938f83298c1dc4af111d1ec4efca050d42d0ae3"}
Sep 17 18:23:02.142882 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:23:02.142718    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zj8jr"
Sep 17 18:23:02.165124 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:23:02.165066    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zj8jr" podStartSLOduration=2.300311165 podStartE2EDuration="6.165054573s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.864743408s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:22:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:23:02.163483194 +0000 UTC m=+2534.667485820" watchObservedRunningTime="2026-09-17 18:23:02.165054573 +0000 UTC m=+2534.669057192"
Sep 17 18:23:03.148372 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:23:03.148334    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zj8jr"
Sep 17 18:23:04.080436 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:23:04.080401    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qx456" podUID="71c71769-a0f6-4c7d-8bf7-a41c4da6b022" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 18:23:09.156877 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:23:09.156851    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zj8jr"
Sep 17 18:23:14.080037 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:23:14.079995    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qx456" podUID="71c71769-a0f6-4c7d-8bf7-a41c4da6b022" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 18:23:24.080098 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:23:24.080045    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qx456" podUID="71c71769-a0f6-4c7d-8bf7-a41c4da6b022" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 18:23:34.080659 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:23:34.080621    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qx456" podUID="71c71769-a0f6-4c7d-8bf7-a41c4da6b022" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 18:23:39.240233 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:23:39.240187    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zj8jr" podUID="ebd5bf44-b1be-43d3-bf96-25df4b31ddea" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 17 18:23:44.080413 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:23:44.080374    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qx456" podUID="71c71769-a0f6-4c7d-8bf7-a41c4da6b022" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 18:23:49.160006 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:23:49.159977    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zj8jr"
Sep 17 18:23:54.082064 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:23:54.082025    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qx456" podUID="71c71769-a0f6-4c7d-8bf7-a41c4da6b022" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 18:23:56.315816 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:23:56.315779    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zj8jr"]
Sep 17 18:23:56.316375 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:23:56.316061    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zj8jr" podUID="ebd5bf44-b1be-43d3-bf96-25df4b31ddea" containerName="kserve-container" containerID="cri-o://7a70af6c74e0fd99c2724bec3938f83298c1dc4af111d1ec4efca050d42d0ae3" gracePeriod=30
Sep 17 18:23:56.316375 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:23:56.316181    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zj8jr" podUID="ebd5bf44-b1be-43d3-bf96-25df4b31ddea" containerName="kube-rbac-proxy" containerID="cri-o://73e3ee87cb1671fe3cb15b512fab521f7993c96104402979b0b7b09809dfcc42" gracePeriod=30
Sep 17 18:23:57.394790 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:23:57.394757    2569 generic.go:356] "Generic (PLEG): container finished" podID="ebd5bf44-b1be-43d3-bf96-25df4b31ddea" containerID="73e3ee87cb1671fe3cb15b512fab521f7993c96104402979b0b7b09809dfcc42" exitCode=2
Sep 17 18:23:57.395161 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:23:57.394834    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zj8jr" event={"ID":"ebd5bf44-b1be-43d3-bf96-25df4b31ddea","Type":"ContainerDied","Data":"73e3ee87cb1671fe3cb15b512fab521f7993c96104402979b0b7b09809dfcc42"}
Sep 17 18:23:59.152756 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:23:59.152717    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zj8jr" podUID="ebd5bf44-b1be-43d3-bf96-25df4b31ddea" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.44:8643/healthz\": dial tcp 10.133.0.44:8643: connect: connection refused"
Sep 17 18:23:59.158156 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:23:59.158130    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zj8jr" podUID="ebd5bf44-b1be-43d3-bf96-25df4b31ddea" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.44:8080/v2/models/isvc-xgboost-v2-runtime/ready\": dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 17 18:24:01.503483 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:01.503441    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qx456" podUID="71c71769-a0f6-4c7d-8bf7-a41c4da6b022" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 18:24:01.514811 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:01.514786    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-fw7lf"]
Sep 17 18:24:01.518329 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:01.518310    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-fw7lf"
Sep 17 18:24:01.520925 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:01.520905    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-predictor-serving-cert\""
Sep 17 18:24:01.521013 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:01.520904    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-kube-rbac-proxy-sar-config\""
Sep 17 18:24:01.528453 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:01.528431    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-fw7lf"]
Sep 17 18:24:01.627636 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:01.627602    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dcbf3e71-59b1-4314-b69f-a71312e253a1-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-fw7lf\" (UID: \"dcbf3e71-59b1-4314-b69f-a71312e253a1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-fw7lf"
Sep 17 18:24:01.627809 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:01.627735    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dcbf3e71-59b1-4314-b69f-a71312e253a1-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-fw7lf\" (UID: \"dcbf3e71-59b1-4314-b69f-a71312e253a1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-fw7lf"
Sep 17 18:24:01.627809 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:01.627773    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p768f\" (UniqueName: \"kubernetes.io/projected/dcbf3e71-59b1-4314-b69f-a71312e253a1-kube-api-access-p768f\") pod \"isvc-xgboost-v2-predictor-fcc668bc-fw7lf\" (UID: \"dcbf3e71-59b1-4314-b69f-a71312e253a1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-fw7lf"
Sep 17 18:24:01.627895 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:01.627818    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dcbf3e71-59b1-4314-b69f-a71312e253a1-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-fw7lf\" (UID: \"dcbf3e71-59b1-4314-b69f-a71312e253a1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-fw7lf"
Sep 17 18:24:01.627895 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:01.627863    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dcbf3e71-59b1-4314-b69f-a71312e253a1-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-fw7lf\" (UID: \"dcbf3e71-59b1-4314-b69f-a71312e253a1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-fw7lf"
Sep 17 18:24:01.729333 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:01.729303    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dcbf3e71-59b1-4314-b69f-a71312e253a1-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-fw7lf\" (UID: \"dcbf3e71-59b1-4314-b69f-a71312e253a1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-fw7lf"
Sep 17 18:24:01.729483 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:01.729409    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dcbf3e71-59b1-4314-b69f-a71312e253a1-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-fw7lf\" (UID: \"dcbf3e71-59b1-4314-b69f-a71312e253a1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-fw7lf"
Sep 17 18:24:01.729483 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:01.729441    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p768f\" (UniqueName: \"kubernetes.io/projected/dcbf3e71-59b1-4314-b69f-a71312e253a1-kube-api-access-p768f\") pod \"isvc-xgboost-v2-predictor-fcc668bc-fw7lf\" (UID: \"dcbf3e71-59b1-4314-b69f-a71312e253a1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-fw7lf"
Sep 17 18:24:01.729483 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:01.729465    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dcbf3e71-59b1-4314-b69f-a71312e253a1-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-fw7lf\" (UID: \"dcbf3e71-59b1-4314-b69f-a71312e253a1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-fw7lf"
Sep 17 18:24:01.729658 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:01.729487    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dcbf3e71-59b1-4314-b69f-a71312e253a1-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-fw7lf\" (UID: \"dcbf3e71-59b1-4314-b69f-a71312e253a1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-fw7lf"
Sep 17 18:24:01.729935 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:01.729908    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dcbf3e71-59b1-4314-b69f-a71312e253a1-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-fw7lf\" (UID: \"dcbf3e71-59b1-4314-b69f-a71312e253a1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-fw7lf"
Sep 17 18:24:01.730045 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:01.730025    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dcbf3e71-59b1-4314-b69f-a71312e253a1-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-fw7lf\" (UID: \"dcbf3e71-59b1-4314-b69f-a71312e253a1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-fw7lf"
Sep 17 18:24:01.730176 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:01.730158    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dcbf3e71-59b1-4314-b69f-a71312e253a1-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-fw7lf\" (UID: \"dcbf3e71-59b1-4314-b69f-a71312e253a1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-fw7lf"
Sep 17 18:24:01.731754 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:01.731731    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dcbf3e71-59b1-4314-b69f-a71312e253a1-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-fw7lf\" (UID: \"dcbf3e71-59b1-4314-b69f-a71312e253a1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-fw7lf"
Sep 17 18:24:01.737559 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:01.737540    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p768f\" (UniqueName: \"kubernetes.io/projected/dcbf3e71-59b1-4314-b69f-a71312e253a1-kube-api-access-p768f\") pod \"isvc-xgboost-v2-predictor-fcc668bc-fw7lf\" (UID: \"dcbf3e71-59b1-4314-b69f-a71312e253a1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-fw7lf"
Sep 17 18:24:01.829589 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:01.829527    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-fw7lf"
Sep 17 18:24:02.104411 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:02.104389    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-fw7lf"]
Sep 17 18:24:02.106025 ip-10-0-131-14 kubenswrapper[2569]: W0917 18:24:02.105995    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcbf3e71_59b1_4314_b69f_a71312e253a1.slice/crio-85105733bf6c81192011ca57cc600e5fe2938d256c967d0171f53b133bdc0532 WatchSource:0}: Error finding container 85105733bf6c81192011ca57cc600e5fe2938d256c967d0171f53b133bdc0532: Status 404 returned error can't find the container with id 85105733bf6c81192011ca57cc600e5fe2938d256c967d0171f53b133bdc0532
Sep 17 18:24:02.262438 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:02.262411    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zj8jr"
Sep 17 18:24:02.335022 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:02.334994    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebd5bf44-b1be-43d3-bf96-25df4b31ddea-proxy-tls\") pod \"ebd5bf44-b1be-43d3-bf96-25df4b31ddea\" (UID: \"ebd5bf44-b1be-43d3-bf96-25df4b31ddea\") "
Sep 17 18:24:02.335193 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:02.335039    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ebd5bf44-b1be-43d3-bf96-25df4b31ddea-cabundle-cert\") pod \"ebd5bf44-b1be-43d3-bf96-25df4b31ddea\" (UID: \"ebd5bf44-b1be-43d3-bf96-25df4b31ddea\") "
Sep 17 18:24:02.335193 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:02.335064    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wb4zz\" (UniqueName: \"kubernetes.io/projected/ebd5bf44-b1be-43d3-bf96-25df4b31ddea-kube-api-access-wb4zz\") pod \"ebd5bf44-b1be-43d3-bf96-25df4b31ddea\" (UID: \"ebd5bf44-b1be-43d3-bf96-25df4b31ddea\") "
Sep 17 18:24:02.335193 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:02.335111    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ebd5bf44-b1be-43d3-bf96-25df4b31ddea-kube-rbac-proxy-sar-config\") pod \"ebd5bf44-b1be-43d3-bf96-25df4b31ddea\" (UID: \"ebd5bf44-b1be-43d3-bf96-25df4b31ddea\") "
Sep 17 18:24:02.335193 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:02.335150    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ebd5bf44-b1be-43d3-bf96-25df4b31ddea-kserve-provision-location\") pod \"ebd5bf44-b1be-43d3-bf96-25df4b31ddea\" (UID: \"ebd5bf44-b1be-43d3-bf96-25df4b31ddea\") "
Sep 17 18:24:02.335496 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:02.335465    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebd5bf44-b1be-43d3-bf96-25df4b31ddea-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ebd5bf44-b1be-43d3-bf96-25df4b31ddea" (UID: "ebd5bf44-b1be-43d3-bf96-25df4b31ddea"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:24:02.335552 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:02.335485    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebd5bf44-b1be-43d3-bf96-25df4b31ddea-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ebd5bf44-b1be-43d3-bf96-25df4b31ddea" (UID: "ebd5bf44-b1be-43d3-bf96-25df4b31ddea"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:24:02.335552 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:02.335519    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebd5bf44-b1be-43d3-bf96-25df4b31ddea-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ebd5bf44-b1be-43d3-bf96-25df4b31ddea" (UID: "ebd5bf44-b1be-43d3-bf96-25df4b31ddea"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:24:02.337164 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:02.337138    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebd5bf44-b1be-43d3-bf96-25df4b31ddea-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ebd5bf44-b1be-43d3-bf96-25df4b31ddea" (UID: "ebd5bf44-b1be-43d3-bf96-25df4b31ddea"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:24:02.337242 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:02.337201    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebd5bf44-b1be-43d3-bf96-25df4b31ddea-kube-api-access-wb4zz" (OuterVolumeSpecName: "kube-api-access-wb4zz") pod "ebd5bf44-b1be-43d3-bf96-25df4b31ddea" (UID: "ebd5bf44-b1be-43d3-bf96-25df4b31ddea"). InnerVolumeSpecName "kube-api-access-wb4zz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:24:02.424926 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:02.424854    2569 generic.go:356] "Generic (PLEG): container finished" podID="ebd5bf44-b1be-43d3-bf96-25df4b31ddea" containerID="7a70af6c74e0fd99c2724bec3938f83298c1dc4af111d1ec4efca050d42d0ae3" exitCode=0
Sep 17 18:24:02.424926 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:02.424913    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zj8jr"
Sep 17 18:24:02.425116 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:02.424941    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zj8jr" event={"ID":"ebd5bf44-b1be-43d3-bf96-25df4b31ddea","Type":"ContainerDied","Data":"7a70af6c74e0fd99c2724bec3938f83298c1dc4af111d1ec4efca050d42d0ae3"}
Sep 17 18:24:02.425116 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:02.425003    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zj8jr" event={"ID":"ebd5bf44-b1be-43d3-bf96-25df4b31ddea","Type":"ContainerDied","Data":"91a4d1b908ed0a981c1798c75112948a22d75ded2c254b72a887baca964cc599"}
Sep 17 18:24:02.425116 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:02.425032    2569 scope.go:117] "RemoveContainer" containerID="73e3ee87cb1671fe3cb15b512fab521f7993c96104402979b0b7b09809dfcc42"
Sep 17 18:24:02.426878 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:02.426855    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-fw7lf" event={"ID":"dcbf3e71-59b1-4314-b69f-a71312e253a1","Type":"ContainerStarted","Data":"dcb78144274b7117560840fa54827da106b53485bbdb5eb9019ecf56cd6781c4"}
Sep 17 18:24:02.426971 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:02.426913    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-fw7lf" event={"ID":"dcbf3e71-59b1-4314-b69f-a71312e253a1","Type":"ContainerStarted","Data":"85105733bf6c81192011ca57cc600e5fe2938d256c967d0171f53b133bdc0532"}
Sep 17 18:24:02.434068 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:02.434050    2569 scope.go:117] "RemoveContainer" containerID="7a70af6c74e0fd99c2724bec3938f83298c1dc4af111d1ec4efca050d42d0ae3"
Sep 17 18:24:02.436198 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:02.436183    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ebd5bf44-b1be-43d3-bf96-25df4b31ddea-cabundle-cert\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:24:02.436275 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:02.436199    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wb4zz\" (UniqueName: \"kubernetes.io/projected/ebd5bf44-b1be-43d3-bf96-25df4b31ddea-kube-api-access-wb4zz\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:24:02.436275 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:02.436209    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ebd5bf44-b1be-43d3-bf96-25df4b31ddea-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:24:02.436275 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:02.436219    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ebd5bf44-b1be-43d3-bf96-25df4b31ddea-kserve-provision-location\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:24:02.436275 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:02.436227    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebd5bf44-b1be-43d3-bf96-25df4b31ddea-proxy-tls\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:24:02.442164 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:02.442139    2569 scope.go:117] "RemoveContainer" containerID="9ea2a60f443053cbd4f339b8dd980120be204d49c34e1f847a5e754979a7e622"
Sep 17 18:24:02.451175 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:02.451154    2569 scope.go:117] "RemoveContainer" containerID="73e3ee87cb1671fe3cb15b512fab521f7993c96104402979b0b7b09809dfcc42"
Sep 17 18:24:02.451482 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:24:02.451464    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73e3ee87cb1671fe3cb15b512fab521f7993c96104402979b0b7b09809dfcc42\": container with ID starting with 73e3ee87cb1671fe3cb15b512fab521f7993c96104402979b0b7b09809dfcc42 not found: ID does not exist" containerID="73e3ee87cb1671fe3cb15b512fab521f7993c96104402979b0b7b09809dfcc42"
Sep 17 18:24:02.451556 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:02.451492    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73e3ee87cb1671fe3cb15b512fab521f7993c96104402979b0b7b09809dfcc42"} err="failed to get container status \"73e3ee87cb1671fe3cb15b512fab521f7993c96104402979b0b7b09809dfcc42\": rpc error: code = NotFound desc = could not find container \"73e3ee87cb1671fe3cb15b512fab521f7993c96104402979b0b7b09809dfcc42\": container with ID starting with 73e3ee87cb1671fe3cb15b512fab521f7993c96104402979b0b7b09809dfcc42 not found: ID does not exist"
Sep 17 18:24:02.451556 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:02.451515    2569 scope.go:117] "RemoveContainer" containerID="7a70af6c74e0fd99c2724bec3938f83298c1dc4af111d1ec4efca050d42d0ae3"
Sep 17 18:24:02.451818 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:24:02.451799    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a70af6c74e0fd99c2724bec3938f83298c1dc4af111d1ec4efca050d42d0ae3\": container with ID starting with 7a70af6c74e0fd99c2724bec3938f83298c1dc4af111d1ec4efca050d42d0ae3 not found: ID does not exist" containerID="7a70af6c74e0fd99c2724bec3938f83298c1dc4af111d1ec4efca050d42d0ae3"
Sep 17 18:24:02.451860 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:02.451828    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a70af6c74e0fd99c2724bec3938f83298c1dc4af111d1ec4efca050d42d0ae3"} err="failed to get container status \"7a70af6c74e0fd99c2724bec3938f83298c1dc4af111d1ec4efca050d42d0ae3\": rpc error: code = NotFound desc = could not find container \"7a70af6c74e0fd99c2724bec3938f83298c1dc4af111d1ec4efca050d42d0ae3\": container with ID starting with 7a70af6c74e0fd99c2724bec3938f83298c1dc4af111d1ec4efca050d42d0ae3 not found: ID does not exist"
Sep 17 18:24:02.451860 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:02.451843    2569 scope.go:117] "RemoveContainer" containerID="9ea2a60f443053cbd4f339b8dd980120be204d49c34e1f847a5e754979a7e622"
Sep 17 18:24:02.452108 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:24:02.452072    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ea2a60f443053cbd4f339b8dd980120be204d49c34e1f847a5e754979a7e622\": container with ID starting with 9ea2a60f443053cbd4f339b8dd980120be204d49c34e1f847a5e754979a7e622 not found: ID does not exist" containerID="9ea2a60f443053cbd4f339b8dd980120be204d49c34e1f847a5e754979a7e622"
Sep 17 18:24:02.452169 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:02.452114    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ea2a60f443053cbd4f339b8dd980120be204d49c34e1f847a5e754979a7e622"} err="failed to get container status \"9ea2a60f443053cbd4f339b8dd980120be204d49c34e1f847a5e754979a7e622\": rpc error: code = NotFound desc = could not find container \"9ea2a60f443053cbd4f339b8dd980120be204d49c34e1f847a5e754979a7e622\": container with ID starting with 9ea2a60f443053cbd4f339b8dd980120be204d49c34e1f847a5e754979a7e622 not found: ID does not exist"
Sep 17 18:24:02.461582 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:02.461563    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zj8jr"]
Sep 17 18:24:02.465417 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:02.465398    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-zj8jr"]
Sep 17 18:24:03.507915 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:03.507875    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebd5bf44-b1be-43d3-bf96-25df4b31ddea" path="/var/lib/kubelet/pods/ebd5bf44-b1be-43d3-bf96-25df4b31ddea/volumes"
Sep 17 18:24:06.449623 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:06.449590    2569 generic.go:356] "Generic (PLEG): container finished" podID="dcbf3e71-59b1-4314-b69f-a71312e253a1" containerID="dcb78144274b7117560840fa54827da106b53485bbdb5eb9019ecf56cd6781c4" exitCode=0
Sep 17 18:24:06.449984 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:06.449634    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-fw7lf" event={"ID":"dcbf3e71-59b1-4314-b69f-a71312e253a1","Type":"ContainerDied","Data":"dcb78144274b7117560840fa54827da106b53485bbdb5eb9019ecf56cd6781c4"}
Sep 17 18:24:07.457602 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:07.457569    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-fw7lf" event={"ID":"dcbf3e71-59b1-4314-b69f-a71312e253a1","Type":"ContainerStarted","Data":"9cb47055d8f6a1930d4af3fc63cd52246aa6148003d0a8b569553d6fe2995c95"}
Sep 17 18:24:07.458003 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:07.457661    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-fw7lf" event={"ID":"dcbf3e71-59b1-4314-b69f-a71312e253a1","Type":"ContainerStarted","Data":"9ec6f499105345b8b8b1342589b1a649d0120d6d2293beef09b1241c0c8c6952"}
Sep 17 18:24:07.458003 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:07.457920    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-fw7lf"
Sep 17 18:24:07.458165 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:07.458066    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-fw7lf"
Sep 17 18:24:07.459162 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:07.459137    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-fw7lf" podUID="dcbf3e71-59b1-4314-b69f-a71312e253a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 17 18:24:07.478631 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:07.478576    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-fw7lf" podStartSLOduration=2.763692802 podStartE2EDuration="6.478561971s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.714869169s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:24:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:24:07.477686288 +0000 UTC m=+2599.981688915" watchObservedRunningTime="2026-09-17 18:24:07.478561971 +0000 UTC m=+2599.982564596"
Sep 17 18:24:08.462863 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:08.462822    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-fw7lf" podUID="dcbf3e71-59b1-4314-b69f-a71312e253a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 17 18:24:11.504227 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:11.504182    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qx456" podUID="71c71769-a0f6-4c7d-8bf7-a41c4da6b022" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 18:24:13.467154 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:13.467123    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-fw7lf"
Sep 17 18:24:13.467610 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:13.467582    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-fw7lf" podUID="dcbf3e71-59b1-4314-b69f-a71312e253a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 17 18:24:21.508138 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:21.508108    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qx456"
Sep 17 18:24:23.468549 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:23.468508    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-fw7lf" podUID="dcbf3e71-59b1-4314-b69f-a71312e253a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 17 18:24:23.852991 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:23.852920    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qx456"]
Sep 17 18:24:23.853183 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:23.853158    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qx456" podUID="71c71769-a0f6-4c7d-8bf7-a41c4da6b022" containerName="kserve-container" containerID="cri-o://6861e9c5dccfb4dadd20b291bcd846e1db2252f795e70b70eed3dcbeef4b8e4a" gracePeriod=30
Sep 17 18:24:23.853280 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:23.853209    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qx456" podUID="71c71769-a0f6-4c7d-8bf7-a41c4da6b022" containerName="kube-rbac-proxy" containerID="cri-o://73e0302b95e04d02b910dea2c9dfe158c37d6be187e5df45d536b4d6665d707c" gracePeriod=30
Sep 17 18:24:24.075377 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:24.075349    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qx456" podUID="71c71769-a0f6-4c7d-8bf7-a41c4da6b022" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.43:8643/healthz\": dial tcp 10.133.0.43:8643: connect: connection refused"
Sep 17 18:24:24.533345 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:24.533317    2569 generic.go:356] "Generic (PLEG): container finished" podID="71c71769-a0f6-4c7d-8bf7-a41c4da6b022" containerID="73e0302b95e04d02b910dea2c9dfe158c37d6be187e5df45d536b4d6665d707c" exitCode=2
Sep 17 18:24:24.533692 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:24.533395    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qx456" event={"ID":"71c71769-a0f6-4c7d-8bf7-a41c4da6b022","Type":"ContainerDied","Data":"73e0302b95e04d02b910dea2c9dfe158c37d6be187e5df45d536b4d6665d707c"}
Sep 17 18:24:28.553836 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:28.553808    2569 generic.go:356] "Generic (PLEG): container finished" podID="71c71769-a0f6-4c7d-8bf7-a41c4da6b022" containerID="6861e9c5dccfb4dadd20b291bcd846e1db2252f795e70b70eed3dcbeef4b8e4a" exitCode=0
Sep 17 18:24:28.554211 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:28.553874    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qx456" event={"ID":"71c71769-a0f6-4c7d-8bf7-a41c4da6b022","Type":"ContainerDied","Data":"6861e9c5dccfb4dadd20b291bcd846e1db2252f795e70b70eed3dcbeef4b8e4a"}
Sep 17 18:24:28.695023 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:28.695002    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qx456"
Sep 17 18:24:28.834673 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:28.834642    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v2pcb\" (UniqueName: \"kubernetes.io/projected/71c71769-a0f6-4c7d-8bf7-a41c4da6b022-kube-api-access-v2pcb\") pod \"71c71769-a0f6-4c7d-8bf7-a41c4da6b022\" (UID: \"71c71769-a0f6-4c7d-8bf7-a41c4da6b022\") "
Sep 17 18:24:28.834673 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:28.834678    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71c71769-a0f6-4c7d-8bf7-a41c4da6b022-proxy-tls\") pod \"71c71769-a0f6-4c7d-8bf7-a41c4da6b022\" (UID: \"71c71769-a0f6-4c7d-8bf7-a41c4da6b022\") "
Sep 17 18:24:28.834865 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:28.834703    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71c71769-a0f6-4c7d-8bf7-a41c4da6b022-cabundle-cert\") pod \"71c71769-a0f6-4c7d-8bf7-a41c4da6b022\" (UID: \"71c71769-a0f6-4c7d-8bf7-a41c4da6b022\") "
Sep 17 18:24:28.834865 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:28.834729    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71c71769-a0f6-4c7d-8bf7-a41c4da6b022-kserve-provision-location\") pod \"71c71769-a0f6-4c7d-8bf7-a41c4da6b022\" (UID: \"71c71769-a0f6-4c7d-8bf7-a41c4da6b022\") "
Sep 17 18:24:28.834865 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:28.834771    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71c71769-a0f6-4c7d-8bf7-a41c4da6b022-kube-rbac-proxy-sar-config\") pod \"71c71769-a0f6-4c7d-8bf7-a41c4da6b022\" (UID: \"71c71769-a0f6-4c7d-8bf7-a41c4da6b022\") "
Sep 17 18:24:28.835145 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:28.835122    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71c71769-a0f6-4c7d-8bf7-a41c4da6b022-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "71c71769-a0f6-4c7d-8bf7-a41c4da6b022" (UID: "71c71769-a0f6-4c7d-8bf7-a41c4da6b022"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:24:28.835219 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:28.835162    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71c71769-a0f6-4c7d-8bf7-a41c4da6b022-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "71c71769-a0f6-4c7d-8bf7-a41c4da6b022" (UID: "71c71769-a0f6-4c7d-8bf7-a41c4da6b022"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:24:28.835219 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:28.835189    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71c71769-a0f6-4c7d-8bf7-a41c4da6b022-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "71c71769-a0f6-4c7d-8bf7-a41c4da6b022" (UID: "71c71769-a0f6-4c7d-8bf7-a41c4da6b022"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:24:28.836683 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:28.836660    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71c71769-a0f6-4c7d-8bf7-a41c4da6b022-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "71c71769-a0f6-4c7d-8bf7-a41c4da6b022" (UID: "71c71769-a0f6-4c7d-8bf7-a41c4da6b022"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:24:28.836782 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:28.836723    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71c71769-a0f6-4c7d-8bf7-a41c4da6b022-kube-api-access-v2pcb" (OuterVolumeSpecName: "kube-api-access-v2pcb") pod "71c71769-a0f6-4c7d-8bf7-a41c4da6b022" (UID: "71c71769-a0f6-4c7d-8bf7-a41c4da6b022"). InnerVolumeSpecName "kube-api-access-v2pcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:24:28.935566 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:28.935506    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2pcb\" (UniqueName: \"kubernetes.io/projected/71c71769-a0f6-4c7d-8bf7-a41c4da6b022-kube-api-access-v2pcb\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:24:28.935566 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:28.935535    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71c71769-a0f6-4c7d-8bf7-a41c4da6b022-proxy-tls\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:24:28.935566 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:28.935545    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71c71769-a0f6-4c7d-8bf7-a41c4da6b022-cabundle-cert\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:24:28.935566 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:28.935554    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71c71769-a0f6-4c7d-8bf7-a41c4da6b022-kserve-provision-location\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:24:28.935566 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:28.935562    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71c71769-a0f6-4c7d-8bf7-a41c4da6b022-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:24:29.032106 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:29.032040    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-w8mcl"]
Sep 17 18:24:29.032356 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:29.032344    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71c71769-a0f6-4c7d-8bf7-a41c4da6b022" containerName="kserve-container"
Sep 17 18:24:29.032399 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:29.032357    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="71c71769-a0f6-4c7d-8bf7-a41c4da6b022" containerName="kserve-container"
Sep 17 18:24:29.032399 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:29.032369    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebd5bf44-b1be-43d3-bf96-25df4b31ddea" containerName="kserve-container"
Sep 17 18:24:29.032399 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:29.032375    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebd5bf44-b1be-43d3-bf96-25df4b31ddea" containerName="kserve-container"
Sep 17 18:24:29.032399 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:29.032384    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71c71769-a0f6-4c7d-8bf7-a41c4da6b022" containerName="kube-rbac-proxy"
Sep 17 18:24:29.032399 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:29.032389    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="71c71769-a0f6-4c7d-8bf7-a41c4da6b022" containerName="kube-rbac-proxy"
Sep 17 18:24:29.032399 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:29.032398    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebd5bf44-b1be-43d3-bf96-25df4b31ddea" containerName="kube-rbac-proxy"
Sep 17 18:24:29.032577 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:29.032403    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebd5bf44-b1be-43d3-bf96-25df4b31ddea" containerName="kube-rbac-proxy"
Sep 17 18:24:29.032577 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:29.032411    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebd5bf44-b1be-43d3-bf96-25df4b31ddea" containerName="storage-initializer"
Sep 17 18:24:29.032577 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:29.032416    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebd5bf44-b1be-43d3-bf96-25df4b31ddea" containerName="storage-initializer"
Sep 17 18:24:29.032577 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:29.032425    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71c71769-a0f6-4c7d-8bf7-a41c4da6b022" containerName="storage-initializer"
Sep 17 18:24:29.032577 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:29.032430    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="71c71769-a0f6-4c7d-8bf7-a41c4da6b022" containerName="storage-initializer"
Sep 17 18:24:29.032577 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:29.032474    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebd5bf44-b1be-43d3-bf96-25df4b31ddea" containerName="kube-rbac-proxy"
Sep 17 18:24:29.032577 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:29.032484    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="71c71769-a0f6-4c7d-8bf7-a41c4da6b022" containerName="kube-rbac-proxy"
Sep 17 18:24:29.032577 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:29.032491    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebd5bf44-b1be-43d3-bf96-25df4b31ddea" containerName="kserve-container"
Sep 17 18:24:29.032577 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:29.032498    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="71c71769-a0f6-4c7d-8bf7-a41c4da6b022" containerName="kserve-container"
Sep 17 18:24:29.035778 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:29.035754    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-w8mcl"
Sep 17 18:24:29.038305 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:29.038286    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-predictor-serving-cert\""
Sep 17 18:24:29.038421 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:29.038307    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 17 18:24:29.046097 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:29.046057    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-w8mcl"]
Sep 17 18:24:29.136656 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:29.136630    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51ecdcc7-9ed7-4de6-a62a-40289ef05c2d-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-w8mcl\" (UID: \"51ecdcc7-9ed7-4de6-a62a-40289ef05c2d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-w8mcl"
Sep 17 18:24:29.136758 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:29.136680    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgm7p\" (UniqueName: \"kubernetes.io/projected/51ecdcc7-9ed7-4de6-a62a-40289ef05c2d-kube-api-access-hgm7p\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-w8mcl\" (UID: \"51ecdcc7-9ed7-4de6-a62a-40289ef05c2d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-w8mcl"
Sep 17 18:24:29.136758 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:29.136722    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51ecdcc7-9ed7-4de6-a62a-40289ef05c2d-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-w8mcl\" (UID: \"51ecdcc7-9ed7-4de6-a62a-40289ef05c2d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-w8mcl"
Sep 17 18:24:29.136849 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:29.136767    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51ecdcc7-9ed7-4de6-a62a-40289ef05c2d-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-w8mcl\" (UID: \"51ecdcc7-9ed7-4de6-a62a-40289ef05c2d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-w8mcl"
Sep 17 18:24:29.136849 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:29.136783    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51ecdcc7-9ed7-4de6-a62a-40289ef05c2d-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-w8mcl\" (UID: \"51ecdcc7-9ed7-4de6-a62a-40289ef05c2d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-w8mcl"
Sep 17 18:24:29.237974 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:29.237945    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51ecdcc7-9ed7-4de6-a62a-40289ef05c2d-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-w8mcl\" (UID: \"51ecdcc7-9ed7-4de6-a62a-40289ef05c2d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-w8mcl"
Sep 17 18:24:29.238114 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:29.237978    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51ecdcc7-9ed7-4de6-a62a-40289ef05c2d-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-w8mcl\" (UID: \"51ecdcc7-9ed7-4de6-a62a-40289ef05c2d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-w8mcl"
Sep 17 18:24:29.238114 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:29.238007    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51ecdcc7-9ed7-4de6-a62a-40289ef05c2d-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-w8mcl\" (UID: \"51ecdcc7-9ed7-4de6-a62a-40289ef05c2d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-w8mcl"
Sep 17 18:24:29.238114 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:29.238063    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgm7p\" (UniqueName: \"kubernetes.io/projected/51ecdcc7-9ed7-4de6-a62a-40289ef05c2d-kube-api-access-hgm7p\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-w8mcl\" (UID: \"51ecdcc7-9ed7-4de6-a62a-40289ef05c2d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-w8mcl"
Sep 17 18:24:29.238287 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:24:29.238238    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-serving-cert: secret "isvc-lightgbm-v2-runtime-predictor-serving-cert" not found
Sep 17 18:24:29.238287 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:29.238271    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51ecdcc7-9ed7-4de6-a62a-40289ef05c2d-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-w8mcl\" (UID: \"51ecdcc7-9ed7-4de6-a62a-40289ef05c2d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-w8mcl"
Sep 17 18:24:29.238392 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:24:29.238297    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51ecdcc7-9ed7-4de6-a62a-40289ef05c2d-proxy-tls podName:51ecdcc7-9ed7-4de6-a62a-40289ef05c2d nodeName:}" failed. No retries permitted until 2026-09-17 18:24:29.738280465 +0000 UTC m=+2622.242283068 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/51ecdcc7-9ed7-4de6-a62a-40289ef05c2d-proxy-tls") pod "isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-w8mcl" (UID: "51ecdcc7-9ed7-4de6-a62a-40289ef05c2d") : secret "isvc-lightgbm-v2-runtime-predictor-serving-cert" not found
Sep 17 18:24:29.238392 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:29.238339    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51ecdcc7-9ed7-4de6-a62a-40289ef05c2d-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-w8mcl\" (UID: \"51ecdcc7-9ed7-4de6-a62a-40289ef05c2d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-w8mcl"
Sep 17 18:24:29.238684 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:29.238665    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51ecdcc7-9ed7-4de6-a62a-40289ef05c2d-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-w8mcl\" (UID: \"51ecdcc7-9ed7-4de6-a62a-40289ef05c2d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-w8mcl"
Sep 17 18:24:29.238771 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:29.238752    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51ecdcc7-9ed7-4de6-a62a-40289ef05c2d-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-w8mcl\" (UID: \"51ecdcc7-9ed7-4de6-a62a-40289ef05c2d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-w8mcl"
Sep 17 18:24:29.246172 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:29.246149    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgm7p\" (UniqueName: \"kubernetes.io/projected/51ecdcc7-9ed7-4de6-a62a-40289ef05c2d-kube-api-access-hgm7p\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-w8mcl\" (UID: \"51ecdcc7-9ed7-4de6-a62a-40289ef05c2d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-w8mcl"
Sep 17 18:24:29.560158 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:29.560134    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qx456" event={"ID":"71c71769-a0f6-4c7d-8bf7-a41c4da6b022","Type":"ContainerDied","Data":"1da8ae2d2b43f2c3a5cbf86cd7cd8a5ee1aa35eb691398bd02b509dbdadc469d"}
Sep 17 18:24:29.560437 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:29.560190    2569 scope.go:117] "RemoveContainer" containerID="73e0302b95e04d02b910dea2c9dfe158c37d6be187e5df45d536b4d6665d707c"
Sep 17 18:24:29.560437 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:29.560147    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qx456"
Sep 17 18:24:29.568042 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:29.567901    2569 scope.go:117] "RemoveContainer" containerID="6861e9c5dccfb4dadd20b291bcd846e1db2252f795e70b70eed3dcbeef4b8e4a"
Sep 17 18:24:29.574891 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:29.574876    2569 scope.go:117] "RemoveContainer" containerID="cf2fb74c35c05b14b097536b1e0483af83838bf13ad1f29378d9c856f7c7179f"
Sep 17 18:24:29.578131 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:29.578109    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qx456"]
Sep 17 18:24:29.584441 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:29.584420    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qx456"]
Sep 17 18:24:29.742339 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:29.742311    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51ecdcc7-9ed7-4de6-a62a-40289ef05c2d-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-w8mcl\" (UID: \"51ecdcc7-9ed7-4de6-a62a-40289ef05c2d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-w8mcl"
Sep 17 18:24:29.744709 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:29.744674    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51ecdcc7-9ed7-4de6-a62a-40289ef05c2d-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-w8mcl\" (UID: \"51ecdcc7-9ed7-4de6-a62a-40289ef05c2d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-w8mcl"
Sep 17 18:24:29.946095 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:29.946055    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-w8mcl"
Sep 17 18:24:30.218416 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:30.218362    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-w8mcl"]
Sep 17 18:24:30.220424 ip-10-0-131-14 kubenswrapper[2569]: W0917 18:24:30.220395    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51ecdcc7_9ed7_4de6_a62a_40289ef05c2d.slice/crio-8a6ba0a50aa97fc64ae5585b79b376834eadfeb521f39cdf456de4e1c9671839 WatchSource:0}: Error finding container 8a6ba0a50aa97fc64ae5585b79b376834eadfeb521f39cdf456de4e1c9671839: Status 404 returned error can't find the container with id 8a6ba0a50aa97fc64ae5585b79b376834eadfeb521f39cdf456de4e1c9671839
Sep 17 18:24:30.567670 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:30.567573    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-w8mcl" event={"ID":"51ecdcc7-9ed7-4de6-a62a-40289ef05c2d","Type":"ContainerStarted","Data":"0a9e66f2c0f402594aa0c214fd444fd7bb6fc42a6f99a2fd4bf33666a453f021"}
Sep 17 18:24:30.567670 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:30.567648    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-w8mcl" event={"ID":"51ecdcc7-9ed7-4de6-a62a-40289ef05c2d","Type":"ContainerStarted","Data":"8a6ba0a50aa97fc64ae5585b79b376834eadfeb521f39cdf456de4e1c9671839"}
Sep 17 18:24:31.512222 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:31.512186    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71c71769-a0f6-4c7d-8bf7-a41c4da6b022" path="/var/lib/kubelet/pods/71c71769-a0f6-4c7d-8bf7-a41c4da6b022/volumes"
Sep 17 18:24:33.468495 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:33.468463    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-fw7lf" podUID="dcbf3e71-59b1-4314-b69f-a71312e253a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 17 18:24:34.587456 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:34.587423    2569 generic.go:356] "Generic (PLEG): container finished" podID="51ecdcc7-9ed7-4de6-a62a-40289ef05c2d" containerID="0a9e66f2c0f402594aa0c214fd444fd7bb6fc42a6f99a2fd4bf33666a453f021" exitCode=0
Sep 17 18:24:34.587902 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:34.587471    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-w8mcl" event={"ID":"51ecdcc7-9ed7-4de6-a62a-40289ef05c2d","Type":"ContainerDied","Data":"0a9e66f2c0f402594aa0c214fd444fd7bb6fc42a6f99a2fd4bf33666a453f021"}
Sep 17 18:24:35.593904 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:35.593872    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-w8mcl" event={"ID":"51ecdcc7-9ed7-4de6-a62a-40289ef05c2d","Type":"ContainerStarted","Data":"3649bdd96b1fef1bb71891c3d8699fd145df4f97a240e373c07658dc4fb1d1b7"}
Sep 17 18:24:35.594322 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:35.593931    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-w8mcl" event={"ID":"51ecdcc7-9ed7-4de6-a62a-40289ef05c2d","Type":"ContainerStarted","Data":"4ee7f6016cb864107c0edab6b562ee2443049288d3d1dcf3690cabfd0093e1b9"}
Sep 17 18:24:35.594322 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:35.594232    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-w8mcl"
Sep 17 18:24:35.594322 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:35.594297    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-w8mcl"
Sep 17 18:24:35.616482 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:35.616433    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-w8mcl" podStartSLOduration=2.83508412 podStartE2EDuration="6.616419136s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.781335016s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:24:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:24:35.615100416 +0000 UTC m=+2628.119103044" watchObservedRunningTime="2026-09-17 18:24:35.616419136 +0000 UTC m=+2628.120421761"
Sep 17 18:24:41.604442 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:41.604410    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-w8mcl"
Sep 17 18:24:43.468018 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:43.467978    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-fw7lf" podUID="dcbf3e71-59b1-4314-b69f-a71312e253a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 17 18:24:53.467844 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:24:53.467798    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-fw7lf" podUID="dcbf3e71-59b1-4314-b69f-a71312e253a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 17 18:25:03.467847 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:03.467808    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-fw7lf" podUID="dcbf3e71-59b1-4314-b69f-a71312e253a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 17 18:25:11.608104 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:11.608057    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-w8mcl"
Sep 17 18:25:13.468196 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:13.468160    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-fw7lf"
Sep 17 18:25:19.105978 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:19.105940    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-w8mcl"]
Sep 17 18:25:19.106489 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:19.106260    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-w8mcl" podUID="51ecdcc7-9ed7-4de6-a62a-40289ef05c2d" containerName="kserve-container" containerID="cri-o://4ee7f6016cb864107c0edab6b562ee2443049288d3d1dcf3690cabfd0093e1b9" gracePeriod=30
Sep 17 18:25:19.106489 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:19.106289    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-w8mcl" podUID="51ecdcc7-9ed7-4de6-a62a-40289ef05c2d" containerName="kube-rbac-proxy" containerID="cri-o://3649bdd96b1fef1bb71891c3d8699fd145df4f97a240e373c07658dc4fb1d1b7" gracePeriod=30
Sep 17 18:25:19.782846 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:19.782812    2569 generic.go:356] "Generic (PLEG): container finished" podID="51ecdcc7-9ed7-4de6-a62a-40289ef05c2d" containerID="3649bdd96b1fef1bb71891c3d8699fd145df4f97a240e373c07658dc4fb1d1b7" exitCode=2
Sep 17 18:25:19.783026 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:19.782887    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-w8mcl" event={"ID":"51ecdcc7-9ed7-4de6-a62a-40289ef05c2d","Type":"ContainerDied","Data":"3649bdd96b1fef1bb71891c3d8699fd145df4f97a240e373c07658dc4fb1d1b7"}
Sep 17 18:25:20.048487 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:20.048466    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-w8mcl"
Sep 17 18:25:20.095590 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:20.095567    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hgm7p\" (UniqueName: \"kubernetes.io/projected/51ecdcc7-9ed7-4de6-a62a-40289ef05c2d-kube-api-access-hgm7p\") pod \"51ecdcc7-9ed7-4de6-a62a-40289ef05c2d\" (UID: \"51ecdcc7-9ed7-4de6-a62a-40289ef05c2d\") "
Sep 17 18:25:20.095708 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:20.095613    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51ecdcc7-9ed7-4de6-a62a-40289ef05c2d-proxy-tls\") pod \"51ecdcc7-9ed7-4de6-a62a-40289ef05c2d\" (UID: \"51ecdcc7-9ed7-4de6-a62a-40289ef05c2d\") "
Sep 17 18:25:20.095708 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:20.095639    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51ecdcc7-9ed7-4de6-a62a-40289ef05c2d-kube-rbac-proxy-sar-config\") pod \"51ecdcc7-9ed7-4de6-a62a-40289ef05c2d\" (UID: \"51ecdcc7-9ed7-4de6-a62a-40289ef05c2d\") "
Sep 17 18:25:20.095708 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:20.095693    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51ecdcc7-9ed7-4de6-a62a-40289ef05c2d-cabundle-cert\") pod \"51ecdcc7-9ed7-4de6-a62a-40289ef05c2d\" (UID: \"51ecdcc7-9ed7-4de6-a62a-40289ef05c2d\") "
Sep 17 18:25:20.095877 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:20.095727    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51ecdcc7-9ed7-4de6-a62a-40289ef05c2d-kserve-provision-location\") pod \"51ecdcc7-9ed7-4de6-a62a-40289ef05c2d\" (UID: \"51ecdcc7-9ed7-4de6-a62a-40289ef05c2d\") "
Sep 17 18:25:20.096045 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:20.096021    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51ecdcc7-9ed7-4de6-a62a-40289ef05c2d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "51ecdcc7-9ed7-4de6-a62a-40289ef05c2d" (UID: "51ecdcc7-9ed7-4de6-a62a-40289ef05c2d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:25:20.096045 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:20.096018    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51ecdcc7-9ed7-4de6-a62a-40289ef05c2d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "51ecdcc7-9ed7-4de6-a62a-40289ef05c2d" (UID: "51ecdcc7-9ed7-4de6-a62a-40289ef05c2d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:25:20.096210 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:20.096063    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51ecdcc7-9ed7-4de6-a62a-40289ef05c2d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "51ecdcc7-9ed7-4de6-a62a-40289ef05c2d" (UID: "51ecdcc7-9ed7-4de6-a62a-40289ef05c2d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:25:20.097647 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:20.097627    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51ecdcc7-9ed7-4de6-a62a-40289ef05c2d-kube-api-access-hgm7p" (OuterVolumeSpecName: "kube-api-access-hgm7p") pod "51ecdcc7-9ed7-4de6-a62a-40289ef05c2d" (UID: "51ecdcc7-9ed7-4de6-a62a-40289ef05c2d"). InnerVolumeSpecName "kube-api-access-hgm7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:25:20.097647 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:20.097637    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51ecdcc7-9ed7-4de6-a62a-40289ef05c2d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "51ecdcc7-9ed7-4de6-a62a-40289ef05c2d" (UID: "51ecdcc7-9ed7-4de6-a62a-40289ef05c2d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:25:20.196303 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:20.196283    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgm7p\" (UniqueName: \"kubernetes.io/projected/51ecdcc7-9ed7-4de6-a62a-40289ef05c2d-kube-api-access-hgm7p\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:25:20.196585 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:20.196303    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51ecdcc7-9ed7-4de6-a62a-40289ef05c2d-proxy-tls\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:25:20.196585 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:20.196318    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51ecdcc7-9ed7-4de6-a62a-40289ef05c2d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:25:20.196585 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:20.196328    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51ecdcc7-9ed7-4de6-a62a-40289ef05c2d-cabundle-cert\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:25:20.196585 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:20.196338    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51ecdcc7-9ed7-4de6-a62a-40289ef05c2d-kserve-provision-location\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:25:20.789425 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:20.789396    2569 generic.go:356] "Generic (PLEG): container finished" podID="51ecdcc7-9ed7-4de6-a62a-40289ef05c2d" containerID="4ee7f6016cb864107c0edab6b562ee2443049288d3d1dcf3690cabfd0093e1b9" exitCode=0
Sep 17 18:25:20.789533 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:20.789447    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-w8mcl"
Sep 17 18:25:20.789533 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:20.789486    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-w8mcl" event={"ID":"51ecdcc7-9ed7-4de6-a62a-40289ef05c2d","Type":"ContainerDied","Data":"4ee7f6016cb864107c0edab6b562ee2443049288d3d1dcf3690cabfd0093e1b9"}
Sep 17 18:25:20.789616 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:20.789532    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-w8mcl" event={"ID":"51ecdcc7-9ed7-4de6-a62a-40289ef05c2d","Type":"ContainerDied","Data":"8a6ba0a50aa97fc64ae5585b79b376834eadfeb521f39cdf456de4e1c9671839"}
Sep 17 18:25:20.789616 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:20.789562    2569 scope.go:117] "RemoveContainer" containerID="3649bdd96b1fef1bb71891c3d8699fd145df4f97a240e373c07658dc4fb1d1b7"
Sep 17 18:25:20.798921 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:20.798905    2569 scope.go:117] "RemoveContainer" containerID="4ee7f6016cb864107c0edab6b562ee2443049288d3d1dcf3690cabfd0093e1b9"
Sep 17 18:25:20.807324 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:20.807251    2569 scope.go:117] "RemoveContainer" containerID="0a9e66f2c0f402594aa0c214fd444fd7bb6fc42a6f99a2fd4bf33666a453f021"
Sep 17 18:25:20.812991 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:20.812968    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-w8mcl"]
Sep 17 18:25:20.815799 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:20.815776    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-w8mcl"]
Sep 17 18:25:20.817708 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:20.817677    2569 scope.go:117] "RemoveContainer" containerID="3649bdd96b1fef1bb71891c3d8699fd145df4f97a240e373c07658dc4fb1d1b7"
Sep 17 18:25:20.818000 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:25:20.817983    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3649bdd96b1fef1bb71891c3d8699fd145df4f97a240e373c07658dc4fb1d1b7\": container with ID starting with 3649bdd96b1fef1bb71891c3d8699fd145df4f97a240e373c07658dc4fb1d1b7 not found: ID does not exist" containerID="3649bdd96b1fef1bb71891c3d8699fd145df4f97a240e373c07658dc4fb1d1b7"
Sep 17 18:25:20.818059 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:20.818008    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3649bdd96b1fef1bb71891c3d8699fd145df4f97a240e373c07658dc4fb1d1b7"} err="failed to get container status \"3649bdd96b1fef1bb71891c3d8699fd145df4f97a240e373c07658dc4fb1d1b7\": rpc error: code = NotFound desc = could not find container \"3649bdd96b1fef1bb71891c3d8699fd145df4f97a240e373c07658dc4fb1d1b7\": container with ID starting with 3649bdd96b1fef1bb71891c3d8699fd145df4f97a240e373c07658dc4fb1d1b7 not found: ID does not exist"
Sep 17 18:25:20.818059 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:20.818025    2569 scope.go:117] "RemoveContainer" containerID="4ee7f6016cb864107c0edab6b562ee2443049288d3d1dcf3690cabfd0093e1b9"
Sep 17 18:25:20.818304 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:25:20.818287    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ee7f6016cb864107c0edab6b562ee2443049288d3d1dcf3690cabfd0093e1b9\": container with ID starting with 4ee7f6016cb864107c0edab6b562ee2443049288d3d1dcf3690cabfd0093e1b9 not found: ID does not exist" containerID="4ee7f6016cb864107c0edab6b562ee2443049288d3d1dcf3690cabfd0093e1b9"
Sep 17 18:25:20.818358 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:20.818309    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ee7f6016cb864107c0edab6b562ee2443049288d3d1dcf3690cabfd0093e1b9"} err="failed to get container status \"4ee7f6016cb864107c0edab6b562ee2443049288d3d1dcf3690cabfd0093e1b9\": rpc error: code = NotFound desc = could not find container \"4ee7f6016cb864107c0edab6b562ee2443049288d3d1dcf3690cabfd0093e1b9\": container with ID starting with 4ee7f6016cb864107c0edab6b562ee2443049288d3d1dcf3690cabfd0093e1b9 not found: ID does not exist"
Sep 17 18:25:20.818358 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:20.818323    2569 scope.go:117] "RemoveContainer" containerID="0a9e66f2c0f402594aa0c214fd444fd7bb6fc42a6f99a2fd4bf33666a453f021"
Sep 17 18:25:20.818556 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:25:20.818540    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a9e66f2c0f402594aa0c214fd444fd7bb6fc42a6f99a2fd4bf33666a453f021\": container with ID starting with 0a9e66f2c0f402594aa0c214fd444fd7bb6fc42a6f99a2fd4bf33666a453f021 not found: ID does not exist" containerID="0a9e66f2c0f402594aa0c214fd444fd7bb6fc42a6f99a2fd4bf33666a453f021"
Sep 17 18:25:20.818597 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:20.818560    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a9e66f2c0f402594aa0c214fd444fd7bb6fc42a6f99a2fd4bf33666a453f021"} err="failed to get container status \"0a9e66f2c0f402594aa0c214fd444fd7bb6fc42a6f99a2fd4bf33666a453f021\": rpc error: code = NotFound desc = could not find container \"0a9e66f2c0f402594aa0c214fd444fd7bb6fc42a6f99a2fd4bf33666a453f021\": container with ID starting with 0a9e66f2c0f402594aa0c214fd444fd7bb6fc42a6f99a2fd4bf33666a453f021 not found: ID does not exist"
Sep 17 18:25:21.507735 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:21.507708    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51ecdcc7-9ed7-4de6-a62a-40289ef05c2d" path="/var/lib/kubelet/pods/51ecdcc7-9ed7-4de6-a62a-40289ef05c2d/volumes"
Sep 17 18:25:22.171369 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:22.171337    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-fw7lf"]
Sep 17 18:25:22.171651 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:22.171627    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-fw7lf" podUID="dcbf3e71-59b1-4314-b69f-a71312e253a1" containerName="kserve-container" containerID="cri-o://9ec6f499105345b8b8b1342589b1a649d0120d6d2293beef09b1241c0c8c6952" gracePeriod=30
Sep 17 18:25:22.171748 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:22.171674    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-fw7lf" podUID="dcbf3e71-59b1-4314-b69f-a71312e253a1" containerName="kube-rbac-proxy" containerID="cri-o://9cb47055d8f6a1930d4af3fc63cd52246aa6148003d0a8b569553d6fe2995c95" gracePeriod=30
Sep 17 18:25:22.801536 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:22.801502    2569 generic.go:356] "Generic (PLEG): container finished" podID="dcbf3e71-59b1-4314-b69f-a71312e253a1" containerID="9cb47055d8f6a1930d4af3fc63cd52246aa6148003d0a8b569553d6fe2995c95" exitCode=2
Sep 17 18:25:22.801902 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:22.801539    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-fw7lf" event={"ID":"dcbf3e71-59b1-4314-b69f-a71312e253a1","Type":"ContainerDied","Data":"9cb47055d8f6a1930d4af3fc63cd52246aa6148003d0a8b569553d6fe2995c95"}
Sep 17 18:25:23.463627 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:23.463588    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-fw7lf" podUID="dcbf3e71-59b1-4314-b69f-a71312e253a1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.45:8643/healthz\": dial tcp 10.133.0.45:8643: connect: connection refused"
Sep 17 18:25:23.467877 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:23.467853    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-fw7lf" podUID="dcbf3e71-59b1-4314-b69f-a71312e253a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 17 18:25:24.284424 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:24.284395    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5ftl2"]
Sep 17 18:25:24.284788 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:24.284675    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51ecdcc7-9ed7-4de6-a62a-40289ef05c2d" containerName="kserve-container"
Sep 17 18:25:24.284788 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:24.284686    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="51ecdcc7-9ed7-4de6-a62a-40289ef05c2d" containerName="kserve-container"
Sep 17 18:25:24.284788 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:24.284695    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51ecdcc7-9ed7-4de6-a62a-40289ef05c2d" containerName="kube-rbac-proxy"
Sep 17 18:25:24.284788 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:24.284701    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="51ecdcc7-9ed7-4de6-a62a-40289ef05c2d" containerName="kube-rbac-proxy"
Sep 17 18:25:24.284788 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:24.284714    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51ecdcc7-9ed7-4de6-a62a-40289ef05c2d" containerName="storage-initializer"
Sep 17 18:25:24.284788 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:24.284720    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="51ecdcc7-9ed7-4de6-a62a-40289ef05c2d" containerName="storage-initializer"
Sep 17 18:25:24.284788 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:24.284764    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="51ecdcc7-9ed7-4de6-a62a-40289ef05c2d" containerName="kube-rbac-proxy"
Sep 17 18:25:24.284788 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:24.284770    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="51ecdcc7-9ed7-4de6-a62a-40289ef05c2d" containerName="kserve-container"
Sep 17 18:25:24.287691 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:24.287677    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5ftl2"
Sep 17 18:25:24.290424 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:24.290404    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-predictor-serving-cert\""
Sep 17 18:25:24.290659 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:24.290645    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 17 18:25:24.297522 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:24.297502    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5ftl2"]
Sep 17 18:25:24.427412 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:24.427386    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51782ea4-5fef-4f89-b155-c948c5145c1a-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5ftl2\" (UID: \"51782ea4-5fef-4f89-b155-c948c5145c1a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5ftl2"
Sep 17 18:25:24.427547 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:24.427452    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51782ea4-5fef-4f89-b155-c948c5145c1a-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5ftl2\" (UID: \"51782ea4-5fef-4f89-b155-c948c5145c1a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5ftl2"
Sep 17 18:25:24.427547 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:24.427483    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51782ea4-5fef-4f89-b155-c948c5145c1a-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5ftl2\" (UID: \"51782ea4-5fef-4f89-b155-c948c5145c1a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5ftl2"
Sep 17 18:25:24.427547 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:24.427509    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnh8k\" (UniqueName: \"kubernetes.io/projected/51782ea4-5fef-4f89-b155-c948c5145c1a-kube-api-access-pnh8k\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5ftl2\" (UID: \"51782ea4-5fef-4f89-b155-c948c5145c1a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5ftl2"
Sep 17 18:25:24.427547 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:24.427532    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51782ea4-5fef-4f89-b155-c948c5145c1a-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5ftl2\" (UID: \"51782ea4-5fef-4f89-b155-c948c5145c1a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5ftl2"
Sep 17 18:25:24.528115 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:24.528064    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51782ea4-5fef-4f89-b155-c948c5145c1a-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5ftl2\" (UID: \"51782ea4-5fef-4f89-b155-c948c5145c1a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5ftl2"
Sep 17 18:25:24.528221 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:24.528119    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnh8k\" (UniqueName: \"kubernetes.io/projected/51782ea4-5fef-4f89-b155-c948c5145c1a-kube-api-access-pnh8k\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5ftl2\" (UID: \"51782ea4-5fef-4f89-b155-c948c5145c1a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5ftl2"
Sep 17 18:25:24.528221 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:24.528137    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51782ea4-5fef-4f89-b155-c948c5145c1a-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5ftl2\" (UID: \"51782ea4-5fef-4f89-b155-c948c5145c1a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5ftl2"
Sep 17 18:25:24.528221 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:24.528192    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51782ea4-5fef-4f89-b155-c948c5145c1a-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5ftl2\" (UID: \"51782ea4-5fef-4f89-b155-c948c5145c1a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5ftl2"
Sep 17 18:25:24.528354 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:24.528229    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51782ea4-5fef-4f89-b155-c948c5145c1a-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5ftl2\" (UID: \"51782ea4-5fef-4f89-b155-c948c5145c1a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5ftl2"
Sep 17 18:25:24.528550 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:24.528535    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51782ea4-5fef-4f89-b155-c948c5145c1a-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5ftl2\" (UID: \"51782ea4-5fef-4f89-b155-c948c5145c1a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5ftl2"
Sep 17 18:25:24.528880 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:24.528857    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51782ea4-5fef-4f89-b155-c948c5145c1a-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5ftl2\" (UID: \"51782ea4-5fef-4f89-b155-c948c5145c1a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5ftl2"
Sep 17 18:25:24.528924 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:24.528864    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51782ea4-5fef-4f89-b155-c948c5145c1a-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5ftl2\" (UID: \"51782ea4-5fef-4f89-b155-c948c5145c1a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5ftl2"
Sep 17 18:25:24.530630 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:24.530606    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51782ea4-5fef-4f89-b155-c948c5145c1a-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5ftl2\" (UID: \"51782ea4-5fef-4f89-b155-c948c5145c1a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5ftl2"
Sep 17 18:25:24.536101 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:24.536026    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnh8k\" (UniqueName: \"kubernetes.io/projected/51782ea4-5fef-4f89-b155-c948c5145c1a-kube-api-access-pnh8k\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5ftl2\" (UID: \"51782ea4-5fef-4f89-b155-c948c5145c1a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5ftl2"
Sep 17 18:25:24.597954 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:24.597933    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5ftl2"
Sep 17 18:25:24.867921 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:24.867895    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5ftl2"]
Sep 17 18:25:24.869479 ip-10-0-131-14 kubenswrapper[2569]: W0917 18:25:24.869447    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51782ea4_5fef_4f89_b155_c948c5145c1a.slice/crio-5acc006d95e4e99bf37f355a7a373a8db3b17edc2ca4ea678cb8f2b4af30ffc5 WatchSource:0}: Error finding container 5acc006d95e4e99bf37f355a7a373a8db3b17edc2ca4ea678cb8f2b4af30ffc5: Status 404 returned error can't find the container with id 5acc006d95e4e99bf37f355a7a373a8db3b17edc2ca4ea678cb8f2b4af30ffc5
Sep 17 18:25:25.607094 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:25.607063    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-fw7lf"
Sep 17 18:25:25.737887 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:25.737852    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p768f\" (UniqueName: \"kubernetes.io/projected/dcbf3e71-59b1-4314-b69f-a71312e253a1-kube-api-access-p768f\") pod \"dcbf3e71-59b1-4314-b69f-a71312e253a1\" (UID: \"dcbf3e71-59b1-4314-b69f-a71312e253a1\") "
Sep 17 18:25:25.738019 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:25.737905    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dcbf3e71-59b1-4314-b69f-a71312e253a1-kserve-provision-location\") pod \"dcbf3e71-59b1-4314-b69f-a71312e253a1\" (UID: \"dcbf3e71-59b1-4314-b69f-a71312e253a1\") "
Sep 17 18:25:25.738019 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:25.738007    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dcbf3e71-59b1-4314-b69f-a71312e253a1-kube-rbac-proxy-sar-config\") pod \"dcbf3e71-59b1-4314-b69f-a71312e253a1\" (UID: \"dcbf3e71-59b1-4314-b69f-a71312e253a1\") "
Sep 17 18:25:25.738154 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:25.738141    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dcbf3e71-59b1-4314-b69f-a71312e253a1-proxy-tls\") pod \"dcbf3e71-59b1-4314-b69f-a71312e253a1\" (UID: \"dcbf3e71-59b1-4314-b69f-a71312e253a1\") "
Sep 17 18:25:25.738220 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:25.738164    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcbf3e71-59b1-4314-b69f-a71312e253a1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dcbf3e71-59b1-4314-b69f-a71312e253a1" (UID: "dcbf3e71-59b1-4314-b69f-a71312e253a1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:25:25.738220 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:25.738178    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dcbf3e71-59b1-4314-b69f-a71312e253a1-cabundle-cert\") pod \"dcbf3e71-59b1-4314-b69f-a71312e253a1\" (UID: \"dcbf3e71-59b1-4314-b69f-a71312e253a1\") "
Sep 17 18:25:25.738356 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:25.738333    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcbf3e71-59b1-4314-b69f-a71312e253a1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dcbf3e71-59b1-4314-b69f-a71312e253a1" (UID: "dcbf3e71-59b1-4314-b69f-a71312e253a1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:25:25.738450 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:25.738432    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dcbf3e71-59b1-4314-b69f-a71312e253a1-kserve-provision-location\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:25:25.738507 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:25.738456    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dcbf3e71-59b1-4314-b69f-a71312e253a1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:25:25.738561 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:25.738538    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcbf3e71-59b1-4314-b69f-a71312e253a1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dcbf3e71-59b1-4314-b69f-a71312e253a1" (UID: "dcbf3e71-59b1-4314-b69f-a71312e253a1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:25:25.739857 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:25.739836    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcbf3e71-59b1-4314-b69f-a71312e253a1-kube-api-access-p768f" (OuterVolumeSpecName: "kube-api-access-p768f") pod "dcbf3e71-59b1-4314-b69f-a71312e253a1" (UID: "dcbf3e71-59b1-4314-b69f-a71312e253a1"). InnerVolumeSpecName "kube-api-access-p768f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:25:25.739985 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:25.739964    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcbf3e71-59b1-4314-b69f-a71312e253a1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dcbf3e71-59b1-4314-b69f-a71312e253a1" (UID: "dcbf3e71-59b1-4314-b69f-a71312e253a1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:25:25.815343 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:25.815313    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5ftl2" event={"ID":"51782ea4-5fef-4f89-b155-c948c5145c1a","Type":"ContainerStarted","Data":"54bc790b108ecb61ebc49afc38d91096c69f3743190b4c29a4e3173b9a805e3c"}
Sep 17 18:25:25.815457 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:25.815374    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5ftl2" event={"ID":"51782ea4-5fef-4f89-b155-c948c5145c1a","Type":"ContainerStarted","Data":"5acc006d95e4e99bf37f355a7a373a8db3b17edc2ca4ea678cb8f2b4af30ffc5"}
Sep 17 18:25:25.817334 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:25.817311    2569 generic.go:356] "Generic (PLEG): container finished" podID="dcbf3e71-59b1-4314-b69f-a71312e253a1" containerID="9ec6f499105345b8b8b1342589b1a649d0120d6d2293beef09b1241c0c8c6952" exitCode=0
Sep 17 18:25:25.817423 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:25.817368    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-fw7lf"
Sep 17 18:25:25.817423 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:25.817393    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-fw7lf" event={"ID":"dcbf3e71-59b1-4314-b69f-a71312e253a1","Type":"ContainerDied","Data":"9ec6f499105345b8b8b1342589b1a649d0120d6d2293beef09b1241c0c8c6952"}
Sep 17 18:25:25.817514 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:25.817445    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-fw7lf" event={"ID":"dcbf3e71-59b1-4314-b69f-a71312e253a1","Type":"ContainerDied","Data":"85105733bf6c81192011ca57cc600e5fe2938d256c967d0171f53b133bdc0532"}
Sep 17 18:25:25.817514 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:25.817471    2569 scope.go:117] "RemoveContainer" containerID="9cb47055d8f6a1930d4af3fc63cd52246aa6148003d0a8b569553d6fe2995c95"
Sep 17 18:25:25.825522 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:25.825506    2569 scope.go:117] "RemoveContainer" containerID="9ec6f499105345b8b8b1342589b1a649d0120d6d2293beef09b1241c0c8c6952"
Sep 17 18:25:25.832625 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:25.832607    2569 scope.go:117] "RemoveContainer" containerID="dcb78144274b7117560840fa54827da106b53485bbdb5eb9019ecf56cd6781c4"
Sep 17 18:25:25.839653 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:25.839632    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p768f\" (UniqueName: \"kubernetes.io/projected/dcbf3e71-59b1-4314-b69f-a71312e253a1-kube-api-access-p768f\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:25:25.839653 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:25.839651    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dcbf3e71-59b1-4314-b69f-a71312e253a1-proxy-tls\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:25:25.839809 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:25.839662    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dcbf3e71-59b1-4314-b69f-a71312e253a1-cabundle-cert\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:25:25.839886 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:25.839829    2569 scope.go:117] "RemoveContainer" containerID="9cb47055d8f6a1930d4af3fc63cd52246aa6148003d0a8b569553d6fe2995c95"
Sep 17 18:25:25.840093 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:25:25.840062    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cb47055d8f6a1930d4af3fc63cd52246aa6148003d0a8b569553d6fe2995c95\": container with ID starting with 9cb47055d8f6a1930d4af3fc63cd52246aa6148003d0a8b569553d6fe2995c95 not found: ID does not exist" containerID="9cb47055d8f6a1930d4af3fc63cd52246aa6148003d0a8b569553d6fe2995c95"
Sep 17 18:25:25.840142 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:25.840095    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cb47055d8f6a1930d4af3fc63cd52246aa6148003d0a8b569553d6fe2995c95"} err="failed to get container status \"9cb47055d8f6a1930d4af3fc63cd52246aa6148003d0a8b569553d6fe2995c95\": rpc error: code = NotFound desc = could not find container \"9cb47055d8f6a1930d4af3fc63cd52246aa6148003d0a8b569553d6fe2995c95\": container with ID starting with 9cb47055d8f6a1930d4af3fc63cd52246aa6148003d0a8b569553d6fe2995c95 not found: ID does not exist"
Sep 17 18:25:25.840142 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:25.840110    2569 scope.go:117] "RemoveContainer" containerID="9ec6f499105345b8b8b1342589b1a649d0120d6d2293beef09b1241c0c8c6952"
Sep 17 18:25:25.840419 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:25:25.840359    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ec6f499105345b8b8b1342589b1a649d0120d6d2293beef09b1241c0c8c6952\": container with ID starting with 9ec6f499105345b8b8b1342589b1a649d0120d6d2293beef09b1241c0c8c6952 not found: ID does not exist" containerID="9ec6f499105345b8b8b1342589b1a649d0120d6d2293beef09b1241c0c8c6952"
Sep 17 18:25:25.840499 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:25.840427    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ec6f499105345b8b8b1342589b1a649d0120d6d2293beef09b1241c0c8c6952"} err="failed to get container status \"9ec6f499105345b8b8b1342589b1a649d0120d6d2293beef09b1241c0c8c6952\": rpc error: code = NotFound desc = could not find container \"9ec6f499105345b8b8b1342589b1a649d0120d6d2293beef09b1241c0c8c6952\": container with ID starting with 9ec6f499105345b8b8b1342589b1a649d0120d6d2293beef09b1241c0c8c6952 not found: ID does not exist"
Sep 17 18:25:25.840499 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:25.840447    2569 scope.go:117] "RemoveContainer" containerID="dcb78144274b7117560840fa54827da106b53485bbdb5eb9019ecf56cd6781c4"
Sep 17 18:25:25.840740 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:25:25.840722    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcb78144274b7117560840fa54827da106b53485bbdb5eb9019ecf56cd6781c4\": container with ID starting with dcb78144274b7117560840fa54827da106b53485bbdb5eb9019ecf56cd6781c4 not found: ID does not exist" containerID="dcb78144274b7117560840fa54827da106b53485bbdb5eb9019ecf56cd6781c4"
Sep 17 18:25:25.840805 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:25.840745    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcb78144274b7117560840fa54827da106b53485bbdb5eb9019ecf56cd6781c4"} err="failed to get container status \"dcb78144274b7117560840fa54827da106b53485bbdb5eb9019ecf56cd6781c4\": rpc error: code = NotFound desc = could not find container \"dcb78144274b7117560840fa54827da106b53485bbdb5eb9019ecf56cd6781c4\": container with ID starting with dcb78144274b7117560840fa54827da106b53485bbdb5eb9019ecf56cd6781c4 not found: ID does not exist"
Sep 17 18:25:25.846060 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:25.846042    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-fw7lf"]
Sep 17 18:25:25.849362 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:25.849343    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-fw7lf"]
Sep 17 18:25:27.362552 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:27.362513    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-m4zmz"]
Sep 17 18:25:27.363032 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:27.362960    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcbf3e71-59b1-4314-b69f-a71312e253a1" containerName="kube-rbac-proxy"
Sep 17 18:25:27.363032 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:27.362979    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcbf3e71-59b1-4314-b69f-a71312e253a1" containerName="kube-rbac-proxy"
Sep 17 18:25:27.363032 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:27.362999    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcbf3e71-59b1-4314-b69f-a71312e253a1" containerName="storage-initializer"
Sep 17 18:25:27.363032 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:27.363007    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcbf3e71-59b1-4314-b69f-a71312e253a1" containerName="storage-initializer"
Sep 17 18:25:27.363032 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:27.363017    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcbf3e71-59b1-4314-b69f-a71312e253a1" containerName="kserve-container"
Sep 17 18:25:27.363032 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:27.363026    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcbf3e71-59b1-4314-b69f-a71312e253a1" containerName="kserve-container"
Sep 17 18:25:27.363409 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:27.363124    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcbf3e71-59b1-4314-b69f-a71312e253a1" containerName="kube-rbac-proxy"
Sep 17 18:25:27.363409 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:27.363139    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcbf3e71-59b1-4314-b69f-a71312e253a1" containerName="kserve-container"
Sep 17 18:25:27.367928 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:27.367904    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-m4zmz"
Sep 17 18:25:27.370539 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:27.370517    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-predictor-serving-cert\""
Sep 17 18:25:27.370657 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:27.370552    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-kube-rbac-proxy-sar-config\""
Sep 17 18:25:27.370737 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:27.370712    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 17 18:25:27.381972 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:27.379670    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-m4zmz"]
Sep 17 18:25:27.452914 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:27.452886    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dacdd83c-048b-4bb2-9c18-f914abd804c5-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-64bc8d7b4d-m4zmz\" (UID: \"dacdd83c-048b-4bb2-9c18-f914abd804c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-m4zmz"
Sep 17 18:25:27.453010 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:27.452925    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dacdd83c-048b-4bb2-9c18-f914abd804c5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-64bc8d7b4d-m4zmz\" (UID: \"dacdd83c-048b-4bb2-9c18-f914abd804c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-m4zmz"
Sep 17 18:25:27.453010 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:27.452947    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dacdd83c-048b-4bb2-9c18-f914abd804c5-proxy-tls\") pod \"isvc-sklearn-s3-predictor-64bc8d7b4d-m4zmz\" (UID: \"dacdd83c-048b-4bb2-9c18-f914abd804c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-m4zmz"
Sep 17 18:25:27.453155 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:27.453017    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5fg6\" (UniqueName: \"kubernetes.io/projected/dacdd83c-048b-4bb2-9c18-f914abd804c5-kube-api-access-t5fg6\") pod \"isvc-sklearn-s3-predictor-64bc8d7b4d-m4zmz\" (UID: \"dacdd83c-048b-4bb2-9c18-f914abd804c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-m4zmz"
Sep 17 18:25:27.507791 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:27.507756    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dcbf3e71-59b1-4314-b69f-a71312e253a1" path="/var/lib/kubelet/pods/dcbf3e71-59b1-4314-b69f-a71312e253a1/volumes"
Sep 17 18:25:27.553589 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:27.553566    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dacdd83c-048b-4bb2-9c18-f914abd804c5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-64bc8d7b4d-m4zmz\" (UID: \"dacdd83c-048b-4bb2-9c18-f914abd804c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-m4zmz"
Sep 17 18:25:27.553708 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:27.553602    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dacdd83c-048b-4bb2-9c18-f914abd804c5-proxy-tls\") pod \"isvc-sklearn-s3-predictor-64bc8d7b4d-m4zmz\" (UID: \"dacdd83c-048b-4bb2-9c18-f914abd804c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-m4zmz"
Sep 17 18:25:27.553708 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:27.553638    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5fg6\" (UniqueName: \"kubernetes.io/projected/dacdd83c-048b-4bb2-9c18-f914abd804c5-kube-api-access-t5fg6\") pod \"isvc-sklearn-s3-predictor-64bc8d7b4d-m4zmz\" (UID: \"dacdd83c-048b-4bb2-9c18-f914abd804c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-m4zmz"
Sep 17 18:25:27.553828 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:27.553716    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dacdd83c-048b-4bb2-9c18-f914abd804c5-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-64bc8d7b4d-m4zmz\" (UID: \"dacdd83c-048b-4bb2-9c18-f914abd804c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-m4zmz"
Sep 17 18:25:27.553828 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:25:27.553729    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-predictor-serving-cert: secret "isvc-sklearn-s3-predictor-serving-cert" not found
Sep 17 18:25:27.553828 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:25:27.553797    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dacdd83c-048b-4bb2-9c18-f914abd804c5-proxy-tls podName:dacdd83c-048b-4bb2-9c18-f914abd804c5 nodeName:}" failed. No retries permitted until 2026-09-17 18:25:28.053776492 +0000 UTC m=+2680.557779097 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/dacdd83c-048b-4bb2-9c18-f914abd804c5-proxy-tls") pod "isvc-sklearn-s3-predictor-64bc8d7b4d-m4zmz" (UID: "dacdd83c-048b-4bb2-9c18-f914abd804c5") : secret "isvc-sklearn-s3-predictor-serving-cert" not found
Sep 17 18:25:27.554099 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:27.554056    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dacdd83c-048b-4bb2-9c18-f914abd804c5-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-64bc8d7b4d-m4zmz\" (UID: \"dacdd83c-048b-4bb2-9c18-f914abd804c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-m4zmz"
Sep 17 18:25:27.554239 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:27.554221    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dacdd83c-048b-4bb2-9c18-f914abd804c5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-64bc8d7b4d-m4zmz\" (UID: \"dacdd83c-048b-4bb2-9c18-f914abd804c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-m4zmz"
Sep 17 18:25:27.561944 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:27.561927    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5fg6\" (UniqueName: \"kubernetes.io/projected/dacdd83c-048b-4bb2-9c18-f914abd804c5-kube-api-access-t5fg6\") pod \"isvc-sklearn-s3-predictor-64bc8d7b4d-m4zmz\" (UID: \"dacdd83c-048b-4bb2-9c18-f914abd804c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-m4zmz"
Sep 17 18:25:28.057069 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:28.057031    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dacdd83c-048b-4bb2-9c18-f914abd804c5-proxy-tls\") pod \"isvc-sklearn-s3-predictor-64bc8d7b4d-m4zmz\" (UID: \"dacdd83c-048b-4bb2-9c18-f914abd804c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-m4zmz"
Sep 17 18:25:28.059591 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:28.059563    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dacdd83c-048b-4bb2-9c18-f914abd804c5-proxy-tls\") pod \"isvc-sklearn-s3-predictor-64bc8d7b4d-m4zmz\" (UID: \"dacdd83c-048b-4bb2-9c18-f914abd804c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-m4zmz"
Sep 17 18:25:28.278328 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:28.278301    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-m4zmz"
Sep 17 18:25:28.553918 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:28.553895    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-m4zmz"]
Sep 17 18:25:28.556583 ip-10-0-131-14 kubenswrapper[2569]: W0917 18:25:28.556556    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddacdd83c_048b_4bb2_9c18_f914abd804c5.slice/crio-3afa01ed5d3ed79d77f79307b1a30854878d71957e6fd4502a526b887ecbb284 WatchSource:0}: Error finding container 3afa01ed5d3ed79d77f79307b1a30854878d71957e6fd4502a526b887ecbb284: Status 404 returned error can't find the container with id 3afa01ed5d3ed79d77f79307b1a30854878d71957e6fd4502a526b887ecbb284
Sep 17 18:25:28.833875 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:28.833843    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-m4zmz" event={"ID":"dacdd83c-048b-4bb2-9c18-f914abd804c5","Type":"ContainerStarted","Data":"f58800afb66ce96693cf69583cac5e08f5ef18f72c37ea3e3211c96faae79f5f"}
Sep 17 18:25:28.833997 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:28.833903    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-m4zmz" event={"ID":"dacdd83c-048b-4bb2-9c18-f914abd804c5","Type":"ContainerStarted","Data":"3afa01ed5d3ed79d77f79307b1a30854878d71957e6fd4502a526b887ecbb284"}
Sep 17 18:25:28.835353 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:28.835331    2569 generic.go:356] "Generic (PLEG): container finished" podID="51782ea4-5fef-4f89-b155-c948c5145c1a" containerID="54bc790b108ecb61ebc49afc38d91096c69f3743190b4c29a4e3173b9a805e3c" exitCode=0
Sep 17 18:25:28.835436 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:28.835400    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5ftl2" event={"ID":"51782ea4-5fef-4f89-b155-c948c5145c1a","Type":"ContainerDied","Data":"54bc790b108ecb61ebc49afc38d91096c69f3743190b4c29a4e3173b9a805e3c"}
Sep 17 18:25:29.843050 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:29.843023    2569 generic.go:356] "Generic (PLEG): container finished" podID="dacdd83c-048b-4bb2-9c18-f914abd804c5" containerID="f58800afb66ce96693cf69583cac5e08f5ef18f72c37ea3e3211c96faae79f5f" exitCode=0
Sep 17 18:25:29.843430 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:29.843113    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-m4zmz" event={"ID":"dacdd83c-048b-4bb2-9c18-f914abd804c5","Type":"ContainerDied","Data":"f58800afb66ce96693cf69583cac5e08f5ef18f72c37ea3e3211c96faae79f5f"}
Sep 17 18:25:29.845614 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:29.845465    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5ftl2" event={"ID":"51782ea4-5fef-4f89-b155-c948c5145c1a","Type":"ContainerStarted","Data":"85b24be7294648e9fc3b5d6b97c233a7745d289130eeec7ecc4dbc1e73571bf9"}
Sep 17 18:25:29.845614 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:29.845527    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5ftl2" event={"ID":"51782ea4-5fef-4f89-b155-c948c5145c1a","Type":"ContainerStarted","Data":"03cc2b68647b96249d4d21a6152a74396dd1675ea4b6b682f876d81f1e15651d"}
Sep 17 18:25:29.845825 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:29.845806    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5ftl2"
Sep 17 18:25:29.879689 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:29.879651    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5ftl2" podStartSLOduration=2.00970035 podStartE2EDuration="5.879637652s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.869937302s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:25:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:25:29.879486633 +0000 UTC m=+2682.383489255" watchObservedRunningTime="2026-09-17 18:25:29.879637652 +0000 UTC m=+2682.383640276"
Sep 17 18:25:30.853408 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:30.853336    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-m4zmz" event={"ID":"dacdd83c-048b-4bb2-9c18-f914abd804c5","Type":"ContainerStarted","Data":"281b051a405cece893755a182ea1cdbbb06db62d133eda7c3ebec9c7d1141389"}
Sep 17 18:25:30.853818 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:30.853415    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-m4zmz" event={"ID":"dacdd83c-048b-4bb2-9c18-f914abd804c5","Type":"ContainerStarted","Data":"7a29682059d7bf540df76e1ecd0011fa71546f1cbbfdac90073f918ec77a1e80"}
Sep 17 18:25:30.853818 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:30.853697    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5ftl2"
Sep 17 18:25:30.855006 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:30.854978    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5ftl2" podUID="51782ea4-5fef-4f89-b155-c948c5145c1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 17 18:25:30.872647 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:30.872597    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-m4zmz" podStartSLOduration=2.741127948 podStartE2EDuration="3.872582704s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.131454756s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:25:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:25:30.871779827 +0000 UTC m=+2683.375782455" watchObservedRunningTime="2026-09-17 18:25:30.872582704 +0000 UTC m=+2683.376585329"
Sep 17 18:25:31.858637 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:31.858603    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5ftl2" podUID="51782ea4-5fef-4f89-b155-c948c5145c1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 17 18:25:31.859176 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:31.858756    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-m4zmz"
Sep 17 18:25:31.859176 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:31.858816    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-m4zmz"
Sep 17 18:25:31.859498 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:31.859478    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-m4zmz" podUID="dacdd83c-048b-4bb2-9c18-f914abd804c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 17 18:25:32.863738 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:32.863701    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-m4zmz" podUID="dacdd83c-048b-4bb2-9c18-f914abd804c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 17 18:25:36.863484 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:36.863458    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5ftl2"
Sep 17 18:25:36.863950 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:36.863926    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5ftl2" podUID="51782ea4-5fef-4f89-b155-c948c5145c1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 17 18:25:37.868966 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:37.868937    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-m4zmz"
Sep 17 18:25:37.869436 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:37.869412    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-m4zmz" podUID="dacdd83c-048b-4bb2-9c18-f914abd804c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 17 18:25:46.865239 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:46.865212    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5ftl2"
Sep 17 18:25:47.869520 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:47.869485    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-m4zmz" podUID="dacdd83c-048b-4bb2-9c18-f914abd804c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 17 18:25:49.667498 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:49.667473    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-t7pqb_b8d76e28-4655-4d21-98d8-0324e4756aaf/console-operator/2.log"
Sep 17 18:25:49.671118 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:49.671095    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-t7pqb_b8d76e28-4655-4d21-98d8-0324e4756aaf/console-operator/2.log"
Sep 17 18:25:49.672749 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:49.672733    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k4mqn_6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff/ovn-acl-logging/0.log"
Sep 17 18:25:49.676651 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:49.676635    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k4mqn_6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff/ovn-acl-logging/0.log"
Sep 17 18:25:49.777665 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:49.777642    2569 scope.go:117] "RemoveContainer" containerID="eaed17ccd88d0a4511729bcb11ecfe227e08e4794de4687d8523a6abb0acbbeb"
Sep 17 18:25:49.785313 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:49.785295    2569 scope.go:117] "RemoveContainer" containerID="b98aba38fc4712ab7aeb5a41d9c93985fd6365cd5318c6aa013ea1c72bceda7a"
Sep 17 18:25:49.792335 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:49.792320    2569 scope.go:117] "RemoveContainer" containerID="3e6bc48a71420810dc8b4bd394c08ab864c8447008778cdcd9ab1d1813d48988"
Sep 17 18:25:49.799197 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:49.799182    2569 scope.go:117] "RemoveContainer" containerID="535d74f12a86e6a47eeae80c328d79bdde951c7a824f84a2883714f9883e532b"
Sep 17 18:25:49.805796 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:49.805784    2569 scope.go:117] "RemoveContainer" containerID="82e54b02948a4048dc1c69e68489fd19df719c6d83fd0404a5d216a333189529"
Sep 17 18:25:49.812384 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:49.812368    2569 scope.go:117] "RemoveContainer" containerID="fd499b5d3e80ca664ed8ee7ec8956f81ce82c7a7d39e2802bee75394590fe0d3"
Sep 17 18:25:54.402224 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:54.402194    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5ftl2"]
Sep 17 18:25:54.402612 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:54.402489    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5ftl2" podUID="51782ea4-5fef-4f89-b155-c948c5145c1a" containerName="kserve-container" containerID="cri-o://03cc2b68647b96249d4d21a6152a74396dd1675ea4b6b682f876d81f1e15651d" gracePeriod=30
Sep 17 18:25:54.402612 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:54.402531    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5ftl2" podUID="51782ea4-5fef-4f89-b155-c948c5145c1a" containerName="kube-rbac-proxy" containerID="cri-o://85b24be7294648e9fc3b5d6b97c233a7745d289130eeec7ecc4dbc1e73571bf9" gracePeriod=30
Sep 17 18:25:54.963649 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:54.963621    2569 generic.go:356] "Generic (PLEG): container finished" podID="51782ea4-5fef-4f89-b155-c948c5145c1a" containerID="85b24be7294648e9fc3b5d6b97c233a7745d289130eeec7ecc4dbc1e73571bf9" exitCode=2
Sep 17 18:25:54.963649 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:54.963644    2569 generic.go:356] "Generic (PLEG): container finished" podID="51782ea4-5fef-4f89-b155-c948c5145c1a" containerID="03cc2b68647b96249d4d21a6152a74396dd1675ea4b6b682f876d81f1e15651d" exitCode=0
Sep 17 18:25:54.963822 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:54.963687    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5ftl2" event={"ID":"51782ea4-5fef-4f89-b155-c948c5145c1a","Type":"ContainerDied","Data":"85b24be7294648e9fc3b5d6b97c233a7745d289130eeec7ecc4dbc1e73571bf9"}
Sep 17 18:25:54.963822 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:54.963743    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5ftl2" event={"ID":"51782ea4-5fef-4f89-b155-c948c5145c1a","Type":"ContainerDied","Data":"03cc2b68647b96249d4d21a6152a74396dd1675ea4b6b682f876d81f1e15651d"}
Sep 17 18:25:55.040198 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:55.040177    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5ftl2"
Sep 17 18:25:55.157887 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:55.157832    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51782ea4-5fef-4f89-b155-c948c5145c1a-kube-rbac-proxy-sar-config\") pod \"51782ea4-5fef-4f89-b155-c948c5145c1a\" (UID: \"51782ea4-5fef-4f89-b155-c948c5145c1a\") "
Sep 17 18:25:55.157992 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:55.157885    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pnh8k\" (UniqueName: \"kubernetes.io/projected/51782ea4-5fef-4f89-b155-c948c5145c1a-kube-api-access-pnh8k\") pod \"51782ea4-5fef-4f89-b155-c948c5145c1a\" (UID: \"51782ea4-5fef-4f89-b155-c948c5145c1a\") "
Sep 17 18:25:55.157992 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:55.157933    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51782ea4-5fef-4f89-b155-c948c5145c1a-kserve-provision-location\") pod \"51782ea4-5fef-4f89-b155-c948c5145c1a\" (UID: \"51782ea4-5fef-4f89-b155-c948c5145c1a\") "
Sep 17 18:25:55.157992 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:55.157974    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51782ea4-5fef-4f89-b155-c948c5145c1a-proxy-tls\") pod \"51782ea4-5fef-4f89-b155-c948c5145c1a\" (UID: \"51782ea4-5fef-4f89-b155-c948c5145c1a\") "
Sep 17 18:25:55.158156 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:55.158005    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51782ea4-5fef-4f89-b155-c948c5145c1a-cabundle-cert\") pod \"51782ea4-5fef-4f89-b155-c948c5145c1a\" (UID: \"51782ea4-5fef-4f89-b155-c948c5145c1a\") "
Sep 17 18:25:55.158252 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:55.158228    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51782ea4-5fef-4f89-b155-c948c5145c1a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "51782ea4-5fef-4f89-b155-c948c5145c1a" (UID: "51782ea4-5fef-4f89-b155-c948c5145c1a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:25:55.158519 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:55.158309    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51782ea4-5fef-4f89-b155-c948c5145c1a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "51782ea4-5fef-4f89-b155-c948c5145c1a" (UID: "51782ea4-5fef-4f89-b155-c948c5145c1a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:25:55.158519 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:55.158400    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51782ea4-5fef-4f89-b155-c948c5145c1a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "51782ea4-5fef-4f89-b155-c948c5145c1a" (UID: "51782ea4-5fef-4f89-b155-c948c5145c1a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:25:55.159961 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:55.159943    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51782ea4-5fef-4f89-b155-c948c5145c1a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "51782ea4-5fef-4f89-b155-c948c5145c1a" (UID: "51782ea4-5fef-4f89-b155-c948c5145c1a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:25:55.160022 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:55.159994    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51782ea4-5fef-4f89-b155-c948c5145c1a-kube-api-access-pnh8k" (OuterVolumeSpecName: "kube-api-access-pnh8k") pod "51782ea4-5fef-4f89-b155-c948c5145c1a" (UID: "51782ea4-5fef-4f89-b155-c948c5145c1a"). InnerVolumeSpecName "kube-api-access-pnh8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:25:55.259558 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:55.259536    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51782ea4-5fef-4f89-b155-c948c5145c1a-proxy-tls\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:25:55.259558 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:55.259556    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51782ea4-5fef-4f89-b155-c948c5145c1a-cabundle-cert\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:25:55.259681 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:55.259566    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51782ea4-5fef-4f89-b155-c948c5145c1a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:25:55.259681 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:55.259574    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnh8k\" (UniqueName: \"kubernetes.io/projected/51782ea4-5fef-4f89-b155-c948c5145c1a-kube-api-access-pnh8k\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:25:55.259681 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:55.259583    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51782ea4-5fef-4f89-b155-c948c5145c1a-kserve-provision-location\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:25:55.970203 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:55.970184    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5ftl2"
Sep 17 18:25:55.970541 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:55.970177    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5ftl2" event={"ID":"51782ea4-5fef-4f89-b155-c948c5145c1a","Type":"ContainerDied","Data":"5acc006d95e4e99bf37f355a7a373a8db3b17edc2ca4ea678cb8f2b4af30ffc5"}
Sep 17 18:25:55.970541 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:55.970335    2569 scope.go:117] "RemoveContainer" containerID="85b24be7294648e9fc3b5d6b97c233a7745d289130eeec7ecc4dbc1e73571bf9"
Sep 17 18:25:55.978431 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:55.978404    2569 scope.go:117] "RemoveContainer" containerID="03cc2b68647b96249d4d21a6152a74396dd1675ea4b6b682f876d81f1e15651d"
Sep 17 18:25:55.985426 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:55.985409    2569 scope.go:117] "RemoveContainer" containerID="54bc790b108ecb61ebc49afc38d91096c69f3743190b4c29a4e3173b9a805e3c"
Sep 17 18:25:55.989373 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:55.989350    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5ftl2"]
Sep 17 18:25:55.996491 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:55.996464    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-5ftl2"]
Sep 17 18:25:57.508730 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:57.508694    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51782ea4-5fef-4f89-b155-c948c5145c1a" path="/var/lib/kubelet/pods/51782ea4-5fef-4f89-b155-c948c5145c1a/volumes"
Sep 17 18:25:57.869617 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:57.869542    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-m4zmz" podUID="dacdd83c-048b-4bb2-9c18-f914abd804c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 17 18:25:59.574409 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:59.574378    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-qcsxp"]
Sep 17 18:25:59.574861 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:59.574791    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51782ea4-5fef-4f89-b155-c948c5145c1a" containerName="storage-initializer"
Sep 17 18:25:59.574861 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:59.574807    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="51782ea4-5fef-4f89-b155-c948c5145c1a" containerName="storage-initializer"
Sep 17 18:25:59.574861 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:59.574833    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51782ea4-5fef-4f89-b155-c948c5145c1a" containerName="kserve-container"
Sep 17 18:25:59.574861 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:59.574840    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="51782ea4-5fef-4f89-b155-c948c5145c1a" containerName="kserve-container"
Sep 17 18:25:59.574861 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:59.574859    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51782ea4-5fef-4f89-b155-c948c5145c1a" containerName="kube-rbac-proxy"
Sep 17 18:25:59.575172 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:59.574868    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="51782ea4-5fef-4f89-b155-c948c5145c1a" containerName="kube-rbac-proxy"
Sep 17 18:25:59.575172 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:59.574943    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="51782ea4-5fef-4f89-b155-c948c5145c1a" containerName="kserve-container"
Sep 17 18:25:59.575172 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:59.574955    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="51782ea4-5fef-4f89-b155-c948c5145c1a" containerName="kube-rbac-proxy"
Sep 17 18:25:59.579441 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:59.579421    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-qcsxp"
Sep 17 18:25:59.582059 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:59.582039    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:25:59.582209 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:59.582151    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 17 18:25:59.582209 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:59.582152    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-predictor-serving-cert\""
Sep 17 18:25:59.582209 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:59.582205    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:25:59.588950 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:59.588930    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-qcsxp"]
Sep 17 18:25:59.688278 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:59.688251    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/227bd8d1-8489-49f6-b586-896a7e8f60eb-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-qcsxp\" (UID: \"227bd8d1-8489-49f6-b586-896a7e8f60eb\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-qcsxp"
Sep 17 18:25:59.688407 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:59.688308    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/227bd8d1-8489-49f6-b586-896a7e8f60eb-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-qcsxp\" (UID: \"227bd8d1-8489-49f6-b586-896a7e8f60eb\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-qcsxp"
Sep 17 18:25:59.688407 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:59.688386    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/227bd8d1-8489-49f6-b586-896a7e8f60eb-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-qcsxp\" (UID: \"227bd8d1-8489-49f6-b586-896a7e8f60eb\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-qcsxp"
Sep 17 18:25:59.688534 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:59.688423    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/227bd8d1-8489-49f6-b586-896a7e8f60eb-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-qcsxp\" (UID: \"227bd8d1-8489-49f6-b586-896a7e8f60eb\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-qcsxp"
Sep 17 18:25:59.688534 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:59.688468    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br7hw\" (UniqueName: \"kubernetes.io/projected/227bd8d1-8489-49f6-b586-896a7e8f60eb-kube-api-access-br7hw\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-qcsxp\" (UID: \"227bd8d1-8489-49f6-b586-896a7e8f60eb\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-qcsxp"
Sep 17 18:25:59.789664 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:59.789637    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/227bd8d1-8489-49f6-b586-896a7e8f60eb-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-qcsxp\" (UID: \"227bd8d1-8489-49f6-b586-896a7e8f60eb\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-qcsxp"
Sep 17 18:25:59.789779 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:59.789672    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/227bd8d1-8489-49f6-b586-896a7e8f60eb-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-qcsxp\" (UID: \"227bd8d1-8489-49f6-b586-896a7e8f60eb\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-qcsxp"
Sep 17 18:25:59.789779 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:59.789703    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br7hw\" (UniqueName: \"kubernetes.io/projected/227bd8d1-8489-49f6-b586-896a7e8f60eb-kube-api-access-br7hw\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-qcsxp\" (UID: \"227bd8d1-8489-49f6-b586-896a7e8f60eb\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-qcsxp"
Sep 17 18:25:59.789779 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:59.789749    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/227bd8d1-8489-49f6-b586-896a7e8f60eb-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-qcsxp\" (UID: \"227bd8d1-8489-49f6-b586-896a7e8f60eb\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-qcsxp"
Sep 17 18:25:59.789779 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:59.789769    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/227bd8d1-8489-49f6-b586-896a7e8f60eb-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-qcsxp\" (UID: \"227bd8d1-8489-49f6-b586-896a7e8f60eb\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-qcsxp"
Sep 17 18:25:59.789992 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:25:59.789876    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-serving-cert: secret "isvc-mlflow-v2-runtime-predictor-serving-cert" not found
Sep 17 18:25:59.789992 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:25:59.789930    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/227bd8d1-8489-49f6-b586-896a7e8f60eb-proxy-tls podName:227bd8d1-8489-49f6-b586-896a7e8f60eb nodeName:}" failed. No retries permitted until 2026-09-17 18:26:00.289911852 +0000 UTC m=+2712.793914454 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/227bd8d1-8489-49f6-b586-896a7e8f60eb-proxy-tls") pod "isvc-mlflow-v2-runtime-predictor-6456f94494-qcsxp" (UID: "227bd8d1-8489-49f6-b586-896a7e8f60eb") : secret "isvc-mlflow-v2-runtime-predictor-serving-cert" not found
Sep 17 18:25:59.790209 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:59.790188    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/227bd8d1-8489-49f6-b586-896a7e8f60eb-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-qcsxp\" (UID: \"227bd8d1-8489-49f6-b586-896a7e8f60eb\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-qcsxp"
Sep 17 18:25:59.790380 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:59.790362    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/227bd8d1-8489-49f6-b586-896a7e8f60eb-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-qcsxp\" (UID: \"227bd8d1-8489-49f6-b586-896a7e8f60eb\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-qcsxp"
Sep 17 18:25:59.790461 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:59.790441    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/227bd8d1-8489-49f6-b586-896a7e8f60eb-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-qcsxp\" (UID: \"227bd8d1-8489-49f6-b586-896a7e8f60eb\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-qcsxp"
Sep 17 18:25:59.797988 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:25:59.797970    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-br7hw\" (UniqueName: \"kubernetes.io/projected/227bd8d1-8489-49f6-b586-896a7e8f60eb-kube-api-access-br7hw\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-qcsxp\" (UID: \"227bd8d1-8489-49f6-b586-896a7e8f60eb\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-qcsxp"
Sep 17 18:26:00.292042 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:00.291983    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/227bd8d1-8489-49f6-b586-896a7e8f60eb-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-qcsxp\" (UID: \"227bd8d1-8489-49f6-b586-896a7e8f60eb\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-qcsxp"
Sep 17 18:26:00.294247 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:00.294230    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/227bd8d1-8489-49f6-b586-896a7e8f60eb-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-qcsxp\" (UID: \"227bd8d1-8489-49f6-b586-896a7e8f60eb\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-qcsxp"
Sep 17 18:26:00.490759 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:00.490733    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-qcsxp"
Sep 17 18:26:00.763839 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:00.763811    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-qcsxp"]
Sep 17 18:26:00.766106 ip-10-0-131-14 kubenswrapper[2569]: W0917 18:26:00.766061    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod227bd8d1_8489_49f6_b586_896a7e8f60eb.slice/crio-d5fad9303f8ddcbfa71708f33a182ca7a30b4b0f353bff92da5597ee2d4c53b3 WatchSource:0}: Error finding container d5fad9303f8ddcbfa71708f33a182ca7a30b4b0f353bff92da5597ee2d4c53b3: Status 404 returned error can't find the container with id d5fad9303f8ddcbfa71708f33a182ca7a30b4b0f353bff92da5597ee2d4c53b3
Sep 17 18:26:00.991036 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:00.991001    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-qcsxp" event={"ID":"227bd8d1-8489-49f6-b586-896a7e8f60eb","Type":"ContainerStarted","Data":"f1eb15de8575fbebe5d048b4fc4be95fc7873b579ffe67e5e0d79455e35c8c6d"}
Sep 17 18:26:00.991228 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:00.991070    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-qcsxp" event={"ID":"227bd8d1-8489-49f6-b586-896a7e8f60eb","Type":"ContainerStarted","Data":"d5fad9303f8ddcbfa71708f33a182ca7a30b4b0f353bff92da5597ee2d4c53b3"}
Sep 17 18:26:05.013535 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:05.013503    2569 generic.go:356] "Generic (PLEG): container finished" podID="227bd8d1-8489-49f6-b586-896a7e8f60eb" containerID="f1eb15de8575fbebe5d048b4fc4be95fc7873b579ffe67e5e0d79455e35c8c6d" exitCode=0
Sep 17 18:26:05.013936 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:05.013547    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-qcsxp" event={"ID":"227bd8d1-8489-49f6-b586-896a7e8f60eb","Type":"ContainerDied","Data":"f1eb15de8575fbebe5d048b4fc4be95fc7873b579ffe67e5e0d79455e35c8c6d"}
Sep 17 18:26:06.019810 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:06.019776    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-qcsxp" event={"ID":"227bd8d1-8489-49f6-b586-896a7e8f60eb","Type":"ContainerStarted","Data":"e372ea41e3748794a7c396ff9cab88ee35bf786e4c18a093ad84a6b845ee27b2"}
Sep 17 18:26:06.020178 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:06.019846    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-qcsxp" event={"ID":"227bd8d1-8489-49f6-b586-896a7e8f60eb","Type":"ContainerStarted","Data":"e7710edb14efebf3fd17e0547d9f8b10492c90ab2dc59f12fb4e388792dff08a"}
Sep 17 18:26:06.020178 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:06.020041    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-qcsxp"
Sep 17 18:26:06.041490 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:06.041440    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-qcsxp" podStartSLOduration=3.039396546 podStartE2EDuration="7.04142072s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.002024174s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:25:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:26:06.03998234 +0000 UTC m=+2718.543984965" watchObservedRunningTime="2026-09-17 18:26:06.04142072 +0000 UTC m=+2718.545423346"
Sep 17 18:26:07.024867 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:07.024834    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-qcsxp"
Sep 17 18:26:07.869654 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:07.869609    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-m4zmz" podUID="dacdd83c-048b-4bb2-9c18-f914abd804c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 17 18:26:13.035109 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:13.035059    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-qcsxp"
Sep 17 18:26:17.869382 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:17.869340    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-m4zmz" podUID="dacdd83c-048b-4bb2-9c18-f914abd804c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 17 18:26:27.869838 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:27.869796    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-m4zmz" podUID="dacdd83c-048b-4bb2-9c18-f914abd804c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 17 18:26:37.869759 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:37.869721    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-m4zmz" podUID="dacdd83c-048b-4bb2-9c18-f914abd804c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 17 18:26:43.039294 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:43.039258    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-qcsxp"
Sep 17 18:26:47.870934 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:47.870898    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-m4zmz"
Sep 17 18:26:49.638791 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:49.638761    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-qcsxp"]
Sep 17 18:26:49.639218 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:49.638976    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-qcsxp" podUID="227bd8d1-8489-49f6-b586-896a7e8f60eb" containerName="kserve-container" containerID="cri-o://e7710edb14efebf3fd17e0547d9f8b10492c90ab2dc59f12fb4e388792dff08a" gracePeriod=30
Sep 17 18:26:49.639218 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:49.639018    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-qcsxp" podUID="227bd8d1-8489-49f6-b586-896a7e8f60eb" containerName="kube-rbac-proxy" containerID="cri-o://e372ea41e3748794a7c396ff9cab88ee35bf786e4c18a093ad84a6b845ee27b2" gracePeriod=30
Sep 17 18:26:50.215182 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:50.215120    2569 generic.go:356] "Generic (PLEG): container finished" podID="227bd8d1-8489-49f6-b586-896a7e8f60eb" containerID="e372ea41e3748794a7c396ff9cab88ee35bf786e4c18a093ad84a6b845ee27b2" exitCode=2
Sep 17 18:26:50.215384 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:50.215188    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-qcsxp" event={"ID":"227bd8d1-8489-49f6-b586-896a7e8f60eb","Type":"ContainerDied","Data":"e372ea41e3748794a7c396ff9cab88ee35bf786e4c18a093ad84a6b845ee27b2"}
Sep 17 18:26:50.687441 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:50.687414    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-qcsxp"
Sep 17 18:26:50.758131 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:50.758107    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/227bd8d1-8489-49f6-b586-896a7e8f60eb-proxy-tls\") pod \"227bd8d1-8489-49f6-b586-896a7e8f60eb\" (UID: \"227bd8d1-8489-49f6-b586-896a7e8f60eb\") "
Sep 17 18:26:50.758255 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:50.758167    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/227bd8d1-8489-49f6-b586-896a7e8f60eb-kube-rbac-proxy-sar-config\") pod \"227bd8d1-8489-49f6-b586-896a7e8f60eb\" (UID: \"227bd8d1-8489-49f6-b586-896a7e8f60eb\") "
Sep 17 18:26:50.758255 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:50.758214    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/227bd8d1-8489-49f6-b586-896a7e8f60eb-cabundle-cert\") pod \"227bd8d1-8489-49f6-b586-896a7e8f60eb\" (UID: \"227bd8d1-8489-49f6-b586-896a7e8f60eb\") "
Sep 17 18:26:50.758255 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:50.758243    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-br7hw\" (UniqueName: \"kubernetes.io/projected/227bd8d1-8489-49f6-b586-896a7e8f60eb-kube-api-access-br7hw\") pod \"227bd8d1-8489-49f6-b586-896a7e8f60eb\" (UID: \"227bd8d1-8489-49f6-b586-896a7e8f60eb\") "
Sep 17 18:26:50.758424 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:50.758276    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/227bd8d1-8489-49f6-b586-896a7e8f60eb-kserve-provision-location\") pod \"227bd8d1-8489-49f6-b586-896a7e8f60eb\" (UID: \"227bd8d1-8489-49f6-b586-896a7e8f60eb\") "
Sep 17 18:26:50.758516 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:50.758486    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/227bd8d1-8489-49f6-b586-896a7e8f60eb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "227bd8d1-8489-49f6-b586-896a7e8f60eb" (UID: "227bd8d1-8489-49f6-b586-896a7e8f60eb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:26:50.758634 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:50.758569    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/227bd8d1-8489-49f6-b586-896a7e8f60eb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "227bd8d1-8489-49f6-b586-896a7e8f60eb" (UID: "227bd8d1-8489-49f6-b586-896a7e8f60eb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:26:50.758687 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:50.758638    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/227bd8d1-8489-49f6-b586-896a7e8f60eb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "227bd8d1-8489-49f6-b586-896a7e8f60eb" (UID: "227bd8d1-8489-49f6-b586-896a7e8f60eb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:26:50.760110 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:50.760089    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/227bd8d1-8489-49f6-b586-896a7e8f60eb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "227bd8d1-8489-49f6-b586-896a7e8f60eb" (UID: "227bd8d1-8489-49f6-b586-896a7e8f60eb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:26:50.760251 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:50.760232    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/227bd8d1-8489-49f6-b586-896a7e8f60eb-kube-api-access-br7hw" (OuterVolumeSpecName: "kube-api-access-br7hw") pod "227bd8d1-8489-49f6-b586-896a7e8f60eb" (UID: "227bd8d1-8489-49f6-b586-896a7e8f60eb"). InnerVolumeSpecName "kube-api-access-br7hw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:26:50.859730 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:50.859667    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/227bd8d1-8489-49f6-b586-896a7e8f60eb-proxy-tls\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:26:50.859730 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:50.859701    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/227bd8d1-8489-49f6-b586-896a7e8f60eb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:26:50.859730 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:50.859711    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/227bd8d1-8489-49f6-b586-896a7e8f60eb-cabundle-cert\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:26:50.859730 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:50.859721    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-br7hw\" (UniqueName: \"kubernetes.io/projected/227bd8d1-8489-49f6-b586-896a7e8f60eb-kube-api-access-br7hw\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:26:50.859730 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:50.859730    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/227bd8d1-8489-49f6-b586-896a7e8f60eb-kserve-provision-location\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:26:51.221032 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:51.221004    2569 generic.go:356] "Generic (PLEG): container finished" podID="227bd8d1-8489-49f6-b586-896a7e8f60eb" containerID="e7710edb14efebf3fd17e0547d9f8b10492c90ab2dc59f12fb4e388792dff08a" exitCode=0
Sep 17 18:26:51.221153 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:51.221043    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-qcsxp" event={"ID":"227bd8d1-8489-49f6-b586-896a7e8f60eb","Type":"ContainerDied","Data":"e7710edb14efebf3fd17e0547d9f8b10492c90ab2dc59f12fb4e388792dff08a"}
Sep 17 18:26:51.221153 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:51.221065    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-qcsxp"
Sep 17 18:26:51.221153 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:51.221111    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-qcsxp" event={"ID":"227bd8d1-8489-49f6-b586-896a7e8f60eb","Type":"ContainerDied","Data":"d5fad9303f8ddcbfa71708f33a182ca7a30b4b0f353bff92da5597ee2d4c53b3"}
Sep 17 18:26:51.221153 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:51.221141    2569 scope.go:117] "RemoveContainer" containerID="e372ea41e3748794a7c396ff9cab88ee35bf786e4c18a093ad84a6b845ee27b2"
Sep 17 18:26:51.229857 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:51.229839    2569 scope.go:117] "RemoveContainer" containerID="e7710edb14efebf3fd17e0547d9f8b10492c90ab2dc59f12fb4e388792dff08a"
Sep 17 18:26:51.237676 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:51.237660    2569 scope.go:117] "RemoveContainer" containerID="f1eb15de8575fbebe5d048b4fc4be95fc7873b579ffe67e5e0d79455e35c8c6d"
Sep 17 18:26:51.244107 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:51.244067    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-qcsxp"]
Sep 17 18:26:51.245219 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:51.245199    2569 scope.go:117] "RemoveContainer" containerID="e372ea41e3748794a7c396ff9cab88ee35bf786e4c18a093ad84a6b845ee27b2"
Sep 17 18:26:51.245543 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:26:51.245520    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e372ea41e3748794a7c396ff9cab88ee35bf786e4c18a093ad84a6b845ee27b2\": container with ID starting with e372ea41e3748794a7c396ff9cab88ee35bf786e4c18a093ad84a6b845ee27b2 not found: ID does not exist" containerID="e372ea41e3748794a7c396ff9cab88ee35bf786e4c18a093ad84a6b845ee27b2"
Sep 17 18:26:51.245656 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:51.245551    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e372ea41e3748794a7c396ff9cab88ee35bf786e4c18a093ad84a6b845ee27b2"} err="failed to get container status \"e372ea41e3748794a7c396ff9cab88ee35bf786e4c18a093ad84a6b845ee27b2\": rpc error: code = NotFound desc = could not find container \"e372ea41e3748794a7c396ff9cab88ee35bf786e4c18a093ad84a6b845ee27b2\": container with ID starting with e372ea41e3748794a7c396ff9cab88ee35bf786e4c18a093ad84a6b845ee27b2 not found: ID does not exist"
Sep 17 18:26:51.245656 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:51.245570    2569 scope.go:117] "RemoveContainer" containerID="e7710edb14efebf3fd17e0547d9f8b10492c90ab2dc59f12fb4e388792dff08a"
Sep 17 18:26:51.246623 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:26:51.246012    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7710edb14efebf3fd17e0547d9f8b10492c90ab2dc59f12fb4e388792dff08a\": container with ID starting with e7710edb14efebf3fd17e0547d9f8b10492c90ab2dc59f12fb4e388792dff08a not found: ID does not exist" containerID="e7710edb14efebf3fd17e0547d9f8b10492c90ab2dc59f12fb4e388792dff08a"
Sep 17 18:26:51.246623 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:51.246049    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7710edb14efebf3fd17e0547d9f8b10492c90ab2dc59f12fb4e388792dff08a"} err="failed to get container status \"e7710edb14efebf3fd17e0547d9f8b10492c90ab2dc59f12fb4e388792dff08a\": rpc error: code = NotFound desc = could not find container \"e7710edb14efebf3fd17e0547d9f8b10492c90ab2dc59f12fb4e388792dff08a\": container with ID starting with e7710edb14efebf3fd17e0547d9f8b10492c90ab2dc59f12fb4e388792dff08a not found: ID does not exist"
Sep 17 18:26:51.246623 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:51.246069    2569 scope.go:117] "RemoveContainer" containerID="f1eb15de8575fbebe5d048b4fc4be95fc7873b579ffe67e5e0d79455e35c8c6d"
Sep 17 18:26:51.246936 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:26:51.246919    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1eb15de8575fbebe5d048b4fc4be95fc7873b579ffe67e5e0d79455e35c8c6d\": container with ID starting with f1eb15de8575fbebe5d048b4fc4be95fc7873b579ffe67e5e0d79455e35c8c6d not found: ID does not exist" containerID="f1eb15de8575fbebe5d048b4fc4be95fc7873b579ffe67e5e0d79455e35c8c6d"
Sep 17 18:26:51.246984 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:51.246942    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1eb15de8575fbebe5d048b4fc4be95fc7873b579ffe67e5e0d79455e35c8c6d"} err="failed to get container status \"f1eb15de8575fbebe5d048b4fc4be95fc7873b579ffe67e5e0d79455e35c8c6d\": rpc error: code = NotFound desc = could not find container \"f1eb15de8575fbebe5d048b4fc4be95fc7873b579ffe67e5e0d79455e35c8c6d\": container with ID starting with f1eb15de8575fbebe5d048b4fc4be95fc7873b579ffe67e5e0d79455e35c8c6d not found: ID does not exist"
Sep 17 18:26:51.248052 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:51.248035    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-qcsxp"]
Sep 17 18:26:51.508360 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:51.508283    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="227bd8d1-8489-49f6-b586-896a7e8f60eb" path="/var/lib/kubelet/pods/227bd8d1-8489-49f6-b586-896a7e8f60eb/volumes"
Sep 17 18:26:54.842550 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:54.842518    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-xpfvz"]
Sep 17 18:26:54.842967 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:54.842804    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="227bd8d1-8489-49f6-b586-896a7e8f60eb" containerName="storage-initializer"
Sep 17 18:26:54.842967 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:54.842815    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="227bd8d1-8489-49f6-b586-896a7e8f60eb" containerName="storage-initializer"
Sep 17 18:26:54.842967 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:54.842828    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="227bd8d1-8489-49f6-b586-896a7e8f60eb" containerName="kube-rbac-proxy"
Sep 17 18:26:54.842967 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:54.842833    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="227bd8d1-8489-49f6-b586-896a7e8f60eb" containerName="kube-rbac-proxy"
Sep 17 18:26:54.842967 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:54.842848    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="227bd8d1-8489-49f6-b586-896a7e8f60eb" containerName="kserve-container"
Sep 17 18:26:54.842967 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:54.842853    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="227bd8d1-8489-49f6-b586-896a7e8f60eb" containerName="kserve-container"
Sep 17 18:26:54.842967 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:54.842912    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="227bd8d1-8489-49f6-b586-896a7e8f60eb" containerName="kserve-container"
Sep 17 18:26:54.842967 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:54.842920    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="227bd8d1-8489-49f6-b586-896a7e8f60eb" containerName="kube-rbac-proxy"
Sep 17 18:26:54.847872 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:54.847853    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-xpfvz"
Sep 17 18:26:54.850544 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:54.850515    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:26:54.850694 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:54.850655    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-predictor-serving-cert\""
Sep 17 18:26:54.850694 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:54.850673    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\""
Sep 17 18:26:54.850861 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:54.850677    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:26:54.857128 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:54.857105    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-xpfvz"]
Sep 17 18:26:54.992301 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:54.992277    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2f49aaf-af15-4acc-b0f8-0258e6e794a4-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-67cd675847-xpfvz\" (UID: \"d2f49aaf-af15-4acc-b0f8-0258e6e794a4\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-xpfvz"
Sep 17 18:26:54.992406 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:54.992312    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2f49aaf-af15-4acc-b0f8-0258e6e794a4-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-67cd675847-xpfvz\" (UID: \"d2f49aaf-af15-4acc-b0f8-0258e6e794a4\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-xpfvz"
Sep 17 18:26:54.992406 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:54.992342    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2f49aaf-af15-4acc-b0f8-0258e6e794a4-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-67cd675847-xpfvz\" (UID: \"d2f49aaf-af15-4acc-b0f8-0258e6e794a4\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-xpfvz"
Sep 17 18:26:54.992507 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:54.992405    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsmcx\" (UniqueName: \"kubernetes.io/projected/d2f49aaf-af15-4acc-b0f8-0258e6e794a4-kube-api-access-lsmcx\") pod \"isvc-sklearn-mcp-predictor-67cd675847-xpfvz\" (UID: \"d2f49aaf-af15-4acc-b0f8-0258e6e794a4\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-xpfvz"
Sep 17 18:26:54.992507 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:54.992435    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2f49aaf-af15-4acc-b0f8-0258e6e794a4-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-67cd675847-xpfvz\" (UID: \"d2f49aaf-af15-4acc-b0f8-0258e6e794a4\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-xpfvz"
Sep 17 18:26:55.093409 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:55.093343    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2f49aaf-af15-4acc-b0f8-0258e6e794a4-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-67cd675847-xpfvz\" (UID: \"d2f49aaf-af15-4acc-b0f8-0258e6e794a4\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-xpfvz"
Sep 17 18:26:55.093409 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:55.093380    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2f49aaf-af15-4acc-b0f8-0258e6e794a4-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-67cd675847-xpfvz\" (UID: \"d2f49aaf-af15-4acc-b0f8-0258e6e794a4\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-xpfvz"
Sep 17 18:26:55.093409 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:55.093409    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2f49aaf-af15-4acc-b0f8-0258e6e794a4-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-67cd675847-xpfvz\" (UID: \"d2f49aaf-af15-4acc-b0f8-0258e6e794a4\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-xpfvz"
Sep 17 18:26:55.093575 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:55.093432    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsmcx\" (UniqueName: \"kubernetes.io/projected/d2f49aaf-af15-4acc-b0f8-0258e6e794a4-kube-api-access-lsmcx\") pod \"isvc-sklearn-mcp-predictor-67cd675847-xpfvz\" (UID: \"d2f49aaf-af15-4acc-b0f8-0258e6e794a4\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-xpfvz"
Sep 17 18:26:55.093575 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:55.093489    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2f49aaf-af15-4acc-b0f8-0258e6e794a4-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-67cd675847-xpfvz\" (UID: \"d2f49aaf-af15-4acc-b0f8-0258e6e794a4\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-xpfvz"
Sep 17 18:26:55.093780 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:55.093761    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2f49aaf-af15-4acc-b0f8-0258e6e794a4-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-67cd675847-xpfvz\" (UID: \"d2f49aaf-af15-4acc-b0f8-0258e6e794a4\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-xpfvz"
Sep 17 18:26:55.094263 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:55.094232    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2f49aaf-af15-4acc-b0f8-0258e6e794a4-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-67cd675847-xpfvz\" (UID: \"d2f49aaf-af15-4acc-b0f8-0258e6e794a4\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-xpfvz"
Sep 17 18:26:55.094362 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:55.094232    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2f49aaf-af15-4acc-b0f8-0258e6e794a4-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-67cd675847-xpfvz\" (UID: \"d2f49aaf-af15-4acc-b0f8-0258e6e794a4\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-xpfvz"
Sep 17 18:26:55.096040 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:55.096020    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2f49aaf-af15-4acc-b0f8-0258e6e794a4-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-67cd675847-xpfvz\" (UID: \"d2f49aaf-af15-4acc-b0f8-0258e6e794a4\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-xpfvz"
Sep 17 18:26:55.102833 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:55.102812    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsmcx\" (UniqueName: \"kubernetes.io/projected/d2f49aaf-af15-4acc-b0f8-0258e6e794a4-kube-api-access-lsmcx\") pod \"isvc-sklearn-mcp-predictor-67cd675847-xpfvz\" (UID: \"d2f49aaf-af15-4acc-b0f8-0258e6e794a4\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-xpfvz"
Sep 17 18:26:55.159382 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:55.159353    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-xpfvz"
Sep 17 18:26:55.432290 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:55.432265    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-xpfvz"]
Sep 17 18:26:55.434098 ip-10-0-131-14 kubenswrapper[2569]: W0917 18:26:55.434052    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2f49aaf_af15_4acc_b0f8_0258e6e794a4.slice/crio-77b1c8af569801e589a261792f19e6fcdefcf6c1a100e6715c5489e63588f4f6 WatchSource:0}: Error finding container 77b1c8af569801e589a261792f19e6fcdefcf6c1a100e6715c5489e63588f4f6: Status 404 returned error can't find the container with id 77b1c8af569801e589a261792f19e6fcdefcf6c1a100e6715c5489e63588f4f6
Sep 17 18:26:56.244828 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:56.244790    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-xpfvz" event={"ID":"d2f49aaf-af15-4acc-b0f8-0258e6e794a4","Type":"ContainerStarted","Data":"a8a74c5fbe3aa92245f0204f14567f207024ce7eed8e8cca39507dff5ad99f0f"}
Sep 17 18:26:56.245272 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:56.244866    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-xpfvz" event={"ID":"d2f49aaf-af15-4acc-b0f8-0258e6e794a4","Type":"ContainerStarted","Data":"77b1c8af569801e589a261792f19e6fcdefcf6c1a100e6715c5489e63588f4f6"}
Sep 17 18:26:57.448424 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:57.448393    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-m4zmz"]
Sep 17 18:26:57.448848 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:57.448612    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-m4zmz" podUID="dacdd83c-048b-4bb2-9c18-f914abd804c5" containerName="kserve-container" containerID="cri-o://7a29682059d7bf540df76e1ecd0011fa71546f1cbbfdac90073f918ec77a1e80" gracePeriod=30
Sep 17 18:26:57.448848 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:57.448662    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-m4zmz" podUID="dacdd83c-048b-4bb2-9c18-f914abd804c5" containerName="kube-rbac-proxy" containerID="cri-o://281b051a405cece893755a182ea1cdbbb06db62d133eda7c3ebec9c7d1141389" gracePeriod=30
Sep 17 18:26:57.864045 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:57.863961    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-m4zmz" podUID="dacdd83c-048b-4bb2-9c18-f914abd804c5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.48:8643/healthz\": dial tcp 10.133.0.48:8643: connect: connection refused"
Sep 17 18:26:57.869500 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:57.869475    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-m4zmz" podUID="dacdd83c-048b-4bb2-9c18-f914abd804c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 17 18:26:58.257132 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:58.257100    2569 generic.go:356] "Generic (PLEG): container finished" podID="dacdd83c-048b-4bb2-9c18-f914abd804c5" containerID="281b051a405cece893755a182ea1cdbbb06db62d133eda7c3ebec9c7d1141389" exitCode=2
Sep 17 18:26:58.257284 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:26:58.257165    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-m4zmz" event={"ID":"dacdd83c-048b-4bb2-9c18-f914abd804c5","Type":"ContainerDied","Data":"281b051a405cece893755a182ea1cdbbb06db62d133eda7c3ebec9c7d1141389"}
Sep 17 18:27:00.266983 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:27:00.266901    2569 generic.go:356] "Generic (PLEG): container finished" podID="d2f49aaf-af15-4acc-b0f8-0258e6e794a4" containerID="a8a74c5fbe3aa92245f0204f14567f207024ce7eed8e8cca39507dff5ad99f0f" exitCode=0
Sep 17 18:27:00.267606 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:27:00.267583    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-xpfvz" event={"ID":"d2f49aaf-af15-4acc-b0f8-0258e6e794a4","Type":"ContainerDied","Data":"a8a74c5fbe3aa92245f0204f14567f207024ce7eed8e8cca39507dff5ad99f0f"}
Sep 17 18:27:00.328523 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:27:00.328502    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:27:01.276505 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:27:01.276467    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-xpfvz" event={"ID":"d2f49aaf-af15-4acc-b0f8-0258e6e794a4","Type":"ContainerStarted","Data":"de67a974ceedf6d2098aa582d9a60d2be51725440d2926475e1e77adc97a886d"}
Sep 17 18:27:01.279282 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:27:01.279174    2569 generic.go:356] "Generic (PLEG): container finished" podID="dacdd83c-048b-4bb2-9c18-f914abd804c5" containerID="7a29682059d7bf540df76e1ecd0011fa71546f1cbbfdac90073f918ec77a1e80" exitCode=0
Sep 17 18:27:01.279282 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:27:01.279220    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-m4zmz" event={"ID":"dacdd83c-048b-4bb2-9c18-f914abd804c5","Type":"ContainerDied","Data":"7a29682059d7bf540df76e1ecd0011fa71546f1cbbfdac90073f918ec77a1e80"}
Sep 17 18:27:01.326627 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:27:01.326588    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-m4zmz"
Sep 17 18:27:01.442802 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:27:01.442000    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dacdd83c-048b-4bb2-9c18-f914abd804c5-proxy-tls\") pod \"dacdd83c-048b-4bb2-9c18-f914abd804c5\" (UID: \"dacdd83c-048b-4bb2-9c18-f914abd804c5\") "
Sep 17 18:27:01.442802 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:27:01.442116    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dacdd83c-048b-4bb2-9c18-f914abd804c5-kube-rbac-proxy-sar-config\") pod \"dacdd83c-048b-4bb2-9c18-f914abd804c5\" (UID: \"dacdd83c-048b-4bb2-9c18-f914abd804c5\") "
Sep 17 18:27:01.442802 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:27:01.442160    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t5fg6\" (UniqueName: \"kubernetes.io/projected/dacdd83c-048b-4bb2-9c18-f914abd804c5-kube-api-access-t5fg6\") pod \"dacdd83c-048b-4bb2-9c18-f914abd804c5\" (UID: \"dacdd83c-048b-4bb2-9c18-f914abd804c5\") "
Sep 17 18:27:01.442802 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:27:01.442206    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dacdd83c-048b-4bb2-9c18-f914abd804c5-kserve-provision-location\") pod \"dacdd83c-048b-4bb2-9c18-f914abd804c5\" (UID: \"dacdd83c-048b-4bb2-9c18-f914abd804c5\") "
Sep 17 18:27:01.443601 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:27:01.443228    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dacdd83c-048b-4bb2-9c18-f914abd804c5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dacdd83c-048b-4bb2-9c18-f914abd804c5" (UID: "dacdd83c-048b-4bb2-9c18-f914abd804c5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:27:01.443601 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:27:01.443568    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dacdd83c-048b-4bb2-9c18-f914abd804c5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dacdd83c-048b-4bb2-9c18-f914abd804c5" (UID: "dacdd83c-048b-4bb2-9c18-f914abd804c5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:27:01.446032 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:27:01.445998    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dacdd83c-048b-4bb2-9c18-f914abd804c5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dacdd83c-048b-4bb2-9c18-f914abd804c5" (UID: "dacdd83c-048b-4bb2-9c18-f914abd804c5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:27:01.446221 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:27:01.446194    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dacdd83c-048b-4bb2-9c18-f914abd804c5-kube-api-access-t5fg6" (OuterVolumeSpecName: "kube-api-access-t5fg6") pod "dacdd83c-048b-4bb2-9c18-f914abd804c5" (UID: "dacdd83c-048b-4bb2-9c18-f914abd804c5"). InnerVolumeSpecName "kube-api-access-t5fg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:27:01.542997 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:27:01.542965    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dacdd83c-048b-4bb2-9c18-f914abd804c5-proxy-tls\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:27:01.542997 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:27:01.542995    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dacdd83c-048b-4bb2-9c18-f914abd804c5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:27:01.543186 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:27:01.543010    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5fg6\" (UniqueName: \"kubernetes.io/projected/dacdd83c-048b-4bb2-9c18-f914abd804c5-kube-api-access-t5fg6\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:27:01.543186 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:27:01.543023    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dacdd83c-048b-4bb2-9c18-f914abd804c5-kserve-provision-location\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:27:02.286382 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:27:02.286309    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-m4zmz" event={"ID":"dacdd83c-048b-4bb2-9c18-f914abd804c5","Type":"ContainerDied","Data":"3afa01ed5d3ed79d77f79307b1a30854878d71957e6fd4502a526b887ecbb284"}
Sep 17 18:27:02.286382 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:27:02.286332    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-m4zmz"
Sep 17 18:27:02.286382 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:27:02.286370    2569 scope.go:117] "RemoveContainer" containerID="281b051a405cece893755a182ea1cdbbb06db62d133eda7c3ebec9c7d1141389"
Sep 17 18:27:02.305666 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:27:02.305644    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-m4zmz"]
Sep 17 18:27:02.307979 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:27:02.307959    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-64bc8d7b4d-m4zmz"]
Sep 17 18:27:02.467468 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:27:02.467271    2569 scope.go:117] "RemoveContainer" containerID="7a29682059d7bf540df76e1ecd0011fa71546f1cbbfdac90073f918ec77a1e80"
Sep 17 18:27:03.059433 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:27:03.059410    2569 scope.go:117] "RemoveContainer" containerID="f58800afb66ce96693cf69583cac5e08f5ef18f72c37ea3e3211c96faae79f5f"
Sep 17 18:27:03.292292 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:27:03.292259    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-xpfvz" event={"ID":"d2f49aaf-af15-4acc-b0f8-0258e6e794a4","Type":"ContainerStarted","Data":"bbfe52f6717f5fed7d250dabdff43124048dbe09d6c55e740fbe2a0b08378793"}
Sep 17 18:27:03.292683 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:27:03.292359    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-xpfvz"
Sep 17 18:27:03.292683 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:27:03.292511    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-xpfvz"
Sep 17 18:27:03.292683 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:27:03.292569    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-xpfvz" event={"ID":"d2f49aaf-af15-4acc-b0f8-0258e6e794a4","Type":"ContainerStarted","Data":"c104eb85b835672f6d287430d7a378dff39e1aa7d86cdb843223e359071933dd"}
Sep 17 18:27:03.292683 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:27:03.292625    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-xpfvz"
Sep 17 18:27:03.317810 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:27:03.317724    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-xpfvz" podStartSLOduration=2.713556518 podStartE2EDuration="9.317703802s" totalImagesPullingTime="2.783976881s" totalInitContainerRuntime="3.820170403s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:26:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:27:03.315397528 +0000 UTC m=+2775.819400166" watchObservedRunningTime="2026-09-17 18:27:03.317703802 +0000 UTC m=+2775.821706426"
Sep 17 18:27:03.508361 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:27:03.508329    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dacdd83c-048b-4bb2-9c18-f914abd804c5" path="/var/lib/kubelet/pods/dacdd83c-048b-4bb2-9c18-f914abd804c5/volumes"
Sep 17 18:27:09.302048 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:27:09.302019    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-xpfvz"
Sep 17 18:27:39.303960 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:27:39.303924    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-xpfvz"
Sep 17 18:28:09.304786 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:09.304754    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-xpfvz"
Sep 17 18:28:14.878768 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:14.878718    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-xpfvz"]
Sep 17 18:28:14.879408 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:14.879025    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-xpfvz" podUID="d2f49aaf-af15-4acc-b0f8-0258e6e794a4" containerName="kserve-container" containerID="cri-o://de67a974ceedf6d2098aa582d9a60d2be51725440d2926475e1e77adc97a886d" gracePeriod=30
Sep 17 18:28:14.879408 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:14.879036    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-xpfvz" podUID="d2f49aaf-af15-4acc-b0f8-0258e6e794a4" containerName="kube-rbac-proxy" containerID="cri-o://bbfe52f6717f5fed7d250dabdff43124048dbe09d6c55e740fbe2a0b08378793" gracePeriod=30
Sep 17 18:28:14.879408 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:14.879049    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-xpfvz" podUID="d2f49aaf-af15-4acc-b0f8-0258e6e794a4" containerName="kserve-agent" containerID="cri-o://c104eb85b835672f6d287430d7a378dff39e1aa7d86cdb843223e359071933dd" gracePeriod=30
Sep 17 18:28:15.592296 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:15.592266    2569 generic.go:356] "Generic (PLEG): container finished" podID="d2f49aaf-af15-4acc-b0f8-0258e6e794a4" containerID="bbfe52f6717f5fed7d250dabdff43124048dbe09d6c55e740fbe2a0b08378793" exitCode=2
Sep 17 18:28:15.592458 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:15.592321    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-xpfvz" event={"ID":"d2f49aaf-af15-4acc-b0f8-0258e6e794a4","Type":"ContainerDied","Data":"bbfe52f6717f5fed7d250dabdff43124048dbe09d6c55e740fbe2a0b08378793"}
Sep 17 18:28:16.598209 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:16.598182    2569 generic.go:356] "Generic (PLEG): container finished" podID="d2f49aaf-af15-4acc-b0f8-0258e6e794a4" containerID="de67a974ceedf6d2098aa582d9a60d2be51725440d2926475e1e77adc97a886d" exitCode=0
Sep 17 18:28:16.598491 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:16.598254    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-xpfvz" event={"ID":"d2f49aaf-af15-4acc-b0f8-0258e6e794a4","Type":"ContainerDied","Data":"de67a974ceedf6d2098aa582d9a60d2be51725440d2926475e1e77adc97a886d"}
Sep 17 18:28:19.297664 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:19.297620    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-xpfvz" podUID="d2f49aaf-af15-4acc-b0f8-0258e6e794a4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.50:8643/healthz\": dial tcp 10.133.0.50:8643: connect: connection refused"
Sep 17 18:28:19.302874 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:19.302841    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-xpfvz" podUID="d2f49aaf-af15-4acc-b0f8-0258e6e794a4" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.50:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 17 18:28:20.067182 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:20.067148    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fbxsc"]
Sep 17 18:28:20.067549 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:20.067532    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dacdd83c-048b-4bb2-9c18-f914abd804c5" containerName="kserve-container"
Sep 17 18:28:20.067643 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:20.067553    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="dacdd83c-048b-4bb2-9c18-f914abd804c5" containerName="kserve-container"
Sep 17 18:28:20.067643 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:20.067571    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dacdd83c-048b-4bb2-9c18-f914abd804c5" containerName="storage-initializer"
Sep 17 18:28:20.067643 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:20.067578    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="dacdd83c-048b-4bb2-9c18-f914abd804c5" containerName="storage-initializer"
Sep 17 18:28:20.067643 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:20.067611    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dacdd83c-048b-4bb2-9c18-f914abd804c5" containerName="kube-rbac-proxy"
Sep 17 18:28:20.067643 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:20.067620    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="dacdd83c-048b-4bb2-9c18-f914abd804c5" containerName="kube-rbac-proxy"
Sep 17 18:28:20.067907 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:20.067697    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="dacdd83c-048b-4bb2-9c18-f914abd804c5" containerName="kserve-container"
Sep 17 18:28:20.067907 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:20.067710    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="dacdd83c-048b-4bb2-9c18-f914abd804c5" containerName="kube-rbac-proxy"
Sep 17 18:28:20.070928 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:20.070907    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fbxsc"
Sep 17 18:28:20.073613 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:20.073594    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-predictor-serving-cert\""
Sep 17 18:28:20.073704 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:20.073665    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-kube-rbac-proxy-sar-config\""
Sep 17 18:28:20.080671 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:20.080649    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fbxsc"]
Sep 17 18:28:20.097347 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:20.097318    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4fc18b2f-e780-4c6d-a956-07b35eb91e1c-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-fbxsc\" (UID: \"4fc18b2f-e780-4c6d-a956-07b35eb91e1c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fbxsc"
Sep 17 18:28:20.097463 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:20.097354    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4fc18b2f-e780-4c6d-a956-07b35eb91e1c-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-fbxsc\" (UID: \"4fc18b2f-e780-4c6d-a956-07b35eb91e1c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fbxsc"
Sep 17 18:28:20.097463 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:20.097432    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4fc18b2f-e780-4c6d-a956-07b35eb91e1c-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-fbxsc\" (UID: \"4fc18b2f-e780-4c6d-a956-07b35eb91e1c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fbxsc"
Sep 17 18:28:20.097582 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:20.097499    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvj97\" (UniqueName: \"kubernetes.io/projected/4fc18b2f-e780-4c6d-a956-07b35eb91e1c-kube-api-access-cvj97\") pod \"isvc-paddle-predictor-6d5c9fd4f6-fbxsc\" (UID: \"4fc18b2f-e780-4c6d-a956-07b35eb91e1c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fbxsc"
Sep 17 18:28:20.097582 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:20.097532    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4fc18b2f-e780-4c6d-a956-07b35eb91e1c-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-fbxsc\" (UID: \"4fc18b2f-e780-4c6d-a956-07b35eb91e1c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fbxsc"
Sep 17 18:28:20.198517 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:20.198495    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4fc18b2f-e780-4c6d-a956-07b35eb91e1c-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-fbxsc\" (UID: \"4fc18b2f-e780-4c6d-a956-07b35eb91e1c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fbxsc"
Sep 17 18:28:20.198612 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:20.198535    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4fc18b2f-e780-4c6d-a956-07b35eb91e1c-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-fbxsc\" (UID: \"4fc18b2f-e780-4c6d-a956-07b35eb91e1c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fbxsc"
Sep 17 18:28:20.198612 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:20.198567    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4fc18b2f-e780-4c6d-a956-07b35eb91e1c-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-fbxsc\" (UID: \"4fc18b2f-e780-4c6d-a956-07b35eb91e1c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fbxsc"
Sep 17 18:28:20.198715 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:20.198627    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4fc18b2f-e780-4c6d-a956-07b35eb91e1c-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-fbxsc\" (UID: \"4fc18b2f-e780-4c6d-a956-07b35eb91e1c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fbxsc"
Sep 17 18:28:20.198715 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:20.198659    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvj97\" (UniqueName: \"kubernetes.io/projected/4fc18b2f-e780-4c6d-a956-07b35eb91e1c-kube-api-access-cvj97\") pod \"isvc-paddle-predictor-6d5c9fd4f6-fbxsc\" (UID: \"4fc18b2f-e780-4c6d-a956-07b35eb91e1c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fbxsc"
Sep 17 18:28:20.198821 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:28:20.198767    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-predictor-serving-cert: secret "isvc-paddle-predictor-serving-cert" not found
Sep 17 18:28:20.198860 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:28:20.198846    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fc18b2f-e780-4c6d-a956-07b35eb91e1c-proxy-tls podName:4fc18b2f-e780-4c6d-a956-07b35eb91e1c nodeName:}" failed. No retries permitted until 2026-09-17 18:28:20.698829998 +0000 UTC m=+2853.202832603 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4fc18b2f-e780-4c6d-a956-07b35eb91e1c-proxy-tls") pod "isvc-paddle-predictor-6d5c9fd4f6-fbxsc" (UID: "4fc18b2f-e780-4c6d-a956-07b35eb91e1c") : secret "isvc-paddle-predictor-serving-cert" not found
Sep 17 18:28:20.198963 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:20.198945    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4fc18b2f-e780-4c6d-a956-07b35eb91e1c-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-fbxsc\" (UID: \"4fc18b2f-e780-4c6d-a956-07b35eb91e1c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fbxsc"
Sep 17 18:28:20.199370 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:20.199350    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4fc18b2f-e780-4c6d-a956-07b35eb91e1c-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-fbxsc\" (UID: \"4fc18b2f-e780-4c6d-a956-07b35eb91e1c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fbxsc"
Sep 17 18:28:20.199416 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:20.199350    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4fc18b2f-e780-4c6d-a956-07b35eb91e1c-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-fbxsc\" (UID: \"4fc18b2f-e780-4c6d-a956-07b35eb91e1c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fbxsc"
Sep 17 18:28:20.207388 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:20.207365    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvj97\" (UniqueName: \"kubernetes.io/projected/4fc18b2f-e780-4c6d-a956-07b35eb91e1c-kube-api-access-cvj97\") pod \"isvc-paddle-predictor-6d5c9fd4f6-fbxsc\" (UID: \"4fc18b2f-e780-4c6d-a956-07b35eb91e1c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fbxsc"
Sep 17 18:28:20.701944 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:20.701914    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4fc18b2f-e780-4c6d-a956-07b35eb91e1c-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-fbxsc\" (UID: \"4fc18b2f-e780-4c6d-a956-07b35eb91e1c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fbxsc"
Sep 17 18:28:20.704300 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:20.704278    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4fc18b2f-e780-4c6d-a956-07b35eb91e1c-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-fbxsc\" (UID: \"4fc18b2f-e780-4c6d-a956-07b35eb91e1c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fbxsc"
Sep 17 18:28:20.981942 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:20.981877    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fbxsc"
Sep 17 18:28:21.256064 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:21.255989    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fbxsc"]
Sep 17 18:28:21.259639 ip-10-0-131-14 kubenswrapper[2569]: W0917 18:28:21.259613    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fc18b2f_e780_4c6d_a956_07b35eb91e1c.slice/crio-9065d0daeee71b15ce43ca1a355779ee6f9ab2e89196d0432b929ccfcaa08a6f WatchSource:0}: Error finding container 9065d0daeee71b15ce43ca1a355779ee6f9ab2e89196d0432b929ccfcaa08a6f: Status 404 returned error can't find the container with id 9065d0daeee71b15ce43ca1a355779ee6f9ab2e89196d0432b929ccfcaa08a6f
Sep 17 18:28:21.621493 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:21.621413    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fbxsc" event={"ID":"4fc18b2f-e780-4c6d-a956-07b35eb91e1c","Type":"ContainerStarted","Data":"0158cd8a2cac50b09c0654c23eeaba4a1e4f2038ce9fe9fc6ef2b1a273a73117"}
Sep 17 18:28:21.621493 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:21.621470    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fbxsc" event={"ID":"4fc18b2f-e780-4c6d-a956-07b35eb91e1c","Type":"ContainerStarted","Data":"9065d0daeee71b15ce43ca1a355779ee6f9ab2e89196d0432b929ccfcaa08a6f"}
Sep 17 18:28:24.298029 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:24.297992    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-xpfvz" podUID="d2f49aaf-af15-4acc-b0f8-0258e6e794a4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.50:8643/healthz\": dial tcp 10.133.0.50:8643: connect: connection refused"
Sep 17 18:28:26.643554 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:26.643522    2569 generic.go:356] "Generic (PLEG): container finished" podID="4fc18b2f-e780-4c6d-a956-07b35eb91e1c" containerID="0158cd8a2cac50b09c0654c23eeaba4a1e4f2038ce9fe9fc6ef2b1a273a73117" exitCode=0
Sep 17 18:28:26.643926 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:26.643595    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fbxsc" event={"ID":"4fc18b2f-e780-4c6d-a956-07b35eb91e1c","Type":"ContainerDied","Data":"0158cd8a2cac50b09c0654c23eeaba4a1e4f2038ce9fe9fc6ef2b1a273a73117"}
Sep 17 18:28:29.298119 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:29.298063    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-xpfvz" podUID="d2f49aaf-af15-4acc-b0f8-0258e6e794a4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.50:8643/healthz\": dial tcp 10.133.0.50:8643: connect: connection refused"
Sep 17 18:28:29.298563 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:29.298215    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-xpfvz"
Sep 17 18:28:29.302776 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:29.302749    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-xpfvz" podUID="d2f49aaf-af15-4acc-b0f8-0258e6e794a4" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.50:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 17 18:28:34.298286 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:34.298173    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-xpfvz" podUID="d2f49aaf-af15-4acc-b0f8-0258e6e794a4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.50:8643/healthz\": dial tcp 10.133.0.50:8643: connect: connection refused"
Sep 17 18:28:38.704731 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:38.704700    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fbxsc" event={"ID":"4fc18b2f-e780-4c6d-a956-07b35eb91e1c","Type":"ContainerStarted","Data":"6306d246625fb6cf35df4cb6ba90bc4df80f4645270e2bafdda3d92c1bb0f837"}
Sep 17 18:28:38.705195 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:38.704793    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fbxsc" event={"ID":"4fc18b2f-e780-4c6d-a956-07b35eb91e1c","Type":"ContainerStarted","Data":"820efc7505c72fdabee926aa8ef86f659fd3ffac8a15bf7c4c67a20b9731e7e9"}
Sep 17 18:28:38.705195 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:38.705054    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fbxsc"
Sep 17 18:28:38.705195 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:38.705123    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fbxsc"
Sep 17 18:28:38.706190 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:38.706162    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fbxsc" podUID="4fc18b2f-e780-4c6d-a956-07b35eb91e1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 17 18:28:38.725554 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:38.725510    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fbxsc" podStartSLOduration=2.680938253 podStartE2EDuration="18.725495725s" totalImagesPullingTime="11.590388812s" totalInitContainerRuntime="4.45416866s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:28:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:28:38.724422556 +0000 UTC m=+2871.228425182" watchObservedRunningTime="2026-09-17 18:28:38.725495725 +0000 UTC m=+2871.229498349"
Sep 17 18:28:39.297415 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:39.297380    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-xpfvz" podUID="d2f49aaf-af15-4acc-b0f8-0258e6e794a4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.50:8643/healthz\": dial tcp 10.133.0.50:8643: connect: connection refused"
Sep 17 18:28:39.302800 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:39.302773    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-xpfvz" podUID="d2f49aaf-af15-4acc-b0f8-0258e6e794a4" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.50:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 17 18:28:39.302933 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:39.302917    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-xpfvz"
Sep 17 18:28:39.708862 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:39.708820    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fbxsc" podUID="4fc18b2f-e780-4c6d-a956-07b35eb91e1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 17 18:28:44.298094 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:44.298050    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-xpfvz" podUID="d2f49aaf-af15-4acc-b0f8-0258e6e794a4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.50:8643/healthz\": dial tcp 10.133.0.50:8643: connect: connection refused"
Sep 17 18:28:44.714267 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:44.714239    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fbxsc"
Sep 17 18:28:44.714758 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:44.714720    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fbxsc" podUID="4fc18b2f-e780-4c6d-a956-07b35eb91e1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 17 18:28:45.068599 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:45.068579    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-xpfvz"
Sep 17 18:28:45.192319 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:45.192276    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2f49aaf-af15-4acc-b0f8-0258e6e794a4-proxy-tls\") pod \"d2f49aaf-af15-4acc-b0f8-0258e6e794a4\" (UID: \"d2f49aaf-af15-4acc-b0f8-0258e6e794a4\") "
Sep 17 18:28:45.192319 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:45.192333    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2f49aaf-af15-4acc-b0f8-0258e6e794a4-kserve-provision-location\") pod \"d2f49aaf-af15-4acc-b0f8-0258e6e794a4\" (UID: \"d2f49aaf-af15-4acc-b0f8-0258e6e794a4\") "
Sep 17 18:28:45.192570 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:45.192369    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2f49aaf-af15-4acc-b0f8-0258e6e794a4-kube-rbac-proxy-sar-config\") pod \"d2f49aaf-af15-4acc-b0f8-0258e6e794a4\" (UID: \"d2f49aaf-af15-4acc-b0f8-0258e6e794a4\") "
Sep 17 18:28:45.192570 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:45.192460    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2f49aaf-af15-4acc-b0f8-0258e6e794a4-cabundle-cert\") pod \"d2f49aaf-af15-4acc-b0f8-0258e6e794a4\" (UID: \"d2f49aaf-af15-4acc-b0f8-0258e6e794a4\") "
Sep 17 18:28:45.192570 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:45.192515    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lsmcx\" (UniqueName: \"kubernetes.io/projected/d2f49aaf-af15-4acc-b0f8-0258e6e794a4-kube-api-access-lsmcx\") pod \"d2f49aaf-af15-4acc-b0f8-0258e6e794a4\" (UID: \"d2f49aaf-af15-4acc-b0f8-0258e6e794a4\") "
Sep 17 18:28:45.192735 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:45.192695    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2f49aaf-af15-4acc-b0f8-0258e6e794a4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d2f49aaf-af15-4acc-b0f8-0258e6e794a4" (UID: "d2f49aaf-af15-4acc-b0f8-0258e6e794a4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:28:45.192775 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:45.192731    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2f49aaf-af15-4acc-b0f8-0258e6e794a4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d2f49aaf-af15-4acc-b0f8-0258e6e794a4" (UID: "d2f49aaf-af15-4acc-b0f8-0258e6e794a4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:28:45.192808 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:45.192798    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2f49aaf-af15-4acc-b0f8-0258e6e794a4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d2f49aaf-af15-4acc-b0f8-0258e6e794a4" (UID: "d2f49aaf-af15-4acc-b0f8-0258e6e794a4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:28:45.194471 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:45.194436    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2f49aaf-af15-4acc-b0f8-0258e6e794a4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d2f49aaf-af15-4acc-b0f8-0258e6e794a4" (UID: "d2f49aaf-af15-4acc-b0f8-0258e6e794a4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:28:45.194602 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:45.194545    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2f49aaf-af15-4acc-b0f8-0258e6e794a4-kube-api-access-lsmcx" (OuterVolumeSpecName: "kube-api-access-lsmcx") pod "d2f49aaf-af15-4acc-b0f8-0258e6e794a4" (UID: "d2f49aaf-af15-4acc-b0f8-0258e6e794a4"). InnerVolumeSpecName "kube-api-access-lsmcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:28:45.294071 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:45.294019    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2f49aaf-af15-4acc-b0f8-0258e6e794a4-cabundle-cert\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:28:45.294071 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:45.294039    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsmcx\" (UniqueName: \"kubernetes.io/projected/d2f49aaf-af15-4acc-b0f8-0258e6e794a4-kube-api-access-lsmcx\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:28:45.294071 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:45.294050    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2f49aaf-af15-4acc-b0f8-0258e6e794a4-proxy-tls\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:28:45.294071 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:45.294059    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2f49aaf-af15-4acc-b0f8-0258e6e794a4-kserve-provision-location\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:28:45.294071 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:45.294068    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2f49aaf-af15-4acc-b0f8-0258e6e794a4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:28:45.733721 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:45.733694    2569 generic.go:356] "Generic (PLEG): container finished" podID="d2f49aaf-af15-4acc-b0f8-0258e6e794a4" containerID="c104eb85b835672f6d287430d7a378dff39e1aa7d86cdb843223e359071933dd" exitCode=0
Sep 17 18:28:45.734063 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:45.733768    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-xpfvz"
Sep 17 18:28:45.734063 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:45.733772    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-xpfvz" event={"ID":"d2f49aaf-af15-4acc-b0f8-0258e6e794a4","Type":"ContainerDied","Data":"c104eb85b835672f6d287430d7a378dff39e1aa7d86cdb843223e359071933dd"}
Sep 17 18:28:45.734063 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:45.733845    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-xpfvz" event={"ID":"d2f49aaf-af15-4acc-b0f8-0258e6e794a4","Type":"ContainerDied","Data":"77b1c8af569801e589a261792f19e6fcdefcf6c1a100e6715c5489e63588f4f6"}
Sep 17 18:28:45.734063 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:45.733878    2569 scope.go:117] "RemoveContainer" containerID="bbfe52f6717f5fed7d250dabdff43124048dbe09d6c55e740fbe2a0b08378793"
Sep 17 18:28:45.741855 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:45.741823    2569 scope.go:117] "RemoveContainer" containerID="c104eb85b835672f6d287430d7a378dff39e1aa7d86cdb843223e359071933dd"
Sep 17 18:28:45.749042 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:45.749025    2569 scope.go:117] "RemoveContainer" containerID="de67a974ceedf6d2098aa582d9a60d2be51725440d2926475e1e77adc97a886d"
Sep 17 18:28:45.754364 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:45.754343    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-xpfvz"]
Sep 17 18:28:45.756822 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:45.756806    2569 scope.go:117] "RemoveContainer" containerID="a8a74c5fbe3aa92245f0204f14567f207024ce7eed8e8cca39507dff5ad99f0f"
Sep 17 18:28:45.757807 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:45.757788    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cd675847-xpfvz"]
Sep 17 18:28:45.763997 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:45.763983    2569 scope.go:117] "RemoveContainer" containerID="bbfe52f6717f5fed7d250dabdff43124048dbe09d6c55e740fbe2a0b08378793"
Sep 17 18:28:45.764266 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:28:45.764249    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbfe52f6717f5fed7d250dabdff43124048dbe09d6c55e740fbe2a0b08378793\": container with ID starting with bbfe52f6717f5fed7d250dabdff43124048dbe09d6c55e740fbe2a0b08378793 not found: ID does not exist" containerID="bbfe52f6717f5fed7d250dabdff43124048dbe09d6c55e740fbe2a0b08378793"
Sep 17 18:28:45.764321 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:45.764274    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbfe52f6717f5fed7d250dabdff43124048dbe09d6c55e740fbe2a0b08378793"} err="failed to get container status \"bbfe52f6717f5fed7d250dabdff43124048dbe09d6c55e740fbe2a0b08378793\": rpc error: code = NotFound desc = could not find container \"bbfe52f6717f5fed7d250dabdff43124048dbe09d6c55e740fbe2a0b08378793\": container with ID starting with bbfe52f6717f5fed7d250dabdff43124048dbe09d6c55e740fbe2a0b08378793 not found: ID does not exist"
Sep 17 18:28:45.764321 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:45.764291    2569 scope.go:117] "RemoveContainer" containerID="c104eb85b835672f6d287430d7a378dff39e1aa7d86cdb843223e359071933dd"
Sep 17 18:28:45.764580 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:28:45.764550    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c104eb85b835672f6d287430d7a378dff39e1aa7d86cdb843223e359071933dd\": container with ID starting with c104eb85b835672f6d287430d7a378dff39e1aa7d86cdb843223e359071933dd not found: ID does not exist" containerID="c104eb85b835672f6d287430d7a378dff39e1aa7d86cdb843223e359071933dd"
Sep 17 18:28:45.764663 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:45.764579    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c104eb85b835672f6d287430d7a378dff39e1aa7d86cdb843223e359071933dd"} err="failed to get container status \"c104eb85b835672f6d287430d7a378dff39e1aa7d86cdb843223e359071933dd\": rpc error: code = NotFound desc = could not find container \"c104eb85b835672f6d287430d7a378dff39e1aa7d86cdb843223e359071933dd\": container with ID starting with c104eb85b835672f6d287430d7a378dff39e1aa7d86cdb843223e359071933dd not found: ID does not exist"
Sep 17 18:28:45.764663 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:45.764601    2569 scope.go:117] "RemoveContainer" containerID="de67a974ceedf6d2098aa582d9a60d2be51725440d2926475e1e77adc97a886d"
Sep 17 18:28:45.764834 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:28:45.764814    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de67a974ceedf6d2098aa582d9a60d2be51725440d2926475e1e77adc97a886d\": container with ID starting with de67a974ceedf6d2098aa582d9a60d2be51725440d2926475e1e77adc97a886d not found: ID does not exist" containerID="de67a974ceedf6d2098aa582d9a60d2be51725440d2926475e1e77adc97a886d"
Sep 17 18:28:45.764878 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:45.764839    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de67a974ceedf6d2098aa582d9a60d2be51725440d2926475e1e77adc97a886d"} err="failed to get container status \"de67a974ceedf6d2098aa582d9a60d2be51725440d2926475e1e77adc97a886d\": rpc error: code = NotFound desc = could not find container \"de67a974ceedf6d2098aa582d9a60d2be51725440d2926475e1e77adc97a886d\": container with ID starting with de67a974ceedf6d2098aa582d9a60d2be51725440d2926475e1e77adc97a886d not found: ID does not exist"
Sep 17 18:28:45.764878 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:45.764855    2569 scope.go:117] "RemoveContainer" containerID="a8a74c5fbe3aa92245f0204f14567f207024ce7eed8e8cca39507dff5ad99f0f"
Sep 17 18:28:45.765087 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:28:45.765057    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8a74c5fbe3aa92245f0204f14567f207024ce7eed8e8cca39507dff5ad99f0f\": container with ID starting with a8a74c5fbe3aa92245f0204f14567f207024ce7eed8e8cca39507dff5ad99f0f not found: ID does not exist" containerID="a8a74c5fbe3aa92245f0204f14567f207024ce7eed8e8cca39507dff5ad99f0f"
Sep 17 18:28:45.765178 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:45.765111    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8a74c5fbe3aa92245f0204f14567f207024ce7eed8e8cca39507dff5ad99f0f"} err="failed to get container status \"a8a74c5fbe3aa92245f0204f14567f207024ce7eed8e8cca39507dff5ad99f0f\": rpc error: code = NotFound desc = could not find container \"a8a74c5fbe3aa92245f0204f14567f207024ce7eed8e8cca39507dff5ad99f0f\": container with ID starting with a8a74c5fbe3aa92245f0204f14567f207024ce7eed8e8cca39507dff5ad99f0f not found: ID does not exist"
Sep 17 18:28:47.508918 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:47.508886    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2f49aaf-af15-4acc-b0f8-0258e6e794a4" path="/var/lib/kubelet/pods/d2f49aaf-af15-4acc-b0f8-0258e6e794a4/volumes"
Sep 17 18:28:54.714977 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:28:54.714936    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fbxsc" podUID="4fc18b2f-e780-4c6d-a956-07b35eb91e1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 17 18:29:04.715068 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:04.715033    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fbxsc" podUID="4fc18b2f-e780-4c6d-a956-07b35eb91e1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 17 18:29:14.714685 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:14.714647    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fbxsc" podUID="4fc18b2f-e780-4c6d-a956-07b35eb91e1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 17 18:29:24.715937 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:24.715907    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fbxsc"
Sep 17 18:29:31.472303 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:31.472270    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fbxsc"]
Sep 17 18:29:31.472790 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:31.472509    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fbxsc" podUID="4fc18b2f-e780-4c6d-a956-07b35eb91e1c" containerName="kserve-container" containerID="cri-o://820efc7505c72fdabee926aa8ef86f659fd3ffac8a15bf7c4c67a20b9731e7e9" gracePeriod=30
Sep 17 18:29:31.472790 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:31.472697    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fbxsc" podUID="4fc18b2f-e780-4c6d-a956-07b35eb91e1c" containerName="kube-rbac-proxy" containerID="cri-o://6306d246625fb6cf35df4cb6ba90bc4df80f4645270e2bafdda3d92c1bb0f837" gracePeriod=30
Sep 17 18:29:31.916897 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:31.916821    2569 generic.go:356] "Generic (PLEG): container finished" podID="4fc18b2f-e780-4c6d-a956-07b35eb91e1c" containerID="6306d246625fb6cf35df4cb6ba90bc4df80f4645270e2bafdda3d92c1bb0f837" exitCode=2
Sep 17 18:29:31.917036 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:31.916887    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fbxsc" event={"ID":"4fc18b2f-e780-4c6d-a956-07b35eb91e1c","Type":"ContainerDied","Data":"6306d246625fb6cf35df4cb6ba90bc4df80f4645270e2bafdda3d92c1bb0f837"}
Sep 17 18:29:33.921541 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:33.921518    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fbxsc"
Sep 17 18:29:33.926335 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:33.926308    2569 generic.go:356] "Generic (PLEG): container finished" podID="4fc18b2f-e780-4c6d-a956-07b35eb91e1c" containerID="820efc7505c72fdabee926aa8ef86f659fd3ffac8a15bf7c4c67a20b9731e7e9" exitCode=0
Sep 17 18:29:33.926454 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:33.926360    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fbxsc"
Sep 17 18:29:33.926454 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:33.926384    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fbxsc" event={"ID":"4fc18b2f-e780-4c6d-a956-07b35eb91e1c","Type":"ContainerDied","Data":"820efc7505c72fdabee926aa8ef86f659fd3ffac8a15bf7c4c67a20b9731e7e9"}
Sep 17 18:29:33.926575 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:33.926469    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fbxsc" event={"ID":"4fc18b2f-e780-4c6d-a956-07b35eb91e1c","Type":"ContainerDied","Data":"9065d0daeee71b15ce43ca1a355779ee6f9ab2e89196d0432b929ccfcaa08a6f"}
Sep 17 18:29:33.926575 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:33.926496    2569 scope.go:117] "RemoveContainer" containerID="6306d246625fb6cf35df4cb6ba90bc4df80f4645270e2bafdda3d92c1bb0f837"
Sep 17 18:29:33.933945 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:33.933928    2569 scope.go:117] "RemoveContainer" containerID="820efc7505c72fdabee926aa8ef86f659fd3ffac8a15bf7c4c67a20b9731e7e9"
Sep 17 18:29:33.940917 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:33.940899    2569 scope.go:117] "RemoveContainer" containerID="0158cd8a2cac50b09c0654c23eeaba4a1e4f2038ce9fe9fc6ef2b1a273a73117"
Sep 17 18:29:33.941025 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:33.941002    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4fc18b2f-e780-4c6d-a956-07b35eb91e1c-kube-rbac-proxy-sar-config\") pod \"4fc18b2f-e780-4c6d-a956-07b35eb91e1c\" (UID: \"4fc18b2f-e780-4c6d-a956-07b35eb91e1c\") "
Sep 17 18:29:33.941104 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:33.941059    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4fc18b2f-e780-4c6d-a956-07b35eb91e1c-cabundle-cert\") pod \"4fc18b2f-e780-4c6d-a956-07b35eb91e1c\" (UID: \"4fc18b2f-e780-4c6d-a956-07b35eb91e1c\") "
Sep 17 18:29:33.941215 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:33.941193    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4fc18b2f-e780-4c6d-a956-07b35eb91e1c-proxy-tls\") pod \"4fc18b2f-e780-4c6d-a956-07b35eb91e1c\" (UID: \"4fc18b2f-e780-4c6d-a956-07b35eb91e1c\") "
Sep 17 18:29:33.941295 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:33.941270    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cvj97\" (UniqueName: \"kubernetes.io/projected/4fc18b2f-e780-4c6d-a956-07b35eb91e1c-kube-api-access-cvj97\") pod \"4fc18b2f-e780-4c6d-a956-07b35eb91e1c\" (UID: \"4fc18b2f-e780-4c6d-a956-07b35eb91e1c\") "
Sep 17 18:29:33.941367 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:33.941295    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4fc18b2f-e780-4c6d-a956-07b35eb91e1c-kserve-provision-location\") pod \"4fc18b2f-e780-4c6d-a956-07b35eb91e1c\" (UID: \"4fc18b2f-e780-4c6d-a956-07b35eb91e1c\") "
Sep 17 18:29:33.941526 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:33.941484    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fc18b2f-e780-4c6d-a956-07b35eb91e1c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4fc18b2f-e780-4c6d-a956-07b35eb91e1c" (UID: "4fc18b2f-e780-4c6d-a956-07b35eb91e1c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:29:33.941969 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:33.941543    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fc18b2f-e780-4c6d-a956-07b35eb91e1c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4fc18b2f-e780-4c6d-a956-07b35eb91e1c" (UID: "4fc18b2f-e780-4c6d-a956-07b35eb91e1c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:29:33.944145 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:33.944101    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fc18b2f-e780-4c6d-a956-07b35eb91e1c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4fc18b2f-e780-4c6d-a956-07b35eb91e1c" (UID: "4fc18b2f-e780-4c6d-a956-07b35eb91e1c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:29:33.944443 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:33.944407    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fc18b2f-e780-4c6d-a956-07b35eb91e1c-kube-api-access-cvj97" (OuterVolumeSpecName: "kube-api-access-cvj97") pod "4fc18b2f-e780-4c6d-a956-07b35eb91e1c" (UID: "4fc18b2f-e780-4c6d-a956-07b35eb91e1c"). InnerVolumeSpecName "kube-api-access-cvj97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:29:33.950978 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:33.950953    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fc18b2f-e780-4c6d-a956-07b35eb91e1c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4fc18b2f-e780-4c6d-a956-07b35eb91e1c" (UID: "4fc18b2f-e780-4c6d-a956-07b35eb91e1c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:29:33.957138 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:33.957116    2569 scope.go:117] "RemoveContainer" containerID="6306d246625fb6cf35df4cb6ba90bc4df80f4645270e2bafdda3d92c1bb0f837"
Sep 17 18:29:33.957413 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:29:33.957391    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6306d246625fb6cf35df4cb6ba90bc4df80f4645270e2bafdda3d92c1bb0f837\": container with ID starting with 6306d246625fb6cf35df4cb6ba90bc4df80f4645270e2bafdda3d92c1bb0f837 not found: ID does not exist" containerID="6306d246625fb6cf35df4cb6ba90bc4df80f4645270e2bafdda3d92c1bb0f837"
Sep 17 18:29:33.957498 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:33.957422    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6306d246625fb6cf35df4cb6ba90bc4df80f4645270e2bafdda3d92c1bb0f837"} err="failed to get container status \"6306d246625fb6cf35df4cb6ba90bc4df80f4645270e2bafdda3d92c1bb0f837\": rpc error: code = NotFound desc = could not find container \"6306d246625fb6cf35df4cb6ba90bc4df80f4645270e2bafdda3d92c1bb0f837\": container with ID starting with 6306d246625fb6cf35df4cb6ba90bc4df80f4645270e2bafdda3d92c1bb0f837 not found: ID does not exist"
Sep 17 18:29:33.957498 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:33.957445    2569 scope.go:117] "RemoveContainer" containerID="820efc7505c72fdabee926aa8ef86f659fd3ffac8a15bf7c4c67a20b9731e7e9"
Sep 17 18:29:33.957737 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:29:33.957711    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"820efc7505c72fdabee926aa8ef86f659fd3ffac8a15bf7c4c67a20b9731e7e9\": container with ID starting with 820efc7505c72fdabee926aa8ef86f659fd3ffac8a15bf7c4c67a20b9731e7e9 not found: ID does not exist" containerID="820efc7505c72fdabee926aa8ef86f659fd3ffac8a15bf7c4c67a20b9731e7e9"
Sep 17 18:29:33.957788 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:33.957744    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"820efc7505c72fdabee926aa8ef86f659fd3ffac8a15bf7c4c67a20b9731e7e9"} err="failed to get container status \"820efc7505c72fdabee926aa8ef86f659fd3ffac8a15bf7c4c67a20b9731e7e9\": rpc error: code = NotFound desc = could not find container \"820efc7505c72fdabee926aa8ef86f659fd3ffac8a15bf7c4c67a20b9731e7e9\": container with ID starting with 820efc7505c72fdabee926aa8ef86f659fd3ffac8a15bf7c4c67a20b9731e7e9 not found: ID does not exist"
Sep 17 18:29:33.957788 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:33.957760    2569 scope.go:117] "RemoveContainer" containerID="0158cd8a2cac50b09c0654c23eeaba4a1e4f2038ce9fe9fc6ef2b1a273a73117"
Sep 17 18:29:33.958001 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:29:33.957982    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0158cd8a2cac50b09c0654c23eeaba4a1e4f2038ce9fe9fc6ef2b1a273a73117\": container with ID starting with 0158cd8a2cac50b09c0654c23eeaba4a1e4f2038ce9fe9fc6ef2b1a273a73117 not found: ID does not exist" containerID="0158cd8a2cac50b09c0654c23eeaba4a1e4f2038ce9fe9fc6ef2b1a273a73117"
Sep 17 18:29:33.958056 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:33.958009    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0158cd8a2cac50b09c0654c23eeaba4a1e4f2038ce9fe9fc6ef2b1a273a73117"} err="failed to get container status \"0158cd8a2cac50b09c0654c23eeaba4a1e4f2038ce9fe9fc6ef2b1a273a73117\": rpc error: code = NotFound desc = could not find container \"0158cd8a2cac50b09c0654c23eeaba4a1e4f2038ce9fe9fc6ef2b1a273a73117\": container with ID starting with 0158cd8a2cac50b09c0654c23eeaba4a1e4f2038ce9fe9fc6ef2b1a273a73117 not found: ID does not exist"
Sep 17 18:29:34.042467 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:34.042411    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4fc18b2f-e780-4c6d-a956-07b35eb91e1c-cabundle-cert\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:29:34.042467 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:34.042433    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4fc18b2f-e780-4c6d-a956-07b35eb91e1c-proxy-tls\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:29:34.042467 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:34.042445    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvj97\" (UniqueName: \"kubernetes.io/projected/4fc18b2f-e780-4c6d-a956-07b35eb91e1c-kube-api-access-cvj97\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:29:34.042467 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:34.042458    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4fc18b2f-e780-4c6d-a956-07b35eb91e1c-kserve-provision-location\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:29:34.042650 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:34.042471    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4fc18b2f-e780-4c6d-a956-07b35eb91e1c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:29:34.248562 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:34.248535    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fbxsc"]
Sep 17 18:29:34.252234 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:34.252212    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fbxsc"]
Sep 17 18:29:35.508130 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:35.508093    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fc18b2f-e780-4c6d-a956-07b35eb91e1c" path="/var/lib/kubelet/pods/4fc18b2f-e780-4c6d-a956-07b35eb91e1c/volumes"
Sep 17 18:29:36.659719 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:36.659679    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b527c"]
Sep 17 18:29:36.660212 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:36.660112    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fc18b2f-e780-4c6d-a956-07b35eb91e1c" containerName="storage-initializer"
Sep 17 18:29:36.660212 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:36.660130    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fc18b2f-e780-4c6d-a956-07b35eb91e1c" containerName="storage-initializer"
Sep 17 18:29:36.660212 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:36.660141    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2f49aaf-af15-4acc-b0f8-0258e6e794a4" containerName="kserve-agent"
Sep 17 18:29:36.660212 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:36.660149    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2f49aaf-af15-4acc-b0f8-0258e6e794a4" containerName="kserve-agent"
Sep 17 18:29:36.660212 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:36.660161    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2f49aaf-af15-4acc-b0f8-0258e6e794a4" containerName="storage-initializer"
Sep 17 18:29:36.660212 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:36.660168    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2f49aaf-af15-4acc-b0f8-0258e6e794a4" containerName="storage-initializer"
Sep 17 18:29:36.660535 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:36.660217    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fc18b2f-e780-4c6d-a956-07b35eb91e1c" containerName="kube-rbac-proxy"
Sep 17 18:29:36.660535 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:36.660226    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fc18b2f-e780-4c6d-a956-07b35eb91e1c" containerName="kube-rbac-proxy"
Sep 17 18:29:36.660535 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:36.660242    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fc18b2f-e780-4c6d-a956-07b35eb91e1c" containerName="kserve-container"
Sep 17 18:29:36.660535 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:36.660250    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fc18b2f-e780-4c6d-a956-07b35eb91e1c" containerName="kserve-container"
Sep 17 18:29:36.660535 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:36.660260    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2f49aaf-af15-4acc-b0f8-0258e6e794a4" containerName="kserve-container"
Sep 17 18:29:36.660535 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:36.660269    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2f49aaf-af15-4acc-b0f8-0258e6e794a4" containerName="kserve-container"
Sep 17 18:29:36.660535 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:36.660279    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2f49aaf-af15-4acc-b0f8-0258e6e794a4" containerName="kube-rbac-proxy"
Sep 17 18:29:36.660535 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:36.660287    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2f49aaf-af15-4acc-b0f8-0258e6e794a4" containerName="kube-rbac-proxy"
Sep 17 18:29:36.660535 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:36.660360    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2f49aaf-af15-4acc-b0f8-0258e6e794a4" containerName="kserve-container"
Sep 17 18:29:36.660535 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:36.660375    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2f49aaf-af15-4acc-b0f8-0258e6e794a4" containerName="kube-rbac-proxy"
Sep 17 18:29:36.660535 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:36.660386    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2f49aaf-af15-4acc-b0f8-0258e6e794a4" containerName="kserve-agent"
Sep 17 18:29:36.660535 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:36.660396    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fc18b2f-e780-4c6d-a956-07b35eb91e1c" containerName="kube-rbac-proxy"
Sep 17 18:29:36.660535 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:36.660405    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fc18b2f-e780-4c6d-a956-07b35eb91e1c" containerName="kserve-container"
Sep 17 18:29:36.665088 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:36.665053    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b527c"
Sep 17 18:29:36.667930 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:36.667910    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:29:36.668137 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:36.668121    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:29:36.669270 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:36.669251    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-predictor-serving-cert\""
Sep 17 18:29:36.669411 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:36.669291    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:29:36.669411 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:36.669349    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:29:36.669411 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:36.669357    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-kube-rbac-proxy-sar-config\""
Sep 17 18:29:36.669411 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:36.669290    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-kkx8f\""
Sep 17 18:29:36.673663 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:36.673632    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b527c"]
Sep 17 18:29:36.760228 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:36.760207    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21f11740-b17b-467e-8de2-4f1ad721382f-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-b527c\" (UID: \"21f11740-b17b-467e-8de2-4f1ad721382f\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b527c"
Sep 17 18:29:36.760337 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:36.760242    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21f11740-b17b-467e-8de2-4f1ad721382f-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-b527c\" (UID: \"21f11740-b17b-467e-8de2-4f1ad721382f\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b527c"
Sep 17 18:29:36.760337 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:36.760261    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21f11740-b17b-467e-8de2-4f1ad721382f-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-b527c\" (UID: \"21f11740-b17b-467e-8de2-4f1ad721382f\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b527c"
Sep 17 18:29:36.760337 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:36.760329    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21f11740-b17b-467e-8de2-4f1ad721382f-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-b527c\" (UID: \"21f11740-b17b-467e-8de2-4f1ad721382f\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b527c"
Sep 17 18:29:36.760450 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:36.760355    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzt2z\" (UniqueName: \"kubernetes.io/projected/21f11740-b17b-467e-8de2-4f1ad721382f-kube-api-access-qzt2z\") pod \"isvc-paddle-runtime-predictor-58b99fff76-b527c\" (UID: \"21f11740-b17b-467e-8de2-4f1ad721382f\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b527c"
Sep 17 18:29:36.860929 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:36.860904    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21f11740-b17b-467e-8de2-4f1ad721382f-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-b527c\" (UID: \"21f11740-b17b-467e-8de2-4f1ad721382f\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b527c"
Sep 17 18:29:36.861012 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:36.860934    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21f11740-b17b-467e-8de2-4f1ad721382f-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-b527c\" (UID: \"21f11740-b17b-467e-8de2-4f1ad721382f\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b527c"
Sep 17 18:29:36.861012 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:36.860962    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21f11740-b17b-467e-8de2-4f1ad721382f-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-b527c\" (UID: \"21f11740-b17b-467e-8de2-4f1ad721382f\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b527c"
Sep 17 18:29:36.861012 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:36.860984    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzt2z\" (UniqueName: \"kubernetes.io/projected/21f11740-b17b-467e-8de2-4f1ad721382f-kube-api-access-qzt2z\") pod \"isvc-paddle-runtime-predictor-58b99fff76-b527c\" (UID: \"21f11740-b17b-467e-8de2-4f1ad721382f\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b527c"
Sep 17 18:29:36.861176 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:36.861026    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21f11740-b17b-467e-8de2-4f1ad721382f-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-b527c\" (UID: \"21f11740-b17b-467e-8de2-4f1ad721382f\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b527c"
Sep 17 18:29:36.861331 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:36.861311    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21f11740-b17b-467e-8de2-4f1ad721382f-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-b527c\" (UID: \"21f11740-b17b-467e-8de2-4f1ad721382f\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b527c"
Sep 17 18:29:36.861584 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:36.861559    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21f11740-b17b-467e-8de2-4f1ad721382f-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-b527c\" (UID: \"21f11740-b17b-467e-8de2-4f1ad721382f\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b527c"
Sep 17 18:29:36.861670 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:36.861645    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21f11740-b17b-467e-8de2-4f1ad721382f-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-b527c\" (UID: \"21f11740-b17b-467e-8de2-4f1ad721382f\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b527c"
Sep 17 18:29:36.863306 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:36.863287    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21f11740-b17b-467e-8de2-4f1ad721382f-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-b527c\" (UID: \"21f11740-b17b-467e-8de2-4f1ad721382f\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b527c"
Sep 17 18:29:36.868804 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:36.868786    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzt2z\" (UniqueName: \"kubernetes.io/projected/21f11740-b17b-467e-8de2-4f1ad721382f-kube-api-access-qzt2z\") pod \"isvc-paddle-runtime-predictor-58b99fff76-b527c\" (UID: \"21f11740-b17b-467e-8de2-4f1ad721382f\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b527c"
Sep 17 18:29:36.977051 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:36.977027    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b527c"
Sep 17 18:29:37.250890 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:37.250863    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b527c"]
Sep 17 18:29:37.253084 ip-10-0-131-14 kubenswrapper[2569]: W0917 18:29:37.253043    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21f11740_b17b_467e_8de2_4f1ad721382f.slice/crio-e1a810ea12e4bd2eef0b8d352cfd303247acf96286dac886820a2a204576572c WatchSource:0}: Error finding container e1a810ea12e4bd2eef0b8d352cfd303247acf96286dac886820a2a204576572c: Status 404 returned error can't find the container with id e1a810ea12e4bd2eef0b8d352cfd303247acf96286dac886820a2a204576572c
Sep 17 18:29:37.949692 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:37.949652    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b527c" event={"ID":"21f11740-b17b-467e-8de2-4f1ad721382f","Type":"ContainerStarted","Data":"583447ee1b33cfb7fbfbff937e59c532676fc05245a741cf6c41ac0d7a165cae"}
Sep 17 18:29:37.950096 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:37.949756    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b527c" event={"ID":"21f11740-b17b-467e-8de2-4f1ad721382f","Type":"ContainerStarted","Data":"e1a810ea12e4bd2eef0b8d352cfd303247acf96286dac886820a2a204576572c"}
Sep 17 18:29:41.968508 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:41.968441    2569 generic.go:356] "Generic (PLEG): container finished" podID="21f11740-b17b-467e-8de2-4f1ad721382f" containerID="583447ee1b33cfb7fbfbff937e59c532676fc05245a741cf6c41ac0d7a165cae" exitCode=0
Sep 17 18:29:41.968829 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:41.968516    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b527c" event={"ID":"21f11740-b17b-467e-8de2-4f1ad721382f","Type":"ContainerDied","Data":"583447ee1b33cfb7fbfbff937e59c532676fc05245a741cf6c41ac0d7a165cae"}
Sep 17 18:29:42.974647 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:42.974617    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b527c" event={"ID":"21f11740-b17b-467e-8de2-4f1ad721382f","Type":"ContainerStarted","Data":"9b3c8666c165807f7b0773a98b69be0d885017766361f130799f75419015a2fb"}
Sep 17 18:29:42.975014 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:42.974684    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b527c" event={"ID":"21f11740-b17b-467e-8de2-4f1ad721382f","Type":"ContainerStarted","Data":"5b30622b1b780af74b9318a6b13d1a5d353c7a09d508c02a3d537410361e1bfe"}
Sep 17 18:29:42.975014 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:42.974876    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b527c"
Sep 17 18:29:42.996757 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:42.996715    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b527c" podStartSLOduration=2.605901711 podStartE2EDuration="6.996702194s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.390800483s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:42.994834634 +0000 UTC m=+2935.498837259" watchObservedRunningTime="2026-09-17 18:29:42.996702194 +0000 UTC m=+2935.500704818"
Sep 17 18:29:43.978069 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:43.978040    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b527c"
Sep 17 18:29:43.979169 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:43.979142    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b527c" podUID="21f11740-b17b-467e-8de2-4f1ad721382f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 17 18:29:44.981456 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:44.981419    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b527c" podUID="21f11740-b17b-467e-8de2-4f1ad721382f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 17 18:29:49.985618 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:49.985595    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b527c"
Sep 17 18:29:49.986053 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:49.986026    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b527c" podUID="21f11740-b17b-467e-8de2-4f1ad721382f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 17 18:29:59.986684 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:29:59.986635    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b527c" podUID="21f11740-b17b-467e-8de2-4f1ad721382f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 17 18:30:09.988163 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:09.988115    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b527c" podUID="21f11740-b17b-467e-8de2-4f1ad721382f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 17 18:30:19.986336 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:19.986281    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b527c" podUID="21f11740-b17b-467e-8de2-4f1ad721382f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 17 18:30:22.072243 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:22.072208    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-7978j"]
Sep 17 18:30:22.075614 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:22.075593    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-7978j"
Sep 17 18:30:22.078269 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:22.078249    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert\""
Sep 17 18:30:22.078503 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:22.078488    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 17 18:30:22.078567 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:22.078489    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\""
Sep 17 18:30:22.086164 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:22.086145    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-7978j"]
Sep 17 18:30:22.177370 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:22.177346    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58a50202-13f2-49b6-acaf-479d2baf2e56-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-7978j\" (UID: \"58a50202-13f2-49b6-acaf-479d2baf2e56\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-7978j"
Sep 17 18:30:22.177478 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:22.177381    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/58a50202-13f2-49b6-acaf-479d2baf2e56-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-7978j\" (UID: \"58a50202-13f2-49b6-acaf-479d2baf2e56\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-7978j"
Sep 17 18:30:22.177478 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:22.177419    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58a50202-13f2-49b6-acaf-479d2baf2e56-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-7978j\" (UID: \"58a50202-13f2-49b6-acaf-479d2baf2e56\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-7978j"
Sep 17 18:30:22.177478 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:22.177476    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/58a50202-13f2-49b6-acaf-479d2baf2e56-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-7978j\" (UID: \"58a50202-13f2-49b6-acaf-479d2baf2e56\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-7978j"
Sep 17 18:30:22.177589 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:22.177506    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8g6m\" (UniqueName: \"kubernetes.io/projected/58a50202-13f2-49b6-acaf-479d2baf2e56-kube-api-access-x8g6m\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-7978j\" (UID: \"58a50202-13f2-49b6-acaf-479d2baf2e56\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-7978j"
Sep 17 18:30:22.278431 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:22.278404    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8g6m\" (UniqueName: \"kubernetes.io/projected/58a50202-13f2-49b6-acaf-479d2baf2e56-kube-api-access-x8g6m\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-7978j\" (UID: \"58a50202-13f2-49b6-acaf-479d2baf2e56\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-7978j"
Sep 17 18:30:22.278538 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:22.278453    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58a50202-13f2-49b6-acaf-479d2baf2e56-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-7978j\" (UID: \"58a50202-13f2-49b6-acaf-479d2baf2e56\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-7978j"
Sep 17 18:30:22.278538 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:22.278485    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/58a50202-13f2-49b6-acaf-479d2baf2e56-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-7978j\" (UID: \"58a50202-13f2-49b6-acaf-479d2baf2e56\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-7978j"
Sep 17 18:30:22.278538 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:22.278529    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58a50202-13f2-49b6-acaf-479d2baf2e56-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-7978j\" (UID: \"58a50202-13f2-49b6-acaf-479d2baf2e56\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-7978j"
Sep 17 18:30:22.278657 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:22.278580    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/58a50202-13f2-49b6-acaf-479d2baf2e56-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-7978j\" (UID: \"58a50202-13f2-49b6-acaf-479d2baf2e56\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-7978j"
Sep 17 18:30:22.279007 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:22.278985    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58a50202-13f2-49b6-acaf-479d2baf2e56-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-7978j\" (UID: \"58a50202-13f2-49b6-acaf-479d2baf2e56\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-7978j"
Sep 17 18:30:22.279296 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:22.279156    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/58a50202-13f2-49b6-acaf-479d2baf2e56-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-7978j\" (UID: \"58a50202-13f2-49b6-acaf-479d2baf2e56\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-7978j"
Sep 17 18:30:22.283115 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:22.279580    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/58a50202-13f2-49b6-acaf-479d2baf2e56-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-7978j\" (UID: \"58a50202-13f2-49b6-acaf-479d2baf2e56\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-7978j"
Sep 17 18:30:22.286008 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:22.285988    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58a50202-13f2-49b6-acaf-479d2baf2e56-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-7978j\" (UID: \"58a50202-13f2-49b6-acaf-479d2baf2e56\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-7978j"
Sep 17 18:30:22.287802 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:22.287781    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8g6m\" (UniqueName: \"kubernetes.io/projected/58a50202-13f2-49b6-acaf-479d2baf2e56-kube-api-access-x8g6m\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-7978j\" (UID: \"58a50202-13f2-49b6-acaf-479d2baf2e56\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-7978j"
Sep 17 18:30:22.385473 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:22.385422    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-7978j"
Sep 17 18:30:22.658875 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:22.658852    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-7978j"]
Sep 17 18:30:22.661111 ip-10-0-131-14 kubenswrapper[2569]: W0917 18:30:22.661068    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58a50202_13f2_49b6_acaf_479d2baf2e56.slice/crio-309d52e830e8a5d70d0ef160eeef00daf2c15a09a0c1d4181a2471a48763d97d WatchSource:0}: Error finding container 309d52e830e8a5d70d0ef160eeef00daf2c15a09a0c1d4181a2471a48763d97d: Status 404 returned error can't find the container with id 309d52e830e8a5d70d0ef160eeef00daf2c15a09a0c1d4181a2471a48763d97d
Sep 17 18:30:23.132091 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:23.132037    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-7978j" event={"ID":"58a50202-13f2-49b6-acaf-479d2baf2e56","Type":"ContainerStarted","Data":"102fc1e540fdf6d95f019388de0feee1706262436985b2c80c07a6443099de2f"}
Sep 17 18:30:23.132525 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:23.132117    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-7978j" event={"ID":"58a50202-13f2-49b6-acaf-479d2baf2e56","Type":"ContainerStarted","Data":"309d52e830e8a5d70d0ef160eeef00daf2c15a09a0c1d4181a2471a48763d97d"}
Sep 17 18:30:24.137461 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:24.137394    2569 generic.go:356] "Generic (PLEG): container finished" podID="58a50202-13f2-49b6-acaf-479d2baf2e56" containerID="102fc1e540fdf6d95f019388de0feee1706262436985b2c80c07a6443099de2f" exitCode=0
Sep 17 18:30:24.137461 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:24.137452    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-7978j" event={"ID":"58a50202-13f2-49b6-acaf-479d2baf2e56","Type":"ContainerDied","Data":"102fc1e540fdf6d95f019388de0feee1706262436985b2c80c07a6443099de2f"}
Sep 17 18:30:25.144475 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:25.144429    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-7978j" event={"ID":"58a50202-13f2-49b6-acaf-479d2baf2e56","Type":"ContainerStarted","Data":"8c44a471fc9fdc2ea291a6bb66eb91e64d6fa19cf43c0928567270cb041cb0c7"}
Sep 17 18:30:25.144946 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:25.144508    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-7978j" event={"ID":"58a50202-13f2-49b6-acaf-479d2baf2e56","Type":"ContainerStarted","Data":"ee05b7286956b634196691ea1ffcf262520200dd19eb96735cc4429ff65dc509"}
Sep 17 18:30:25.144946 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:25.144653    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-7978j"
Sep 17 18:30:25.164194 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:25.164150    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-7978j" podStartSLOduration=2.117736933 podStartE2EDuration="3.164137285s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.046400352s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:25.16377929 +0000 UTC m=+2977.667781917" watchObservedRunningTime="2026-09-17 18:30:25.164137285 +0000 UTC m=+2977.668139911"
Sep 17 18:30:26.148592 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:26.148561    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-7978j"
Sep 17 18:30:26.149710 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:26.149680    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-7978j" podUID="58a50202-13f2-49b6-acaf-479d2baf2e56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 17 18:30:27.152794 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:27.152757    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-7978j" podUID="58a50202-13f2-49b6-acaf-479d2baf2e56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 17 18:30:29.986838 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:29.986808    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b527c"
Sep 17 18:30:32.158426 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:32.158401    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-7978j"
Sep 17 18:30:32.158904 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:32.158886    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-7978j" podUID="58a50202-13f2-49b6-acaf-479d2baf2e56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 17 18:30:38.061705 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:38.061666    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b527c"]
Sep 17 18:30:38.062204 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:38.061900    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b527c" podUID="21f11740-b17b-467e-8de2-4f1ad721382f" containerName="kserve-container" containerID="cri-o://5b30622b1b780af74b9318a6b13d1a5d353c7a09d508c02a3d537410361e1bfe" gracePeriod=30
Sep 17 18:30:38.062204 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:38.061962    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b527c" podUID="21f11740-b17b-467e-8de2-4f1ad721382f" containerName="kube-rbac-proxy" containerID="cri-o://9b3c8666c165807f7b0773a98b69be0d885017766361f130799f75419015a2fb" gracePeriod=30
Sep 17 18:30:38.201261 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:38.201238    2569 generic.go:356] "Generic (PLEG): container finished" podID="21f11740-b17b-467e-8de2-4f1ad721382f" containerID="9b3c8666c165807f7b0773a98b69be0d885017766361f130799f75419015a2fb" exitCode=2
Sep 17 18:30:38.201373 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:38.201309    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b527c" event={"ID":"21f11740-b17b-467e-8de2-4f1ad721382f","Type":"ContainerDied","Data":"9b3c8666c165807f7b0773a98b69be0d885017766361f130799f75419015a2fb"}
Sep 17 18:30:39.982478 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:39.982430    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b527c" podUID="21f11740-b17b-467e-8de2-4f1ad721382f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.52:8643/healthz\": dial tcp 10.133.0.52:8643: connect: connection refused"
Sep 17 18:30:39.986706 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:39.986682    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b527c" podUID="21f11740-b17b-467e-8de2-4f1ad721382f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 17 18:30:40.600770 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:40.600747    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b527c"
Sep 17 18:30:40.711634 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:40.711570    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qzt2z\" (UniqueName: \"kubernetes.io/projected/21f11740-b17b-467e-8de2-4f1ad721382f-kube-api-access-qzt2z\") pod \"21f11740-b17b-467e-8de2-4f1ad721382f\" (UID: \"21f11740-b17b-467e-8de2-4f1ad721382f\") "
Sep 17 18:30:40.711634 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:40.711618    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21f11740-b17b-467e-8de2-4f1ad721382f-kube-rbac-proxy-sar-config\") pod \"21f11740-b17b-467e-8de2-4f1ad721382f\" (UID: \"21f11740-b17b-467e-8de2-4f1ad721382f\") "
Sep 17 18:30:40.711802 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:40.711700    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21f11740-b17b-467e-8de2-4f1ad721382f-proxy-tls\") pod \"21f11740-b17b-467e-8de2-4f1ad721382f\" (UID: \"21f11740-b17b-467e-8de2-4f1ad721382f\") "
Sep 17 18:30:40.711802 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:40.711738    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21f11740-b17b-467e-8de2-4f1ad721382f-kserve-provision-location\") pod \"21f11740-b17b-467e-8de2-4f1ad721382f\" (UID: \"21f11740-b17b-467e-8de2-4f1ad721382f\") "
Sep 17 18:30:40.711802 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:40.711798    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21f11740-b17b-467e-8de2-4f1ad721382f-cabundle-cert\") pod \"21f11740-b17b-467e-8de2-4f1ad721382f\" (UID: \"21f11740-b17b-467e-8de2-4f1ad721382f\") "
Sep 17 18:30:40.711973 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:40.711893    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21f11740-b17b-467e-8de2-4f1ad721382f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "21f11740-b17b-467e-8de2-4f1ad721382f" (UID: "21f11740-b17b-467e-8de2-4f1ad721382f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:30:40.712151 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:40.712128    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21f11740-b17b-467e-8de2-4f1ad721382f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:30:40.712267 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:40.712162    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21f11740-b17b-467e-8de2-4f1ad721382f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "21f11740-b17b-467e-8de2-4f1ad721382f" (UID: "21f11740-b17b-467e-8de2-4f1ad721382f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:30:40.713605 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:40.713582    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21f11740-b17b-467e-8de2-4f1ad721382f-kube-api-access-qzt2z" (OuterVolumeSpecName: "kube-api-access-qzt2z") pod "21f11740-b17b-467e-8de2-4f1ad721382f" (UID: "21f11740-b17b-467e-8de2-4f1ad721382f"). InnerVolumeSpecName "kube-api-access-qzt2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:30:40.713696 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:40.713666    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21f11740-b17b-467e-8de2-4f1ad721382f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "21f11740-b17b-467e-8de2-4f1ad721382f" (UID: "21f11740-b17b-467e-8de2-4f1ad721382f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:30:40.721391 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:40.721370    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21f11740-b17b-467e-8de2-4f1ad721382f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "21f11740-b17b-467e-8de2-4f1ad721382f" (UID: "21f11740-b17b-467e-8de2-4f1ad721382f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:30:40.813223 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:40.813200    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21f11740-b17b-467e-8de2-4f1ad721382f-cabundle-cert\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:30:40.813223 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:40.813220    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzt2z\" (UniqueName: \"kubernetes.io/projected/21f11740-b17b-467e-8de2-4f1ad721382f-kube-api-access-qzt2z\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:30:40.813342 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:40.813230    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21f11740-b17b-467e-8de2-4f1ad721382f-proxy-tls\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:30:40.813342 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:40.813241    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21f11740-b17b-467e-8de2-4f1ad721382f-kserve-provision-location\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:30:41.221536 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:41.221505    2569 generic.go:356] "Generic (PLEG): container finished" podID="21f11740-b17b-467e-8de2-4f1ad721382f" containerID="5b30622b1b780af74b9318a6b13d1a5d353c7a09d508c02a3d537410361e1bfe" exitCode=0
Sep 17 18:30:41.221874 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:41.221560    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b527c"
Sep 17 18:30:41.221874 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:41.221593    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b527c" event={"ID":"21f11740-b17b-467e-8de2-4f1ad721382f","Type":"ContainerDied","Data":"5b30622b1b780af74b9318a6b13d1a5d353c7a09d508c02a3d537410361e1bfe"}
Sep 17 18:30:41.221874 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:41.221658    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b527c" event={"ID":"21f11740-b17b-467e-8de2-4f1ad721382f","Type":"ContainerDied","Data":"e1a810ea12e4bd2eef0b8d352cfd303247acf96286dac886820a2a204576572c"}
Sep 17 18:30:41.221874 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:41.221686    2569 scope.go:117] "RemoveContainer" containerID="9b3c8666c165807f7b0773a98b69be0d885017766361f130799f75419015a2fb"
Sep 17 18:30:41.230609 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:41.230590    2569 scope.go:117] "RemoveContainer" containerID="5b30622b1b780af74b9318a6b13d1a5d353c7a09d508c02a3d537410361e1bfe"
Sep 17 18:30:41.238604 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:41.238585    2569 scope.go:117] "RemoveContainer" containerID="583447ee1b33cfb7fbfbff937e59c532676fc05245a741cf6c41ac0d7a165cae"
Sep 17 18:30:41.245543 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:41.245485    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b527c"]
Sep 17 18:30:41.246436 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:41.246417    2569 scope.go:117] "RemoveContainer" containerID="9b3c8666c165807f7b0773a98b69be0d885017766361f130799f75419015a2fb"
Sep 17 18:30:41.246721 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:30:41.246701    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b3c8666c165807f7b0773a98b69be0d885017766361f130799f75419015a2fb\": container with ID starting with 9b3c8666c165807f7b0773a98b69be0d885017766361f130799f75419015a2fb not found: ID does not exist" containerID="9b3c8666c165807f7b0773a98b69be0d885017766361f130799f75419015a2fb"
Sep 17 18:30:41.246850 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:41.246729    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b3c8666c165807f7b0773a98b69be0d885017766361f130799f75419015a2fb"} err="failed to get container status \"9b3c8666c165807f7b0773a98b69be0d885017766361f130799f75419015a2fb\": rpc error: code = NotFound desc = could not find container \"9b3c8666c165807f7b0773a98b69be0d885017766361f130799f75419015a2fb\": container with ID starting with 9b3c8666c165807f7b0773a98b69be0d885017766361f130799f75419015a2fb not found: ID does not exist"
Sep 17 18:30:41.246850 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:41.246746    2569 scope.go:117] "RemoveContainer" containerID="5b30622b1b780af74b9318a6b13d1a5d353c7a09d508c02a3d537410361e1bfe"
Sep 17 18:30:41.247012 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:30:41.246990    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b30622b1b780af74b9318a6b13d1a5d353c7a09d508c02a3d537410361e1bfe\": container with ID starting with 5b30622b1b780af74b9318a6b13d1a5d353c7a09d508c02a3d537410361e1bfe not found: ID does not exist" containerID="5b30622b1b780af74b9318a6b13d1a5d353c7a09d508c02a3d537410361e1bfe"
Sep 17 18:30:41.247096 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:41.247015    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b30622b1b780af74b9318a6b13d1a5d353c7a09d508c02a3d537410361e1bfe"} err="failed to get container status \"5b30622b1b780af74b9318a6b13d1a5d353c7a09d508c02a3d537410361e1bfe\": rpc error: code = NotFound desc = could not find container \"5b30622b1b780af74b9318a6b13d1a5d353c7a09d508c02a3d537410361e1bfe\": container with ID starting with 5b30622b1b780af74b9318a6b13d1a5d353c7a09d508c02a3d537410361e1bfe not found: ID does not exist"
Sep 17 18:30:41.247096 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:41.247031    2569 scope.go:117] "RemoveContainer" containerID="583447ee1b33cfb7fbfbff937e59c532676fc05245a741cf6c41ac0d7a165cae"
Sep 17 18:30:41.247328 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:30:41.247308    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"583447ee1b33cfb7fbfbff937e59c532676fc05245a741cf6c41ac0d7a165cae\": container with ID starting with 583447ee1b33cfb7fbfbff937e59c532676fc05245a741cf6c41ac0d7a165cae not found: ID does not exist" containerID="583447ee1b33cfb7fbfbff937e59c532676fc05245a741cf6c41ac0d7a165cae"
Sep 17 18:30:41.247403 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:41.247331    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"583447ee1b33cfb7fbfbff937e59c532676fc05245a741cf6c41ac0d7a165cae"} err="failed to get container status \"583447ee1b33cfb7fbfbff937e59c532676fc05245a741cf6c41ac0d7a165cae\": rpc error: code = NotFound desc = could not find container \"583447ee1b33cfb7fbfbff937e59c532676fc05245a741cf6c41ac0d7a165cae\": container with ID starting with 583447ee1b33cfb7fbfbff937e59c532676fc05245a741cf6c41ac0d7a165cae not found: ID does not exist"
Sep 17 18:30:41.247712 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:41.247689    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b527c"]
Sep 17 18:30:41.508399 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:41.508340    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21f11740-b17b-467e-8de2-4f1ad721382f" path="/var/lib/kubelet/pods/21f11740-b17b-467e-8de2-4f1ad721382f/volumes"
Sep 17 18:30:42.159701 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:42.159665    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-7978j" podUID="58a50202-13f2-49b6-acaf-479d2baf2e56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 17 18:30:43.275780 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:43.275749    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nb4dx"]
Sep 17 18:30:43.276252 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:43.276051    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21f11740-b17b-467e-8de2-4f1ad721382f" containerName="kube-rbac-proxy"
Sep 17 18:30:43.276252 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:43.276063    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="21f11740-b17b-467e-8de2-4f1ad721382f" containerName="kube-rbac-proxy"
Sep 17 18:30:43.276252 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:43.276086    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21f11740-b17b-467e-8de2-4f1ad721382f" containerName="storage-initializer"
Sep 17 18:30:43.276252 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:43.276092    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="21f11740-b17b-467e-8de2-4f1ad721382f" containerName="storage-initializer"
Sep 17 18:30:43.276252 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:43.276104    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21f11740-b17b-467e-8de2-4f1ad721382f" containerName="kserve-container"
Sep 17 18:30:43.276252 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:43.276112    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="21f11740-b17b-467e-8de2-4f1ad721382f" containerName="kserve-container"
Sep 17 18:30:43.276252 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:43.276172    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="21f11740-b17b-467e-8de2-4f1ad721382f" containerName="kserve-container"
Sep 17 18:30:43.276252 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:43.276182    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="21f11740-b17b-467e-8de2-4f1ad721382f" containerName="kube-rbac-proxy"
Sep 17 18:30:43.280908 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:43.280893    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nb4dx"
Sep 17 18:30:43.283737 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:43.283713    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 17 18:30:43.283737 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:43.283725    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:30:43.283904 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:43.283749    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-predictor-serving-cert\""
Sep 17 18:30:43.289275 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:43.289255    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nb4dx"]
Sep 17 18:30:43.432844 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:43.432817    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-nb4dx\" (UID: \"b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nb4dx"
Sep 17 18:30:43.432966 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:43.432852    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-nb4dx\" (UID: \"b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nb4dx"
Sep 17 18:30:43.432966 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:43.432872    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-nb4dx\" (UID: \"b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nb4dx"
Sep 17 18:30:43.432966 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:43.432942    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcvjx\" (UniqueName: \"kubernetes.io/projected/b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1-kube-api-access-dcvjx\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-nb4dx\" (UID: \"b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nb4dx"
Sep 17 18:30:43.433100 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:43.432986    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-nb4dx\" (UID: \"b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nb4dx"
Sep 17 18:30:43.534323 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:43.534261    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-nb4dx\" (UID: \"b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nb4dx"
Sep 17 18:30:43.534323 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:43.534293    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-nb4dx\" (UID: \"b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nb4dx"
Sep 17 18:30:43.534323 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:43.534312    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-nb4dx\" (UID: \"b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nb4dx"
Sep 17 18:30:43.534545 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:43.534432    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcvjx\" (UniqueName: \"kubernetes.io/projected/b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1-kube-api-access-dcvjx\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-nb4dx\" (UID: \"b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nb4dx"
Sep 17 18:30:43.534545 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:43.534461    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-nb4dx\" (UID: \"b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nb4dx"
Sep 17 18:30:43.534628 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:30:43.534573    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-serving-cert: secret "isvc-paddle-v2-kserve-predictor-serving-cert" not found
Sep 17 18:30:43.534672 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:30:43.534639    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1-proxy-tls podName:b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:44.034617713 +0000 UTC m=+2996.538620320 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1-proxy-tls") pod "isvc-paddle-v2-kserve-predictor-64d4c74b4d-nb4dx" (UID: "b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1") : secret "isvc-paddle-v2-kserve-predictor-serving-cert" not found
Sep 17 18:30:43.534672 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:43.534647    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-nb4dx\" (UID: \"b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nb4dx"
Sep 17 18:30:43.534991 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:43.534975    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-nb4dx\" (UID: \"b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nb4dx"
Sep 17 18:30:43.535176 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:43.535047    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-nb4dx\" (UID: \"b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nb4dx"
Sep 17 18:30:43.545328 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:43.545309    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcvjx\" (UniqueName: \"kubernetes.io/projected/b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1-kube-api-access-dcvjx\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-nb4dx\" (UID: \"b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nb4dx"
Sep 17 18:30:44.037961 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:44.037933    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-nb4dx\" (UID: \"b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nb4dx"
Sep 17 18:30:44.040377 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:44.040354    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-nb4dx\" (UID: \"b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nb4dx"
Sep 17 18:30:44.192250 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:44.192225    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nb4dx"
Sep 17 18:30:44.461732 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:44.461709    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nb4dx"]
Sep 17 18:30:44.463430 ip-10-0-131-14 kubenswrapper[2569]: W0917 18:30:44.463401    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9a5688b_bbe7_4405_9b6b_6bea7b50d9f1.slice/crio-c4ad106f78500141870361f66eaf3b2e30bcc70861fa0b3c87a35db03de827fe WatchSource:0}: Error finding container c4ad106f78500141870361f66eaf3b2e30bcc70861fa0b3c87a35db03de827fe: Status 404 returned error can't find the container with id c4ad106f78500141870361f66eaf3b2e30bcc70861fa0b3c87a35db03de827fe
Sep 17 18:30:45.240512 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:45.240477    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nb4dx" event={"ID":"b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1","Type":"ContainerStarted","Data":"50052cc9ee501c5d85efa8bbff6d213fe54d198b4eb19a283f515f693c88b63e"}
Sep 17 18:30:45.240700 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:45.240564    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nb4dx" event={"ID":"b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1","Type":"ContainerStarted","Data":"c4ad106f78500141870361f66eaf3b2e30bcc70861fa0b3c87a35db03de827fe"}
Sep 17 18:30:49.258647 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:49.258615    2569 generic.go:356] "Generic (PLEG): container finished" podID="b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1" containerID="50052cc9ee501c5d85efa8bbff6d213fe54d198b4eb19a283f515f693c88b63e" exitCode=0
Sep 17 18:30:49.259005 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:49.258689    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nb4dx" event={"ID":"b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1","Type":"ContainerDied","Data":"50052cc9ee501c5d85efa8bbff6d213fe54d198b4eb19a283f515f693c88b63e"}
Sep 17 18:30:49.694031 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:49.693949    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-t7pqb_b8d76e28-4655-4d21-98d8-0324e4756aaf/console-operator/2.log"
Sep 17 18:30:49.699567 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:49.699543    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k4mqn_6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff/ovn-acl-logging/0.log"
Sep 17 18:30:49.699879 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:49.699859    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-t7pqb_b8d76e28-4655-4d21-98d8-0324e4756aaf/console-operator/2.log"
Sep 17 18:30:49.705406 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:49.705392    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k4mqn_6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff/ovn-acl-logging/0.log"
Sep 17 18:30:50.264901 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:50.264867    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nb4dx" event={"ID":"b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1","Type":"ContainerStarted","Data":"1168e429f7ea4de6f235d4989b07a4386709ed9fde00f0d6c3ff42c3deb26d33"}
Sep 17 18:30:50.265296 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:50.264955    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nb4dx" event={"ID":"b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1","Type":"ContainerStarted","Data":"bab07d0539cf8bfe459362f3f4e41eea7caae2e516e4dfe24f5ae6b90d33b6d5"}
Sep 17 18:30:50.265296 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:50.265215    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nb4dx"
Sep 17 18:30:50.265405 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:50.265363    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nb4dx"
Sep 17 18:30:50.266639 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:50.266616    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nb4dx" podUID="b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 17 18:30:50.285962 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:50.285909    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nb4dx" podStartSLOduration=2.9964134590000002 podStartE2EDuration="7.285894854s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.289481395s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:50.284035488 +0000 UTC m=+3002.788038115" watchObservedRunningTime="2026-09-17 18:30:50.285894854 +0000 UTC m=+3002.789897481"
Sep 17 18:30:51.269313 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:51.269270    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nb4dx" podUID="b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 17 18:30:52.158966 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:52.158931    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-7978j" podUID="58a50202-13f2-49b6-acaf-479d2baf2e56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 17 18:30:56.274409 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:56.274377    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nb4dx"
Sep 17 18:30:56.274919 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:30:56.274892    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nb4dx" podUID="b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 17 18:31:02.159502 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:02.159466    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-7978j" podUID="58a50202-13f2-49b6-acaf-479d2baf2e56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 17 18:31:06.275756 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:06.275713    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nb4dx" podUID="b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 17 18:31:12.159734 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:12.159689    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-7978j" podUID="58a50202-13f2-49b6-acaf-479d2baf2e56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 17 18:31:16.275760 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:16.275721    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nb4dx" podUID="b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 17 18:31:22.159175 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:22.159135    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-7978j" podUID="58a50202-13f2-49b6-acaf-479d2baf2e56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 17 18:31:26.274995 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:26.274949    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nb4dx" podUID="b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 17 18:31:32.159746 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:32.159657    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-7978j" podUID="58a50202-13f2-49b6-acaf-479d2baf2e56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 17 18:31:36.276042 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:36.276002    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nb4dx"
Sep 17 18:31:42.160263 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:42.160225    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-7978j"
Sep 17 18:31:44.970987 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:44.970956    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nb4dx"]
Sep 17 18:31:44.971407 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:44.971263    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nb4dx" podUID="b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1" containerName="kserve-container" containerID="cri-o://bab07d0539cf8bfe459362f3f4e41eea7caae2e516e4dfe24f5ae6b90d33b6d5" gracePeriod=30
Sep 17 18:31:44.971407 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:44.971278    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nb4dx" podUID="b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1" containerName="kube-rbac-proxy" containerID="cri-o://1168e429f7ea4de6f235d4989b07a4386709ed9fde00f0d6c3ff42c3deb26d33" gracePeriod=30
Sep 17 18:31:45.490139 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:45.490104    2569 generic.go:356] "Generic (PLEG): container finished" podID="b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1" containerID="1168e429f7ea4de6f235d4989b07a4386709ed9fde00f0d6c3ff42c3deb26d33" exitCode=2
Sep 17 18:31:45.490309 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:45.490159    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nb4dx" event={"ID":"b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1","Type":"ContainerDied","Data":"1168e429f7ea4de6f235d4989b07a4386709ed9fde00f0d6c3ff42c3deb26d33"}
Sep 17 18:31:46.269710 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:46.269664    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nb4dx" podUID="b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.54:8643/healthz\": dial tcp 10.133.0.54:8643: connect: connection refused"
Sep 17 18:31:46.275711 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:46.275689    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nb4dx" podUID="b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 17 18:31:47.500179 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:47.500152    2569 generic.go:356] "Generic (PLEG): container finished" podID="b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1" containerID="bab07d0539cf8bfe459362f3f4e41eea7caae2e516e4dfe24f5ae6b90d33b6d5" exitCode=0
Sep 17 18:31:47.500508 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:47.500195    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nb4dx" event={"ID":"b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1","Type":"ContainerDied","Data":"bab07d0539cf8bfe459362f3f4e41eea7caae2e516e4dfe24f5ae6b90d33b6d5"}
Sep 17 18:31:47.500508 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:47.500285    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nb4dx" event={"ID":"b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1","Type":"ContainerDied","Data":"c4ad106f78500141870361f66eaf3b2e30bcc70861fa0b3c87a35db03de827fe"}
Sep 17 18:31:47.500508 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:47.500310    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4ad106f78500141870361f66eaf3b2e30bcc70861fa0b3c87a35db03de827fe"
Sep 17 18:31:47.504065 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:47.504044    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nb4dx"
Sep 17 18:31:47.596575 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:47.596549    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1-kserve-provision-location\") pod \"b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1\" (UID: \"b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1\") "
Sep 17 18:31:47.596698 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:47.596598    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1-kube-rbac-proxy-sar-config\") pod \"b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1\" (UID: \"b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1\") "
Sep 17 18:31:47.596698 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:47.596635    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dcvjx\" (UniqueName: \"kubernetes.io/projected/b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1-kube-api-access-dcvjx\") pod \"b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1\" (UID: \"b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1\") "
Sep 17 18:31:47.596698 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:47.596652    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1-proxy-tls\") pod \"b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1\" (UID: \"b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1\") "
Sep 17 18:31:47.596698 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:47.596680    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1-cabundle-cert\") pod \"b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1\" (UID: \"b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1\") "
Sep 17 18:31:47.597118 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:47.597069    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1" (UID: "b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:31:47.597118 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:47.597103    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1" (UID: "b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:31:47.598692 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:47.598669    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1-kube-api-access-dcvjx" (OuterVolumeSpecName: "kube-api-access-dcvjx") pod "b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1" (UID: "b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1"). InnerVolumeSpecName "kube-api-access-dcvjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:31:47.598797 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:47.598777    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1" (UID: "b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:31:47.606325 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:47.606302    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1" (UID: "b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:31:47.697356 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:47.697335    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1-kserve-provision-location\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:31:47.697356 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:47.697355    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:31:47.697473 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:47.697364    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcvjx\" (UniqueName: \"kubernetes.io/projected/b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1-kube-api-access-dcvjx\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:31:47.697473 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:47.697393    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1-proxy-tls\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:31:47.697473 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:47.697403    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1-cabundle-cert\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:31:48.504490 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:48.504460    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nb4dx"
Sep 17 18:31:48.526260 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:48.526233    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nb4dx"]
Sep 17 18:31:48.528043 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:48.528024    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nb4dx"]
Sep 17 18:31:49.508524 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:49.508494    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1" path="/var/lib/kubelet/pods/b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1/volumes"
Sep 17 18:31:50.179564 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:50.179530    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9k4fm"]
Sep 17 18:31:50.179829 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:50.179817    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1" containerName="kube-rbac-proxy"
Sep 17 18:31:50.179877 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:50.179831    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1" containerName="kube-rbac-proxy"
Sep 17 18:31:50.179877 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:50.179838    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1" containerName="storage-initializer"
Sep 17 18:31:50.179877 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:50.179844    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1" containerName="storage-initializer"
Sep 17 18:31:50.179877 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:50.179859    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1" containerName="kserve-container"
Sep 17 18:31:50.179877 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:50.179865    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1" containerName="kserve-container"
Sep 17 18:31:50.180031 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:50.179910    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1" containerName="kserve-container"
Sep 17 18:31:50.180031 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:50.179919    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9a5688b-bbe7-4405-9b6b-6bea7b50d9f1" containerName="kube-rbac-proxy"
Sep 17 18:31:50.184300 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:50.184282    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9k4fm"
Sep 17 18:31:50.186994 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:50.186975    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-predictor-serving-cert\""
Sep 17 18:31:50.187129 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:50.186995    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:31:50.187129 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:50.186976    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-kube-rbac-proxy-sar-config\""
Sep 17 18:31:50.193226 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:50.193203    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9k4fm"]
Sep 17 18:31:50.316348 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:50.316329    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/449c83c1-8f6b-4820-a739-6bd9937355c9-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-9k4fm\" (UID: \"449c83c1-8f6b-4820-a739-6bd9937355c9\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9k4fm"
Sep 17 18:31:50.316450 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:50.316359    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/449c83c1-8f6b-4820-a739-6bd9937355c9-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-9k4fm\" (UID: \"449c83c1-8f6b-4820-a739-6bd9937355c9\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9k4fm"
Sep 17 18:31:50.316450 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:50.316376    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/449c83c1-8f6b-4820-a739-6bd9937355c9-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-9k4fm\" (UID: \"449c83c1-8f6b-4820-a739-6bd9937355c9\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9k4fm"
Sep 17 18:31:50.316450 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:50.316442    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqnnt\" (UniqueName: \"kubernetes.io/projected/449c83c1-8f6b-4820-a739-6bd9937355c9-kube-api-access-gqnnt\") pod \"isvc-pmml-predictor-8674795f59-9k4fm\" (UID: \"449c83c1-8f6b-4820-a739-6bd9937355c9\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9k4fm"
Sep 17 18:31:50.316552 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:50.316473    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/449c83c1-8f6b-4820-a739-6bd9937355c9-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-9k4fm\" (UID: \"449c83c1-8f6b-4820-a739-6bd9937355c9\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9k4fm"
Sep 17 18:31:50.417613 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:50.417589    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/449c83c1-8f6b-4820-a739-6bd9937355c9-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-9k4fm\" (UID: \"449c83c1-8f6b-4820-a739-6bd9937355c9\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9k4fm"
Sep 17 18:31:50.417715 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:50.417629    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/449c83c1-8f6b-4820-a739-6bd9937355c9-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-9k4fm\" (UID: \"449c83c1-8f6b-4820-a739-6bd9937355c9\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9k4fm"
Sep 17 18:31:50.417715 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:50.417648    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/449c83c1-8f6b-4820-a739-6bd9937355c9-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-9k4fm\" (UID: \"449c83c1-8f6b-4820-a739-6bd9937355c9\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9k4fm"
Sep 17 18:31:50.417715 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:50.417691    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqnnt\" (UniqueName: \"kubernetes.io/projected/449c83c1-8f6b-4820-a739-6bd9937355c9-kube-api-access-gqnnt\") pod \"isvc-pmml-predictor-8674795f59-9k4fm\" (UID: \"449c83c1-8f6b-4820-a739-6bd9937355c9\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9k4fm"
Sep 17 18:31:50.417871 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:50.417717    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/449c83c1-8f6b-4820-a739-6bd9937355c9-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-9k4fm\" (UID: \"449c83c1-8f6b-4820-a739-6bd9937355c9\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9k4fm"
Sep 17 18:31:50.418157 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:50.418139    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/449c83c1-8f6b-4820-a739-6bd9937355c9-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-9k4fm\" (UID: \"449c83c1-8f6b-4820-a739-6bd9937355c9\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9k4fm"
Sep 17 18:31:50.418460 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:50.418439    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/449c83c1-8f6b-4820-a739-6bd9937355c9-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-9k4fm\" (UID: \"449c83c1-8f6b-4820-a739-6bd9937355c9\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9k4fm"
Sep 17 18:31:50.418504 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:50.418439    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/449c83c1-8f6b-4820-a739-6bd9937355c9-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-9k4fm\" (UID: \"449c83c1-8f6b-4820-a739-6bd9937355c9\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9k4fm"
Sep 17 18:31:50.420183 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:50.420164    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/449c83c1-8f6b-4820-a739-6bd9937355c9-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-9k4fm\" (UID: \"449c83c1-8f6b-4820-a739-6bd9937355c9\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9k4fm"
Sep 17 18:31:50.426167 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:50.426147    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqnnt\" (UniqueName: \"kubernetes.io/projected/449c83c1-8f6b-4820-a739-6bd9937355c9-kube-api-access-gqnnt\") pod \"isvc-pmml-predictor-8674795f59-9k4fm\" (UID: \"449c83c1-8f6b-4820-a739-6bd9937355c9\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9k4fm"
Sep 17 18:31:50.494646 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:50.494609    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9k4fm"
Sep 17 18:31:50.765669 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:50.765647    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9k4fm"]
Sep 17 18:31:50.767589 ip-10-0-131-14 kubenswrapper[2569]: W0917 18:31:50.767559    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod449c83c1_8f6b_4820_a739_6bd9937355c9.slice/crio-1fb986929715c446a9672d03b4a54f536181abe8882a410ddfcf2bb001a310f3 WatchSource:0}: Error finding container 1fb986929715c446a9672d03b4a54f536181abe8882a410ddfcf2bb001a310f3: Status 404 returned error can't find the container with id 1fb986929715c446a9672d03b4a54f536181abe8882a410ddfcf2bb001a310f3
Sep 17 18:31:51.517287 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:51.517257    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9k4fm" event={"ID":"449c83c1-8f6b-4820-a739-6bd9937355c9","Type":"ContainerStarted","Data":"32ed7ab9465d4db344d8700685944f7e1dc5cd4e6f047fcd54d07d465c3f1311"}
Sep 17 18:31:51.517501 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:51.517318    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9k4fm" event={"ID":"449c83c1-8f6b-4820-a739-6bd9937355c9","Type":"ContainerStarted","Data":"1fb986929715c446a9672d03b4a54f536181abe8882a410ddfcf2bb001a310f3"}
Sep 17 18:31:52.090181 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:52.090148    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-7978j"]
Sep 17 18:31:52.090607 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:52.090368    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-7978j" podUID="58a50202-13f2-49b6-acaf-479d2baf2e56" containerName="kserve-container" containerID="cri-o://ee05b7286956b634196691ea1ffcf262520200dd19eb96735cc4429ff65dc509" gracePeriod=30
Sep 17 18:31:52.090607 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:52.090417    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-7978j" podUID="58a50202-13f2-49b6-acaf-479d2baf2e56" containerName="kube-rbac-proxy" containerID="cri-o://8c44a471fc9fdc2ea291a6bb66eb91e64d6fa19cf43c0928567270cb041cb0c7" gracePeriod=30
Sep 17 18:31:52.153404 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:52.153373    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-7978j" podUID="58a50202-13f2-49b6-acaf-479d2baf2e56" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.53:8643/healthz\": dial tcp 10.133.0.53:8643: connect: connection refused"
Sep 17 18:31:52.523173 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:52.523144    2569 generic.go:356] "Generic (PLEG): container finished" podID="58a50202-13f2-49b6-acaf-479d2baf2e56" containerID="8c44a471fc9fdc2ea291a6bb66eb91e64d6fa19cf43c0928567270cb041cb0c7" exitCode=2
Sep 17 18:31:52.523343 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:52.523216    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-7978j" event={"ID":"58a50202-13f2-49b6-acaf-479d2baf2e56","Type":"ContainerDied","Data":"8c44a471fc9fdc2ea291a6bb66eb91e64d6fa19cf43c0928567270cb041cb0c7"}
Sep 17 18:31:55.535773 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:55.535737    2569 generic.go:356] "Generic (PLEG): container finished" podID="449c83c1-8f6b-4820-a739-6bd9937355c9" containerID="32ed7ab9465d4db344d8700685944f7e1dc5cd4e6f047fcd54d07d465c3f1311" exitCode=0
Sep 17 18:31:55.536191 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:55.535799    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9k4fm" event={"ID":"449c83c1-8f6b-4820-a739-6bd9937355c9","Type":"ContainerDied","Data":"32ed7ab9465d4db344d8700685944f7e1dc5cd4e6f047fcd54d07d465c3f1311"}
Sep 17 18:31:55.931317 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:55.931297    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-7978j"
Sep 17 18:31:56.063815 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:56.063031    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58a50202-13f2-49b6-acaf-479d2baf2e56-proxy-tls\") pod \"58a50202-13f2-49b6-acaf-479d2baf2e56\" (UID: \"58a50202-13f2-49b6-acaf-479d2baf2e56\") "
Sep 17 18:31:56.063815 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:56.063069    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/58a50202-13f2-49b6-acaf-479d2baf2e56-cabundle-cert\") pod \"58a50202-13f2-49b6-acaf-479d2baf2e56\" (UID: \"58a50202-13f2-49b6-acaf-479d2baf2e56\") "
Sep 17 18:31:56.063815 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:56.063117    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x8g6m\" (UniqueName: \"kubernetes.io/projected/58a50202-13f2-49b6-acaf-479d2baf2e56-kube-api-access-x8g6m\") pod \"58a50202-13f2-49b6-acaf-479d2baf2e56\" (UID: \"58a50202-13f2-49b6-acaf-479d2baf2e56\") "
Sep 17 18:31:56.063815 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:56.063231    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/58a50202-13f2-49b6-acaf-479d2baf2e56-kube-rbac-proxy-sar-config\") pod \"58a50202-13f2-49b6-acaf-479d2baf2e56\" (UID: \"58a50202-13f2-49b6-acaf-479d2baf2e56\") "
Sep 17 18:31:56.063815 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:56.063265    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58a50202-13f2-49b6-acaf-479d2baf2e56-kserve-provision-location\") pod \"58a50202-13f2-49b6-acaf-479d2baf2e56\" (UID: \"58a50202-13f2-49b6-acaf-479d2baf2e56\") "
Sep 17 18:31:56.065320 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:56.065059    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58a50202-13f2-49b6-acaf-479d2baf2e56-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "58a50202-13f2-49b6-acaf-479d2baf2e56" (UID: "58a50202-13f2-49b6-acaf-479d2baf2e56"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:31:56.065320 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:56.065259    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58a50202-13f2-49b6-acaf-479d2baf2e56-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "58a50202-13f2-49b6-acaf-479d2baf2e56" (UID: "58a50202-13f2-49b6-acaf-479d2baf2e56"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:31:56.065660 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:56.065629    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58a50202-13f2-49b6-acaf-479d2baf2e56-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "58a50202-13f2-49b6-acaf-479d2baf2e56" (UID: "58a50202-13f2-49b6-acaf-479d2baf2e56"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:31:56.071911 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:56.071879    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58a50202-13f2-49b6-acaf-479d2baf2e56-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "58a50202-13f2-49b6-acaf-479d2baf2e56" (UID: "58a50202-13f2-49b6-acaf-479d2baf2e56"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:31:56.072916 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:56.072878    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58a50202-13f2-49b6-acaf-479d2baf2e56-kube-api-access-x8g6m" (OuterVolumeSpecName: "kube-api-access-x8g6m") pod "58a50202-13f2-49b6-acaf-479d2baf2e56" (UID: "58a50202-13f2-49b6-acaf-479d2baf2e56"). InnerVolumeSpecName "kube-api-access-x8g6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:31:56.164463 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:56.164437    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/58a50202-13f2-49b6-acaf-479d2baf2e56-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:31:56.164463 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:56.164468    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58a50202-13f2-49b6-acaf-479d2baf2e56-kserve-provision-location\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:31:56.164664 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:56.164482    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58a50202-13f2-49b6-acaf-479d2baf2e56-proxy-tls\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:31:56.164664 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:56.164497    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/58a50202-13f2-49b6-acaf-479d2baf2e56-cabundle-cert\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:31:56.164664 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:56.164511    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8g6m\" (UniqueName: \"kubernetes.io/projected/58a50202-13f2-49b6-acaf-479d2baf2e56-kube-api-access-x8g6m\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:31:56.546659 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:56.546297    2569 generic.go:356] "Generic (PLEG): container finished" podID="58a50202-13f2-49b6-acaf-479d2baf2e56" containerID="ee05b7286956b634196691ea1ffcf262520200dd19eb96735cc4429ff65dc509" exitCode=0
Sep 17 18:31:56.546659 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:56.546415    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-7978j" event={"ID":"58a50202-13f2-49b6-acaf-479d2baf2e56","Type":"ContainerDied","Data":"ee05b7286956b634196691ea1ffcf262520200dd19eb96735cc4429ff65dc509"}
Sep 17 18:31:56.546659 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:56.546474    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-7978j" event={"ID":"58a50202-13f2-49b6-acaf-479d2baf2e56","Type":"ContainerDied","Data":"309d52e830e8a5d70d0ef160eeef00daf2c15a09a0c1d4181a2471a48763d97d"}
Sep 17 18:31:56.546659 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:56.546513    2569 scope.go:117] "RemoveContainer" containerID="8c44a471fc9fdc2ea291a6bb66eb91e64d6fa19cf43c0928567270cb041cb0c7"
Sep 17 18:31:56.547319 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:56.546708    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-7978j"
Sep 17 18:31:56.561069 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:56.560506    2569 scope.go:117] "RemoveContainer" containerID="ee05b7286956b634196691ea1ffcf262520200dd19eb96735cc4429ff65dc509"
Sep 17 18:31:56.572388 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:56.572372    2569 scope.go:117] "RemoveContainer" containerID="102fc1e540fdf6d95f019388de0feee1706262436985b2c80c07a6443099de2f"
Sep 17 18:31:56.576871 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:56.576821    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-7978j"]
Sep 17 18:31:56.581061 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:56.581039    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5685b5dd95-7978j"]
Sep 17 18:31:56.587742 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:56.587723    2569 scope.go:117] "RemoveContainer" containerID="8c44a471fc9fdc2ea291a6bb66eb91e64d6fa19cf43c0928567270cb041cb0c7"
Sep 17 18:31:56.588120 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:31:56.588093    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c44a471fc9fdc2ea291a6bb66eb91e64d6fa19cf43c0928567270cb041cb0c7\": container with ID starting with 8c44a471fc9fdc2ea291a6bb66eb91e64d6fa19cf43c0928567270cb041cb0c7 not found: ID does not exist" containerID="8c44a471fc9fdc2ea291a6bb66eb91e64d6fa19cf43c0928567270cb041cb0c7"
Sep 17 18:31:56.588212 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:56.588131    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c44a471fc9fdc2ea291a6bb66eb91e64d6fa19cf43c0928567270cb041cb0c7"} err="failed to get container status \"8c44a471fc9fdc2ea291a6bb66eb91e64d6fa19cf43c0928567270cb041cb0c7\": rpc error: code = NotFound desc = could not find container \"8c44a471fc9fdc2ea291a6bb66eb91e64d6fa19cf43c0928567270cb041cb0c7\": container with ID starting with 8c44a471fc9fdc2ea291a6bb66eb91e64d6fa19cf43c0928567270cb041cb0c7 not found: ID does not exist"
Sep 17 18:31:56.588212 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:56.588155    2569 scope.go:117] "RemoveContainer" containerID="ee05b7286956b634196691ea1ffcf262520200dd19eb96735cc4429ff65dc509"
Sep 17 18:31:56.588768 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:31:56.588642    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee05b7286956b634196691ea1ffcf262520200dd19eb96735cc4429ff65dc509\": container with ID starting with ee05b7286956b634196691ea1ffcf262520200dd19eb96735cc4429ff65dc509 not found: ID does not exist" containerID="ee05b7286956b634196691ea1ffcf262520200dd19eb96735cc4429ff65dc509"
Sep 17 18:31:56.588768 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:56.588672    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee05b7286956b634196691ea1ffcf262520200dd19eb96735cc4429ff65dc509"} err="failed to get container status \"ee05b7286956b634196691ea1ffcf262520200dd19eb96735cc4429ff65dc509\": rpc error: code = NotFound desc = could not find container \"ee05b7286956b634196691ea1ffcf262520200dd19eb96735cc4429ff65dc509\": container with ID starting with ee05b7286956b634196691ea1ffcf262520200dd19eb96735cc4429ff65dc509 not found: ID does not exist"
Sep 17 18:31:56.588768 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:56.588692    2569 scope.go:117] "RemoveContainer" containerID="102fc1e540fdf6d95f019388de0feee1706262436985b2c80c07a6443099de2f"
Sep 17 18:31:56.589209 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:31:56.589148    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"102fc1e540fdf6d95f019388de0feee1706262436985b2c80c07a6443099de2f\": container with ID starting with 102fc1e540fdf6d95f019388de0feee1706262436985b2c80c07a6443099de2f not found: ID does not exist" containerID="102fc1e540fdf6d95f019388de0feee1706262436985b2c80c07a6443099de2f"
Sep 17 18:31:56.589209 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:56.589179    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"102fc1e540fdf6d95f019388de0feee1706262436985b2c80c07a6443099de2f"} err="failed to get container status \"102fc1e540fdf6d95f019388de0feee1706262436985b2c80c07a6443099de2f\": rpc error: code = NotFound desc = could not find container \"102fc1e540fdf6d95f019388de0feee1706262436985b2c80c07a6443099de2f\": container with ID starting with 102fc1e540fdf6d95f019388de0feee1706262436985b2c80c07a6443099de2f not found: ID does not exist"
Sep 17 18:31:57.510665 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:31:57.510619    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58a50202-13f2-49b6-acaf-479d2baf2e56" path="/var/lib/kubelet/pods/58a50202-13f2-49b6-acaf-479d2baf2e56/volumes"
Sep 17 18:32:03.584940 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:32:03.584900    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9k4fm" event={"ID":"449c83c1-8f6b-4820-a739-6bd9937355c9","Type":"ContainerStarted","Data":"3b6a29fe15ad494b500634821ec80322014f4af4a867653e4904c01152031c68"}
Sep 17 18:32:03.585407 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:32:03.584961    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9k4fm" event={"ID":"449c83c1-8f6b-4820-a739-6bd9937355c9","Type":"ContainerStarted","Data":"e520e8c20a7001cb495de4176dbb9752a7a89a0b859aaa0864566291868073b9"}
Sep 17 18:32:03.585407 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:32:03.585208    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9k4fm"
Sep 17 18:32:03.609198 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:32:03.609150    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9k4fm" podStartSLOduration=2.617986758 podStartE2EDuration="13.609137903s" totalImagesPullingTime="7.21576844s" totalInitContainerRuntime="3.775382705s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:31:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:32:03.607900265 +0000 UTC m=+3076.111902891" watchObservedRunningTime="2026-09-17 18:32:03.609137903 +0000 UTC m=+3076.113140541"
Sep 17 18:32:04.588808 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:32:04.588769    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9k4fm"
Sep 17 18:32:04.590183 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:32:04.590156    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9k4fm" podUID="449c83c1-8f6b-4820-a739-6bd9937355c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 17 18:32:05.592934 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:32:05.592887    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9k4fm" podUID="449c83c1-8f6b-4820-a739-6bd9937355c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 17 18:32:10.598230 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:32:10.598201    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9k4fm"
Sep 17 18:32:10.598635 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:32:10.598610    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9k4fm" podUID="449c83c1-8f6b-4820-a739-6bd9937355c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 17 18:32:20.599207 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:32:20.599163    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9k4fm" podUID="449c83c1-8f6b-4820-a739-6bd9937355c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 17 18:32:30.598657 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:32:30.598610    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9k4fm" podUID="449c83c1-8f6b-4820-a739-6bd9937355c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 17 18:32:40.598737 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:32:40.598697    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9k4fm" podUID="449c83c1-8f6b-4820-a739-6bd9937355c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 17 18:32:50.598767 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:32:50.598728    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9k4fm" podUID="449c83c1-8f6b-4820-a739-6bd9937355c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 17 18:33:00.598865 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:00.598826    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9k4fm" podUID="449c83c1-8f6b-4820-a739-6bd9937355c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 17 18:33:10.599688 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:10.599649    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9k4fm" podUID="449c83c1-8f6b-4820-a739-6bd9937355c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 17 18:33:20.599402 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:20.599358    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9k4fm" podUID="449c83c1-8f6b-4820-a739-6bd9937355c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 17 18:33:25.507579 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:25.507551    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9k4fm"
Sep 17 18:33:31.872999 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:31.872969    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9k4fm"]
Sep 17 18:33:31.873465 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:31.873231    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9k4fm" podUID="449c83c1-8f6b-4820-a739-6bd9937355c9" containerName="kserve-container" containerID="cri-o://e520e8c20a7001cb495de4176dbb9752a7a89a0b859aaa0864566291868073b9" gracePeriod=30
Sep 17 18:33:31.873465 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:31.873284    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9k4fm" podUID="449c83c1-8f6b-4820-a739-6bd9937355c9" containerName="kube-rbac-proxy" containerID="cri-o://3b6a29fe15ad494b500634821ec80322014f4af4a867653e4904c01152031c68" gracePeriod=30
Sep 17 18:33:32.957320 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:32.957283    2569 generic.go:356] "Generic (PLEG): container finished" podID="449c83c1-8f6b-4820-a739-6bd9937355c9" containerID="3b6a29fe15ad494b500634821ec80322014f4af4a867653e4904c01152031c68" exitCode=2
Sep 17 18:33:32.957694 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:32.957351    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9k4fm" event={"ID":"449c83c1-8f6b-4820-a739-6bd9937355c9","Type":"ContainerDied","Data":"3b6a29fe15ad494b500634821ec80322014f4af4a867653e4904c01152031c68"}
Sep 17 18:33:35.503353 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:35.503306    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9k4fm" podUID="449c83c1-8f6b-4820-a739-6bd9937355c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 17 18:33:35.593268 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:35.593224    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9k4fm" podUID="449c83c1-8f6b-4820-a739-6bd9937355c9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.55:8643/healthz\": dial tcp 10.133.0.55:8643: connect: connection refused"
Sep 17 18:33:36.217911 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:36.217885    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9k4fm"
Sep 17 18:33:36.323205 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:36.323115    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/449c83c1-8f6b-4820-a739-6bd9937355c9-cabundle-cert\") pod \"449c83c1-8f6b-4820-a739-6bd9937355c9\" (UID: \"449c83c1-8f6b-4820-a739-6bd9937355c9\") "
Sep 17 18:33:36.323205 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:36.323162    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gqnnt\" (UniqueName: \"kubernetes.io/projected/449c83c1-8f6b-4820-a739-6bd9937355c9-kube-api-access-gqnnt\") pod \"449c83c1-8f6b-4820-a739-6bd9937355c9\" (UID: \"449c83c1-8f6b-4820-a739-6bd9937355c9\") "
Sep 17 18:33:36.323205 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:36.323195    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/449c83c1-8f6b-4820-a739-6bd9937355c9-kube-rbac-proxy-sar-config\") pod \"449c83c1-8f6b-4820-a739-6bd9937355c9\" (UID: \"449c83c1-8f6b-4820-a739-6bd9937355c9\") "
Sep 17 18:33:36.323505 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:36.323212    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/449c83c1-8f6b-4820-a739-6bd9937355c9-proxy-tls\") pod \"449c83c1-8f6b-4820-a739-6bd9937355c9\" (UID: \"449c83c1-8f6b-4820-a739-6bd9937355c9\") "
Sep 17 18:33:36.323505 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:36.323237    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/449c83c1-8f6b-4820-a739-6bd9937355c9-kserve-provision-location\") pod \"449c83c1-8f6b-4820-a739-6bd9937355c9\" (UID: \"449c83c1-8f6b-4820-a739-6bd9937355c9\") "
Sep 17 18:33:36.323612 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:36.323519    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/449c83c1-8f6b-4820-a739-6bd9937355c9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "449c83c1-8f6b-4820-a739-6bd9937355c9" (UID: "449c83c1-8f6b-4820-a739-6bd9937355c9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:33:36.323612 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:36.323593    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/449c83c1-8f6b-4820-a739-6bd9937355c9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "449c83c1-8f6b-4820-a739-6bd9937355c9" (UID: "449c83c1-8f6b-4820-a739-6bd9937355c9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:33:36.323696 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:36.323613    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/449c83c1-8f6b-4820-a739-6bd9937355c9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "449c83c1-8f6b-4820-a739-6bd9937355c9" (UID: "449c83c1-8f6b-4820-a739-6bd9937355c9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:33:36.325386 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:36.325361    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/449c83c1-8f6b-4820-a739-6bd9937355c9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "449c83c1-8f6b-4820-a739-6bd9937355c9" (UID: "449c83c1-8f6b-4820-a739-6bd9937355c9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:33:36.325496 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:36.325464    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/449c83c1-8f6b-4820-a739-6bd9937355c9-kube-api-access-gqnnt" (OuterVolumeSpecName: "kube-api-access-gqnnt") pod "449c83c1-8f6b-4820-a739-6bd9937355c9" (UID: "449c83c1-8f6b-4820-a739-6bd9937355c9"). InnerVolumeSpecName "kube-api-access-gqnnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:33:36.424016 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:36.423983    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/449c83c1-8f6b-4820-a739-6bd9937355c9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:33:36.424016 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:36.424012    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/449c83c1-8f6b-4820-a739-6bd9937355c9-proxy-tls\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:33:36.424250 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:36.424025    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/449c83c1-8f6b-4820-a739-6bd9937355c9-kserve-provision-location\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:33:36.424250 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:36.424034    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/449c83c1-8f6b-4820-a739-6bd9937355c9-cabundle-cert\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:33:36.424250 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:36.424043    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqnnt\" (UniqueName: \"kubernetes.io/projected/449c83c1-8f6b-4820-a739-6bd9937355c9-kube-api-access-gqnnt\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:33:36.976781 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:36.976747    2569 generic.go:356] "Generic (PLEG): container finished" podID="449c83c1-8f6b-4820-a739-6bd9937355c9" containerID="e520e8c20a7001cb495de4176dbb9752a7a89a0b859aaa0864566291868073b9" exitCode=0
Sep 17 18:33:36.977197 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:36.976806    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9k4fm"
Sep 17 18:33:36.977197 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:36.976828    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9k4fm" event={"ID":"449c83c1-8f6b-4820-a739-6bd9937355c9","Type":"ContainerDied","Data":"e520e8c20a7001cb495de4176dbb9752a7a89a0b859aaa0864566291868073b9"}
Sep 17 18:33:36.977197 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:36.976890    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9k4fm" event={"ID":"449c83c1-8f6b-4820-a739-6bd9937355c9","Type":"ContainerDied","Data":"1fb986929715c446a9672d03b4a54f536181abe8882a410ddfcf2bb001a310f3"}
Sep 17 18:33:36.977197 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:36.976916    2569 scope.go:117] "RemoveContainer" containerID="3b6a29fe15ad494b500634821ec80322014f4af4a867653e4904c01152031c68"
Sep 17 18:33:36.986376 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:36.986356    2569 scope.go:117] "RemoveContainer" containerID="e520e8c20a7001cb495de4176dbb9752a7a89a0b859aaa0864566291868073b9"
Sep 17 18:33:36.994104 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:36.994066    2569 scope.go:117] "RemoveContainer" containerID="32ed7ab9465d4db344d8700685944f7e1dc5cd4e6f047fcd54d07d465c3f1311"
Sep 17 18:33:37.005140 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:37.005116    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9k4fm"]
Sep 17 18:33:37.005461 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:37.005435    2569 scope.go:117] "RemoveContainer" containerID="3b6a29fe15ad494b500634821ec80322014f4af4a867653e4904c01152031c68"
Sep 17 18:33:37.005884 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:33:37.005840    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b6a29fe15ad494b500634821ec80322014f4af4a867653e4904c01152031c68\": container with ID starting with 3b6a29fe15ad494b500634821ec80322014f4af4a867653e4904c01152031c68 not found: ID does not exist" containerID="3b6a29fe15ad494b500634821ec80322014f4af4a867653e4904c01152031c68"
Sep 17 18:33:37.005994 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:37.005894    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b6a29fe15ad494b500634821ec80322014f4af4a867653e4904c01152031c68"} err="failed to get container status \"3b6a29fe15ad494b500634821ec80322014f4af4a867653e4904c01152031c68\": rpc error: code = NotFound desc = could not find container \"3b6a29fe15ad494b500634821ec80322014f4af4a867653e4904c01152031c68\": container with ID starting with 3b6a29fe15ad494b500634821ec80322014f4af4a867653e4904c01152031c68 not found: ID does not exist"
Sep 17 18:33:37.005994 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:37.005920    2569 scope.go:117] "RemoveContainer" containerID="e520e8c20a7001cb495de4176dbb9752a7a89a0b859aaa0864566291868073b9"
Sep 17 18:33:37.006570 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:33:37.006544    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e520e8c20a7001cb495de4176dbb9752a7a89a0b859aaa0864566291868073b9\": container with ID starting with e520e8c20a7001cb495de4176dbb9752a7a89a0b859aaa0864566291868073b9 not found: ID does not exist" containerID="e520e8c20a7001cb495de4176dbb9752a7a89a0b859aaa0864566291868073b9"
Sep 17 18:33:37.006683 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:37.006578    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e520e8c20a7001cb495de4176dbb9752a7a89a0b859aaa0864566291868073b9"} err="failed to get container status \"e520e8c20a7001cb495de4176dbb9752a7a89a0b859aaa0864566291868073b9\": rpc error: code = NotFound desc = could not find container \"e520e8c20a7001cb495de4176dbb9752a7a89a0b859aaa0864566291868073b9\": container with ID starting with e520e8c20a7001cb495de4176dbb9752a7a89a0b859aaa0864566291868073b9 not found: ID does not exist"
Sep 17 18:33:37.006683 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:37.006599    2569 scope.go:117] "RemoveContainer" containerID="32ed7ab9465d4db344d8700685944f7e1dc5cd4e6f047fcd54d07d465c3f1311"
Sep 17 18:33:37.006911 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:33:37.006890    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32ed7ab9465d4db344d8700685944f7e1dc5cd4e6f047fcd54d07d465c3f1311\": container with ID starting with 32ed7ab9465d4db344d8700685944f7e1dc5cd4e6f047fcd54d07d465c3f1311 not found: ID does not exist" containerID="32ed7ab9465d4db344d8700685944f7e1dc5cd4e6f047fcd54d07d465c3f1311"
Sep 17 18:33:37.006952 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:37.006919    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32ed7ab9465d4db344d8700685944f7e1dc5cd4e6f047fcd54d07d465c3f1311"} err="failed to get container status \"32ed7ab9465d4db344d8700685944f7e1dc5cd4e6f047fcd54d07d465c3f1311\": rpc error: code = NotFound desc = could not find container \"32ed7ab9465d4db344d8700685944f7e1dc5cd4e6f047fcd54d07d465c3f1311\": container with ID starting with 32ed7ab9465d4db344d8700685944f7e1dc5cd4e6f047fcd54d07d465c3f1311 not found: ID does not exist"
Sep 17 18:33:37.008519 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:37.008501    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9k4fm"]
Sep 17 18:33:37.084821 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:37.084792    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-6a05e5-predictor-84867f98d4-2m2f5"]
Sep 17 18:33:37.085091 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:37.085067    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="449c83c1-8f6b-4820-a739-6bd9937355c9" containerName="storage-initializer"
Sep 17 18:33:37.085140 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:37.085096    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="449c83c1-8f6b-4820-a739-6bd9937355c9" containerName="storage-initializer"
Sep 17 18:33:37.085140 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:37.085109    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="449c83c1-8f6b-4820-a739-6bd9937355c9" containerName="kube-rbac-proxy"
Sep 17 18:33:37.085140 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:37.085117    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="449c83c1-8f6b-4820-a739-6bd9937355c9" containerName="kube-rbac-proxy"
Sep 17 18:33:37.085140 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:37.085127    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58a50202-13f2-49b6-acaf-479d2baf2e56" containerName="kube-rbac-proxy"
Sep 17 18:33:37.085140 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:37.085136    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="58a50202-13f2-49b6-acaf-479d2baf2e56" containerName="kube-rbac-proxy"
Sep 17 18:33:37.085295 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:37.085149    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="449c83c1-8f6b-4820-a739-6bd9937355c9" containerName="kserve-container"
Sep 17 18:33:37.085295 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:37.085154    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="449c83c1-8f6b-4820-a739-6bd9937355c9" containerName="kserve-container"
Sep 17 18:33:37.085295 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:37.085185    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58a50202-13f2-49b6-acaf-479d2baf2e56" containerName="storage-initializer"
Sep 17 18:33:37.085295 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:37.085190    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="58a50202-13f2-49b6-acaf-479d2baf2e56" containerName="storage-initializer"
Sep 17 18:33:37.085295 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:37.085203    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58a50202-13f2-49b6-acaf-479d2baf2e56" containerName="kserve-container"
Sep 17 18:33:37.085295 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:37.085208    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="58a50202-13f2-49b6-acaf-479d2baf2e56" containerName="kserve-container"
Sep 17 18:33:37.085295 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:37.085264    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="58a50202-13f2-49b6-acaf-479d2baf2e56" containerName="kube-rbac-proxy"
Sep 17 18:33:37.085295 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:37.085272    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="449c83c1-8f6b-4820-a739-6bd9937355c9" containerName="kserve-container"
Sep 17 18:33:37.085295 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:37.085279    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="58a50202-13f2-49b6-acaf-479d2baf2e56" containerName="kserve-container"
Sep 17 18:33:37.085295 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:37.085286    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="449c83c1-8f6b-4820-a739-6bd9937355c9" containerName="kube-rbac-proxy"
Sep 17 18:33:37.089736 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:37.089719    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-6a05e5-predictor-84867f98d4-2m2f5"
Sep 17 18:33:37.092372 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:37.092344    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-6a05e5-predictor-serving-cert\""
Sep 17 18:33:37.092491 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:37.092443    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-6a05e5-kube-rbac-proxy-sar-config\""
Sep 17 18:33:37.092491 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:37.092466    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:33:37.092608 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:37.092491    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:33:37.092608 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:37.092601    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-kkx8f\""
Sep 17 18:33:37.092753 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:37.092738    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:33:37.092826 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:37.092812    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:33:37.098360 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:37.098337    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-6a05e5-predictor-84867f98d4-2m2f5"]
Sep 17 18:33:37.230110 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:37.230007    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/258355ba-6e6e-4c20-97df-7f293355c428-cabundle-cert\") pod \"isvc-primary-6a05e5-predictor-84867f98d4-2m2f5\" (UID: \"258355ba-6e6e-4c20-97df-7f293355c428\") " pod="kserve-ci-e2e-test/isvc-primary-6a05e5-predictor-84867f98d4-2m2f5"
Sep 17 18:33:37.230110 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:37.230050    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5s4d\" (UniqueName: \"kubernetes.io/projected/258355ba-6e6e-4c20-97df-7f293355c428-kube-api-access-f5s4d\") pod \"isvc-primary-6a05e5-predictor-84867f98d4-2m2f5\" (UID: \"258355ba-6e6e-4c20-97df-7f293355c428\") " pod="kserve-ci-e2e-test/isvc-primary-6a05e5-predictor-84867f98d4-2m2f5"
Sep 17 18:33:37.230110 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:37.230101    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/258355ba-6e6e-4c20-97df-7f293355c428-kube-rbac-proxy-sar-config\") pod \"isvc-primary-6a05e5-predictor-84867f98d4-2m2f5\" (UID: \"258355ba-6e6e-4c20-97df-7f293355c428\") " pod="kserve-ci-e2e-test/isvc-primary-6a05e5-predictor-84867f98d4-2m2f5"
Sep 17 18:33:37.230344 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:37.230128    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/258355ba-6e6e-4c20-97df-7f293355c428-proxy-tls\") pod \"isvc-primary-6a05e5-predictor-84867f98d4-2m2f5\" (UID: \"258355ba-6e6e-4c20-97df-7f293355c428\") " pod="kserve-ci-e2e-test/isvc-primary-6a05e5-predictor-84867f98d4-2m2f5"
Sep 17 18:33:37.230344 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:37.230158    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/258355ba-6e6e-4c20-97df-7f293355c428-kserve-provision-location\") pod \"isvc-primary-6a05e5-predictor-84867f98d4-2m2f5\" (UID: \"258355ba-6e6e-4c20-97df-7f293355c428\") " pod="kserve-ci-e2e-test/isvc-primary-6a05e5-predictor-84867f98d4-2m2f5"
Sep 17 18:33:37.330595 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:37.330555    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/258355ba-6e6e-4c20-97df-7f293355c428-kube-rbac-proxy-sar-config\") pod \"isvc-primary-6a05e5-predictor-84867f98d4-2m2f5\" (UID: \"258355ba-6e6e-4c20-97df-7f293355c428\") " pod="kserve-ci-e2e-test/isvc-primary-6a05e5-predictor-84867f98d4-2m2f5"
Sep 17 18:33:37.330774 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:37.330606    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/258355ba-6e6e-4c20-97df-7f293355c428-proxy-tls\") pod \"isvc-primary-6a05e5-predictor-84867f98d4-2m2f5\" (UID: \"258355ba-6e6e-4c20-97df-7f293355c428\") " pod="kserve-ci-e2e-test/isvc-primary-6a05e5-predictor-84867f98d4-2m2f5"
Sep 17 18:33:37.330774 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:37.330647    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/258355ba-6e6e-4c20-97df-7f293355c428-kserve-provision-location\") pod \"isvc-primary-6a05e5-predictor-84867f98d4-2m2f5\" (UID: \"258355ba-6e6e-4c20-97df-7f293355c428\") " pod="kserve-ci-e2e-test/isvc-primary-6a05e5-predictor-84867f98d4-2m2f5"
Sep 17 18:33:37.330774 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:37.330691    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/258355ba-6e6e-4c20-97df-7f293355c428-cabundle-cert\") pod \"isvc-primary-6a05e5-predictor-84867f98d4-2m2f5\" (UID: \"258355ba-6e6e-4c20-97df-7f293355c428\") " pod="kserve-ci-e2e-test/isvc-primary-6a05e5-predictor-84867f98d4-2m2f5"
Sep 17 18:33:37.330774 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:37.330723    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5s4d\" (UniqueName: \"kubernetes.io/projected/258355ba-6e6e-4c20-97df-7f293355c428-kube-api-access-f5s4d\") pod \"isvc-primary-6a05e5-predictor-84867f98d4-2m2f5\" (UID: \"258355ba-6e6e-4c20-97df-7f293355c428\") " pod="kserve-ci-e2e-test/isvc-primary-6a05e5-predictor-84867f98d4-2m2f5"
Sep 17 18:33:37.331241 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:37.331217    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/258355ba-6e6e-4c20-97df-7f293355c428-kserve-provision-location\") pod \"isvc-primary-6a05e5-predictor-84867f98d4-2m2f5\" (UID: \"258355ba-6e6e-4c20-97df-7f293355c428\") " pod="kserve-ci-e2e-test/isvc-primary-6a05e5-predictor-84867f98d4-2m2f5"
Sep 17 18:33:37.331468 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:37.331448    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/258355ba-6e6e-4c20-97df-7f293355c428-kube-rbac-proxy-sar-config\") pod \"isvc-primary-6a05e5-predictor-84867f98d4-2m2f5\" (UID: \"258355ba-6e6e-4c20-97df-7f293355c428\") " pod="kserve-ci-e2e-test/isvc-primary-6a05e5-predictor-84867f98d4-2m2f5"
Sep 17 18:33:37.331573 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:37.331550    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/258355ba-6e6e-4c20-97df-7f293355c428-cabundle-cert\") pod \"isvc-primary-6a05e5-predictor-84867f98d4-2m2f5\" (UID: \"258355ba-6e6e-4c20-97df-7f293355c428\") " pod="kserve-ci-e2e-test/isvc-primary-6a05e5-predictor-84867f98d4-2m2f5"
Sep 17 18:33:37.333408 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:37.333392    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/258355ba-6e6e-4c20-97df-7f293355c428-proxy-tls\") pod \"isvc-primary-6a05e5-predictor-84867f98d4-2m2f5\" (UID: \"258355ba-6e6e-4c20-97df-7f293355c428\") " pod="kserve-ci-e2e-test/isvc-primary-6a05e5-predictor-84867f98d4-2m2f5"
Sep 17 18:33:37.339879 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:37.339682    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5s4d\" (UniqueName: \"kubernetes.io/projected/258355ba-6e6e-4c20-97df-7f293355c428-kube-api-access-f5s4d\") pod \"isvc-primary-6a05e5-predictor-84867f98d4-2m2f5\" (UID: \"258355ba-6e6e-4c20-97df-7f293355c428\") " pod="kserve-ci-e2e-test/isvc-primary-6a05e5-predictor-84867f98d4-2m2f5"
Sep 17 18:33:37.400818 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:37.400777    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-6a05e5-predictor-84867f98d4-2m2f5"
Sep 17 18:33:37.508537 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:37.508462    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="449c83c1-8f6b-4820-a739-6bd9937355c9" path="/var/lib/kubelet/pods/449c83c1-8f6b-4820-a739-6bd9937355c9/volumes"
Sep 17 18:33:37.671959 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:37.671906    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-6a05e5-predictor-84867f98d4-2m2f5"]
Sep 17 18:33:37.674252 ip-10-0-131-14 kubenswrapper[2569]: W0917 18:33:37.674225    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod258355ba_6e6e_4c20_97df_7f293355c428.slice/crio-f6ff682222b1c263f20784373f7a00004c70cbd255a139e7d2156595d7e7f681 WatchSource:0}: Error finding container f6ff682222b1c263f20784373f7a00004c70cbd255a139e7d2156595d7e7f681: Status 404 returned error can't find the container with id f6ff682222b1c263f20784373f7a00004c70cbd255a139e7d2156595d7e7f681
Sep 17 18:33:37.983184 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:37.983142    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-6a05e5-predictor-84867f98d4-2m2f5" event={"ID":"258355ba-6e6e-4c20-97df-7f293355c428","Type":"ContainerStarted","Data":"25cb612bbb4cde3fa66e4904a8a7d2cca0b02b32dee32b64724d247a7bb8b4a4"}
Sep 17 18:33:37.983707 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:37.983221    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-6a05e5-predictor-84867f98d4-2m2f5" event={"ID":"258355ba-6e6e-4c20-97df-7f293355c428","Type":"ContainerStarted","Data":"f6ff682222b1c263f20784373f7a00004c70cbd255a139e7d2156595d7e7f681"}
Sep 17 18:33:42.002612 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:42.002583    2569 generic.go:356] "Generic (PLEG): container finished" podID="258355ba-6e6e-4c20-97df-7f293355c428" containerID="25cb612bbb4cde3fa66e4904a8a7d2cca0b02b32dee32b64724d247a7bb8b4a4" exitCode=0
Sep 17 18:33:42.002988 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:42.002661    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-6a05e5-predictor-84867f98d4-2m2f5" event={"ID":"258355ba-6e6e-4c20-97df-7f293355c428","Type":"ContainerDied","Data":"25cb612bbb4cde3fa66e4904a8a7d2cca0b02b32dee32b64724d247a7bb8b4a4"}
Sep 17 18:33:43.008457 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:43.008418    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-6a05e5-predictor-84867f98d4-2m2f5" event={"ID":"258355ba-6e6e-4c20-97df-7f293355c428","Type":"ContainerStarted","Data":"f2ee9ff98f42ff48fc23e7404528838d4f966f17398eef4741192874d36cf672"}
Sep 17 18:33:43.008870 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:43.008515    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-6a05e5-predictor-84867f98d4-2m2f5" event={"ID":"258355ba-6e6e-4c20-97df-7f293355c428","Type":"ContainerStarted","Data":"6031f362ef11a8f0a9511089c96b80624bedd76abeb5fadc0823b834258f2d5b"}
Sep 17 18:33:43.008870 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:43.008783    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-6a05e5-predictor-84867f98d4-2m2f5"
Sep 17 18:33:43.008965 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:43.008945    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-6a05e5-predictor-84867f98d4-2m2f5"
Sep 17 18:33:43.010333 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:43.010305    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-6a05e5-predictor-84867f98d4-2m2f5" podUID="258355ba-6e6e-4c20-97df-7f293355c428" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Sep 17 18:33:43.029540 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:43.029496    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-6a05e5-predictor-84867f98d4-2m2f5" podStartSLOduration=2.225666965 podStartE2EDuration="6.029482603s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.803815638s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:33:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:33:43.028688249 +0000 UTC m=+3175.532690877" watchObservedRunningTime="2026-09-17 18:33:43.029482603 +0000 UTC m=+3175.533485228"
Sep 17 18:33:44.012807 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:44.012767    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-6a05e5-predictor-84867f98d4-2m2f5" podUID="258355ba-6e6e-4c20-97df-7f293355c428" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Sep 17 18:33:49.017092 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:49.017049    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-6a05e5-predictor-84867f98d4-2m2f5"
Sep 17 18:33:49.017571 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:49.017547    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-6a05e5-predictor-84867f98d4-2m2f5" podUID="258355ba-6e6e-4c20-97df-7f293355c428" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Sep 17 18:33:51.126004 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:51.125964    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62"]
Sep 17 18:33:51.130777 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:51.130758    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62"
Sep 17 18:33:51.133208 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:51.133190    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-predictor-serving-cert\""
Sep 17 18:33:51.133297 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:51.133223    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 17 18:33:51.137135 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:51.137067    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmz6j\" (UniqueName: \"kubernetes.io/projected/09d0b7c9-a4ab-46f3-a5cd-1148b58866a3-kube-api-access-tmz6j\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62\" (UID: \"09d0b7c9-a4ab-46f3-a5cd-1148b58866a3\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62"
Sep 17 18:33:51.137135 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:51.137118    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09d0b7c9-a4ab-46f3-a5cd-1148b58866a3-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62\" (UID: \"09d0b7c9-a4ab-46f3-a5cd-1148b58866a3\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62"
Sep 17 18:33:51.137348 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:51.137144    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09d0b7c9-a4ab-46f3-a5cd-1148b58866a3-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62\" (UID: \"09d0b7c9-a4ab-46f3-a5cd-1148b58866a3\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62"
Sep 17 18:33:51.137348 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:51.137163    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09d0b7c9-a4ab-46f3-a5cd-1148b58866a3-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62\" (UID: \"09d0b7c9-a4ab-46f3-a5cd-1148b58866a3\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62"
Sep 17 18:33:51.137348 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:51.137257    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09d0b7c9-a4ab-46f3-a5cd-1148b58866a3-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62\" (UID: \"09d0b7c9-a4ab-46f3-a5cd-1148b58866a3\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62"
Sep 17 18:33:51.139139 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:51.139114    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62"]
Sep 17 18:33:51.238167 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:51.238136    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09d0b7c9-a4ab-46f3-a5cd-1148b58866a3-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62\" (UID: \"09d0b7c9-a4ab-46f3-a5cd-1148b58866a3\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62"
Sep 17 18:33:51.238346 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:51.238299    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmz6j\" (UniqueName: \"kubernetes.io/projected/09d0b7c9-a4ab-46f3-a5cd-1148b58866a3-kube-api-access-tmz6j\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62\" (UID: \"09d0b7c9-a4ab-46f3-a5cd-1148b58866a3\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62"
Sep 17 18:33:51.238346 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:51.238335    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09d0b7c9-a4ab-46f3-a5cd-1148b58866a3-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62\" (UID: \"09d0b7c9-a4ab-46f3-a5cd-1148b58866a3\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62"
Sep 17 18:33:51.238441 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:51.238375    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09d0b7c9-a4ab-46f3-a5cd-1148b58866a3-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62\" (UID: \"09d0b7c9-a4ab-46f3-a5cd-1148b58866a3\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62"
Sep 17 18:33:51.238441 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:51.238404    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09d0b7c9-a4ab-46f3-a5cd-1148b58866a3-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62\" (UID: \"09d0b7c9-a4ab-46f3-a5cd-1148b58866a3\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62"
Sep 17 18:33:51.238552 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:33:51.238530    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-serving-cert: secret "isvc-pmml-v2-kserve-predictor-serving-cert" not found
Sep 17 18:33:51.238631 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:33:51.238615    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09d0b7c9-a4ab-46f3-a5cd-1148b58866a3-proxy-tls podName:09d0b7c9-a4ab-46f3-a5cd-1148b58866a3 nodeName:}" failed. No retries permitted until 2026-09-17 18:33:51.738594744 +0000 UTC m=+3184.242597366 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/09d0b7c9-a4ab-46f3-a5cd-1148b58866a3-proxy-tls") pod "isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62" (UID: "09d0b7c9-a4ab-46f3-a5cd-1148b58866a3") : secret "isvc-pmml-v2-kserve-predictor-serving-cert" not found
Sep 17 18:33:51.238856 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:51.238832    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09d0b7c9-a4ab-46f3-a5cd-1148b58866a3-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62\" (UID: \"09d0b7c9-a4ab-46f3-a5cd-1148b58866a3\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62"
Sep 17 18:33:51.239043 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:51.239021    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09d0b7c9-a4ab-46f3-a5cd-1148b58866a3-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62\" (UID: \"09d0b7c9-a4ab-46f3-a5cd-1148b58866a3\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62"
Sep 17 18:33:51.239104 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:51.239064    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09d0b7c9-a4ab-46f3-a5cd-1148b58866a3-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62\" (UID: \"09d0b7c9-a4ab-46f3-a5cd-1148b58866a3\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62"
Sep 17 18:33:51.251553 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:51.251532    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmz6j\" (UniqueName: \"kubernetes.io/projected/09d0b7c9-a4ab-46f3-a5cd-1148b58866a3-kube-api-access-tmz6j\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62\" (UID: \"09d0b7c9-a4ab-46f3-a5cd-1148b58866a3\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62"
Sep 17 18:33:51.743170 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:51.743134    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09d0b7c9-a4ab-46f3-a5cd-1148b58866a3-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62\" (UID: \"09d0b7c9-a4ab-46f3-a5cd-1148b58866a3\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62"
Sep 17 18:33:51.745635 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:51.745611    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09d0b7c9-a4ab-46f3-a5cd-1148b58866a3-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62\" (UID: \"09d0b7c9-a4ab-46f3-a5cd-1148b58866a3\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62"
Sep 17 18:33:52.041489 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:52.041405    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62"
Sep 17 18:33:52.317859 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:52.317766    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62"]
Sep 17 18:33:52.320730 ip-10-0-131-14 kubenswrapper[2569]: W0917 18:33:52.320707    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09d0b7c9_a4ab_46f3_a5cd_1148b58866a3.slice/crio-ac385430d9f48fbdf42bc2b96bddf089336c846232fa039cd041b477a8c3b539 WatchSource:0}: Error finding container ac385430d9f48fbdf42bc2b96bddf089336c846232fa039cd041b477a8c3b539: Status 404 returned error can't find the container with id ac385430d9f48fbdf42bc2b96bddf089336c846232fa039cd041b477a8c3b539
Sep 17 18:33:53.058148 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:53.058107    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62" event={"ID":"09d0b7c9-a4ab-46f3-a5cd-1148b58866a3","Type":"ContainerStarted","Data":"c24ba4daab64af04368ee0bcb6a3a323e4f3573d1ef2c03966844f623ac964c7"}
Sep 17 18:33:53.058355 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:53.058171    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62" event={"ID":"09d0b7c9-a4ab-46f3-a5cd-1148b58866a3","Type":"ContainerStarted","Data":"ac385430d9f48fbdf42bc2b96bddf089336c846232fa039cd041b477a8c3b539"}
Sep 17 18:33:57.077622 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:57.077584    2569 generic.go:356] "Generic (PLEG): container finished" podID="09d0b7c9-a4ab-46f3-a5cd-1148b58866a3" containerID="c24ba4daab64af04368ee0bcb6a3a323e4f3573d1ef2c03966844f623ac964c7" exitCode=0
Sep 17 18:33:57.078004 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:57.077655    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62" event={"ID":"09d0b7c9-a4ab-46f3-a5cd-1148b58866a3","Type":"ContainerDied","Data":"c24ba4daab64af04368ee0bcb6a3a323e4f3573d1ef2c03966844f623ac964c7"}
Sep 17 18:33:58.084385 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:58.084348    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62" event={"ID":"09d0b7c9-a4ab-46f3-a5cd-1148b58866a3","Type":"ContainerStarted","Data":"1abe3a113991a9b121c787888dd1fa62275f779028a269b8593efeb65bdb56a7"}
Sep 17 18:33:58.084797 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:58.084408    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62" event={"ID":"09d0b7c9-a4ab-46f3-a5cd-1148b58866a3","Type":"ContainerStarted","Data":"bd5b5d3ba70838182174109578d1c99ef588cbf1030eed5e3fa5457d022c6e82"}
Sep 17 18:33:58.084797 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:58.084601    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62"
Sep 17 18:33:58.103522 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:58.103433    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62" podStartSLOduration=3.310722034 podStartE2EDuration="7.103411839s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.792689805s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:33:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:33:58.102680264 +0000 UTC m=+3190.606682884" watchObservedRunningTime="2026-09-17 18:33:58.103411839 +0000 UTC m=+3190.607414465"
Sep 17 18:33:59.017953 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:59.017913    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-6a05e5-predictor-84867f98d4-2m2f5" podUID="258355ba-6e6e-4c20-97df-7f293355c428" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Sep 17 18:33:59.089264 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:59.089237    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62"
Sep 17 18:33:59.090690 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:33:59.090663    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62" podUID="09d0b7c9-a4ab-46f3-a5cd-1148b58866a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 17 18:34:00.093091 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:34:00.093045    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62" podUID="09d0b7c9-a4ab-46f3-a5cd-1148b58866a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 17 18:34:05.097960 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:34:05.097929    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62"
Sep 17 18:34:05.098571 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:34:05.098546    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62" podUID="09d0b7c9-a4ab-46f3-a5cd-1148b58866a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 17 18:34:09.018062 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:34:09.018028    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-6a05e5-predictor-84867f98d4-2m2f5" podUID="258355ba-6e6e-4c20-97df-7f293355c428" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Sep 17 18:34:15.098762 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:34:15.098715    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62" podUID="09d0b7c9-a4ab-46f3-a5cd-1148b58866a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 17 18:34:19.018179 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:34:19.018139    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-6a05e5-predictor-84867f98d4-2m2f5" podUID="258355ba-6e6e-4c20-97df-7f293355c428" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Sep 17 18:34:25.099298 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:34:25.099259    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62" podUID="09d0b7c9-a4ab-46f3-a5cd-1148b58866a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 17 18:34:29.017545 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:34:29.017506    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-6a05e5-predictor-84867f98d4-2m2f5" podUID="258355ba-6e6e-4c20-97df-7f293355c428" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Sep 17 18:34:35.098543 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:34:35.098458    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62" podUID="09d0b7c9-a4ab-46f3-a5cd-1148b58866a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 17 18:34:39.017782 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:34:39.017740    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-6a05e5-predictor-84867f98d4-2m2f5" podUID="258355ba-6e6e-4c20-97df-7f293355c428" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Sep 17 18:34:45.098917 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:34:45.098872    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62" podUID="09d0b7c9-a4ab-46f3-a5cd-1148b58866a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 17 18:34:49.017697 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:34:49.017662    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-6a05e5-predictor-84867f98d4-2m2f5" podUID="258355ba-6e6e-4c20-97df-7f293355c428" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Sep 17 18:34:53.504221 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:34:53.504172    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-6a05e5-predictor-84867f98d4-2m2f5" podUID="258355ba-6e6e-4c20-97df-7f293355c428" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Sep 17 18:34:55.099172 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:34:55.099136    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62" podUID="09d0b7c9-a4ab-46f3-a5cd-1148b58866a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 17 18:35:03.508421 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:03.508392    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-6a05e5-predictor-84867f98d4-2m2f5"
Sep 17 18:35:05.098733 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:05.098695    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62" podUID="09d0b7c9-a4ab-46f3-a5cd-1148b58866a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 17 18:35:15.098884 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:15.098841    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62" podUID="09d0b7c9-a4ab-46f3-a5cd-1148b58866a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 17 18:35:16.503283 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:16.503250    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62" podUID="09d0b7c9-a4ab-46f3-a5cd-1148b58866a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 17 18:35:26.504798 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:26.504765    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62"
Sep 17 18:35:32.933115 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:32.933066    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62"]
Sep 17 18:35:32.933616 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:32.933350    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62" podUID="09d0b7c9-a4ab-46f3-a5cd-1148b58866a3" containerName="kserve-container" containerID="cri-o://bd5b5d3ba70838182174109578d1c99ef588cbf1030eed5e3fa5457d022c6e82" gracePeriod=30
Sep 17 18:35:32.933616 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:32.933351    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62" podUID="09d0b7c9-a4ab-46f3-a5cd-1148b58866a3" containerName="kube-rbac-proxy" containerID="cri-o://1abe3a113991a9b121c787888dd1fa62275f779028a269b8593efeb65bdb56a7" gracePeriod=30
Sep 17 18:35:33.476614 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:33.476579    2569 generic.go:356] "Generic (PLEG): container finished" podID="09d0b7c9-a4ab-46f3-a5cd-1148b58866a3" containerID="1abe3a113991a9b121c787888dd1fa62275f779028a269b8593efeb65bdb56a7" exitCode=2
Sep 17 18:35:33.476614 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:33.476617    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62" event={"ID":"09d0b7c9-a4ab-46f3-a5cd-1148b58866a3","Type":"ContainerDied","Data":"1abe3a113991a9b121c787888dd1fa62275f779028a269b8593efeb65bdb56a7"}
Sep 17 18:35:35.093994 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:35.093949    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62" podUID="09d0b7c9-a4ab-46f3-a5cd-1148b58866a3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.57:8643/healthz\": dial tcp 10.133.0.57:8643: connect: connection refused"
Sep 17 18:35:36.504158 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:36.504120    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62" podUID="09d0b7c9-a4ab-46f3-a5cd-1148b58866a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 17 18:35:37.175709 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:37.175686    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62"
Sep 17 18:35:37.322854 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:37.322748    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tmz6j\" (UniqueName: \"kubernetes.io/projected/09d0b7c9-a4ab-46f3-a5cd-1148b58866a3-kube-api-access-tmz6j\") pod \"09d0b7c9-a4ab-46f3-a5cd-1148b58866a3\" (UID: \"09d0b7c9-a4ab-46f3-a5cd-1148b58866a3\") "
Sep 17 18:35:37.322854 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:37.322811    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09d0b7c9-a4ab-46f3-a5cd-1148b58866a3-proxy-tls\") pod \"09d0b7c9-a4ab-46f3-a5cd-1148b58866a3\" (UID: \"09d0b7c9-a4ab-46f3-a5cd-1148b58866a3\") "
Sep 17 18:35:37.323163 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:37.322859    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09d0b7c9-a4ab-46f3-a5cd-1148b58866a3-kserve-provision-location\") pod \"09d0b7c9-a4ab-46f3-a5cd-1148b58866a3\" (UID: \"09d0b7c9-a4ab-46f3-a5cd-1148b58866a3\") "
Sep 17 18:35:37.323163 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:37.322889    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09d0b7c9-a4ab-46f3-a5cd-1148b58866a3-kube-rbac-proxy-sar-config\") pod \"09d0b7c9-a4ab-46f3-a5cd-1148b58866a3\" (UID: \"09d0b7c9-a4ab-46f3-a5cd-1148b58866a3\") "
Sep 17 18:35:37.323163 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:37.322927    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09d0b7c9-a4ab-46f3-a5cd-1148b58866a3-cabundle-cert\") pod \"09d0b7c9-a4ab-46f3-a5cd-1148b58866a3\" (UID: \"09d0b7c9-a4ab-46f3-a5cd-1148b58866a3\") "
Sep 17 18:35:37.323337 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:37.323222    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09d0b7c9-a4ab-46f3-a5cd-1148b58866a3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "09d0b7c9-a4ab-46f3-a5cd-1148b58866a3" (UID: "09d0b7c9-a4ab-46f3-a5cd-1148b58866a3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:35:37.323337 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:37.323284    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09d0b7c9-a4ab-46f3-a5cd-1148b58866a3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "09d0b7c9-a4ab-46f3-a5cd-1148b58866a3" (UID: "09d0b7c9-a4ab-46f3-a5cd-1148b58866a3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:35:37.323337 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:37.323299    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09d0b7c9-a4ab-46f3-a5cd-1148b58866a3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "09d0b7c9-a4ab-46f3-a5cd-1148b58866a3" (UID: "09d0b7c9-a4ab-46f3-a5cd-1148b58866a3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:35:37.323337 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:37.323302    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09d0b7c9-a4ab-46f3-a5cd-1148b58866a3-kserve-provision-location\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:35:37.325042 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:37.325022    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09d0b7c9-a4ab-46f3-a5cd-1148b58866a3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "09d0b7c9-a4ab-46f3-a5cd-1148b58866a3" (UID: "09d0b7c9-a4ab-46f3-a5cd-1148b58866a3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:35:37.325118 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:37.325050    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09d0b7c9-a4ab-46f3-a5cd-1148b58866a3-kube-api-access-tmz6j" (OuterVolumeSpecName: "kube-api-access-tmz6j") pod "09d0b7c9-a4ab-46f3-a5cd-1148b58866a3" (UID: "09d0b7c9-a4ab-46f3-a5cd-1148b58866a3"). InnerVolumeSpecName "kube-api-access-tmz6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:35:37.424477 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:37.424443    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09d0b7c9-a4ab-46f3-a5cd-1148b58866a3-cabundle-cert\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:35:37.424477 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:37.424472    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmz6j\" (UniqueName: \"kubernetes.io/projected/09d0b7c9-a4ab-46f3-a5cd-1148b58866a3-kube-api-access-tmz6j\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:35:37.424477 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:37.424482    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09d0b7c9-a4ab-46f3-a5cd-1148b58866a3-proxy-tls\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:35:37.424709 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:37.424493    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09d0b7c9-a4ab-46f3-a5cd-1148b58866a3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:35:37.494449 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:37.494411    2569 generic.go:356] "Generic (PLEG): container finished" podID="09d0b7c9-a4ab-46f3-a5cd-1148b58866a3" containerID="bd5b5d3ba70838182174109578d1c99ef588cbf1030eed5e3fa5457d022c6e82" exitCode=0
Sep 17 18:35:37.494599 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:37.494470    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62"
Sep 17 18:35:37.494599 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:37.494480    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62" event={"ID":"09d0b7c9-a4ab-46f3-a5cd-1148b58866a3","Type":"ContainerDied","Data":"bd5b5d3ba70838182174109578d1c99ef588cbf1030eed5e3fa5457d022c6e82"}
Sep 17 18:35:37.494599 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:37.494537    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62" event={"ID":"09d0b7c9-a4ab-46f3-a5cd-1148b58866a3","Type":"ContainerDied","Data":"ac385430d9f48fbdf42bc2b96bddf089336c846232fa039cd041b477a8c3b539"}
Sep 17 18:35:37.494599 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:37.494565    2569 scope.go:117] "RemoveContainer" containerID="1abe3a113991a9b121c787888dd1fa62275f779028a269b8593efeb65bdb56a7"
Sep 17 18:35:37.503100 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:37.503063    2569 scope.go:117] "RemoveContainer" containerID="bd5b5d3ba70838182174109578d1c99ef588cbf1030eed5e3fa5457d022c6e82"
Sep 17 18:35:37.511793 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:37.511776    2569 scope.go:117] "RemoveContainer" containerID="c24ba4daab64af04368ee0bcb6a3a323e4f3573d1ef2c03966844f623ac964c7"
Sep 17 18:35:37.516706 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:37.516688    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62"]
Sep 17 18:35:37.519671 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:37.519654    2569 scope.go:117] "RemoveContainer" containerID="1abe3a113991a9b121c787888dd1fa62275f779028a269b8593efeb65bdb56a7"
Sep 17 18:35:37.519950 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:35:37.519932    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1abe3a113991a9b121c787888dd1fa62275f779028a269b8593efeb65bdb56a7\": container with ID starting with 1abe3a113991a9b121c787888dd1fa62275f779028a269b8593efeb65bdb56a7 not found: ID does not exist" containerID="1abe3a113991a9b121c787888dd1fa62275f779028a269b8593efeb65bdb56a7"
Sep 17 18:35:37.520000 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:37.519960    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1abe3a113991a9b121c787888dd1fa62275f779028a269b8593efeb65bdb56a7"} err="failed to get container status \"1abe3a113991a9b121c787888dd1fa62275f779028a269b8593efeb65bdb56a7\": rpc error: code = NotFound desc = could not find container \"1abe3a113991a9b121c787888dd1fa62275f779028a269b8593efeb65bdb56a7\": container with ID starting with 1abe3a113991a9b121c787888dd1fa62275f779028a269b8593efeb65bdb56a7 not found: ID does not exist"
Sep 17 18:35:37.520000 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:37.519977    2569 scope.go:117] "RemoveContainer" containerID="bd5b5d3ba70838182174109578d1c99ef588cbf1030eed5e3fa5457d022c6e82"
Sep 17 18:35:37.520247 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:35:37.520221    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd5b5d3ba70838182174109578d1c99ef588cbf1030eed5e3fa5457d022c6e82\": container with ID starting with bd5b5d3ba70838182174109578d1c99ef588cbf1030eed5e3fa5457d022c6e82 not found: ID does not exist" containerID="bd5b5d3ba70838182174109578d1c99ef588cbf1030eed5e3fa5457d022c6e82"
Sep 17 18:35:37.520247 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:37.520240    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd5b5d3ba70838182174109578d1c99ef588cbf1030eed5e3fa5457d022c6e82"} err="failed to get container status \"bd5b5d3ba70838182174109578d1c99ef588cbf1030eed5e3fa5457d022c6e82\": rpc error: code = NotFound desc = could not find container \"bd5b5d3ba70838182174109578d1c99ef588cbf1030eed5e3fa5457d022c6e82\": container with ID starting with bd5b5d3ba70838182174109578d1c99ef588cbf1030eed5e3fa5457d022c6e82 not found: ID does not exist"
Sep 17 18:35:37.520338 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:37.520253    2569 scope.go:117] "RemoveContainer" containerID="c24ba4daab64af04368ee0bcb6a3a323e4f3573d1ef2c03966844f623ac964c7"
Sep 17 18:35:37.520467 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:35:37.520449    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c24ba4daab64af04368ee0bcb6a3a323e4f3573d1ef2c03966844f623ac964c7\": container with ID starting with c24ba4daab64af04368ee0bcb6a3a323e4f3573d1ef2c03966844f623ac964c7 not found: ID does not exist" containerID="c24ba4daab64af04368ee0bcb6a3a323e4f3573d1ef2c03966844f623ac964c7"
Sep 17 18:35:37.520511 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:37.520472    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c24ba4daab64af04368ee0bcb6a3a323e4f3573d1ef2c03966844f623ac964c7"} err="failed to get container status \"c24ba4daab64af04368ee0bcb6a3a323e4f3573d1ef2c03966844f623ac964c7\": rpc error: code = NotFound desc = could not find container \"c24ba4daab64af04368ee0bcb6a3a323e4f3573d1ef2c03966844f623ac964c7\": container with ID starting with c24ba4daab64af04368ee0bcb6a3a323e4f3573d1ef2c03966844f623ac964c7 not found: ID does not exist"
Sep 17 18:35:37.520762 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:37.520745    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fgs62"]
Sep 17 18:35:39.507948 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:39.507915    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09d0b7c9-a4ab-46f3-a5cd-1148b58866a3" path="/var/lib/kubelet/pods/09d0b7c9-a4ab-46f3-a5cd-1148b58866a3/volumes"
Sep 17 18:35:42.346634 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:42.346594    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-6a05e5-predictor-84867f98d4-2m2f5"]
Sep 17 18:35:42.347121 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:42.346889    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-6a05e5-predictor-84867f98d4-2m2f5" podUID="258355ba-6e6e-4c20-97df-7f293355c428" containerName="kserve-container" containerID="cri-o://6031f362ef11a8f0a9511089c96b80624bedd76abeb5fadc0823b834258f2d5b" gracePeriod=30
Sep 17 18:35:42.347121 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:42.346937    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-6a05e5-predictor-84867f98d4-2m2f5" podUID="258355ba-6e6e-4c20-97df-7f293355c428" containerName="kube-rbac-proxy" containerID="cri-o://f2ee9ff98f42ff48fc23e7404528838d4f966f17398eef4741192874d36cf672" gracePeriod=30
Sep 17 18:35:42.518773 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:42.518743    2569 generic.go:356] "Generic (PLEG): container finished" podID="258355ba-6e6e-4c20-97df-7f293355c428" containerID="f2ee9ff98f42ff48fc23e7404528838d4f966f17398eef4741192874d36cf672" exitCode=2
Sep 17 18:35:42.518947 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:42.518814    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-6a05e5-predictor-84867f98d4-2m2f5" event={"ID":"258355ba-6e6e-4c20-97df-7f293355c428","Type":"ContainerDied","Data":"f2ee9ff98f42ff48fc23e7404528838d4f966f17398eef4741192874d36cf672"}
Sep 17 18:35:43.504738 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:43.504699    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-6a05e5-predictor-84867f98d4-2m2f5" podUID="258355ba-6e6e-4c20-97df-7f293355c428" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Sep 17 18:35:44.013873 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:44.013819    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-6a05e5-predictor-84867f98d4-2m2f5" podUID="258355ba-6e6e-4c20-97df-7f293355c428" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.56:8643/healthz\": dial tcp 10.133.0.56:8643: connect: connection refused"
Sep 17 18:35:47.095765 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:47.095738    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-6a05e5-predictor-84867f98d4-2m2f5"
Sep 17 18:35:47.206287 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:47.206205    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/258355ba-6e6e-4c20-97df-7f293355c428-kube-rbac-proxy-sar-config\") pod \"258355ba-6e6e-4c20-97df-7f293355c428\" (UID: \"258355ba-6e6e-4c20-97df-7f293355c428\") "
Sep 17 18:35:47.206287 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:47.206240    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/258355ba-6e6e-4c20-97df-7f293355c428-proxy-tls\") pod \"258355ba-6e6e-4c20-97df-7f293355c428\" (UID: \"258355ba-6e6e-4c20-97df-7f293355c428\") "
Sep 17 18:35:47.206287 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:47.206286    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/258355ba-6e6e-4c20-97df-7f293355c428-cabundle-cert\") pod \"258355ba-6e6e-4c20-97df-7f293355c428\" (UID: \"258355ba-6e6e-4c20-97df-7f293355c428\") "
Sep 17 18:35:47.206586 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:47.206310    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f5s4d\" (UniqueName: \"kubernetes.io/projected/258355ba-6e6e-4c20-97df-7f293355c428-kube-api-access-f5s4d\") pod \"258355ba-6e6e-4c20-97df-7f293355c428\" (UID: \"258355ba-6e6e-4c20-97df-7f293355c428\") "
Sep 17 18:35:47.206586 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:47.206363    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/258355ba-6e6e-4c20-97df-7f293355c428-kserve-provision-location\") pod \"258355ba-6e6e-4c20-97df-7f293355c428\" (UID: \"258355ba-6e6e-4c20-97df-7f293355c428\") "
Sep 17 18:35:47.206693 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:47.206665    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/258355ba-6e6e-4c20-97df-7f293355c428-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "258355ba-6e6e-4c20-97df-7f293355c428" (UID: "258355ba-6e6e-4c20-97df-7f293355c428"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:35:47.206746 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:47.206609    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/258355ba-6e6e-4c20-97df-7f293355c428-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "258355ba-6e6e-4c20-97df-7f293355c428" (UID: "258355ba-6e6e-4c20-97df-7f293355c428"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:35:47.206783 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:47.206756    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/258355ba-6e6e-4c20-97df-7f293355c428-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "258355ba-6e6e-4c20-97df-7f293355c428" (UID: "258355ba-6e6e-4c20-97df-7f293355c428"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:35:47.208356 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:47.208335    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/258355ba-6e6e-4c20-97df-7f293355c428-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "258355ba-6e6e-4c20-97df-7f293355c428" (UID: "258355ba-6e6e-4c20-97df-7f293355c428"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:35:47.208504 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:47.208485    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/258355ba-6e6e-4c20-97df-7f293355c428-kube-api-access-f5s4d" (OuterVolumeSpecName: "kube-api-access-f5s4d") pod "258355ba-6e6e-4c20-97df-7f293355c428" (UID: "258355ba-6e6e-4c20-97df-7f293355c428"). InnerVolumeSpecName "kube-api-access-f5s4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:35:47.307930 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:47.307893    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/258355ba-6e6e-4c20-97df-7f293355c428-kserve-provision-location\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:35:47.307930 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:47.307923    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/258355ba-6e6e-4c20-97df-7f293355c428-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:35:47.307930 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:47.307934    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/258355ba-6e6e-4c20-97df-7f293355c428-proxy-tls\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:35:47.308205 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:47.307945    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/258355ba-6e6e-4c20-97df-7f293355c428-cabundle-cert\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:35:47.308205 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:47.307953    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5s4d\" (UniqueName: \"kubernetes.io/projected/258355ba-6e6e-4c20-97df-7f293355c428-kube-api-access-f5s4d\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:35:47.542206 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:47.542176    2569 generic.go:356] "Generic (PLEG): container finished" podID="258355ba-6e6e-4c20-97df-7f293355c428" containerID="6031f362ef11a8f0a9511089c96b80624bedd76abeb5fadc0823b834258f2d5b" exitCode=0
Sep 17 18:35:47.542356 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:47.542228    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-6a05e5-predictor-84867f98d4-2m2f5"
Sep 17 18:35:47.542356 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:47.542241    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-6a05e5-predictor-84867f98d4-2m2f5" event={"ID":"258355ba-6e6e-4c20-97df-7f293355c428","Type":"ContainerDied","Data":"6031f362ef11a8f0a9511089c96b80624bedd76abeb5fadc0823b834258f2d5b"}
Sep 17 18:35:47.542356 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:47.542291    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-6a05e5-predictor-84867f98d4-2m2f5" event={"ID":"258355ba-6e6e-4c20-97df-7f293355c428","Type":"ContainerDied","Data":"f6ff682222b1c263f20784373f7a00004c70cbd255a139e7d2156595d7e7f681"}
Sep 17 18:35:47.542356 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:47.542316    2569 scope.go:117] "RemoveContainer" containerID="f2ee9ff98f42ff48fc23e7404528838d4f966f17398eef4741192874d36cf672"
Sep 17 18:35:47.550317 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:47.550307    2569 scope.go:117] "RemoveContainer" containerID="6031f362ef11a8f0a9511089c96b80624bedd76abeb5fadc0823b834258f2d5b"
Sep 17 18:35:47.557453 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:47.557435    2569 scope.go:117] "RemoveContainer" containerID="25cb612bbb4cde3fa66e4904a8a7d2cca0b02b32dee32b64724d247a7bb8b4a4"
Sep 17 18:35:47.562049 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:47.562031    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-6a05e5-predictor-84867f98d4-2m2f5"]
Sep 17 18:35:47.567440 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:47.567419    2569 scope.go:117] "RemoveContainer" containerID="f2ee9ff98f42ff48fc23e7404528838d4f966f17398eef4741192874d36cf672"
Sep 17 18:35:47.567759 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:47.567735    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-6a05e5-predictor-84867f98d4-2m2f5"]
Sep 17 18:35:47.567861 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:35:47.567780    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2ee9ff98f42ff48fc23e7404528838d4f966f17398eef4741192874d36cf672\": container with ID starting with f2ee9ff98f42ff48fc23e7404528838d4f966f17398eef4741192874d36cf672 not found: ID does not exist" containerID="f2ee9ff98f42ff48fc23e7404528838d4f966f17398eef4741192874d36cf672"
Sep 17 18:35:47.567861 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:47.567817    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2ee9ff98f42ff48fc23e7404528838d4f966f17398eef4741192874d36cf672"} err="failed to get container status \"f2ee9ff98f42ff48fc23e7404528838d4f966f17398eef4741192874d36cf672\": rpc error: code = NotFound desc = could not find container \"f2ee9ff98f42ff48fc23e7404528838d4f966f17398eef4741192874d36cf672\": container with ID starting with f2ee9ff98f42ff48fc23e7404528838d4f966f17398eef4741192874d36cf672 not found: ID does not exist"
Sep 17 18:35:47.567861 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:47.567839    2569 scope.go:117] "RemoveContainer" containerID="6031f362ef11a8f0a9511089c96b80624bedd76abeb5fadc0823b834258f2d5b"
Sep 17 18:35:47.568261 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:35:47.568243    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6031f362ef11a8f0a9511089c96b80624bedd76abeb5fadc0823b834258f2d5b\": container with ID starting with 6031f362ef11a8f0a9511089c96b80624bedd76abeb5fadc0823b834258f2d5b not found: ID does not exist" containerID="6031f362ef11a8f0a9511089c96b80624bedd76abeb5fadc0823b834258f2d5b"
Sep 17 18:35:47.568324 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:47.568266    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6031f362ef11a8f0a9511089c96b80624bedd76abeb5fadc0823b834258f2d5b"} err="failed to get container status \"6031f362ef11a8f0a9511089c96b80624bedd76abeb5fadc0823b834258f2d5b\": rpc error: code = NotFound desc = could not find container \"6031f362ef11a8f0a9511089c96b80624bedd76abeb5fadc0823b834258f2d5b\": container with ID starting with 6031f362ef11a8f0a9511089c96b80624bedd76abeb5fadc0823b834258f2d5b not found: ID does not exist"
Sep 17 18:35:47.568324 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:47.568281    2569 scope.go:117] "RemoveContainer" containerID="25cb612bbb4cde3fa66e4904a8a7d2cca0b02b32dee32b64724d247a7bb8b4a4"
Sep 17 18:35:47.568517 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:35:47.568502    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25cb612bbb4cde3fa66e4904a8a7d2cca0b02b32dee32b64724d247a7bb8b4a4\": container with ID starting with 25cb612bbb4cde3fa66e4904a8a7d2cca0b02b32dee32b64724d247a7bb8b4a4 not found: ID does not exist" containerID="25cb612bbb4cde3fa66e4904a8a7d2cca0b02b32dee32b64724d247a7bb8b4a4"
Sep 17 18:35:47.568559 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:47.568521    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25cb612bbb4cde3fa66e4904a8a7d2cca0b02b32dee32b64724d247a7bb8b4a4"} err="failed to get container status \"25cb612bbb4cde3fa66e4904a8a7d2cca0b02b32dee32b64724d247a7bb8b4a4\": rpc error: code = NotFound desc = could not find container \"25cb612bbb4cde3fa66e4904a8a7d2cca0b02b32dee32b64724d247a7bb8b4a4\": container with ID starting with 25cb612bbb4cde3fa66e4904a8a7d2cca0b02b32dee32b64724d247a7bb8b4a4 not found: ID does not exist"
Sep 17 18:35:48.874357 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:48.874321    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-456fl/must-gather-tmt9k"]
Sep 17 18:35:48.874743 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:48.874614    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09d0b7c9-a4ab-46f3-a5cd-1148b58866a3" containerName="kube-rbac-proxy"
Sep 17 18:35:48.874743 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:48.874624    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="09d0b7c9-a4ab-46f3-a5cd-1148b58866a3" containerName="kube-rbac-proxy"
Sep 17 18:35:48.874743 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:48.874635    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="258355ba-6e6e-4c20-97df-7f293355c428" containerName="kube-rbac-proxy"
Sep 17 18:35:48.874743 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:48.874641    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="258355ba-6e6e-4c20-97df-7f293355c428" containerName="kube-rbac-proxy"
Sep 17 18:35:48.874743 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:48.874658    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="258355ba-6e6e-4c20-97df-7f293355c428" containerName="kserve-container"
Sep 17 18:35:48.874743 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:48.874663    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="258355ba-6e6e-4c20-97df-7f293355c428" containerName="kserve-container"
Sep 17 18:35:48.874743 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:48.874673    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09d0b7c9-a4ab-46f3-a5cd-1148b58866a3" containerName="kserve-container"
Sep 17 18:35:48.874743 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:48.874678    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="09d0b7c9-a4ab-46f3-a5cd-1148b58866a3" containerName="kserve-container"
Sep 17 18:35:48.874743 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:48.874684    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="258355ba-6e6e-4c20-97df-7f293355c428" containerName="storage-initializer"
Sep 17 18:35:48.874743 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:48.874689    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="258355ba-6e6e-4c20-97df-7f293355c428" containerName="storage-initializer"
Sep 17 18:35:48.874743 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:48.874697    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09d0b7c9-a4ab-46f3-a5cd-1148b58866a3" containerName="storage-initializer"
Sep 17 18:35:48.874743 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:48.874702    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="09d0b7c9-a4ab-46f3-a5cd-1148b58866a3" containerName="storage-initializer"
Sep 17 18:35:48.874743 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:48.874744    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="258355ba-6e6e-4c20-97df-7f293355c428" containerName="kserve-container"
Sep 17 18:35:48.875289 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:48.874751    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="09d0b7c9-a4ab-46f3-a5cd-1148b58866a3" containerName="kserve-container"
Sep 17 18:35:48.875289 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:48.874760    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="258355ba-6e6e-4c20-97df-7f293355c428" containerName="kube-rbac-proxy"
Sep 17 18:35:48.875289 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:48.874766    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="09d0b7c9-a4ab-46f3-a5cd-1148b58866a3" containerName="kube-rbac-proxy"
Sep 17 18:35:48.879072 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:48.879056    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-456fl/must-gather-tmt9k"
Sep 17 18:35:48.881708 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:48.881684    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-456fl\"/\"openshift-service-ca.crt\""
Sep 17 18:35:48.881850 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:48.881727    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-456fl\"/\"kube-root-ca.crt\""
Sep 17 18:35:48.887097 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:48.887063    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-456fl/must-gather-tmt9k"]
Sep 17 18:35:48.921280 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:48.921253    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bknvk\" (UniqueName: \"kubernetes.io/projected/7e65e864-768d-41f2-8261-49b905c3774d-kube-api-access-bknvk\") pod \"must-gather-tmt9k\" (UID: \"7e65e864-768d-41f2-8261-49b905c3774d\") " pod="openshift-must-gather-456fl/must-gather-tmt9k"
Sep 17 18:35:48.921398 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:48.921287    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7e65e864-768d-41f2-8261-49b905c3774d-must-gather-output\") pod \"must-gather-tmt9k\" (UID: \"7e65e864-768d-41f2-8261-49b905c3774d\") " pod="openshift-must-gather-456fl/must-gather-tmt9k"
Sep 17 18:35:49.022229 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:49.022200    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bknvk\" (UniqueName: \"kubernetes.io/projected/7e65e864-768d-41f2-8261-49b905c3774d-kube-api-access-bknvk\") pod \"must-gather-tmt9k\" (UID: \"7e65e864-768d-41f2-8261-49b905c3774d\") " pod="openshift-must-gather-456fl/must-gather-tmt9k"
Sep 17 18:35:49.022433 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:49.022249    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7e65e864-768d-41f2-8261-49b905c3774d-must-gather-output\") pod \"must-gather-tmt9k\" (UID: \"7e65e864-768d-41f2-8261-49b905c3774d\") " pod="openshift-must-gather-456fl/must-gather-tmt9k"
Sep 17 18:35:49.022665 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:49.022642    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7e65e864-768d-41f2-8261-49b905c3774d-must-gather-output\") pod \"must-gather-tmt9k\" (UID: \"7e65e864-768d-41f2-8261-49b905c3774d\") " pod="openshift-must-gather-456fl/must-gather-tmt9k"
Sep 17 18:35:49.029922 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:49.029904    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bknvk\" (UniqueName: \"kubernetes.io/projected/7e65e864-768d-41f2-8261-49b905c3774d-kube-api-access-bknvk\") pod \"must-gather-tmt9k\" (UID: \"7e65e864-768d-41f2-8261-49b905c3774d\") " pod="openshift-must-gather-456fl/must-gather-tmt9k"
Sep 17 18:35:49.206786 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:49.206755    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-456fl/must-gather-tmt9k"
Sep 17 18:35:49.475053 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:49.475018    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-456fl/must-gather-tmt9k"]
Sep 17 18:35:49.477357 ip-10-0-131-14 kubenswrapper[2569]: W0917 18:35:49.477332    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e65e864_768d_41f2_8261_49b905c3774d.slice/crio-1be9f50d1f21f0930bfa2e290bd3038a2abe16968d1c9c0c7005d3af2df2c3cd WatchSource:0}: Error finding container 1be9f50d1f21f0930bfa2e290bd3038a2abe16968d1c9c0c7005d3af2df2c3cd: Status 404 returned error can't find the container with id 1be9f50d1f21f0930bfa2e290bd3038a2abe16968d1c9c0c7005d3af2df2c3cd
Sep 17 18:35:49.479058 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:49.479041    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:35:49.508165 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:49.508138    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="258355ba-6e6e-4c20-97df-7f293355c428" path="/var/lib/kubelet/pods/258355ba-6e6e-4c20-97df-7f293355c428/volumes"
Sep 17 18:35:49.551768 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:49.551734    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-456fl/must-gather-tmt9k" event={"ID":"7e65e864-768d-41f2-8261-49b905c3774d","Type":"ContainerStarted","Data":"1be9f50d1f21f0930bfa2e290bd3038a2abe16968d1c9c0c7005d3af2df2c3cd"}
Sep 17 18:35:49.721226 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:49.721196    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-t7pqb_b8d76e28-4655-4d21-98d8-0324e4756aaf/console-operator/2.log"
Sep 17 18:35:49.726561 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:49.726483    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k4mqn_6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff/ovn-acl-logging/0.log"
Sep 17 18:35:49.727328 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:49.727307    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-t7pqb_b8d76e28-4655-4d21-98d8-0324e4756aaf/console-operator/2.log"
Sep 17 18:35:49.732943 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:49.732922    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k4mqn_6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff/ovn-acl-logging/0.log"
Sep 17 18:35:56.588546 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:56.587680    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-456fl/must-gather-tmt9k" event={"ID":"7e65e864-768d-41f2-8261-49b905c3774d","Type":"ContainerStarted","Data":"58ff13d5746bd0eebb34767c413d1f74086d03f901b7e6dd499019a51e4a303f"}
Sep 17 18:35:56.588546 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:56.587734    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-456fl/must-gather-tmt9k" event={"ID":"7e65e864-768d-41f2-8261-49b905c3774d","Type":"ContainerStarted","Data":"c00dd843b4cab9c221ed59e26827b807c434029c0be3e588899da38def3f77de"}
Sep 17 18:35:56.605649 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:35:56.605596    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-456fl/must-gather-tmt9k" podStartSLOduration=2.455001856 podStartE2EDuration="8.605578041s" totalImagesPullingTime="6.150576185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:35:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:35:56.60536056 +0000 UTC m=+3309.109363208" watchObservedRunningTime="2026-09-17 18:35:56.605578041 +0000 UTC m=+3309.109580667"
Sep 17 18:36:24.711686 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:24.711649    2569 generic.go:356] "Generic (PLEG): container finished" podID="7e65e864-768d-41f2-8261-49b905c3774d" containerID="c00dd843b4cab9c221ed59e26827b807c434029c0be3e588899da38def3f77de" exitCode=0
Sep 17 18:36:24.712148 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:24.711722    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-456fl/must-gather-tmt9k" event={"ID":"7e65e864-768d-41f2-8261-49b905c3774d","Type":"ContainerDied","Data":"c00dd843b4cab9c221ed59e26827b807c434029c0be3e588899da38def3f77de"}
Sep 17 18:36:24.712148 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:24.712034    2569 scope.go:117] "RemoveContainer" containerID="c00dd843b4cab9c221ed59e26827b807c434029c0be3e588899da38def3f77de"
Sep 17 18:36:25.190496 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:25.190415    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-456fl_must-gather-tmt9k_7e65e864-768d-41f2-8261-49b905c3774d/gather/0.log"
Sep 17 18:36:28.747375 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:28.747345    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-cfb52_3f18fbf0-4d7d-4621-9260-199ef7cc0e0d/global-pull-secret-syncer/0.log"
Sep 17 18:36:28.928627 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:28.928599    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-qn6lp_372b5295-ddc7-4999-9b48-c6cfb84d8e83/konnectivity-agent/0.log"
Sep 17 18:36:28.951525 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:28.951503    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-14.ec2.internal_73796eafa3737317490091fb30921865/haproxy/0.log"
Sep 17 18:36:30.750438 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:30.750402    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-456fl/must-gather-tmt9k"]
Sep 17 18:36:30.750856 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:30.750608    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-456fl/must-gather-tmt9k" podUID="7e65e864-768d-41f2-8261-49b905c3774d" containerName="copy" containerID="cri-o://58ff13d5746bd0eebb34767c413d1f74086d03f901b7e6dd499019a51e4a303f" gracePeriod=2
Sep 17 18:36:30.755246 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:30.755220    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-456fl/must-gather-tmt9k"]
Sep 17 18:36:30.978118 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:30.978067    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-456fl_must-gather-tmt9k_7e65e864-768d-41f2-8261-49b905c3774d/copy/0.log"
Sep 17 18:36:30.980658 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:30.980640    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-456fl/must-gather-tmt9k"
Sep 17 18:36:30.983053 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:36:30.983030    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-tmt9k\" is forbidden: User \"system:node:ip-10-0-131-14.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-456fl\": no relationship found between node 'ip-10-0-131-14.ec2.internal' and this object" podUID="7e65e864-768d-41f2-8261-49b905c3774d" pod="openshift-must-gather-456fl/must-gather-tmt9k"
Sep 17 18:36:31.086571 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:31.086491    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7e65e864-768d-41f2-8261-49b905c3774d-must-gather-output\") pod \"7e65e864-768d-41f2-8261-49b905c3774d\" (UID: \"7e65e864-768d-41f2-8261-49b905c3774d\") "
Sep 17 18:36:31.086711 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:31.086592    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bknvk\" (UniqueName: \"kubernetes.io/projected/7e65e864-768d-41f2-8261-49b905c3774d-kube-api-access-bknvk\") pod \"7e65e864-768d-41f2-8261-49b905c3774d\" (UID: \"7e65e864-768d-41f2-8261-49b905c3774d\") "
Sep 17 18:36:31.087943 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:31.087916    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e65e864-768d-41f2-8261-49b905c3774d-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "7e65e864-768d-41f2-8261-49b905c3774d" (UID: "7e65e864-768d-41f2-8261-49b905c3774d"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:36:31.088792 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:31.088768    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e65e864-768d-41f2-8261-49b905c3774d-kube-api-access-bknvk" (OuterVolumeSpecName: "kube-api-access-bknvk") pod "7e65e864-768d-41f2-8261-49b905c3774d" (UID: "7e65e864-768d-41f2-8261-49b905c3774d"). InnerVolumeSpecName "kube-api-access-bknvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:36:31.187503 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:31.187471    2569 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7e65e864-768d-41f2-8261-49b905c3774d-must-gather-output\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:36:31.187503 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:31.187499    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bknvk\" (UniqueName: \"kubernetes.io/projected/7e65e864-768d-41f2-8261-49b905c3774d-kube-api-access-bknvk\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\""
Sep 17 18:36:31.508153 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:31.508120    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e65e864-768d-41f2-8261-49b905c3774d" path="/var/lib/kubelet/pods/7e65e864-768d-41f2-8261-49b905c3774d/volumes"
Sep 17 18:36:31.740980 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:31.740951    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-456fl_must-gather-tmt9k_7e65e864-768d-41f2-8261-49b905c3774d/copy/0.log"
Sep 17 18:36:31.741495 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:31.741473    2569 generic.go:356] "Generic (PLEG): container finished" podID="7e65e864-768d-41f2-8261-49b905c3774d" containerID="58ff13d5746bd0eebb34767c413d1f74086d03f901b7e6dd499019a51e4a303f" exitCode=143
Sep 17 18:36:31.741606 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:31.741517    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-456fl/must-gather-tmt9k"
Sep 17 18:36:31.741606 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:31.741569    2569 scope.go:117] "RemoveContainer" containerID="58ff13d5746bd0eebb34767c413d1f74086d03f901b7e6dd499019a51e4a303f"
Sep 17 18:36:31.749282 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:31.749252    2569 scope.go:117] "RemoveContainer" containerID="c00dd843b4cab9c221ed59e26827b807c434029c0be3e588899da38def3f77de"
Sep 17 18:36:31.762342 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:31.762131    2569 scope.go:117] "RemoveContainer" containerID="58ff13d5746bd0eebb34767c413d1f74086d03f901b7e6dd499019a51e4a303f"
Sep 17 18:36:31.762587 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:36:31.762434    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58ff13d5746bd0eebb34767c413d1f74086d03f901b7e6dd499019a51e4a303f\": container with ID starting with 58ff13d5746bd0eebb34767c413d1f74086d03f901b7e6dd499019a51e4a303f not found: ID does not exist" containerID="58ff13d5746bd0eebb34767c413d1f74086d03f901b7e6dd499019a51e4a303f"
Sep 17 18:36:31.762587 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:31.762461    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58ff13d5746bd0eebb34767c413d1f74086d03f901b7e6dd499019a51e4a303f"} err="failed to get container status \"58ff13d5746bd0eebb34767c413d1f74086d03f901b7e6dd499019a51e4a303f\": rpc error: code = NotFound desc = could not find container \"58ff13d5746bd0eebb34767c413d1f74086d03f901b7e6dd499019a51e4a303f\": container with ID starting with 58ff13d5746bd0eebb34767c413d1f74086d03f901b7e6dd499019a51e4a303f not found: ID does not exist"
Sep 17 18:36:31.762587 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:31.762479    2569 scope.go:117] "RemoveContainer" containerID="c00dd843b4cab9c221ed59e26827b807c434029c0be3e588899da38def3f77de"
Sep 17 18:36:31.762719 ip-10-0-131-14 kubenswrapper[2569]: E0917 18:36:31.762704    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c00dd843b4cab9c221ed59e26827b807c434029c0be3e588899da38def3f77de\": container with ID starting with c00dd843b4cab9c221ed59e26827b807c434029c0be3e588899da38def3f77de not found: ID does not exist" containerID="c00dd843b4cab9c221ed59e26827b807c434029c0be3e588899da38def3f77de"
Sep 17 18:36:31.762765 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:31.762724    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c00dd843b4cab9c221ed59e26827b807c434029c0be3e588899da38def3f77de"} err="failed to get container status \"c00dd843b4cab9c221ed59e26827b807c434029c0be3e588899da38def3f77de\": rpc error: code = NotFound desc = could not find container \"c00dd843b4cab9c221ed59e26827b807c434029c0be3e588899da38def3f77de\": container with ID starting with c00dd843b4cab9c221ed59e26827b807c434029c0be3e588899da38def3f77de not found: ID does not exist"
Sep 17 18:36:32.889531 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:32.889505    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-74d659dd7f-k5rqx_f91a3eda-11bd-4add-a5e7-a5491083081e/metrics-server/0.log"
Sep 17 18:36:33.151605 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:33.151525    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kfnnp_c5650e98-c78e-42cf-8c37-f83e002791aa/node-exporter/0.log"
Sep 17 18:36:33.173031 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:33.173011    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kfnnp_c5650e98-c78e-42cf-8c37-f83e002791aa/kube-rbac-proxy/0.log"
Sep 17 18:36:33.195374 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:33.195343    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kfnnp_c5650e98-c78e-42cf-8c37-f83e002791aa/init-textfile/0.log"
Sep 17 18:36:33.327710 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:33.327687    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cfe07e07-2803-45c5-b12c-460185459d32/prometheus/0.log"
Sep 17 18:36:33.349123 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:33.349051    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cfe07e07-2803-45c5-b12c-460185459d32/config-reloader/0.log"
Sep 17 18:36:33.385128 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:33.385100    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cfe07e07-2803-45c5-b12c-460185459d32/thanos-sidecar/0.log"
Sep 17 18:36:33.418139 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:33.418048    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cfe07e07-2803-45c5-b12c-460185459d32/kube-rbac-proxy-web/0.log"
Sep 17 18:36:33.446150 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:33.446128    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cfe07e07-2803-45c5-b12c-460185459d32/kube-rbac-proxy/0.log"
Sep 17 18:36:33.470252 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:33.470225    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cfe07e07-2803-45c5-b12c-460185459d32/kube-rbac-proxy-thanos/0.log"
Sep 17 18:36:33.492662 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:33.492642    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cfe07e07-2803-45c5-b12c-460185459d32/init-config-reloader/0.log"
Sep 17 18:36:33.520338 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:33.520312    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-c7xwv_4c85ebb0-dc17-4937-9dca-9d87547620f3/prometheus-operator/0.log"
Sep 17 18:36:33.539807 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:33.539785    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-c7xwv_4c85ebb0-dc17-4937-9dca-9d87547620f3/kube-rbac-proxy/0.log"
Sep 17 18:36:33.578731 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:33.578707    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5c6747dc7b-7m22j_b54d572b-971f-4e9b-9f00-c1b8b3eb9188/prometheus-operator-admission-webhook/0.log"
Sep 17 18:36:35.741416 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:35.741395    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-t7pqb_b8d76e28-4655-4d21-98d8-0324e4756aaf/console-operator/2.log"
Sep 17 18:36:35.745013 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:35.744985    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-t7pqb_b8d76e28-4655-4d21-98d8-0324e4756aaf/console-operator/3.log"
Sep 17 18:36:37.157179 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:37.157144    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bc5r6/perf-node-gather-daemonset-9hl45"]
Sep 17 18:36:37.157586 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:37.157424    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e65e864-768d-41f2-8261-49b905c3774d" containerName="copy"
Sep 17 18:36:37.157586 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:37.157434    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e65e864-768d-41f2-8261-49b905c3774d" containerName="copy"
Sep 17 18:36:37.157586 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:37.157450    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e65e864-768d-41f2-8261-49b905c3774d" containerName="gather"
Sep 17 18:36:37.157586 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:37.157456    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e65e864-768d-41f2-8261-49b905c3774d" containerName="gather"
Sep 17 18:36:37.157586 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:37.157505    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e65e864-768d-41f2-8261-49b905c3774d" containerName="gather"
Sep 17 18:36:37.157586 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:37.157512    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e65e864-768d-41f2-8261-49b905c3774d" containerName="copy"
Sep 17 18:36:37.162595 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:37.162579    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bc5r6/perf-node-gather-daemonset-9hl45"
Sep 17 18:36:37.165362 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:37.165341    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bc5r6\"/\"openshift-service-ca.crt\""
Sep 17 18:36:37.166617 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:37.166596    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bc5r6\"/\"default-dockercfg-mqmws\""
Sep 17 18:36:37.166749 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:37.166616    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bc5r6\"/\"kube-root-ca.crt\""
Sep 17 18:36:37.168532 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:37.168510    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bc5r6/perf-node-gather-daemonset-9hl45"]
Sep 17 18:36:37.230426 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:37.230391    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm5nw\" (UniqueName: \"kubernetes.io/projected/d28f03d2-f10a-4626-8df4-41c75297cf9e-kube-api-access-cm5nw\") pod \"perf-node-gather-daemonset-9hl45\" (UID: \"d28f03d2-f10a-4626-8df4-41c75297cf9e\") " pod="openshift-must-gather-bc5r6/perf-node-gather-daemonset-9hl45"
Sep 17 18:36:37.230426 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:37.230431    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d28f03d2-f10a-4626-8df4-41c75297cf9e-sys\") pod \"perf-node-gather-daemonset-9hl45\" (UID: \"d28f03d2-f10a-4626-8df4-41c75297cf9e\") " pod="openshift-must-gather-bc5r6/perf-node-gather-daemonset-9hl45"
Sep 17 18:36:37.230635 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:37.230457    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d28f03d2-f10a-4626-8df4-41c75297cf9e-podres\") pod \"perf-node-gather-daemonset-9hl45\" (UID: \"d28f03d2-f10a-4626-8df4-41c75297cf9e\") " pod="openshift-must-gather-bc5r6/perf-node-gather-daemonset-9hl45"
Sep 17 18:36:37.230635 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:37.230498    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d28f03d2-f10a-4626-8df4-41c75297cf9e-lib-modules\") pod \"perf-node-gather-daemonset-9hl45\" (UID: \"d28f03d2-f10a-4626-8df4-41c75297cf9e\") " pod="openshift-must-gather-bc5r6/perf-node-gather-daemonset-9hl45"
Sep 17 18:36:37.230635 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:37.230521    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d28f03d2-f10a-4626-8df4-41c75297cf9e-proc\") pod \"perf-node-gather-daemonset-9hl45\" (UID: \"d28f03d2-f10a-4626-8df4-41c75297cf9e\") " pod="openshift-must-gather-bc5r6/perf-node-gather-daemonset-9hl45"
Sep 17 18:36:37.331039 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:37.331000    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d28f03d2-f10a-4626-8df4-41c75297cf9e-sys\") pod \"perf-node-gather-daemonset-9hl45\" (UID: \"d28f03d2-f10a-4626-8df4-41c75297cf9e\") " pod="openshift-must-gather-bc5r6/perf-node-gather-daemonset-9hl45"
Sep 17 18:36:37.331039 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:37.331043    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d28f03d2-f10a-4626-8df4-41c75297cf9e-podres\") pod \"perf-node-gather-daemonset-9hl45\" (UID: \"d28f03d2-f10a-4626-8df4-41c75297cf9e\") " pod="openshift-must-gather-bc5r6/perf-node-gather-daemonset-9hl45"
Sep 17 18:36:37.331270 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:37.331103    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d28f03d2-f10a-4626-8df4-41c75297cf9e-lib-modules\") pod \"perf-node-gather-daemonset-9hl45\" (UID: \"d28f03d2-f10a-4626-8df4-41c75297cf9e\") " pod="openshift-must-gather-bc5r6/perf-node-gather-daemonset-9hl45"
Sep 17 18:36:37.331270 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:37.331128    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d28f03d2-f10a-4626-8df4-41c75297cf9e-proc\") pod \"perf-node-gather-daemonset-9hl45\" (UID: \"d28f03d2-f10a-4626-8df4-41c75297cf9e\") " pod="openshift-must-gather-bc5r6/perf-node-gather-daemonset-9hl45"
Sep 17 18:36:37.331270 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:37.331168    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d28f03d2-f10a-4626-8df4-41c75297cf9e-sys\") pod \"perf-node-gather-daemonset-9hl45\" (UID: \"d28f03d2-f10a-4626-8df4-41c75297cf9e\") " pod="openshift-must-gather-bc5r6/perf-node-gather-daemonset-9hl45"
Sep 17 18:36:37.331270 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:37.331203    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm5nw\" (UniqueName: \"kubernetes.io/projected/d28f03d2-f10a-4626-8df4-41c75297cf9e-kube-api-access-cm5nw\") pod \"perf-node-gather-daemonset-9hl45\" (UID: \"d28f03d2-f10a-4626-8df4-41c75297cf9e\") " pod="openshift-must-gather-bc5r6/perf-node-gather-daemonset-9hl45"
Sep 17 18:36:37.331270 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:37.331231    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d28f03d2-f10a-4626-8df4-41c75297cf9e-proc\") pod \"perf-node-gather-daemonset-9hl45\" (UID: \"d28f03d2-f10a-4626-8df4-41c75297cf9e\") " pod="openshift-must-gather-bc5r6/perf-node-gather-daemonset-9hl45"
Sep 17 18:36:37.331270 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:37.331234    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d28f03d2-f10a-4626-8df4-41c75297cf9e-podres\") pod \"perf-node-gather-daemonset-9hl45\" (UID: \"d28f03d2-f10a-4626-8df4-41c75297cf9e\") " pod="openshift-must-gather-bc5r6/perf-node-gather-daemonset-9hl45"
Sep 17 18:36:37.331270 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:37.331257    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d28f03d2-f10a-4626-8df4-41c75297cf9e-lib-modules\") pod \"perf-node-gather-daemonset-9hl45\" (UID: \"d28f03d2-f10a-4626-8df4-41c75297cf9e\") " pod="openshift-must-gather-bc5r6/perf-node-gather-daemonset-9hl45"
Sep 17 18:36:37.339191 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:37.339163    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm5nw\" (UniqueName: \"kubernetes.io/projected/d28f03d2-f10a-4626-8df4-41c75297cf9e-kube-api-access-cm5nw\") pod \"perf-node-gather-daemonset-9hl45\" (UID: \"d28f03d2-f10a-4626-8df4-41c75297cf9e\") " pod="openshift-must-gather-bc5r6/perf-node-gather-daemonset-9hl45"
Sep 17 18:36:37.472959 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:37.472926    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bc5r6/perf-node-gather-daemonset-9hl45"
Sep 17 18:36:37.743565 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:37.743539    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bc5r6/perf-node-gather-daemonset-9hl45"]
Sep 17 18:36:37.745140 ip-10-0-131-14 kubenswrapper[2569]: W0917 18:36:37.745111    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd28f03d2_f10a_4626_8df4_41c75297cf9e.slice/crio-936a1ddf3440be610db7821259f1df36e708832e00ebe42c1ae7752fc7557129 WatchSource:0}: Error finding container 936a1ddf3440be610db7821259f1df36e708832e00ebe42c1ae7752fc7557129: Status 404 returned error can't find the container with id 936a1ddf3440be610db7821259f1df36e708832e00ebe42c1ae7752fc7557129
Sep 17 18:36:37.748015 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:37.747996    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6lkgm_023f4b84-91bb-4103-97cb-5123dc08e459/dns/0.log"
Sep 17 18:36:37.768149 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:37.768119    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bc5r6/perf-node-gather-daemonset-9hl45" event={"ID":"d28f03d2-f10a-4626-8df4-41c75297cf9e","Type":"ContainerStarted","Data":"936a1ddf3440be610db7821259f1df36e708832e00ebe42c1ae7752fc7557129"}
Sep 17 18:36:37.771223 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:37.771197    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6lkgm_023f4b84-91bb-4103-97cb-5123dc08e459/kube-rbac-proxy/0.log"
Sep 17 18:36:37.892608 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:37.892576    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qn9k7_42dd1d14-0348-40e6-824e-90b1763291f1/dns-node-resolver/0.log"
Sep 17 18:36:38.419879 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:38.419847    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nbmpx_1b398422-1bbe-4f75-a7e6-a641cce3f5e8/node-ca/0.log"
Sep 17 18:36:38.774735 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:38.774700    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bc5r6/perf-node-gather-daemonset-9hl45" event={"ID":"d28f03d2-f10a-4626-8df4-41c75297cf9e","Type":"ContainerStarted","Data":"8fff762c4cbe4331c6ecf66f4b5adb726cd2a34c92056617a93b66c76dfbf7c2"}
Sep 17 18:36:38.774930 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:38.774814    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bc5r6/perf-node-gather-daemonset-9hl45"
Sep 17 18:36:38.793005 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:38.792958    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bc5r6/perf-node-gather-daemonset-9hl45" podStartSLOduration=1.792942371 podStartE2EDuration="1.792942371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:36:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:36:38.791025295 +0000 UTC m=+3351.295027932" watchObservedRunningTime="2026-09-17 18:36:38.792942371 +0000 UTC m=+3351.296944996"
Sep 17 18:36:39.171989 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:39.171913    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-55f9595458-br8n7_efc7ba59-612a-4b8a-9935-f2228ec44b30/router/0.log"
Sep 17 18:36:39.541042 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:39.541013    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-78r7n_5b1aeab9-ca3c-4ae4-82dc-fae81696a23a/serve-healthcheck-canary/0.log"
Sep 17 18:36:39.916683 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:39.916604    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-sx8ns_3de5fb99-4d6c-4241-a85b-902f36b0a49a/insights-operator/0.log"
Sep 17 18:36:39.918195 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:39.918175    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-sx8ns_3de5fb99-4d6c-4241-a85b-902f36b0a49a/insights-operator/1.log"
Sep 17 18:36:39.938033 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:39.937998    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ggclv_642e943b-2b5e-410d-8a32-c1eaf28b5fb4/kube-rbac-proxy/0.log"
Sep 17 18:36:39.958537 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:39.958512    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ggclv_642e943b-2b5e-410d-8a32-c1eaf28b5fb4/exporter/0.log"
Sep 17 18:36:39.978859 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:39.978840    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ggclv_642e943b-2b5e-410d-8a32-c1eaf28b5fb4/extractor/0.log"
Sep 17 18:36:43.014493 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:43.014465    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-bz49k_4d2cfbc9-260a-4bb3-98d0-f70eea33c6a4/s3-init/0.log"
Sep 17 18:36:43.032933 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:43.032906    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-bz49k_4d2cfbc9-260a-4bb3-98d0-f70eea33c6a4/download-hf-model/0.log"
Sep 17 18:36:43.096232 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:43.096202    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-zdrc5_5ddb7c37-dbbc-49e3-afaf-6dc48ac7b76c/s3-tls-init-custom/0.log"
Sep 17 18:36:43.115595 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:43.115573    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-zdrc5_5ddb7c37-dbbc-49e3-afaf-6dc48ac7b76c/download-hf-model/0.log"
Sep 17 18:36:43.143828 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:43.143798    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-97l7q_cac6dd97-231b-4c91-b35e-051390a8bc12/s3-tls-init-serving/0.log"
Sep 17 18:36:43.160366 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:43.160340    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-97l7q_cac6dd97-231b-4c91-b35e-051390a8bc12/download-hf-model/0.log"
Sep 17 18:36:44.788714 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:44.788681    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bc5r6/perf-node-gather-daemonset-9hl45"
Sep 17 18:36:47.471514 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:47.471463    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-x8vvv_d0fa8652-da03-451f-bc9d-bf02e2264e2f/migrator/0.log"
Sep 17 18:36:47.491652 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:47.491620    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-x8vvv_d0fa8652-da03-451f-bc9d-bf02e2264e2f/graceful-termination/0.log"
Sep 17 18:36:48.992815 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:48.992782    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mwh5v_911990d2-9984-4b04-979c-1380a8bdb7a2/kube-multus-additional-cni-plugins/0.log"
Sep 17 18:36:49.017436 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:49.017414    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mwh5v_911990d2-9984-4b04-979c-1380a8bdb7a2/egress-router-binary-copy/0.log"
Sep 17 18:36:49.038703 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:49.038680    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mwh5v_911990d2-9984-4b04-979c-1380a8bdb7a2/cni-plugins/0.log"
Sep 17 18:36:49.059012 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:49.058950    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mwh5v_911990d2-9984-4b04-979c-1380a8bdb7a2/bond-cni-plugin/0.log"
Sep 17 18:36:49.080141 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:49.080068    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mwh5v_911990d2-9984-4b04-979c-1380a8bdb7a2/routeoverride-cni/0.log"
Sep 17 18:36:49.102238 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:49.102194    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mwh5v_911990d2-9984-4b04-979c-1380a8bdb7a2/whereabouts-cni-bincopy/0.log"
Sep 17 18:36:49.127362 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:49.127327    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mwh5v_911990d2-9984-4b04-979c-1380a8bdb7a2/whereabouts-cni/0.log"
Sep 17 18:36:49.319130 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:49.319028    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-fjfrk_f847c3c6-f2ab-43a0-b467-27db788f5b9f/kube-multus/0.log"
Sep 17 18:36:49.454636 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:49.454606    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-rzvhs_caf785f7-c156-4b6f-8c82-4c1efe5cd710/network-metrics-daemon/0.log"
Sep 17 18:36:49.473431 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:49.473408    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-rzvhs_caf785f7-c156-4b6f-8c82-4c1efe5cd710/kube-rbac-proxy/0.log"
Sep 17 18:36:49.949400 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:49.949367    2569 scope.go:117] "RemoveContainer" containerID="1168e429f7ea4de6f235d4989b07a4386709ed9fde00f0d6c3ff42c3deb26d33"
Sep 17 18:36:49.960948 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:49.960926    2569 scope.go:117] "RemoveContainer" containerID="bab07d0539cf8bfe459362f3f4e41eea7caae2e516e4dfe24f5ae6b90d33b6d5"
Sep 17 18:36:49.968368 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:49.968348    2569 scope.go:117] "RemoveContainer" containerID="50052cc9ee501c5d85efa8bbff6d213fe54d198b4eb19a283f515f693c88b63e"
Sep 17 18:36:50.670745 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:50.670718    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k4mqn_6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff/ovn-controller/0.log"
Sep 17 18:36:50.689281 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:50.689260    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k4mqn_6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff/ovn-acl-logging/1.log"
Sep 17 18:36:50.689436 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:50.689335    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k4mqn_6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff/ovn-acl-logging/0.log"
Sep 17 18:36:50.707968 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:50.707945    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k4mqn_6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff/kube-rbac-proxy-node/0.log"
Sep 17 18:36:50.729285 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:50.729251    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k4mqn_6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff/kube-rbac-proxy-ovn-metrics/0.log"
Sep 17 18:36:50.749689 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:50.749668    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k4mqn_6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff/northd/0.log"
Sep 17 18:36:50.770662 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:50.770620    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k4mqn_6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff/nbdb/0.log"
Sep 17 18:36:50.790737 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:50.790710    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k4mqn_6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff/sbdb/0.log"
Sep 17 18:36:50.833345 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:50.833319    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k4mqn_6c1d1c67-f1d1-4385-9d8b-d2b0e8ed13ff/ovnkube-controller/0.log"
Sep 17 18:36:51.813407 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:51.813378    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-7cbc45fddc-g9gvb_65730b96-9ce9-4d61-9fd6-1ee7b9e6ebf7/check-endpoints/0.log"
Sep 17 18:36:51.882485 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:51.882457    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-ngfbr_bf919600-b3d9-49db-b561-1e02cb809474/network-check-target-container/0.log"
Sep 17 18:36:52.755588 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:52.755563    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-hfw2c_bc71d7ac-eae9-4ef5-a2f8-f0292bdc51b6/iptables-alerter/0.log"
Sep 17 18:36:53.437434 ip-10-0-131-14 kubenswrapper[2569]: I0917 18:36:53.437338    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-csv2f_d40ba619-04c7-482d-b46f-ec561e79987c/tuned/0.log"