Sep 18 16:48:15.930747 ip-10-0-143-5 systemd[1]: Starting Kubernetes Kubelet...
Sep 18 16:48:19.694552 ip-10-0-143-5 kubenswrapper[2585]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 16:48:19.694552 ip-10-0-143-5 kubenswrapper[2585]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 18 16:48:19.694552 ip-10-0-143-5 kubenswrapper[2585]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 16:48:19.694552 ip-10-0-143-5 kubenswrapper[2585]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 18 16:48:19.694552 ip-10-0-143-5 kubenswrapper[2585]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 16:48:19.695516 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.694646    2585 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 18 16:48:19.699223 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699195    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 16:48:19.699223 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699218    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 16:48:19.699223 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699225    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 16:48:19.699223 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699229    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 16:48:19.699223 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699233    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 16:48:19.699596 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699238    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 16:48:19.699596 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699243    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 16:48:19.699596 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699247    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 16:48:19.699596 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699259    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 16:48:19.699596 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699263    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 16:48:19.699596 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699267    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 16:48:19.699596 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699271    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 16:48:19.699596 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699275    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 16:48:19.699596 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699280    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 16:48:19.699596 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699284    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 16:48:19.699596 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699288    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 16:48:19.699596 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699293    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 16:48:19.699596 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699315    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 16:48:19.699596 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699319    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 16:48:19.699596 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699324    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 16:48:19.699596 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699328    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 16:48:19.699596 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699333    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 16:48:19.699596 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699337    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 16:48:19.699596 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699341    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 16:48:19.700470 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699345    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 16:48:19.700470 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699350    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 16:48:19.700470 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699354    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 16:48:19.700470 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699358    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 16:48:19.700470 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699365    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 16:48:19.700470 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699369    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 16:48:19.700470 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699376    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 16:48:19.700470 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699382    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 16:48:19.700470 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699388    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 16:48:19.700470 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699393    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 16:48:19.700470 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699397    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 16:48:19.700470 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699401    2585 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 16:48:19.700470 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699405    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 16:48:19.700470 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699410    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 16:48:19.700470 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699415    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 16:48:19.700470 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699422    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 16:48:19.700470 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699429    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 16:48:19.700470 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699435    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 16:48:19.700470 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699440    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 16:48:19.701363 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699445    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 16:48:19.701363 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699450    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 16:48:19.701363 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699455    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 16:48:19.701363 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699459    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 16:48:19.701363 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699463    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 16:48:19.701363 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699468    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 16:48:19.701363 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699473    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 16:48:19.701363 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699477    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 16:48:19.701363 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699483    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 16:48:19.701363 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699487    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 16:48:19.701363 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699492    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 16:48:19.701363 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699496    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 16:48:19.701363 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699501    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 16:48:19.701363 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699505    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 16:48:19.701363 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699510    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 16:48:19.701363 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699515    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 16:48:19.701363 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699522    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 16:48:19.701363 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699527    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 16:48:19.701363 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699532    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 16:48:19.701363 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699537    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 16:48:19.702096 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699541    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 16:48:19.702096 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699546    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 16:48:19.702096 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699551    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 16:48:19.702096 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699555    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 16:48:19.702096 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699560    2585 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 16:48:19.702096 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699565    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 16:48:19.702096 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699569    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 16:48:19.702096 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699574    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 16:48:19.702096 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699579    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 16:48:19.702096 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699584    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 16:48:19.702096 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699590    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 16:48:19.702096 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699597    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 16:48:19.702096 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699603    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 16:48:19.702096 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699608    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 16:48:19.702096 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699614    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 16:48:19.702096 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699619    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 16:48:19.702096 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699623    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 16:48:19.702096 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699628    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 16:48:19.702096 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699632    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 16:48:19.702096 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699637    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 16:48:19.702870 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699642    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 16:48:19.702870 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699647    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 16:48:19.702870 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699651    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 16:48:19.702870 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699656    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 16:48:19.702870 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699660    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 16:48:19.702870 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699665    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 16:48:19.702870 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699669    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 16:48:19.702870 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699674    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 16:48:19.702870 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699678    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 16:48:19.702870 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699683    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 16:48:19.702870 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699688    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 16:48:19.702870 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699692    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 16:48:19.702870 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699697    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 16:48:19.702870 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699701    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 16:48:19.702870 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699705    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 16:48:19.702870 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.699710    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 16:48:19.702870 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701095    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 16:48:19.702870 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701109    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 16:48:19.702870 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701116    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 16:48:19.702870 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701123    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 16:48:19.703877 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701130    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 16:48:19.703877 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701135    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 16:48:19.703877 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701141    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 16:48:19.703877 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701146    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 16:48:19.703877 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701153    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 16:48:19.703877 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701159    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 16:48:19.703877 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701164    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 16:48:19.703877 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701169    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 16:48:19.703877 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701174    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 16:48:19.703877 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701179    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 16:48:19.703877 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701184    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 16:48:19.703877 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701189    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 16:48:19.703877 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701193    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 16:48:19.703877 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701198    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 16:48:19.703877 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701203    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 16:48:19.703877 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701207    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 16:48:19.703877 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701212    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 16:48:19.703877 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701219    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 16:48:19.703877 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701226    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 16:48:19.704813 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701230    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 16:48:19.704813 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701235    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 16:48:19.704813 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701239    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 16:48:19.704813 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701244    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 16:48:19.704813 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701249    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 16:48:19.704813 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701255    2585 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 16:48:19.704813 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701259    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 16:48:19.704813 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701264    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 16:48:19.704813 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701272    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 16:48:19.704813 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701277    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 16:48:19.704813 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701282    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 16:48:19.704813 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701289    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 16:48:19.704813 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701316    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 16:48:19.704813 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701321    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 16:48:19.704813 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701325    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 16:48:19.704813 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701330    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 16:48:19.704813 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701334    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 16:48:19.704813 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701339    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 16:48:19.704813 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701344    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 16:48:19.705649 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701349    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 16:48:19.705649 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701354    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 16:48:19.705649 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701358    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 16:48:19.705649 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701365    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 16:48:19.705649 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701371    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 16:48:19.705649 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701375    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 16:48:19.705649 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701380    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 16:48:19.705649 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701384    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 16:48:19.705649 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701390    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 16:48:19.705649 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701394    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 16:48:19.705649 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701399    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 16:48:19.705649 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701403    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 16:48:19.705649 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701407    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 16:48:19.705649 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701412    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 16:48:19.705649 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701417    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 16:48:19.705649 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701422    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 16:48:19.705649 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701429    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 16:48:19.705649 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701434    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 16:48:19.705649 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701440    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 16:48:19.706208 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701445    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 16:48:19.706208 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701449    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 16:48:19.706208 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701454    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 16:48:19.706208 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701459    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 16:48:19.706208 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701463    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 16:48:19.706208 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701469    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 16:48:19.706208 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701474    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 16:48:19.706208 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701479    2585 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 16:48:19.706208 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701483    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 16:48:19.706208 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701488    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 16:48:19.706208 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701493    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 16:48:19.706208 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701497    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 16:48:19.706208 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701502    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 16:48:19.706208 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701508    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 16:48:19.706208 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701513    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 16:48:19.706208 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701517    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 16:48:19.706208 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701521    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 16:48:19.706208 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701525    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 16:48:19.706208 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701530    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 16:48:19.706208 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701534    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 16:48:19.706208 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701539    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 16:48:19.706867 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701543    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 16:48:19.706867 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701547    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 16:48:19.706867 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701552    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 16:48:19.706867 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701557    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 16:48:19.706867 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701561    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 16:48:19.706867 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701566    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 16:48:19.706867 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701571    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 16:48:19.706867 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701575    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 16:48:19.706867 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701579    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 16:48:19.706867 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701584    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 16:48:19.706867 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701589    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 16:48:19.706867 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701594    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 16:48:19.706867 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701598    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 16:48:19.706867 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701603    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 16:48:19.706867 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701608    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 16:48:19.706867 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701612    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 16:48:19.706867 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.701618    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 16:48:19.706867 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.702950    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 16:48:19.706867 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.702962    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 16:48:19.706867 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.702967    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 16:48:19.707600 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.702974    2585 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 16:48:19.707600 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.702980    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 16:48:19.707600 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.702984    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 16:48:19.707600 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.702989    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 16:48:19.707600 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.702995    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 16:48:19.707600 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703000    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 16:48:19.707600 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703005    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 16:48:19.707600 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703009    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 16:48:19.707600 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703014    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 16:48:19.707600 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703019    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 16:48:19.707600 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703023    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 16:48:19.707600 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703028    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 16:48:19.707600 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703032    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 16:48:19.707600 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703037    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 16:48:19.707600 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703041    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 16:48:19.707600 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703046    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 16:48:19.707600 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703050    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 16:48:19.707600 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703055    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 16:48:19.707600 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703059    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 16:48:19.707600 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703064    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 16:48:19.708149 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703069    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 16:48:19.708149 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703073    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 16:48:19.708149 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703078    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 16:48:19.708149 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703083    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 16:48:19.708149 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703088    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 16:48:19.708149 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703092    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 16:48:19.708149 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703097    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 16:48:19.708149 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703102    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 16:48:19.708149 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703107    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 16:48:19.708149 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703112    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 16:48:19.708149 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703116    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 16:48:19.708149 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703120    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 16:48:19.708149 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703124    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 16:48:19.708149 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703129    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 16:48:19.708149 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703134    2585 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 16:48:19.708149 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703138    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 16:48:19.708149 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703145    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 16:48:19.708149 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703152    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 16:48:19.708149 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703157    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 16:48:19.708149 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703162    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 16:48:19.708792 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703167    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 16:48:19.708792 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703171    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 16:48:19.708792 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703177    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 16:48:19.708792 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703183    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 16:48:19.708792 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703189    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 16:48:19.708792 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703194    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 16:48:19.708792 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703199    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 16:48:19.708792 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703204    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 16:48:19.708792 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703209    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 16:48:19.708792 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703215    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 16:48:19.708792 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703219    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 16:48:19.708792 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703227    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 16:48:19.708792 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703233    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 16:48:19.708792 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703238    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 16:48:19.708792 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703243    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 16:48:19.708792 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703249    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 16:48:19.708792 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703254    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 16:48:19.708792 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703259    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 16:48:19.708792 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703263    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 16:48:19.709365 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703268    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 16:48:19.709365 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703273    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 16:48:19.709365 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703277    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 16:48:19.709365 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703282    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 16:48:19.709365 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703286    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 16:48:19.709365 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703293    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 16:48:19.709365 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703316    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 16:48:19.709365 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703320    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 16:48:19.709365 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703326    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 16:48:19.709365 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703330    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 16:48:19.709365 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703335    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 16:48:19.709365 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703340    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 16:48:19.709365 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703345    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 16:48:19.709365 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703349    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 16:48:19.709365 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703353    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 16:48:19.709365 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703358    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 16:48:19.709365 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703366    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 16:48:19.709365 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703371    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 16:48:19.709365 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703376    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 16:48:19.709365 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703381    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 16:48:19.709902 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703386    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 16:48:19.709902 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703391    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 16:48:19.709902 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703395    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 16:48:19.709902 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703399    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 16:48:19.709902 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703403    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 16:48:19.709902 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703409    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 16:48:19.709902 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703414    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 16:48:19.709902 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703419    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 16:48:19.709902 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703424    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 16:48:19.709902 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703428    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 16:48:19.709902 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703433    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 16:48:19.709902 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703445    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 16:48:19.709902 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703450    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 16:48:19.709902 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703454    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 16:48:19.709902 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703459    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 16:48:19.709902 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703463    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 16:48:19.709902 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.703468    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 16:48:19.709902 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.703585    2585 flags.go:64] FLAG: --address="0.0.0.0"
Sep 18 16:48:19.709902 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.703652    2585 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 18 16:48:19.709902 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.703666    2585 flags.go:64] FLAG: --anonymous-auth="true"
Sep 18 16:48:19.710492 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.703674    2585 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 18 16:48:19.710492 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.703682    2585 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 18 16:48:19.710492 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.703689    2585 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 18 16:48:19.710492 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.703697    2585 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 18 16:48:19.710492 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.703710    2585 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 18 16:48:19.710492 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.703716    2585 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 18 16:48:19.710492 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.703721    2585 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 18 16:48:19.710492 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.703728    2585 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 18 16:48:19.710492 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.703734    2585 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 18 16:48:19.710492 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.703739    2585 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 18 16:48:19.710492 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.703744    2585 flags.go:64] FLAG: --cgroup-root=""
Sep 18 16:48:19.710492 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.703749    2585 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 18 16:48:19.710492 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.703755    2585 flags.go:64] FLAG: --client-ca-file=""
Sep 18 16:48:19.710492 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.703760    2585 flags.go:64] FLAG: --cloud-provider="external"
Sep 18 16:48:19.710492 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.703766    2585 flags.go:64] FLAG: --cluster-dns="[]"
Sep 18 16:48:19.710492 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.703773    2585 flags.go:64] FLAG: --cluster-domain=""
Sep 18 16:48:19.710492 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.703778    2585 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 18 16:48:19.710492 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.703784    2585 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 18 16:48:19.710492 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.703790    2585 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 18 16:48:19.710492 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.703797    2585 flags.go:64] FLAG: --container-log-max-files="5"
Sep 18 16:48:19.710492 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.703804    2585 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 18 16:48:19.710492 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.703811    2585 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 18 16:48:19.710492 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.703817    2585 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 18 16:48:19.711091 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.703823    2585 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 18 16:48:19.711091 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.703829    2585 flags.go:64] FLAG: --contention-profiling="false"
Sep 18 16:48:19.711091 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.703835    2585 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 18 16:48:19.711091 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.703840    2585 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 18 16:48:19.711091 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.703846    2585 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 18 16:48:19.711091 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.703852    2585 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 18 16:48:19.711091 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.703859    2585 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 18 16:48:19.711091 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.703865    2585 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 18 16:48:19.711091 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.703870    2585 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 18 16:48:19.711091 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.703875    2585 flags.go:64] FLAG: --enable-load-reader="false"
Sep 18 16:48:19.711091 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.703880    2585 flags.go:64] FLAG: --enable-server="true"
Sep 18 16:48:19.711091 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.703885    2585 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 18 16:48:19.711091 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.703893    2585 flags.go:64] FLAG: --event-burst="100"
Sep 18 16:48:19.711091 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.703898    2585 flags.go:64] FLAG: --event-qps="50"
Sep 18 16:48:19.711091 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.703905    2585 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 18 16:48:19.711091 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.703910    2585 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 18 16:48:19.711091 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.703916    2585 flags.go:64] FLAG: --eviction-hard=""
Sep 18 16:48:19.711091 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.703922    2585 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 18 16:48:19.711091 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.703928    2585 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 18 16:48:19.711091 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.703933    2585 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 18 16:48:19.711091 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.703939    2585 flags.go:64] FLAG: --eviction-soft=""
Sep 18 16:48:19.711091 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.703944    2585 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 18 16:48:19.711091 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.703949    2585 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 18 16:48:19.711091 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.703954    2585 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 18 16:48:19.711091 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.703959    2585 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 18 16:48:19.711884 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.703964    2585 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 18 16:48:19.711884 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.703970    2585 flags.go:64] FLAG: --fail-swap-on="true"
Sep 18 16:48:19.711884 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.703975    2585 flags.go:64] FLAG: --feature-gates=""
Sep 18 16:48:19.711884 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.703982    2585 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 18 16:48:19.711884 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.703988    2585 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 18 16:48:19.711884 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.703993    2585 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 18 16:48:19.711884 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704000    2585 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 18 16:48:19.711884 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704005    2585 flags.go:64] FLAG: --healthz-port="10248"
Sep 18 16:48:19.711884 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704011    2585 flags.go:64] FLAG: --help="false"
Sep 18 16:48:19.711884 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704016    2585 flags.go:64] FLAG: --hostname-override="ip-10-0-143-5.ec2.internal"
Sep 18 16:48:19.711884 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704022    2585 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 18 16:48:19.711884 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704027    2585 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 18 16:48:19.711884 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704032    2585 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 18 16:48:19.711884 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704038    2585 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 18 16:48:19.711884 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704044    2585 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 18 16:48:19.711884 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704049    2585 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 18 16:48:19.711884 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704055    2585 flags.go:64] FLAG: --image-service-endpoint=""
Sep 18 16:48:19.711884 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704060    2585 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 18 16:48:19.711884 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704065    2585 flags.go:64] FLAG: --kube-api-burst="100"
Sep 18 16:48:19.711884 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704070    2585 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 18 16:48:19.711884 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704076    2585 flags.go:64] FLAG: --kube-api-qps="50"
Sep 18 16:48:19.711884 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704081    2585 flags.go:64] FLAG: --kube-reserved=""
Sep 18 16:48:19.711884 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704088    2585 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 18 16:48:19.711884 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704093    2585 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 18 16:48:19.713030 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704099    2585 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 18 16:48:19.713030 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704104    2585 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 18 16:48:19.713030 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704109    2585 flags.go:64] FLAG: --lock-file=""
Sep 18 16:48:19.713030 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704114    2585 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 18 16:48:19.713030 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704119    2585 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 18 16:48:19.713030 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704124    2585 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 18 16:48:19.713030 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704134    2585 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 18 16:48:19.713030 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704141    2585 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 18 16:48:19.713030 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704147    2585 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 18 16:48:19.713030 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704152    2585 flags.go:64] FLAG: --logging-format="text"
Sep 18 16:48:19.713030 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704158    2585 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 18 16:48:19.713030 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704164    2585 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 18 16:48:19.713030 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704169    2585 flags.go:64] FLAG: --manifest-url=""
Sep 18 16:48:19.713030 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704174    2585 flags.go:64] FLAG: --manifest-url-header=""
Sep 18 16:48:19.713030 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704182    2585 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 18 16:48:19.713030 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704188    2585 flags.go:64] FLAG: --max-open-files="1000000"
Sep 18 16:48:19.713030 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704196    2585 flags.go:64] FLAG: --max-pods="110"
Sep 18 16:48:19.713030 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704202    2585 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 18 16:48:19.713030 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704207    2585 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 18 16:48:19.713030 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704212    2585 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 18 16:48:19.713030 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704217    2585 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 18 16:48:19.713030 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704222    2585 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 18 16:48:19.713030 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704227    2585 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 18 16:48:19.713030 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704233    2585 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 18 16:48:19.713030 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704246    2585 flags.go:64] FLAG: --node-status-max-images="50"
Sep 18 16:48:19.713899 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704251    2585 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 18 16:48:19.713899 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704257    2585 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 18 16:48:19.713899 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704262    2585 flags.go:64] FLAG: --pod-cidr=""
Sep 18 16:48:19.713899 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704267    2585 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 18 16:48:19.713899 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704277    2585 flags.go:64] FLAG: --pod-manifest-path=""
Sep 18 16:48:19.713899 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704282    2585 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 18 16:48:19.713899 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704288    2585 flags.go:64] FLAG: --pods-per-core="0"
Sep 18 16:48:19.713899 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704337    2585 flags.go:64] FLAG: --port="10250"
Sep 18 16:48:19.713899 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704345    2585 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 18 16:48:19.713899 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704351    2585 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-011eca0bb8cdaad48"
Sep 18 16:48:19.713899 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704357    2585 flags.go:64] FLAG: --qos-reserved=""
Sep 18 16:48:19.713899 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704363    2585 flags.go:64] FLAG: --read-only-port="10255"
Sep 18 16:48:19.713899 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704369    2585 flags.go:64] FLAG: --register-node="true"
Sep 18 16:48:19.713899 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704374    2585 flags.go:64] FLAG: --register-with-taints=""
Sep 18 16:48:19.713899 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704381    2585 flags.go:64] FLAG: --registry-burst="10"
Sep 18 16:48:19.713899 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704390    2585 flags.go:64] FLAG: --registry-qps="5"
Sep 18 16:48:19.713899 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704399    2585 flags.go:64] FLAG: --reserved-cpus=""
Sep 18 16:48:19.713899 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704404    2585 flags.go:64] FLAG: --reserved-memory=""
Sep 18 16:48:19.713899 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704411    2585 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 18 16:48:19.713899 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704417    2585 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 18 16:48:19.713899 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704422    2585 flags.go:64] FLAG: --rotate-certificates="false"
Sep 18 16:48:19.713899 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704428    2585 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 18 16:48:19.713899 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704433    2585 flags.go:64] FLAG: --runonce="false"
Sep 18 16:48:19.713899 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704438    2585 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 18 16:48:19.713899 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704443    2585 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 18 16:48:19.714685 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704448    2585 flags.go:64] FLAG: --seccomp-default="false"
Sep 18 16:48:19.714685 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704453    2585 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 18 16:48:19.714685 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704459    2585 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 18 16:48:19.714685 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704464    2585 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 18 16:48:19.714685 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704470    2585 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 18 16:48:19.714685 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704475    2585 flags.go:64] FLAG: --storage-driver-password="root"
Sep 18 16:48:19.714685 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704480    2585 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 18 16:48:19.714685 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704485    2585 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 18 16:48:19.714685 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704491    2585 flags.go:64] FLAG: --storage-driver-user="root"
Sep 18 16:48:19.714685 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704496    2585 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 18 16:48:19.714685 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704501    2585 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 18 16:48:19.714685 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704506    2585 flags.go:64] FLAG: --system-cgroups=""
Sep 18 16:48:19.714685 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704511    2585 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 18 16:48:19.714685 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704522    2585 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 18 16:48:19.714685 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704529    2585 flags.go:64] FLAG: --tls-cert-file=""
Sep 18 16:48:19.714685 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704534    2585 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 18 16:48:19.714685 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704541    2585 flags.go:64] FLAG: --tls-min-version=""
Sep 18 16:48:19.714685 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704546    2585 flags.go:64] FLAG: --tls-private-key-file=""
Sep 18 16:48:19.714685 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704551    2585 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 18 16:48:19.714685 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704557    2585 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 18 16:48:19.714685 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704562    2585 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 18 16:48:19.714685 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704567    2585 flags.go:64] FLAG: --v="2"
Sep 18 16:48:19.714685 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704576    2585 flags.go:64] FLAG: --version="false"
Sep 18 16:48:19.714685 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704585    2585 flags.go:64] FLAG: --vmodule=""
Sep 18 16:48:19.714685 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704592    2585 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 18 16:48:19.715388 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.704598    2585 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 18 16:48:19.715388 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.704802    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 16:48:19.715388 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.704809    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 16:48:19.715388 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.704815    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 16:48:19.715388 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.704820    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 16:48:19.715388 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.704825    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 16:48:19.715388 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.704830    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 16:48:19.715388 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.704835    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 16:48:19.715388 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.704839    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 16:48:19.715388 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.704844    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 16:48:19.715388 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.704848    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 16:48:19.715388 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.704853    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 16:48:19.715388 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.704858    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 16:48:19.715388 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.704862    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 16:48:19.715388 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.704867    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 16:48:19.715388 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.704871    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 16:48:19.715388 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.704876    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 16:48:19.715388 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.704881    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 16:48:19.715388 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.704886    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 16:48:19.715388 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.704891    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 16:48:19.715922 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.704895    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 16:48:19.715922 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.704899    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 16:48:19.715922 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.704905    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 16:48:19.715922 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.704910    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 16:48:19.715922 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.704915    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 16:48:19.715922 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.704920    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 16:48:19.715922 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.704925    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 16:48:19.715922 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.704930    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 16:48:19.715922 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.704934    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 16:48:19.715922 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.704941    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 16:48:19.715922 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.704948    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 16:48:19.715922 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.704953    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 16:48:19.715922 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.704958    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 16:48:19.715922 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.704962    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 16:48:19.715922 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.704967    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 16:48:19.715922 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.704972    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 16:48:19.715922 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.704977    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 16:48:19.715922 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.704981    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 16:48:19.715922 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.704986    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 16:48:19.716476 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.704990    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 16:48:19.716476 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.704998    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 16:48:19.716476 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.705004    2585 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 16:48:19.716476 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.705008    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 16:48:19.716476 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.705013    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 16:48:19.716476 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.705018    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 16:48:19.716476 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.705023    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 16:48:19.716476 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.705027    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 16:48:19.716476 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.705031    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 16:48:19.716476 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.705036    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 16:48:19.716476 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.705040    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 16:48:19.716476 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.705045    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 16:48:19.716476 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.705049    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 16:48:19.716476 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.705054    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 16:48:19.716476 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.705058    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 16:48:19.716476 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.705063    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 16:48:19.716476 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.705069    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 16:48:19.716476 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.705073    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 16:48:19.716476 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.705078    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 16:48:19.716476 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.705082    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 16:48:19.717022 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.705087    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 16:48:19.717022 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.705092    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 16:48:19.717022 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.705098    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 16:48:19.717022 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.705105    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 16:48:19.717022 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.705110    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 16:48:19.717022 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.705114    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 16:48:19.717022 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.705119    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 16:48:19.717022 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.705123    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 16:48:19.717022 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.705127    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 16:48:19.717022 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.705132    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 16:48:19.717022 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.705136    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 16:48:19.717022 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.705141    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 16:48:19.717022 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.705146    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 16:48:19.717022 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.705154    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 16:48:19.717022 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.705160    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 16:48:19.717022 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.705165    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 16:48:19.717022 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.705170    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 16:48:19.717022 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.705175    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 16:48:19.717022 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.705179    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 16:48:19.717548 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.705184    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 16:48:19.717548 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.705189    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 16:48:19.717548 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.705194    2585 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 16:48:19.717548 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.705199    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 16:48:19.717548 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.705203    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 16:48:19.717548 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.705208    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 16:48:19.717548 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.705212    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 16:48:19.717548 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.705217    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 16:48:19.717548 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.705222    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 16:48:19.717548 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.705226    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 16:48:19.717548 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.705232    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 16:48:19.717548 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.705237    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 16:48:19.717548 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.705241    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 16:48:19.717548 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.705246    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 16:48:19.717548 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.705250    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 16:48:19.717548 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.705259    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 16:48:19.717548 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.705268    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 16:48:19.717548 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.705273    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 16:48:19.717548 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.705277    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 16:48:19.717548 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.705282    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 16:48:19.718074 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.705287    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 16:48:19.718074 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.705292    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 16:48:19.718074 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.705319    2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 16:48:19.718074 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.715377    2585 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 18 16:48:19.718074 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.715399    2585 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 18 16:48:19.718074 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.715423    2585 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 18 16:48:19.718074 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.715430    2585 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 18 16:48:19.718074 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715494    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 16:48:19.718074 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715499    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 16:48:19.718074 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715503    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 16:48:19.718074 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715507    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 16:48:19.718074 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715511    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 16:48:19.718074 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715514    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 16:48:19.718074 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715517    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 16:48:19.718074 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715520    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 16:48:19.718546 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715524    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 16:48:19.718546 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715526    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 16:48:19.718546 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715529    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 16:48:19.718546 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715532    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 16:48:19.718546 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715535    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 16:48:19.718546 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715538    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 16:48:19.718546 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715541    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 16:48:19.718546 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715544    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 16:48:19.718546 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715547    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 16:48:19.718546 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715550    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 16:48:19.718546 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715553    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 16:48:19.718546 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715556    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 16:48:19.718546 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715559    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 16:48:19.718546 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715562    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 16:48:19.718546 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715564    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 16:48:19.718546 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715567    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 16:48:19.718546 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715570    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 16:48:19.718546 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715573    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 16:48:19.718546 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715576    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 16:48:19.718546 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715579    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 16:48:19.719073 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715584    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 16:48:19.719073 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715590    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 16:48:19.719073 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715593    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 16:48:19.719073 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715596    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 16:48:19.719073 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715599    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 16:48:19.719073 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715602    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 16:48:19.719073 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715605    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 16:48:19.719073 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715608    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 16:48:19.719073 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715612    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 16:48:19.719073 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715616    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 16:48:19.719073 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715619    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 16:48:19.719073 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715623    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 16:48:19.719073 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715626    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 16:48:19.719073 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715629    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 16:48:19.719073 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715633    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 16:48:19.719073 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715637    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 16:48:19.719073 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715640    2585 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 16:48:19.719073 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715643    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 16:48:19.719073 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715647    2585 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 16:48:19.719596 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715649    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 16:48:19.719596 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715652    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 16:48:19.719596 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715655    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 16:48:19.719596 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715658    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 16:48:19.719596 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715661    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 16:48:19.719596 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715664    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 16:48:19.719596 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715666    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 16:48:19.719596 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715669    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 16:48:19.719596 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715672    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 16:48:19.719596 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715675    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 16:48:19.719596 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715678    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 16:48:19.719596 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715680    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 16:48:19.719596 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715684    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 16:48:19.719596 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715687    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 16:48:19.719596 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715690    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 16:48:19.719596 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715695    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 16:48:19.719596 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715698    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 16:48:19.719596 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715701    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 16:48:19.719596 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715704    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 16:48:19.719596 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715706    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 16:48:19.720121 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715709    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 16:48:19.720121 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715712    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 16:48:19.720121 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715715    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 16:48:19.720121 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715718    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 16:48:19.720121 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715720    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 16:48:19.720121 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715723    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 16:48:19.720121 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715726    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 16:48:19.720121 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715729    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 16:48:19.720121 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715732    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 16:48:19.720121 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715734    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 16:48:19.720121 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715737    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 16:48:19.720121 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715740    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 16:48:19.720121 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715744    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 16:48:19.720121 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715746    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 16:48:19.720121 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715749    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 16:48:19.720121 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715752    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 16:48:19.720121 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715755    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 16:48:19.720121 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715758    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 16:48:19.720121 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715761    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 16:48:19.720735 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715764    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 16:48:19.720735 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715766    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 16:48:19.720735 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715769    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 16:48:19.720735 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715772    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 16:48:19.720735 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715775    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 16:48:19.720735 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715778    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 16:48:19.720735 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715781    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 16:48:19.720735 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715785    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 16:48:19.720735 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715788    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 16:48:19.720735 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715791    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 16:48:19.720735 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715794    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 16:48:19.720735 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715799    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 16:48:19.720735 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715802    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 16:48:19.720735 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.715809    2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 16:48:19.721130 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715936    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 16:48:19.721130 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715943    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 16:48:19.721130 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715946    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 16:48:19.721130 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715950    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 16:48:19.721130 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715953    2585 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 16:48:19.721130 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715956    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 16:48:19.721130 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715960    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 16:48:19.721130 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715965    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 16:48:19.721130 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715968    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 16:48:19.721130 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715972    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 16:48:19.721130 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715975    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 16:48:19.721130 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715978    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 16:48:19.721130 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715982    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 16:48:19.721130 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715986    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 16:48:19.721130 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715988    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 16:48:19.721130 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715991    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 16:48:19.721130 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715994    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 16:48:19.721130 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.715997    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 16:48:19.721130 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716000    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 16:48:19.721130 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716003    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 16:48:19.721669 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716006    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 16:48:19.721669 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716009    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 16:48:19.721669 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716012    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 16:48:19.721669 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716015    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 16:48:19.721669 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716018    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 16:48:19.721669 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716022    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 16:48:19.721669 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716025    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 16:48:19.721669 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716028    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 16:48:19.721669 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716031    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 16:48:19.721669 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716034    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 16:48:19.721669 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716037    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 16:48:19.721669 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716040    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 16:48:19.721669 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716043    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 16:48:19.721669 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716045    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 16:48:19.721669 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716048    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 16:48:19.721669 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716051    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 16:48:19.721669 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716054    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 16:48:19.721669 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716057    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 16:48:19.721669 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716060    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 16:48:19.722170 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716063    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 16:48:19.722170 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716065    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 16:48:19.722170 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716068    2585 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 16:48:19.722170 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716071    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 16:48:19.722170 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716073    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 16:48:19.722170 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716076    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 16:48:19.722170 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716081    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 16:48:19.722170 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716084    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 16:48:19.722170 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716088    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 16:48:19.722170 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716091    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 16:48:19.722170 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716094    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 16:48:19.722170 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716097    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 16:48:19.722170 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716100    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 16:48:19.722170 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716103    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 16:48:19.722170 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716106    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 16:48:19.722170 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716109    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 16:48:19.722170 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716112    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 16:48:19.722170 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716115    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 16:48:19.722170 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716118    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 16:48:19.722689 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716122    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 16:48:19.722689 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716125    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 16:48:19.722689 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716128    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 16:48:19.722689 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716131    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 16:48:19.722689 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716134    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 16:48:19.722689 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716137    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 16:48:19.722689 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716139    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 16:48:19.722689 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716142    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 16:48:19.722689 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716145    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 16:48:19.722689 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716149    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 16:48:19.722689 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716153    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 16:48:19.722689 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716156    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 16:48:19.722689 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716160    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 16:48:19.722689 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716163    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 16:48:19.722689 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716165    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 16:48:19.722689 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716168    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 16:48:19.722689 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716171    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 16:48:19.722689 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716173    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 16:48:19.722689 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716176    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 16:48:19.722689 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716179    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 16:48:19.723209 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716183    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 16:48:19.723209 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716186    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 16:48:19.723209 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716189    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 16:48:19.723209 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716192    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 16:48:19.723209 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716194    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 16:48:19.723209 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716197    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 16:48:19.723209 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716200    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 16:48:19.723209 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716203    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 16:48:19.723209 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716206    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 16:48:19.723209 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716208    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 16:48:19.723209 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716212    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 16:48:19.723209 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716214    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 16:48:19.723209 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716218    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 16:48:19.723209 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716221    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 16:48:19.723209 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716224    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 16:48:19.723209 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716227    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 16:48:19.723209 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716230    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 16:48:19.723209 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716233    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 16:48:19.723209 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716236    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 16:48:19.723209 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716239    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 16:48:19.723755 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:19.716241    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 16:48:19.723755 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.716247    2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 16:48:19.723755 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.716414    2585 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 18 16:48:19.723755 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.718825    2585 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 18 16:48:19.723755 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.719992    2585 server.go:1019] "Starting client certificate rotation"
Sep 18 16:48:19.723755 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.720109    2585 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 16:48:19.723755 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.720155    2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 16:48:19.723755 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.722754    2585 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 16:48:19.724077 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.724054    2585 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 16:48:19.732049 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.732024    2585 log.go:25] "Validated CRI v1 runtime API"
Sep 18 16:48:19.734827 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.734807    2585 log.go:25] "Validated CRI v1 image API"
Sep 18 16:48:19.735773 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.735756    2585 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 18 16:48:19.738593 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.738572    2585 fs.go:135] Filesystem UUIDs: map[18cba501-6d4f-4ae6-a804-e7bd5d0b9af1:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 9fa1ed9b-90e2-4524-8c3f-10f506817803:/dev/nvme0n1p3]
Sep 18 16:48:19.738656 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.738592    2585 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 18 16:48:19.746557 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.746404    2585 manager.go:216] Machine: {Timestamp:2026-09-18 16:48:19.745838013 +0000 UTC m=+2.214822963 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098451 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e9ce9752f95ac695d61d46ab181ac SystemUUID:ec2e9ce9-752f-95ac-695d-61d46ab181ac BootID:5e4568ea-458b-411b-95aa-a763cc7314be Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ce:e9:e5:d3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ce:e9:e5:d3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:de:a0:b8:7c:b0:40 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 18 16:48:19.746557 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.746538    2585 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 18 16:48:19.746780 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.746677    2585 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 18 16:48:19.748565 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.748534    2585 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 18 16:48:19.748757 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.748570    2585 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-5.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 18 16:48:19.749981 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.749966    2585 topology_manager.go:138] "Creating topology manager with none policy"
Sep 18 16:48:19.750044 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.749990    2585 container_manager_linux.go:309] "Creating device plugin manager"
Sep 18 16:48:19.750044 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.750003    2585 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 16:48:19.750044 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.750020    2585 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 16:48:19.750244 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.750132    2585 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 18 16:48:19.752265 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:19.752226    2585 certificate_manager.go:596] "Failed while requesting a signed certificate from the control plane" err="cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User \"system:serviceaccount:openshift-machine-config-operator:node-bootstrapper\" cannot create resource \"certificatesigningrequests\" in API group \"certificates.k8s.io\" at the cluster scope" logger="kubernetes.io/kube-apiserver-client-kubelet.UnhandledError"
Sep 18 16:48:19.753992 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.753974    2585 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 16:48:19.754216 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.754202    2585 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 18 16:48:19.754280 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.754252    2585 kubelet.go:486] "Attempting to sync node with API server"
Sep 18 16:48:19.754280 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.754267    2585 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 18 16:48:19.754412 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.754290    2585 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 18 16:48:19.754412 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.754343    2585 kubelet.go:398] "Adding apiserver pod source"
Sep 18 16:48:19.754412 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.754360    2585 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 18 16:48:19.764401 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.754806    2585 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 16:48:19.764401 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.754832    2585 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 16:48:19.765938 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.765913    2585 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 18 16:48:19.768747 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:19.768708    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 18 16:48:19.768846 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.768831    2585 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 18 16:48:19.768896 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.768874    2585 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 18 16:48:19.769117 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:19.769098    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 18 16:48:19.770249 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.770234    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 18 16:48:19.770313 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.770255    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 18 16:48:19.770313 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.770261    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 18 16:48:19.770313 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.770269    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 18 16:48:19.770313 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.770279    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 18 16:48:19.770313 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.770286    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 18 16:48:19.770463 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.770316    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 18 16:48:19.770463 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.770326    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 18 16:48:19.770463 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.770337    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 18 16:48:19.770463 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.770343    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 18 16:48:19.770463 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.770359    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 18 16:48:19.770463 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.770365    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 18 16:48:19.770463 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.770394    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 18 16:48:19.770463 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.770400    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 18 16:48:19.774855 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.774837    2585 server.go:1268] "Started kubelet"
Sep 18 16:48:19.774999 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.774965    2585 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 18 16:48:19.775128 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.774966    2585 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 18 16:48:19.775197 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.775179    2585 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 18 16:48:19.776056 ip-10-0-143-5 systemd[1]: Started Kubernetes Kubelet.
Sep 18 16:48:19.776626 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.776412    2585 server.go:310] "Adding debug handlers to kubelet server"
Sep 18 16:48:19.776626 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.776481    2585 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 18 16:48:19.778847 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.778826    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:48:19.783983 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.783962    2585 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 18 16:48:19.784455 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:19.784439    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-5.ec2.internal\" not found"
Sep 18 16:48:19.784865 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.784849    2585 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 18 16:48:19.784904 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.784869    2585 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 18 16:48:19.786098 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.784145    2585 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 18 16:48:19.786365 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.786346    2585 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 18 16:48:19.786491 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.786440    2585 reconstruct.go:97] "Volume reconstruction finished"
Sep 18 16:48:19.786596 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.786583    2585 reconciler.go:29] "Reconciler: start to sync state"
Sep 18 16:48:19.787549 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:19.787524    2585 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 18 16:48:19.787676 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.787659    2585 factory.go:55] Registering systemd factory
Sep 18 16:48:19.787754 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.787681    2585 factory.go:223] Registration of the systemd container factory successfully
Sep 18 16:48:19.787930 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.787895    2585 factory.go:153] Registering CRI-O factory
Sep 18 16:48:19.787930 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.787927    2585 factory.go:223] Registration of the crio container factory successfully
Sep 18 16:48:19.788059 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.787992    2585 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 18 16:48:19.788059 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.788013    2585 factory.go:103] Registering Raw factory
Sep 18 16:48:19.788059 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.788029    2585 manager.go:1196] Started watching for new ooms in manager
Sep 18 16:48:19.788823 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.788807    2585 manager.go:349] Starting recovery of all containers
Sep 18 16:48:19.808174 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:19.808118    2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 18 16:48:19.809281 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:19.809249    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 18 16:48:19.816677 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:19.815573    2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-5.ec2.internal.18d6783bb5cbfbd1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-5.ec2.internal,UID:ip-10-0-143-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:48:19.774798801 +0000 UTC m=+2.243783749,LastTimestamp:2026-09-18 16:48:19.774798801 +0000 UTC m=+2.243783749,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:48:19.820465 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.820248    2585 manager.go:354] Recovery completed
Sep 18 16:48:19.825058 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.825041    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:48:19.827234 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.827192    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:48:19.827324 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.827246    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:48:19.827324 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.827262    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:48:19.827924 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.827901    2585 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 18 16:48:19.827924 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.827914    2585 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 18 16:48:19.828049 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.827931    2585 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 16:48:19.828049 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.827995    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:48:19.828842 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.828817    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:48:19.828894 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.828845    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:48:19.828894 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.828856    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:48:19.829610 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.829597    2585 policy_none.go:49] "None policy: Start"
Sep 18 16:48:19.829656 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.829616    2585 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 18 16:48:19.829656 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.829627    2585 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 18 16:48:19.830254 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:19.830129    2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-5.ec2.internal.18d6783bb8ebd49f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-5.ec2.internal,UID:ip-10-0-143-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-5.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:48:19.827217567 +0000 UTC m=+2.296202516,LastTimestamp:2026-09-18 16:48:19.827217567 +0000 UTC m=+2.296202516,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:48:19.830922 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.830908    2585 policy_none.go:47] "Start"
Sep 18 16:48:19.843367 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:19.843277    2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-5.ec2.internal.18d6783bb8ec61ac  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-5.ec2.internal,UID:ip-10-0-143-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-5.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:48:19.827253676 +0000 UTC m=+2.296238628,LastTimestamp:2026-09-18 16:48:19.827253676 +0000 UTC m=+2.296238628,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:48:19.854447 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:19.854337    2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-5.ec2.internal.18d6783bb8ec9528  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-5.ec2.internal,UID:ip-10-0-143-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-143-5.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:48:19.827266856 +0000 UTC m=+2.296251804,LastTimestamp:2026-09-18 16:48:19.827266856 +0000 UTC m=+2.296251804,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:48:19.869807 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:19.857368    2585 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-5.ec2.internal.18d6783bb8ebd49f\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-5.ec2.internal.18d6783bb8ebd49f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-5.ec2.internal,UID:ip-10-0-143-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-5.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:48:19.827217567 +0000 UTC m=+2.296202516,LastTimestamp:2026-09-18 16:48:19.828831999 +0000 UTC m=+2.297816948,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:48:19.869807 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.862017    2585 manager.go:337] "Starting Device Plugin manager"
Sep 18 16:48:19.869807 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:19.862064    2585 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 18 16:48:19.869807 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.862079    2585 server.go:87] "Starting device plugin registration server"
Sep 18 16:48:19.869807 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.862918    2585 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 18 16:48:19.869807 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.862948    2585 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 18 16:48:19.869807 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.863067    2585 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 18 16:48:19.869807 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.863157    2585 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 18 16:48:19.869807 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.863166    2585 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 18 16:48:19.869807 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:19.863914    2585 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 18 16:48:19.869807 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:19.863977    2585 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-5.ec2.internal\" not found"
Sep 18 16:48:19.869807 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:19.867433    2585 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-5.ec2.internal.18d6783bb8ec61ac\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-5.ec2.internal.18d6783bb8ec61ac  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-5.ec2.internal,UID:ip-10-0-143-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-5.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:48:19.827253676 +0000 UTC m=+2.296238628,LastTimestamp:2026-09-18 16:48:19.828850504 +0000 UTC m=+2.297835454,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:48:19.878930 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:19.878833    2585 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-5.ec2.internal.18d6783bb8ec9528\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-5.ec2.internal.18d6783bb8ec9528  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-5.ec2.internal,UID:ip-10-0-143-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-143-5.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:48:19.827266856 +0000 UTC m=+2.296251804,LastTimestamp:2026-09-18 16:48:19.828860091 +0000 UTC m=+2.297845040,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:48:19.890013 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:19.889914    2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-5.ec2.internal.18d6783bbb2c0392  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-5.ec2.internal,UID:ip-10-0-143-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:48:19.864978322 +0000 UTC m=+2.333963262,LastTimestamp:2026-09-18 16:48:19.864978322 +0000 UTC m=+2.333963262,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:48:19.908931 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.908895    2585 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 18 16:48:19.910187 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.910168    2585 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 18 16:48:19.910256 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.910229    2585 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 18 16:48:19.910299 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.910265    2585 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 18 16:48:19.910364 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:19.910322    2585 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 18 16:48:19.922964 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:19.922921    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass"
Sep 18 16:48:19.964849 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.964756    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:48:19.966177 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.966156    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:48:19.966264 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.966198    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:48:19.966264 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.966216    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:48:19.966264 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:19.966264    2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:48:19.976041 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:19.975948    2585 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-5.ec2.internal.18d6783bb8ebd49f\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-5.ec2.internal.18d6783bb8ebd49f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-5.ec2.internal,UID:ip-10-0-143-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-5.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:48:19.827217567 +0000 UTC m=+2.296202516,LastTimestamp:2026-09-18 16:48:19.966178357 +0000 UTC m=+2.435163311,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:48:19.987256 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:19.987211    2585 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:48:19.987429 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:19.987265    2585 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-5.ec2.internal.18d6783bb8ec61ac\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-5.ec2.internal.18d6783bb8ec61ac  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-5.ec2.internal,UID:ip-10-0-143-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-5.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:48:19.827253676 +0000 UTC m=+2.296238628,LastTimestamp:2026-09-18 16:48:19.966206235 +0000 UTC m=+2.435191185,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:48:19.998288 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:19.998187    2585 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-5.ec2.internal.18d6783bb8ec9528\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-5.ec2.internal.18d6783bb8ec9528  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-5.ec2.internal,UID:ip-10-0-143-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-143-5.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:48:19.827266856 +0000 UTC m=+2.296251804,LastTimestamp:2026-09-18 16:48:19.966221671 +0000 UTC m=+2.435206619,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:48:20.011433 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:20.010652    2585 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-5.ec2.internal"]
Sep 18 16:48:20.011549 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:20.011493    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:48:20.012655 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:20.012636    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:48:20.012734 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:20.012671    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:48:20.012734 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:20.012684    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:48:20.014117 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:20.014095    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:48:20.014244 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:20.014213    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal"
Sep 18 16:48:20.014316 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:20.014255    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:48:20.014943 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:20.014923    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:48:20.015056 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:20.014948    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:48:20.015056 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:20.014959    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:48:20.015056 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:20.014926    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:48:20.015056 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:20.015030    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:48:20.015056 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:20.015041    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:48:20.016747 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:20.016730    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-5.ec2.internal"
Sep 18 16:48:20.016860 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:20.016760    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:48:20.017580 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:20.017557    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:48:20.017680 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:20.017594    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:48:20.017680 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:20.017608    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:48:20.017927 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:20.017905    2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="400ms"
Sep 18 16:48:20.018037 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:20.017937    2585 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-5.ec2.internal.18d6783bb8ebd49f\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-5.ec2.internal.18d6783bb8ebd49f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-5.ec2.internal,UID:ip-10-0-143-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-5.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:48:19.827217567 +0000 UTC m=+2.296202516,LastTimestamp:2026-09-18 16:48:20.012655149 +0000 UTC m=+2.481640098,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:48:20.023176 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:20.023102    2585 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-5.ec2.internal.18d6783bb8ec61ac\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-5.ec2.internal.18d6783bb8ec61ac  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-5.ec2.internal,UID:ip-10-0-143-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-5.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:48:19.827253676 +0000 UTC m=+2.296238628,LastTimestamp:2026-09-18 16:48:20.012677037 +0000 UTC m=+2.481661986,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:48:20.026840 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:20.026761    2585 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-5.ec2.internal.18d6783bb8ec9528\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-5.ec2.internal.18d6783bb8ec9528  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-5.ec2.internal,UID:ip-10-0-143-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-143-5.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:48:19.827266856 +0000 UTC m=+2.296251804,LastTimestamp:2026-09-18 16:48:20.012688327 +0000 UTC m=+2.481673276,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:48:20.032527 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:20.032443    2585 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-5.ec2.internal.18d6783bb8ebd49f\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-5.ec2.internal.18d6783bb8ebd49f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-5.ec2.internal,UID:ip-10-0-143-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-5.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:48:19.827217567 +0000 UTC m=+2.296202516,LastTimestamp:2026-09-18 16:48:20.014938698 +0000 UTC m=+2.483923650,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:48:20.037545 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:20.037469    2585 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-5.ec2.internal.18d6783bb8ec61ac\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-5.ec2.internal.18d6783bb8ec61ac  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-5.ec2.internal,UID:ip-10-0-143-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-5.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:48:19.827253676 +0000 UTC m=+2.296238628,LastTimestamp:2026-09-18 16:48:20.014953643 +0000 UTC m=+2.483938593,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:48:20.042096 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:20.042020    2585 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-5.ec2.internal.18d6783bb8ec9528\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-5.ec2.internal.18d6783bb8ec9528  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-5.ec2.internal,UID:ip-10-0-143-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-143-5.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:48:19.827266856 +0000 UTC m=+2.296251804,LastTimestamp:2026-09-18 16:48:20.01496326 +0000 UTC m=+2.483948208,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:48:20.043481 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:20.043460    2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-5.ec2.internal\" not found" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:48:20.048067 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:20.047965    2585 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-5.ec2.internal.18d6783bb8ebd49f\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-5.ec2.internal.18d6783bb8ebd49f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-5.ec2.internal,UID:ip-10-0-143-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-5.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:48:19.827217567 +0000 UTC m=+2.296202516,LastTimestamp:2026-09-18 16:48:20.015016496 +0000 UTC m=+2.484001449,Count:6,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:48:20.049003 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:20.048986    2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-5.ec2.internal\" not found" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:48:20.056461 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:20.056378    2585 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-5.ec2.internal.18d6783bb8ec61ac\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-5.ec2.internal.18d6783bb8ec61ac  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-5.ec2.internal,UID:ip-10-0-143-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-5.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:48:19.827253676 +0000 UTC m=+2.296238628,LastTimestamp:2026-09-18 16:48:20.015035522 +0000 UTC m=+2.484020472,Count:6,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:48:20.066279 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:20.066179    2585 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-5.ec2.internal.18d6783bb8ec9528\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-5.ec2.internal.18d6783bb8ec9528  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-5.ec2.internal,UID:ip-10-0-143-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-143-5.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:48:19.827266856 +0000 UTC m=+2.296251804,LastTimestamp:2026-09-18 16:48:20.01504565 +0000 UTC m=+2.484030598,Count:6,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:48:20.077047 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:20.076958    2585 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-5.ec2.internal.18d6783bb8ebd49f\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-5.ec2.internal.18d6783bb8ebd49f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-5.ec2.internal,UID:ip-10-0-143-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-5.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:48:19.827217567 +0000 UTC m=+2.296202516,LastTimestamp:2026-09-18 16:48:20.017577824 +0000 UTC m=+2.486562773,Count:7,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:48:20.087153 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:20.087067    2585 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-5.ec2.internal.18d6783bb8ec61ac\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-5.ec2.internal.18d6783bb8ec61ac  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-5.ec2.internal,UID:ip-10-0-143-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-5.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:48:19.827253676 +0000 UTC m=+2.296238628,LastTimestamp:2026-09-18 16:48:20.017602055 +0000 UTC m=+2.486587008,Count:7,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:48:20.087894 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:20.087875    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/640796da303be256712f04b36adf26a6-config\") pod \"kube-apiserver-proxy-ip-10-0-143-5.ec2.internal\" (UID: \"640796da303be256712f04b36adf26a6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-5.ec2.internal"
Sep 18 16:48:20.087957 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:20.087901    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a3127caef50db0be093e96af172c433b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal\" (UID: \"a3127caef50db0be093e96af172c433b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal"
Sep 18 16:48:20.087957 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:20.087922    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a3127caef50db0be093e96af172c433b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal\" (UID: \"a3127caef50db0be093e96af172c433b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal"
Sep 18 16:48:20.096572 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:20.096488    2585 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-5.ec2.internal.18d6783bb8ec9528\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-5.ec2.internal.18d6783bb8ec9528  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-5.ec2.internal,UID:ip-10-0-143-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-143-5.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:48:19.827266856 +0000 UTC m=+2.296251804,LastTimestamp:2026-09-18 16:48:20.0176128 +0000 UTC m=+2.486597748,Count:7,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:48:20.192922 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:20.192869    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:48:20.192922 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:20.192913    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a3127caef50db0be093e96af172c433b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal\" (UID: \"a3127caef50db0be093e96af172c433b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal"
Sep 18 16:48:20.193141 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:20.192962    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a3127caef50db0be093e96af172c433b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal\" (UID: \"a3127caef50db0be093e96af172c433b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal"
Sep 18 16:48:20.193141 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:20.192980    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/640796da303be256712f04b36adf26a6-config\") pod \"kube-apiserver-proxy-ip-10-0-143-5.ec2.internal\" (UID: \"640796da303be256712f04b36adf26a6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-5.ec2.internal"
Sep 18 16:48:20.193141 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:20.193021    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/640796da303be256712f04b36adf26a6-config\") pod \"kube-apiserver-proxy-ip-10-0-143-5.ec2.internal\" (UID: \"640796da303be256712f04b36adf26a6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-5.ec2.internal"
Sep 18 16:48:20.193141 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:20.192959    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a3127caef50db0be093e96af172c433b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal\" (UID: \"a3127caef50db0be093e96af172c433b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal"
Sep 18 16:48:20.193141 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:20.193098    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a3127caef50db0be093e96af172c433b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal\" (UID: \"a3127caef50db0be093e96af172c433b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal"
Sep 18 16:48:20.194104 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:20.194089    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:48:20.194179 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:20.194120    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:48:20.194179 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:20.194131    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:48:20.194179 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:20.194164    2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:48:20.204585 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:20.204465    2585 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-5.ec2.internal.18d6783bb8ebd49f\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-5.ec2.internal.18d6783bb8ebd49f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-5.ec2.internal,UID:ip-10-0-143-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-5.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:48:19.827217567 +0000 UTC m=+2.296202516,LastTimestamp:2026-09-18 16:48:20.194105219 +0000 UTC m=+2.663090168,Count:8,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:48:20.215447 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:20.215244    2585 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-5.ec2.internal.18d6783bb8ec61ac\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-5.ec2.internal.18d6783bb8ec61ac  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-5.ec2.internal,UID:ip-10-0-143-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-5.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:48:19.827253676 +0000 UTC m=+2.296238628,LastTimestamp:2026-09-18 16:48:20.194125479 +0000 UTC m=+2.663110427,Count:8,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:48:20.215447 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:20.215390    2585 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:48:20.351430 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:20.351384    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal"
Sep 18 16:48:20.353728 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:20.353708    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-5.ec2.internal"
Sep 18 16:48:20.427522 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:20.427486    2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="800ms"
Sep 18 16:48:20.616600 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:20.616500    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:48:20.617617 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:20.617593    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:48:20.617771 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:20.617632    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:48:20.617771 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:20.617682    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:48:20.617771 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:20.617720    2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:48:20.631124 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:20.631080    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 18 16:48:20.640173 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:20.640138    2585 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:48:20.696208 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:20.696169    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 18 16:48:20.789516 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:20.789484    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:48:20.994682 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:20.994638    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3127caef50db0be093e96af172c433b.slice/crio-cf961e79c5b92a2e43a70c6a29822bf8a5a2069147989bc7b9bb0b53e33609ad WatchSource:0}: Error finding container cf961e79c5b92a2e43a70c6a29822bf8a5a2069147989bc7b9bb0b53e33609ad: Status 404 returned error can't find the container with id cf961e79c5b92a2e43a70c6a29822bf8a5a2069147989bc7b9bb0b53e33609ad
Sep 18 16:48:20.995055 ip-10-0-143-5 kubenswrapper[2585]: W0918 16:48:20.995032    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod640796da303be256712f04b36adf26a6.slice/crio-dbf63fb8dec529c8aaf37fe2ae42085a132de70d014ee88ca7d9f1a65f3d237c WatchSource:0}: Error finding container dbf63fb8dec529c8aaf37fe2ae42085a132de70d014ee88ca7d9f1a65f3d237c: Status 404 returned error can't find the container with id dbf63fb8dec529c8aaf37fe2ae42085a132de70d014ee88ca7d9f1a65f3d237c
Sep 18 16:48:21.000732 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:21.000709    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 16:48:21.011267 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:21.011237    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:48:21.012498 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:21.012475    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:48:21.012643 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:21.012514    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:48:21.012643 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:21.012528    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:48:21.015199 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:21.015114    2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d6783bfee293ae  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal,UID:a3127caef50db0be093e96af172c433b,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Pulling,Message:Pulling image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:beacea521116b2422086941236135068766b54396a322e81fd4104d55c0282df\",Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:48:21.001016238 +0000 UTC m=+3.470001170,LastTimestamp:2026-09-18 16:48:21.001016238 +0000 UTC m=+3.470001170,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:48:21.024361 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:21.024322    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 18 16:48:21.024675 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:21.024599    2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"kube-system\"" event="&Event{ObjectMeta:{kube-apiserver-proxy-ip-10-0-143-5.ec2.internal.18d6783bfee387be  kube-system    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-apiserver-proxy-ip-10-0-143-5.ec2.internal,UID:640796da303be256712f04b36adf26a6,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{haproxy},},Reason:Pulling,Message:Pulling image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eaa467bc105be2f7695ecfea0f0c53f4c5cd52f1f705da378c7a33d13f1d102f\",Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:48:21.001078718 +0000 UTC m=+3.470063658,LastTimestamp:2026-09-18 16:48:21.001078718 +0000 UTC m=+3.470063658,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:48:21.118596 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:21.118561    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass"
Sep 18 16:48:21.236688 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:21.236586    2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="1.6s"
Sep 18 16:48:21.440771 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:21.440713    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:48:21.441839 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:21.441804    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:48:21.441839 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:21.441845    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:48:21.442025 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:21.441856    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:48:21.442025 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:21.441886    2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:48:21.460917 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:21.460876    2585 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:48:21.794827 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:21.794791    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:48:21.857096 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:21.857040    2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 16:48:21.861400 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:21.861288    2585 certificate_manager.go:596] "Failed while requesting a signed certificate from the control plane" err="cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User \"system:serviceaccount:openshift-machine-config-operator:node-bootstrapper\" cannot create resource \"certificatesigningrequests\" in API group \"certificates.k8s.io\" at the cluster scope" logger="kubernetes.io/kube-apiserver-client-kubelet.UnhandledError"
Sep 18 16:48:21.918501 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:21.918438    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-5.ec2.internal" event={"ID":"640796da303be256712f04b36adf26a6","Type":"ContainerStarted","Data":"dbf63fb8dec529c8aaf37fe2ae42085a132de70d014ee88ca7d9f1a65f3d237c"}
Sep 18 16:48:21.920657 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:21.920619    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" event={"ID":"a3127caef50db0be093e96af172c433b","Type":"ContainerStarted","Data":"cf961e79c5b92a2e43a70c6a29822bf8a5a2069147989bc7b9bb0b53e33609ad"}
Sep 18 16:48:22.645488 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:22.645455    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 18 16:48:22.789068 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:22.789032    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:48:22.847411 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:22.847365    2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="3.2s"
Sep 18 16:48:22.915268 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:22.915161    2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d6783c70675cdb  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal,UID:a3127caef50db0be093e96af172c433b,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Pulled,Message:Successfully pulled image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:beacea521116b2422086941236135068766b54396a322e81fd4104d55c0282df\" in 1.904s (1.904s including waiting). Image size: 281584844 bytes.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:48:22.905543899 +0000 UTC m=+5.374528845,LastTimestamp:2026-09-18 16:48:22.905543899 +0000 UTC m=+5.374528845,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:48:22.937076 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:22.936968    2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"kube-system\"" event="&Event{ObjectMeta:{kube-apiserver-proxy-ip-10-0-143-5.ec2.internal.18d6783c71b0a7d8  kube-system    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-apiserver-proxy-ip-10-0-143-5.ec2.internal,UID:640796da303be256712f04b36adf26a6,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{haproxy},},Reason:Pulled,Message:Successfully pulled image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eaa467bc105be2f7695ecfea0f0c53f4c5cd52f1f705da378c7a33d13f1d102f\" in 1.926s (1.926s including waiting). Image size: 320421125 bytes.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:48:22.92712444 +0000 UTC m=+5.396109386,LastTimestamp:2026-09-18 16:48:22.92712444 +0000 UTC m=+5.396109386,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:48:22.991127 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:22.991046    2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"kube-system\"" event="&Event{ObjectMeta:{kube-apiserver-proxy-ip-10-0-143-5.ec2.internal.18d6783c74f09156  kube-system    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-apiserver-proxy-ip-10-0-143-5.ec2.internal,UID:640796da303be256712f04b36adf26a6,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{haproxy},},Reason:Created,Message:Created container: haproxy,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:48:22.98164463 +0000 UTC m=+5.450629578,LastTimestamp:2026-09-18 16:48:22.98164463 +0000 UTC m=+5.450629578,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:48:23.000141 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:23.000044    2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"kube-system\"" event="&Event{ObjectMeta:{kube-apiserver-proxy-ip-10-0-143-5.ec2.internal.18d6783c755c3824  kube-system    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-apiserver-proxy-ip-10-0-143-5.ec2.internal,UID:640796da303be256712f04b36adf26a6,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{haproxy},},Reason:Started,Message:Started container haproxy,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:48:22.988699684 +0000 UTC m=+5.457684638,LastTimestamp:2026-09-18 16:48:22.988699684 +0000 UTC m=+5.457684638,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:48:23.061330 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:23.061279    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:48:23.062555 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:23.062532    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:48:23.062661 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:23.062572    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:48:23.062661 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:23.062587    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:48:23.062661 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:23.062625    2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:48:23.088543 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:23.088500    2585 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:48:23.150292 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:23.150153    2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d6783c7e467f59  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal,UID:a3127caef50db0be093e96af172c433b,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Created,Message:Created container: setup,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:48:23.138271065 +0000 UTC m=+5.607256021,LastTimestamp:2026-09-18 16:48:23.138271065 +0000 UTC m=+5.607256021,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:48:23.161883 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:23.161756    2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d6783c7ebf6662  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal,UID:a3127caef50db0be093e96af172c433b,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Started,Message:Started container setup,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:48:23.14619453 +0000 UTC m=+5.615179485,LastTimestamp:2026-09-18 16:48:23.14619453 +0000 UTC m=+5.615179485,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:48:23.302837 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:23.302800    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:48:23.303913 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:23.303895    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:48:23.303966 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:23.303935    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:48:23.303966 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:23.303947    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:48:23.440873 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:23.440786    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 18 16:48:23.658674 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:23.658636    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 18 16:48:23.788732 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:23.788660    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:48:23.925802 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:23.925766    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-5.ec2.internal" event={"ID":"640796da303be256712f04b36adf26a6","Type":"ContainerStarted","Data":"68652b312a27cbc529589bd0117cc66944e3c6e26e46573fe955ea6b9147cbe4"}
Sep 18 16:48:23.926550 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:23.925783    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:48:23.926978 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:23.926954    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:48:23.927114 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:23.926995    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:48:23.927114 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:23.927011    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:48:23.927244 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:23.927196    2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-5.ec2.internal\" not found" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:48:23.927497 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:23.927467    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" event={"ID":"a3127caef50db0be093e96af172c433b","Type":"ContainerStarted","Data":"53099c762e45e17f593f5422d3ac57b10420999fef8fa675e0c713a71f83ee24"}
Sep 18 16:48:23.927559 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:23.927518    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:48:23.928434 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:23.928417    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:48:23.928507 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:23.928446    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:48:23.928507 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:23.928456    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:48:23.928610 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:23.928598    2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-5.ec2.internal\" not found" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:48:24.033297 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:24.033267    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass"
Sep 18 16:48:24.789715 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:24.789686    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:48:24.929391 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:24.929365    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:48:24.929805 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:24.929365    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:48:24.930328 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:24.930311    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:48:24.930432 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:24.930324    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:48:24.930432 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:24.930341    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:48:24.930432 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:24.930349    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:48:24.930432 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:24.930360    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:48:24.930432 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:24.930351    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:48:24.930606 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:24.930596    2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-5.ec2.internal\" not found" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:48:24.930743 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:24.930726    2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-5.ec2.internal\" not found" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:48:25.788134 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:25.788104    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:48:25.950186 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:25.950146    2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 16:48:25.953394 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:25.953365    2585 certificate_manager.go:596] "Failed while requesting a signed certificate from the control plane" err="cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User \"system:serviceaccount:openshift-machine-config-operator:node-bootstrapper\" cannot create resource \"certificatesigningrequests\" in API group \"certificates.k8s.io\" at the cluster scope" logger="kubernetes.io/kube-apiserver-client-kubelet.UnhandledError"
Sep 18 16:48:26.057773 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:26.057682    2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="6.4s"
Sep 18 16:48:26.288965 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:26.288920    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:48:26.289966 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:26.289947    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:48:26.290074 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:26.289985    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:48:26.290074 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:26.289999    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:48:26.290074 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:26.290039    2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:48:26.306574 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:26.306543    2585 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:48:26.792323 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:26.792285    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:48:27.357684 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:27.357653    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 18 16:48:27.514348 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:27.514302    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:48:27.515469 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:27.515441    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:48:27.515598 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:27.515479    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:48:27.515598 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:27.515493    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:48:27.790782 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:27.790702    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:48:27.843101 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:27.843071    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 18 16:48:28.586450 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:28.586416    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass"
Sep 18 16:48:28.626691 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:28.626660    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 18 16:48:28.788567 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:28.788535    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:48:29.787789 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:29.787751    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:48:29.864282 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:29.864237    2585 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-5.ec2.internal\" not found"
Sep 18 16:48:30.788412 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:30.788380    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:48:31.788907 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:31.788719    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:48:32.468109 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:32.468075    2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 18 16:48:32.707014 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:32.706973    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:48:32.708104 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:32.708085    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:48:32.708169 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:32.708119    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:48:32.708169 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:32.708130    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:48:32.708169 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:32.708160    2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:48:32.726454 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:32.726386    2585 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:48:32.787766 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:32.787740    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:48:33.799113 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:33.799078    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:48:34.327098 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:34.327033    2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 16:48:34.330071 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:34.330042    2585 certificate_manager.go:596] "Failed while requesting a signed certificate from the control plane" err="cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User \"system:serviceaccount:openshift-machine-config-operator:node-bootstrapper\" cannot create resource \"certificatesigningrequests\" in API group \"certificates.k8s.io\" at the cluster scope" logger="kubernetes.io/kube-apiserver-client-kubelet.UnhandledError"
Sep 18 16:48:34.787663 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:34.787634    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:48:34.946254 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:34.946210    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_a3127caef50db0be093e96af172c433b/setup/0.log"
Sep 18 16:48:34.946672 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:34.946264    2585 generic.go:356] "Generic (PLEG): container finished" podID="a3127caef50db0be093e96af172c433b" containerID="53099c762e45e17f593f5422d3ac57b10420999fef8fa675e0c713a71f83ee24" exitCode=1
Sep 18 16:48:34.946672 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:34.946293    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" event={"ID":"a3127caef50db0be093e96af172c433b","Type":"ContainerDied","Data":"53099c762e45e17f593f5422d3ac57b10420999fef8fa675e0c713a71f83ee24"}
Sep 18 16:48:34.946672 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:34.946417    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:48:34.947907 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:34.947887    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:48:34.948027 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:34.947916    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:48:34.948027 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:34.947928    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:48:34.948135 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:34.948111    2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-5.ec2.internal\" not found" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:48:34.989241 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:34.989134    2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d6783f402451da  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal,UID:a3127caef50db0be093e96af172c433b,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Pulled,Message:Container image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:beacea521116b2422086941236135068766b54396a322e81fd4104d55c0282df\" already present on machine,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:48:34.98074569 +0000 UTC m=+17.449730684,LastTimestamp:2026-09-18 16:48:34.98074569 +0000 UTC m=+17.449730684,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:48:35.075323 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:35.075218    2585 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d6783c7e467f59\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d6783c7e467f59  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal,UID:a3127caef50db0be093e96af172c433b,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Created,Message:Created container: setup,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:48:23.138271065 +0000 UTC m=+5.607256021,LastTimestamp:2026-09-18 16:48:35.06433119 +0000 UTC m=+17.533316136,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:48:35.085348 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:35.085256    2585 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d6783c7ebf6662\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d6783c7ebf6662  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal,UID:a3127caef50db0be093e96af172c433b,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Started,Message:Started container setup,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:48:23.14619453 +0000 UTC m=+5.615179485,LastTimestamp:2026-09-18 16:48:35.072286964 +0000 UTC m=+17.541271920,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:48:35.611200 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:35.611166    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass"
Sep 18 16:48:35.788832 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:35.788796    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:48:35.950169 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:35.950093    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_a3127caef50db0be093e96af172c433b/setup/0.log"
Sep 18 16:48:35.950169 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:35.950141    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" event={"ID":"a3127caef50db0be093e96af172c433b","Type":"ContainerStarted","Data":"b9732217a3fdd6c6de2493b48fa4c93d162128a4fc6fcf261a26ce1334e266f4"}
Sep 18 16:48:35.950582 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:35.950278    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:48:35.951120 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:35.951101    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:48:35.951209 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:35.951132    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:48:35.951209 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:35.951144    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:48:35.951354 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:35.951342    2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-5.ec2.internal\" not found" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:48:36.086609 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:36.086580    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 18 16:48:36.233355 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:36.233252    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:48:36.234373 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:36.234355    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:48:36.234505 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:36.234390    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:48:36.234505 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:36.234402    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:48:36.788981 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:36.788947    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:48:37.357570 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:37.357533    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 18 16:48:37.787962 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:37.787875    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:48:38.789462 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:38.789424    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:48:39.230976 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:39.230883    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 18 16:48:39.477377 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:39.477338    2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 18 16:48:39.727515 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:39.727471    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:48:39.728600 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:39.728577    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:48:39.728689 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:39.728618    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:48:39.728689 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:39.728630    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:48:39.728689 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:39.728658    2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:48:39.750858 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:39.750823    2585 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:48:39.787817 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:39.787782    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:48:39.864791 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:39.864734    2585 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-5.ec2.internal\" not found"
Sep 18 16:48:40.788589 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:40.788562    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:48:41.787483 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:41.787450    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:48:42.788522 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:42.788492    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:48:43.788943 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:43.788910    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:48:44.791849 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:44.791812    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:48:45.787747 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:45.787713    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:48:46.487891 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:46.487787    2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 18 16:48:46.751254 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:46.751143    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:48:46.752271 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:46.752254    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:48:46.752329 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:46.752286    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:48:46.752329 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:46.752297    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:48:46.752404 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:46.752355    2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:48:46.769471 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:46.769437    2585 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:48:46.790318 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:46.790289    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:48:46.971013 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:46.970984    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_a3127caef50db0be093e96af172c433b/setup/1.log"
Sep 18 16:48:46.971468 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:46.971452    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_a3127caef50db0be093e96af172c433b/setup/0.log"
Sep 18 16:48:46.971546 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:46.971486    2585 generic.go:356] "Generic (PLEG): container finished" podID="a3127caef50db0be093e96af172c433b" containerID="b9732217a3fdd6c6de2493b48fa4c93d162128a4fc6fcf261a26ce1334e266f4" exitCode=1
Sep 18 16:48:46.971546 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:46.971512    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" event={"ID":"a3127caef50db0be093e96af172c433b","Type":"ContainerDied","Data":"b9732217a3fdd6c6de2493b48fa4c93d162128a4fc6fcf261a26ce1334e266f4"}
Sep 18 16:48:46.971618 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:46.971583    2585 scope.go:117] "RemoveContainer" containerID="53099c762e45e17f593f5422d3ac57b10420999fef8fa675e0c713a71f83ee24"
Sep 18 16:48:46.971669 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:46.971655    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:48:46.972531 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:46.972515    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:48:46.972598 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:46.972550    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:48:46.972598 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:46.972562    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:48:46.972930 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:46.972748    2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-5.ec2.internal\" not found" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:48:46.972930 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:46.972795    2585 scope.go:117] "RemoveContainer" containerID="53099c762e45e17f593f5422d3ac57b10420999fef8fa675e0c713a71f83ee24"
Sep 18 16:48:46.980776 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:46.980743    2585 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator_a3127caef50db0be093e96af172c433b_0 in pod sandbox cf961e79c5b92a2e43a70c6a29822bf8a5a2069147989bc7b9bb0b53e33609ad from index: no such id: '53099c762e45e17f593f5422d3ac57b10420999fef8fa675e0c713a71f83ee24'" containerID="53099c762e45e17f593f5422d3ac57b10420999fef8fa675e0c713a71f83ee24"
Sep 18 16:48:46.980857 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:46.980795    2585 kuberuntime_container.go:962] "Unhandled Error" err="failed to remove pod init container \"setup\": rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator_a3127caef50db0be093e96af172c433b_0 in pod sandbox cf961e79c5b92a2e43a70c6a29822bf8a5a2069147989bc7b9bb0b53e33609ad from index: no such id: '53099c762e45e17f593f5422d3ac57b10420999fef8fa675e0c713a71f83ee24'; Skipping pod \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b)\"" logger="UnhandledError"
Sep 18 16:48:46.980959 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:46.980940    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" podUID="a3127caef50db0be093e96af172c433b"
Sep 18 16:48:46.989148 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:46.989062    2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d678420b68270a  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal,UID:a3127caef50db0be093e96af172c433b,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:BackOff,Message:Back-off restarting failed container setup in pod kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b),Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:48:46.980900618 +0000 UTC m=+29.449885565,LastTimestamp:2026-09-18 16:48:46.980900618 +0000 UTC m=+29.449885565,Count:1,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:48:47.788351 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:47.788311    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:48:47.974811 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:47.974785    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_a3127caef50db0be093e96af172c433b/setup/1.log"
Sep 18 16:48:48.788423 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:48.788389    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:48:49.789947 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:49.789919    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:48:49.865912 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:49.865873    2585 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-5.ec2.internal\" not found"
Sep 18 16:48:50.788406 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:50.788370    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:48:51.441465 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:51.441422    2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 16:48:51.443948 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:51.443922    2585 certificate_manager.go:596] "Failed while requesting a signed certificate from the control plane" err="cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User \"system:serviceaccount:openshift-machine-config-operator:node-bootstrapper\" cannot create resource \"certificatesigningrequests\" in API group \"certificates.k8s.io\" at the cluster scope" logger="kubernetes.io/kube-apiserver-client-kubelet.UnhandledError"
Sep 18 16:48:51.444047 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:51.443951    2585 certificate_manager.go:461] "Reached backoff limit, still unable to rotate certs" err="timed out waiting for the condition" logger="kubernetes.io/kube-apiserver-client-kubelet.UnhandledError"
Sep 18 16:48:51.789333 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:51.789299    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:48:52.137704 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:52.137612    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 18 16:48:52.788038 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:52.788013    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:48:52.819916 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:52.819879    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass"
Sep 18 16:48:52.866878 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:52.866849    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 18 16:48:53.498034 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:53.497998    2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 18 16:48:53.769771 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:53.769743    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:48:53.770848 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:53.770822    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:48:53.770977 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:53.770865    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:48:53.770977 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:53.770881    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:48:53.770977 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:53.770915    2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:48:53.787575 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:53.787548    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:48:53.787575 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:53.787558    2585 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:48:54.775315 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:54.775276    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:48:54.776486 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:54.776468    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:48:54.776570 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:54.776502    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:48:54.776570 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:54.776513    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:48:54.788447 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:54.788420    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:48:55.788237 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:55.788203    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:48:56.789022 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:56.788987    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:48:57.788684 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:57.788649    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:48:58.789192 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:58.789156    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:48:59.788099 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:59.788062    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:48:59.866530 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:59.866498    2585 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-5.ec2.internal\" not found"
Sep 18 16:48:59.910508 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:59.910457    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:48:59.911438 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:59.911419    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:48:59.911505 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:59.911454    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:48:59.911505 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:59.911469    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:48:59.911715 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:59.911700    2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-5.ec2.internal\" not found" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:48:59.921610 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:59.921477    2585 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d6783f402451da\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d6783f402451da  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal,UID:a3127caef50db0be093e96af172c433b,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Pulled,Message:Container image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:beacea521116b2422086941236135068766b54396a322e81fd4104d55c0282df\" already present on machine,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:48:34.98074569 +0000 UTC m=+17.449730684,LastTimestamp:2026-09-18 16:48:59.912535337 +0000 UTC m=+42.381520269,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:48:59.994623 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:59.994530    2585 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d6783c7e467f59\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d6783c7e467f59  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal,UID:a3127caef50db0be093e96af172c433b,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Created,Message:Created container: setup,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:48:23.138271065 +0000 UTC m=+5.607256021,LastTimestamp:2026-09-18 16:48:59.983991479 +0000 UTC m=+42.452976425,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:48:59.995072 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:59.995053    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_a3127caef50db0be093e96af172c433b/setup/1.log"
Sep 18 16:48:59.995164 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:59.995096    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" event={"ID":"a3127caef50db0be093e96af172c433b","Type":"ContainerStarted","Data":"dc571beb5e6d2207915036946db1b3388d523d776aa78f7030c8cafeeb62d62a"}
Sep 18 16:48:59.995212 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:59.995202    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:48:59.995902 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:59.995884    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:48:59.995984 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:59.995914    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:48:59.995984 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:48:59.995925    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:48:59.996113 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:48:59.996100    2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-5.ec2.internal\" not found" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:49:00.002459 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:49:00.002374    2585 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d6783c7ebf6662\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d6783c7ebf6662  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal,UID:a3127caef50db0be093e96af172c433b,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Started,Message:Started container setup,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:48:23.14619453 +0000 UTC m=+5.615179485,LastTimestamp:2026-09-18 16:48:59.992604933 +0000 UTC m=+42.461589889,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:49:00.508467 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:49:00.508423    2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 18 16:49:00.787824 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:00.787782    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:49:00.788649 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:00.788626    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:49:00.788805 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:00.788774    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:49:00.788864 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:00.788812    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:49:00.788864 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:00.788828    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:49:00.788953 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:00.788866    2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:49:00.804979 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:49:00.804943    2585 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:49:01.788778 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:01.788740    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:49:01.984339 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:49:01.984299    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 18 16:49:02.789792 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:02.789760    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:49:03.788886 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:03.788850    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:49:04.797673 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:04.797640    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:49:05.788626 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:05.788594    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:49:06.788863 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:06.788832    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:49:07.517128 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:49:07.517099    2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 18 16:49:07.787189 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:07.787107    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:49:07.805284 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:07.805257    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:49:07.806292 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:07.806274    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:49:07.806363 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:07.806307    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:49:07.806363 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:07.806318    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:49:07.806363 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:07.806348    2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:49:07.823772 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:49:07.823736    2585 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:49:08.788172 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:08.788140    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:49:09.791541 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:09.791508    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:49:09.867068 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:49:09.867023    2585 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-5.ec2.internal\" not found"
Sep 18 16:49:10.789036 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:10.788999    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:49:11.788086 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:11.788056    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:49:12.017498 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:12.017472    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_a3127caef50db0be093e96af172c433b/setup/2.log"
Sep 18 16:49:12.017880 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:12.017865    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_a3127caef50db0be093e96af172c433b/setup/1.log"
Sep 18 16:49:12.017939 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:12.017897    2585 generic.go:356] "Generic (PLEG): container finished" podID="a3127caef50db0be093e96af172c433b" containerID="dc571beb5e6d2207915036946db1b3388d523d776aa78f7030c8cafeeb62d62a" exitCode=1
Sep 18 16:49:12.017939 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:12.017925    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" event={"ID":"a3127caef50db0be093e96af172c433b","Type":"ContainerDied","Data":"dc571beb5e6d2207915036946db1b3388d523d776aa78f7030c8cafeeb62d62a"}
Sep 18 16:49:12.018021 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:12.017998    2585 scope.go:117] "RemoveContainer" containerID="b9732217a3fdd6c6de2493b48fa4c93d162128a4fc6fcf261a26ce1334e266f4"
Sep 18 16:49:12.018106 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:12.018091    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:49:12.019264 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:12.018901    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:49:12.019264 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:12.018937    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:49:12.019264 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:12.018954    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:49:12.019264 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:49:12.019189    2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-5.ec2.internal\" not found" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:49:12.019416 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:12.019270    2585 scope.go:117] "RemoveContainer" containerID="b9732217a3fdd6c6de2493b48fa4c93d162128a4fc6fcf261a26ce1334e266f4"
Sep 18 16:49:12.027796 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:49:12.027762    2585 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator_a3127caef50db0be093e96af172c433b_1 in pod sandbox cf961e79c5b92a2e43a70c6a29822bf8a5a2069147989bc7b9bb0b53e33609ad from index: no such id: 'b9732217a3fdd6c6de2493b48fa4c93d162128a4fc6fcf261a26ce1334e266f4'" containerID="b9732217a3fdd6c6de2493b48fa4c93d162128a4fc6fcf261a26ce1334e266f4"
Sep 18 16:49:12.027905 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:49:12.027828    2585 kuberuntime_container.go:962] "Unhandled Error" err="failed to remove pod init container \"setup\": rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator_a3127caef50db0be093e96af172c433b_1 in pod sandbox cf961e79c5b92a2e43a70c6a29822bf8a5a2069147989bc7b9bb0b53e33609ad from index: no such id: 'b9732217a3fdd6c6de2493b48fa4c93d162128a4fc6fcf261a26ce1334e266f4'; Skipping pod \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b)\"" logger="UnhandledError"
Sep 18 16:49:12.027989 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:49:12.027971    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" podUID="a3127caef50db0be093e96af172c433b"
Sep 18 16:49:12.038948 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:49:12.038817    2585 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d678420b68270a\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d678420b68270a  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal,UID:a3127caef50db0be093e96af172c433b,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:BackOff,Message:Back-off restarting failed container setup in pod kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b),Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:48:46.980900618 +0000 UTC m=+29.449885565,LastTimestamp:2026-09-18 16:49:12.02793553 +0000 UTC m=+54.496920476,Count:2,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:49:12.787967 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:12.787935    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:49:13.021499 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:13.021470    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_a3127caef50db0be093e96af172c433b/setup/2.log"
Sep 18 16:49:13.787937 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:13.787904    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:49:14.526598 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:49:14.526570    2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 18 16:49:14.788083 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:14.788001    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:49:14.824369 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:14.824338    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:49:14.825629 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:14.825609    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:49:14.825720 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:14.825643    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:49:14.825720 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:14.825653    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:49:14.825720 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:14.825681    2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:49:14.842844 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:49:14.842812    2585 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:49:15.787940 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:15.787908    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:49:16.787961 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:16.787926    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:49:17.789922 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:17.789893    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:49:18.787273 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:18.787242    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:49:19.790466 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:19.790430    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:49:19.867774 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:49:19.867734    2585 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-5.ec2.internal\" not found"
Sep 18 16:49:20.790302 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:20.790274    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:49:21.536455 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:49:21.536420    2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 18 16:49:21.788099 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:21.788021    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:49:21.843571 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:21.843526    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:49:21.845447 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:21.845430    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:49:21.845550 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:21.845463    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:49:21.845550 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:21.845481    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:49:21.845550 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:21.845516    2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:49:21.863638 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:49:21.863604    2585 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:49:22.789616 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:22.789586    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:49:23.445867 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:23.445816    2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 16:49:23.454024 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:49:23.452496    2585 certificate_manager.go:596] "Failed while requesting a signed certificate from the control plane" err="cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User \"system:serviceaccount:openshift-machine-config-operator:node-bootstrapper\" cannot create resource \"certificatesigningrequests\" in API group \"certificates.k8s.io\" at the cluster scope" logger="kubernetes.io/kube-apiserver-client-kubelet.UnhandledError"
Sep 18 16:49:23.787962 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:23.787924    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:49:23.911554 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:23.911503    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:49:23.916710 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:23.916684    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:49:23.916825 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:23.916727    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:49:23.916825 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:23.916743    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:49:23.916995 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:49:23.916979    2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-5.ec2.internal\" not found" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:49:23.917189 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:49:23.917171    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" podUID="a3127caef50db0be093e96af172c433b"
Sep 18 16:49:23.927290 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:49:23.927180    2585 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d678420b68270a\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d678420b68270a  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal,UID:a3127caef50db0be093e96af172c433b,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:BackOff,Message:Back-off restarting failed container setup in pod kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b),Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:48:46.980900618 +0000 UTC m=+29.449885565,LastTimestamp:2026-09-18 16:49:23.917134883 +0000 UTC m=+66.386119828,Count:3,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:49:24.787163 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:24.787136    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:49:25.743013 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:49:25.742973    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass"
Sep 18 16:49:25.788773 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:25.788737    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:49:26.787628 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:26.787591    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:49:27.787028 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:27.786996    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:49:28.546748 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:49:28.546716    2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 18 16:49:28.787878 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:28.787849    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:49:28.864602 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:28.864534    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:49:28.865631 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:28.865609    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:49:28.865733 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:28.865643    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:49:28.865733 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:28.865654    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:49:28.865733 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:28.865682    2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:49:28.883487 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:49:28.883447    2585 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:49:29.789135 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:29.789098    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:49:29.868721 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:49:29.868669    2585 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-5.ec2.internal\" not found"
Sep 18 16:49:30.787633 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:30.787603    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:49:31.159120 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:31.159014    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:49:31.160436 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:31.160419    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:49:31.160523 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:31.160452    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:49:31.160523 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:31.160462    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:49:31.790424 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:31.790396    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:49:32.787676 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:32.787643    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:49:33.790301 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:33.790269    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:49:34.788983 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:34.788949    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:49:35.555758 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:49:35.555718    2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 18 16:49:35.788784 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:35.788752    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:49:35.883952 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:35.883863    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:49:35.884917 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:35.884897    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:49:35.885057 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:35.884929    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:49:35.885057 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:35.884939    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:49:35.885057 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:35.884968    2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:49:35.902453 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:49:35.902415    2585 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:49:36.788905 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:36.788875    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:49:37.789936 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:37.789901    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:49:37.911351 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:37.911318    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:49:37.912329 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:37.912307    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:49:37.912437 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:37.912340    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:49:37.912437 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:37.912353    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:49:37.912592 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:49:37.912580    2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-5.ec2.internal\" not found" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:49:37.921062 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:49:37.920952    2585 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d6783f402451da\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d6783f402451da  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal,UID:a3127caef50db0be093e96af172c433b,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Pulled,Message:Container image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:beacea521116b2422086941236135068766b54396a322e81fd4104d55c0282df\" already present on machine,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:48:34.98074569 +0000 UTC m=+17.449730684,LastTimestamp:2026-09-18 16:49:37.913424139 +0000 UTC m=+80.382409072,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:49:37.995255 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:49:37.995160    2585 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d6783c7e467f59\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d6783c7e467f59  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal,UID:a3127caef50db0be093e96af172c433b,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Created,Message:Created container: setup,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:48:23.138271065 +0000 UTC m=+5.607256021,LastTimestamp:2026-09-18 16:49:37.985555489 +0000 UTC m=+80.454540439,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:49:38.003434 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:49:38.003350    2585 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d6783c7ebf6662\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d6783c7ebf6662  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal,UID:a3127caef50db0be093e96af172c433b,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Started,Message:Started container setup,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:48:23.14619453 +0000 UTC m=+5.615179485,LastTimestamp:2026-09-18 16:49:37.993941632 +0000 UTC m=+80.462926569,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:49:38.064731 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:38.064661    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_a3127caef50db0be093e96af172c433b/setup/2.log"
Sep 18 16:49:38.064731 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:38.064708    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" event={"ID":"a3127caef50db0be093e96af172c433b","Type":"ContainerStarted","Data":"f5a696b38646d2cd6f51600b226699cfe4f124f7feb1ce2785ccee1699da3c0c"}
Sep 18 16:49:38.064889 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:38.064820    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:49:38.065668 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:38.065651    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:49:38.065754 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:38.065679    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:49:38.065754 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:38.065688    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:49:38.065880 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:49:38.065867    2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-5.ec2.internal\" not found" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:49:38.789496 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:38.789465    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:49:39.789889 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:39.789851    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:49:39.869242 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:49:39.869182    2585 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-5.ec2.internal\" not found"
Sep 18 16:49:40.343668 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:49:40.343623    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 18 16:49:40.789546 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:40.789516    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:49:41.449035 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:49:41.448992    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 18 16:49:41.686216 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:49:41.686172    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 18 16:49:41.786382 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:41.786355    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:49:42.567968 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:49:42.567932    2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 18 16:49:42.787473 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:42.787436    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:49:42.903309 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:42.903245    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:49:42.905533 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:42.905512    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:49:42.905601 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:42.905547    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:49:42.905601 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:42.905560    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:49:42.905601 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:42.905588    2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:49:42.926782 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:49:42.926743    2585 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:49:43.788216 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:43.788184    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:49:44.788997 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:44.788963    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:49:45.793891 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:45.793858    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:49:46.787928 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:46.787898    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:49:47.789130 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:47.789100    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:49:48.790380 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:48.790344    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:49:49.083163 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:49.083082    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_a3127caef50db0be093e96af172c433b/setup/3.log"
Sep 18 16:49:49.083516 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:49.083497    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_a3127caef50db0be093e96af172c433b/setup/2.log"
Sep 18 16:49:49.083580 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:49.083537    2585 generic.go:356] "Generic (PLEG): container finished" podID="a3127caef50db0be093e96af172c433b" containerID="f5a696b38646d2cd6f51600b226699cfe4f124f7feb1ce2785ccee1699da3c0c" exitCode=1
Sep 18 16:49:49.083580 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:49.083570    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" event={"ID":"a3127caef50db0be093e96af172c433b","Type":"ContainerDied","Data":"f5a696b38646d2cd6f51600b226699cfe4f124f7feb1ce2785ccee1699da3c0c"}
Sep 18 16:49:49.083669 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:49.083655    2585 scope.go:117] "RemoveContainer" containerID="dc571beb5e6d2207915036946db1b3388d523d776aa78f7030c8cafeeb62d62a"
Sep 18 16:49:49.083757 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:49.083744    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:49:49.084922 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:49.084560    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:49:49.084922 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:49.084589    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:49:49.084922 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:49.084601    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:49:49.084922 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:49:49.084811    2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-5.ec2.internal\" not found" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:49:49.084922 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:49.084868    2585 scope.go:117] "RemoveContainer" containerID="dc571beb5e6d2207915036946db1b3388d523d776aa78f7030c8cafeeb62d62a"
Sep 18 16:49:49.093701 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:49:49.093666    2585 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator_a3127caef50db0be093e96af172c433b_2 in pod sandbox cf961e79c5b92a2e43a70c6a29822bf8a5a2069147989bc7b9bb0b53e33609ad from index: no such id: 'dc571beb5e6d2207915036946db1b3388d523d776aa78f7030c8cafeeb62d62a'" containerID="dc571beb5e6d2207915036946db1b3388d523d776aa78f7030c8cafeeb62d62a"
Sep 18 16:49:49.093791 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:49:49.093721    2585 kuberuntime_container.go:962] "Unhandled Error" err="failed to remove pod init container \"setup\": rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator_a3127caef50db0be093e96af172c433b_2 in pod sandbox cf961e79c5b92a2e43a70c6a29822bf8a5a2069147989bc7b9bb0b53e33609ad from index: no such id: 'dc571beb5e6d2207915036946db1b3388d523d776aa78f7030c8cafeeb62d62a'; Skipping pod \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b)\"" logger="UnhandledError"
Sep 18 16:49:49.093906 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:49:49.093886    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 40s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" podUID="a3127caef50db0be093e96af172c433b"
Sep 18 16:49:49.104788 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:49:49.104703    2585 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d678420b68270a\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d678420b68270a  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal,UID:a3127caef50db0be093e96af172c433b,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:BackOff,Message:Back-off restarting failed container setup in pod kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b),Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:48:46.980900618 +0000 UTC m=+29.449885565,LastTimestamp:2026-09-18 16:49:49.093846124 +0000 UTC m=+91.562831069,Count:4,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:49:49.577450 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:49:49.577414    2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 18 16:49:49.790963 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:49.790933    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:49:49.869592 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:49:49.869482    2585 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-5.ec2.internal\" not found"
Sep 18 16:49:49.927724 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:49.927670    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:49:49.928778 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:49.928752    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:49:49.928933 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:49.928790    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:49:49.928933 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:49.928800    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:49:49.928933 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:49.928828    2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:49:49.948274 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:49:49.948217    2585 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:49:50.086179 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:50.086152    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_a3127caef50db0be093e96af172c433b/setup/3.log"
Sep 18 16:49:50.789009 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:50.788975    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:49:51.787931 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:51.787899    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:49:52.787453 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:52.787420    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:49:53.789734 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:53.789705    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:49:53.910736 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:53.910694    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:49:53.913017 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:53.912996    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:49:53.913146 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:53.913038    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:49:53.913146 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:53.913052    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:49:53.913299 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:49:53.913286    2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-5.ec2.internal\" not found" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:49:54.788173 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:54.788139    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:49:55.446180 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:55.446139    2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 16:49:55.449251 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:49:55.449212    2585 certificate_manager.go:596] "Failed while requesting a signed certificate from the control plane" err="cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User \"system:serviceaccount:openshift-machine-config-operator:node-bootstrapper\" cannot create resource \"certificatesigningrequests\" in API group \"certificates.k8s.io\" at the cluster scope" logger="kubernetes.io/kube-apiserver-client-kubelet.UnhandledError"
Sep 18 16:49:55.788212 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:55.788187    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:49:56.589475 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:49:56.589441    2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 18 16:49:56.790303 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:56.790271    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:49:56.949418 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:56.949331    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:49:56.950466 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:56.950443    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:49:56.950591 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:56.950484    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:49:56.950591 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:56.950498    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:49:56.950591 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:56.950532    2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:49:56.968167 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:49:56.968130    2585 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:49:57.787548 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:57.787512    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:49:58.787573 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:58.787537    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:49:59.789988 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:49:59.789957    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:49:59.870457 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:49:59.870416    2585 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-5.ec2.internal\" not found"
Sep 18 16:50:00.789787 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:00.789752    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:50:01.788081 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:01.788048    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:50:02.794014 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:02.793974    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:50:03.600775 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:50:03.600737    2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 18 16:50:03.790088 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:03.790056    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:50:03.911321 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:03.911238    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:50:03.913366 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:03.913346    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:50:03.913496 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:03.913378    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:50:03.913496 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:03.913394    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:50:03.913643 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:50:03.913627    2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-5.ec2.internal\" not found" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:50:03.913824 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:50:03.913805    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 40s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" podUID="a3127caef50db0be093e96af172c433b"
Sep 18 16:50:03.926318 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:50:03.926204    2585 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d678420b68270a\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d678420b68270a  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal,UID:a3127caef50db0be093e96af172c433b,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:BackOff,Message:Back-off restarting failed container setup in pod kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b),Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:48:46.980900618 +0000 UTC m=+29.449885565,LastTimestamp:2026-09-18 16:50:03.913769273 +0000 UTC m=+106.382754219,Count:5,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:50:03.968323 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:03.968278    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:50:03.969412 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:03.969391    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:50:03.969532 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:03.969430    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:50:03.969532 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:03.969447    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:50:03.969532 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:03.969482    2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:50:03.988049 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:50:03.988013    2585 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:50:04.789745 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:04.789713    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:50:05.715389 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:50:05.715351    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass"
Sep 18 16:50:05.787665 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:05.787634    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:50:06.789473 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:06.789440    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:50:07.788775 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:07.788745    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:50:08.787849 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:08.787813    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:50:09.789137 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:09.789101    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:50:09.870923 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:50:09.870872    2585 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-5.ec2.internal\" not found"
Sep 18 16:50:10.610602 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:50:10.610559    2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 18 16:50:10.788562 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:10.788535    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:50:10.988350 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:10.988263    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:50:10.989340 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:10.989322    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:50:10.989411 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:10.989355    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:50:10.989411 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:10.989365    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:50:10.989411 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:10.989392    2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:50:11.005949 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:50:11.005917    2585 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:50:11.787427 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:11.787396    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:50:12.792316 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:12.792282    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:50:13.789728 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:13.789697    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:50:14.788626 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:14.788593    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:50:14.910713 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:14.910679    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:50:14.911663 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:14.911643    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:50:14.911797 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:14.911682    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:50:14.911797 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:14.911695    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:50:14.911987 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:50:14.911970    2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-5.ec2.internal\" not found" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:50:14.912165 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:50:14.912145    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 40s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" podUID="a3127caef50db0be093e96af172c433b"
Sep 18 16:50:14.921921 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:50:14.921827    2585 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d678420b68270a\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d678420b68270a  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal,UID:a3127caef50db0be093e96af172c433b,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:BackOff,Message:Back-off restarting failed container setup in pod kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b),Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:48:46.980900618 +0000 UTC m=+29.449885565,LastTimestamp:2026-09-18 16:50:14.912110973 +0000 UTC m=+117.381095919,Count:6,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:50:15.789333 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:15.789304    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:50:16.787848 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:16.787816    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:50:17.620156 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:50:17.620118    2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 18 16:50:17.788721 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:17.788686    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:50:18.006855 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:18.006759    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:50:18.007862 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:18.007840    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:50:18.007987 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:18.007876    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:50:18.007987 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:18.007889    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:50:18.007987 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:18.007923    2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:50:18.025974 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:50:18.025940    2585 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:50:18.787601 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:18.787570    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:50:19.788619 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:19.788586    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:50:19.871170 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:50:19.871129    2585 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-5.ec2.internal\" not found"
Sep 18 16:50:20.789233 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:20.789185    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:50:21.789688 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:21.789652    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:50:22.787847 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:22.787816    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:50:23.789810 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:23.789769    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:50:24.630356 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:50:24.630319    2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 18 16:50:24.788613 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:24.788579    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:50:25.026309 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:25.026279    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:50:25.028211 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:25.028190    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:50:25.028336 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:25.028236    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:50:25.028336 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:25.028249    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:50:25.028336 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:25.028276    2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:50:25.045621 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:50:25.045591    2585 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:50:25.788054 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:25.788020    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:50:26.585794 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:50:26.585756    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 18 16:50:26.788245 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:26.788204    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:50:26.911345 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:26.911258    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:50:26.912311 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:26.912292    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:50:26.912426 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:26.912325    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:50:26.912426 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:26.912341    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:50:26.912578 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:50:26.912565    2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-5.ec2.internal\" not found" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:50:26.912733 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:50:26.912717    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 40s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" podUID="a3127caef50db0be093e96af172c433b"
Sep 18 16:50:26.922392 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:50:26.922287    2585 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d678420b68270a\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d678420b68270a  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal,UID:a3127caef50db0be093e96af172c433b,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:BackOff,Message:Back-off restarting failed container setup in pod kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b),Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:48:46.980900618 +0000 UTC m=+29.449885565,LastTimestamp:2026-09-18 16:50:26.912687855 +0000 UTC m=+129.381672802,Count:7,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:50:27.445470 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:27.445424    2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 16:50:27.448168 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:50:27.448140    2585 certificate_manager.go:596] "Failed while requesting a signed certificate from the control plane" err="cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User \"system:serviceaccount:openshift-machine-config-operator:node-bootstrapper\" cannot create resource \"certificatesigningrequests\" in API group \"certificates.k8s.io\" at the cluster scope" logger="kubernetes.io/kube-apiserver-client-kubelet.UnhandledError"
Sep 18 16:50:27.790755 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:27.790726    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:50:28.789697 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:28.789666    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:50:29.788031 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:29.787994    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:50:29.871845 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:50:29.871795    2585 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-5.ec2.internal\" not found"
Sep 18 16:50:30.792540 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:30.792502    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:50:31.647085 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:50:31.647045    2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 18 16:50:31.787020 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:31.786992    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:50:32.046564 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:32.046525    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:50:32.047605 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:32.047582    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:50:32.047706 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:32.047619    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:50:32.047706 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:32.047641    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:50:32.047706 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:32.047671    2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:50:32.068152 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:50:32.068121    2585 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:50:32.788420 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:32.788386    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:50:33.790054 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:33.790022    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:50:34.130005 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:50:34.129914    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 18 16:50:34.790361 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:34.790326    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:50:35.789139 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:35.789107    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:50:36.788383 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:36.788350    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:50:37.787437 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:37.787407    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:50:38.658157 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:50:38.658125    2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 18 16:50:38.789629 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:38.789594    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:50:39.068850 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:39.068819    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:50:39.069894 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:39.069876    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:50:39.070011 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:39.069906    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:50:39.070011 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:39.069919    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:50:39.070011 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:39.069950    2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:50:39.088396 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:50:39.088358    2585 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:50:39.530513 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:50:39.530479    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 18 16:50:39.788027 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:39.787942    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:50:39.872699 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:50:39.872642    2585 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-5.ec2.internal\" not found"
Sep 18 16:50:40.788191 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:40.788159    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:50:40.910615 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:40.910573    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:50:40.911619 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:40.911596    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:50:40.911751 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:40.911628    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:50:40.911751 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:40.911639    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:50:40.913853 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:50:40.912259    2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-5.ec2.internal\" not found" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:50:40.924939 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:50:40.924822    2585 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d6783f402451da\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d6783f402451da  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal,UID:a3127caef50db0be093e96af172c433b,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Pulled,Message:Container image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:beacea521116b2422086941236135068766b54396a322e81fd4104d55c0282df\" already present on machine,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:48:34.98074569 +0000 UTC m=+17.449730684,LastTimestamp:2026-09-18 16:50:40.915631501 +0000 UTC m=+143.384616455,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:50:41.003638 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:50:41.003540    2585 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d6783c7e467f59\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d6783c7e467f59  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal,UID:a3127caef50db0be093e96af172c433b,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Created,Message:Created container: setup,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:48:23.138271065 +0000 UTC m=+5.607256021,LastTimestamp:2026-09-18 16:50:40.995172502 +0000 UTC m=+143.464157453,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:50:41.014331 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:50:41.014238    2585 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d6783c7ebf6662\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d6783c7ebf6662  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal,UID:a3127caef50db0be093e96af172c433b,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Started,Message:Started container setup,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:48:23.14619453 +0000 UTC m=+5.615179485,LastTimestamp:2026-09-18 16:50:41.004337671 +0000 UTC m=+143.473322626,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:50:41.163036 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:41.162958    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_a3127caef50db0be093e96af172c433b/setup/3.log"
Sep 18 16:50:41.163036 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:41.163005    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" event={"ID":"a3127caef50db0be093e96af172c433b","Type":"ContainerStarted","Data":"63a9509d3c692b2ec712d63129a2bd2ab30bb1290872c793380270831125e694"}
Sep 18 16:50:41.163331 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:41.163148    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:50:41.164003 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:41.163986    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:50:41.164096 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:41.164014    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:50:41.164096 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:41.164024    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:50:41.164213 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:50:41.164201    2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-5.ec2.internal\" not found" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:50:41.414263 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:41.414150    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:50:41.415277 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:41.415247    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:50:41.415407 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:41.415284    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:50:41.415407 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:41.415302    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:50:41.787455 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:41.787431    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:50:42.790053 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:42.790022    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:50:43.789495 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:43.789461    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:50:44.791199 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:44.791166    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:50:45.668004 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:50:45.667963    2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 18 16:50:45.788059 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:45.788027    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:50:46.088745 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:46.088703    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:50:46.089821 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:46.089802    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:50:46.089881 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:46.089835    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:50:46.089881 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:46.089847    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:50:46.089881 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:46.089875    2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:50:46.107742 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:50:46.107709    2585 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:50:46.787830 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:46.787798    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:50:47.787468 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:47.787435    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:50:48.789529 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:48.789492    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:50:49.789684 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:49.789648    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:50:49.873107 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:50:49.873060    2585 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-5.ec2.internal\" not found"
Sep 18 16:50:50.790513 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:50.790481    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:50:51.790438 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:51.790405    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:50:52.182516 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:52.182439    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_a3127caef50db0be093e96af172c433b/setup/4.log"
Sep 18 16:50:52.182907 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:52.182793    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_a3127caef50db0be093e96af172c433b/setup/3.log"
Sep 18 16:50:52.182907 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:52.182823    2585 generic.go:356] "Generic (PLEG): container finished" podID="a3127caef50db0be093e96af172c433b" containerID="63a9509d3c692b2ec712d63129a2bd2ab30bb1290872c793380270831125e694" exitCode=1
Sep 18 16:50:52.182907 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:52.182854    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" event={"ID":"a3127caef50db0be093e96af172c433b","Type":"ContainerDied","Data":"63a9509d3c692b2ec712d63129a2bd2ab30bb1290872c793380270831125e694"}
Sep 18 16:50:52.182907 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:52.182898    2585 scope.go:117] "RemoveContainer" containerID="f5a696b38646d2cd6f51600b226699cfe4f124f7feb1ce2785ccee1699da3c0c"
Sep 18 16:50:52.183098 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:52.183006    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:50:52.186449 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:52.186428    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:50:52.186572 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:52.186462    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:50:52.186572 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:52.186473    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:50:52.186676 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:50:52.186666    2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-5.ec2.internal\" not found" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:50:52.186729 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:52.186717    2585 scope.go:117] "RemoveContainer" containerID="f5a696b38646d2cd6f51600b226699cfe4f124f7feb1ce2785ccee1699da3c0c"
Sep 18 16:50:52.195091 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:50:52.195061    2585 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator_a3127caef50db0be093e96af172c433b_3 in pod sandbox cf961e79c5b92a2e43a70c6a29822bf8a5a2069147989bc7b9bb0b53e33609ad from index: no such id: 'f5a696b38646d2cd6f51600b226699cfe4f124f7feb1ce2785ccee1699da3c0c'" containerID="f5a696b38646d2cd6f51600b226699cfe4f124f7feb1ce2785ccee1699da3c0c"
Sep 18 16:50:52.195192 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:50:52.195110    2585 kuberuntime_container.go:962] "Unhandled Error" err="failed to remove pod init container \"setup\": rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator_a3127caef50db0be093e96af172c433b_3 in pod sandbox cf961e79c5b92a2e43a70c6a29822bf8a5a2069147989bc7b9bb0b53e33609ad from index: no such id: 'f5a696b38646d2cd6f51600b226699cfe4f124f7feb1ce2785ccee1699da3c0c'; Skipping pod \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b)\"" logger="UnhandledError"
Sep 18 16:50:52.195298 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:50:52.195279    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" podUID="a3127caef50db0be093e96af172c433b"
Sep 18 16:50:52.205403 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:50:52.205314    2585 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d678420b68270a\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d678420b68270a  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal,UID:a3127caef50db0be093e96af172c433b,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:BackOff,Message:Back-off restarting failed container setup in pod kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b),Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:48:46.980900618 +0000 UTC m=+29.449885565,LastTimestamp:2026-09-18 16:50:52.195243362 +0000 UTC m=+154.664228308,Count:8,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:50:52.677633 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:50:52.677607    2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 18 16:50:52.788631 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:52.788602    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:50:53.108716 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:53.108680    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:50:53.109699 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:53.109678    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:50:53.109807 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:53.109709    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:50:53.109807 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:53.109718    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:50:53.109807 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:53.109749    2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:50:53.125269 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:50:53.125236    2585 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:50:53.186146 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:53.186122    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_a3127caef50db0be093e96af172c433b/setup/4.log"
Sep 18 16:50:53.788868 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:53.788841    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:50:54.788170 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:54.788137    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:50:55.787156 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:55.787124    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:50:56.791938 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:56.791903    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:50:57.788313 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:57.788277    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:50:58.786971 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:58.786941    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:50:59.290296 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:50:59.290266    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass"
Sep 18 16:50:59.445890 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:59.445845    2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 16:50:59.448622 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:50:59.448598    2585 certificate_manager.go:596] "Failed while requesting a signed certificate from the control plane" err="cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User \"system:serviceaccount:openshift-machine-config-operator:node-bootstrapper\" cannot create resource \"certificatesigningrequests\" in API group \"certificates.k8s.io\" at the cluster scope" logger="kubernetes.io/kube-apiserver-client-kubelet.UnhandledError"
Sep 18 16:50:59.686456 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:50:59.686387    2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 18 16:50:59.787435 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:50:59.787405    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:50:59.873468 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:50:59.873433    2585 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-5.ec2.internal\" not found"
Sep 18 16:51:00.126106 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:00.126062    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:51:00.127104 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:00.127082    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:51:00.127255 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:00.127119    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:51:00.127255 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:00.127133    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:51:00.127255 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:00.127162    2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:51:00.144986 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:51:00.144946    2585 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:51:00.787259 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:00.787207    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:51:01.788845 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:01.788816    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:51:02.788655 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:02.788624    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:51:03.788519 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:03.788480    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:51:04.790473 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:04.790440    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:51:04.911091 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:04.911058    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:51:04.912064 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:04.912047    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:51:04.912143 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:04.912081    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:51:04.912143 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:04.912096    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:51:04.912320 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:51:04.912305    2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-5.ec2.internal\" not found" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:51:04.912473 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:51:04.912456    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" podUID="a3127caef50db0be093e96af172c433b"
Sep 18 16:51:04.921770 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:51:04.921678    2585 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d678420b68270a\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d678420b68270a  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal,UID:a3127caef50db0be093e96af172c433b,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:BackOff,Message:Back-off restarting failed container setup in pod kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b),Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:48:46.980900618 +0000 UTC m=+29.449885565,LastTimestamp:2026-09-18 16:51:04.912429395 +0000 UTC m=+167.381414347,Count:9,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:51:05.790908 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:05.790877    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:51:06.697595 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:51:06.697550    2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 18 16:51:06.789657 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:06.789627    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:51:07.145673 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:07.145631    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:51:07.146814 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:07.146795    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:51:07.146900 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:07.146830    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:51:07.146900 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:07.146841    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:51:07.146900 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:07.146869    2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:51:07.164001 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:51:07.163970    2585 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:51:07.787426 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:07.787395    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:51:08.788636 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:08.788608    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:51:09.791021 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:09.790983    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:51:09.874293 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:51:09.874253    2585 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-5.ec2.internal\" not found"
Sep 18 16:51:10.786598 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:10.786567    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:51:11.787075 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:11.787043    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:51:12.789113 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:12.789086    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:51:13.707103 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:51:13.707071    2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 18 16:51:13.787016 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:13.786987    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:51:14.164811 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:14.164762    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:51:14.166082 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:14.166063    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:51:14.166180 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:14.166094    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:51:14.166180 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:14.166105    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:51:14.166180 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:14.166136    2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:51:14.183914 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:51:14.183878    2585 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:51:14.788395 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:14.788365    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:51:15.789445 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:15.789413    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:51:16.787925 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:16.787894    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:51:16.910884 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:16.910831    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:51:16.912029 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:16.912007    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:51:16.912133 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:16.912041    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:51:16.912133 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:16.912054    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:51:16.912313 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:51:16.912299    2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-5.ec2.internal\" not found" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:51:17.790218 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:17.790188    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:51:18.792362 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:18.792328    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:51:19.790296 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:19.790264    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:51:19.875130 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:51:19.875092    2585 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-5.ec2.internal\" not found"
Sep 18 16:51:19.911552 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:19.911517    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:51:19.912489 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:19.912471    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:51:19.912606 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:19.912500    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:51:19.912606 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:19.912513    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:51:19.912753 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:51:19.912741    2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-5.ec2.internal\" not found" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:51:19.912914 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:51:19.912897    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" podUID="a3127caef50db0be093e96af172c433b"
Sep 18 16:51:19.922213 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:51:19.922121    2585 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d678420b68270a\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d678420b68270a  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal,UID:a3127caef50db0be093e96af172c433b,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:BackOff,Message:Back-off restarting failed container setup in pod kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b),Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:48:46.980900618 +0000 UTC m=+29.449885565,LastTimestamp:2026-09-18 16:51:19.912865817 +0000 UTC m=+182.381850768,Count:10,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:51:20.563179 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:51:20.563145    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 18 16:51:20.719115 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:51:20.719075    2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 18 16:51:20.788653 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:20.788622    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:51:21.184732 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:21.184690    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:51:21.185688 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:21.185668    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:51:21.185796 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:21.185703    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:51:21.185796 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:21.185716    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:51:21.185796 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:21.185745    2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:51:21.207270 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:51:21.207215    2585 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:51:21.790111 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:21.790073    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:51:22.790510 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:22.790473    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:51:23.787497 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:23.787465    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:51:24.789344 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:24.789313    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:51:25.436038 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:51:25.436002    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 18 16:51:25.788821 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:25.788793    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:51:26.179810 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:51:26.179720    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 18 16:51:26.789316 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:26.789283    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:51:27.729002 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:51:27.728966    2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 18 16:51:27.787655 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:27.787630    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:51:28.208042 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:28.208008    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:51:28.209928 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:28.209903    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:51:28.209928 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:28.209935    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:51:28.210085 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:28.209945    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:51:28.210085 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:28.209969    2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:51:28.229001 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:51:28.228966    2585 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:51:28.788397 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:28.788365    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:51:29.789848 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:29.789817    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:51:29.876262 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:51:29.876193    2585 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-5.ec2.internal\" not found"
Sep 18 16:51:30.787694 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:30.787662    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:51:31.445686 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:31.445639    2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 16:51:31.448659 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:51:31.448635    2585 certificate_manager.go:596] "Failed while requesting a signed certificate from the control plane" err="cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User \"system:serviceaccount:openshift-machine-config-operator:node-bootstrapper\" cannot create resource \"certificatesigningrequests\" in API group \"certificates.k8s.io\" at the cluster scope" logger="kubernetes.io/kube-apiserver-client-kubelet.UnhandledError"
Sep 18 16:51:31.789744 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:31.789716    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:51:31.910559 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:31.910528    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:51:31.911539 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:31.911520    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:51:31.911637 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:31.911548    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:51:31.911637 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:31.911558    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:51:31.911799 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:51:31.911785    2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-5.ec2.internal\" not found" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:51:31.911945 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:51:31.911928    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" podUID="a3127caef50db0be093e96af172c433b"
Sep 18 16:51:31.922425 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:51:31.922332    2585 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d678420b68270a\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d678420b68270a  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal,UID:a3127caef50db0be093e96af172c433b,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:BackOff,Message:Back-off restarting failed container setup in pod kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b),Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:48:46.980900618 +0000 UTC m=+29.449885565,LastTimestamp:2026-09-18 16:51:31.911901412 +0000 UTC m=+194.380886359,Count:11,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:51:32.788459 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:32.788423    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:51:33.789412 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:33.789378    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:51:34.738919 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:51:34.738883    2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 18 16:51:34.789289 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:34.789259    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:51:35.229660 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:35.229623    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:51:35.230861 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:35.230834    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:51:35.230960 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:35.230871    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:51:35.230960 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:35.230882    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:51:35.230960 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:35.230912    2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:51:35.250767 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:51:35.250732    2585 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:51:35.788429 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:35.788399    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:51:36.788198 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:36.788163    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:51:37.788939 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:37.788904    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:51:38.789805 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:38.789767    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:51:39.788383 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:39.788352    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:51:39.877026 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:51:39.876977    2585 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-5.ec2.internal\" not found"
Sep 18 16:51:40.789415 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:40.789381    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:51:41.747193 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:51:41.747154    2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 18 16:51:41.787000 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:41.786967    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:51:42.250864 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:42.250815    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:51:42.253652 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:42.253629    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:51:42.253791 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:42.253668    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:51:42.253791 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:42.253683    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:51:42.253791 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:42.253719    2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:51:42.273288 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:51:42.273248    2585 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:51:42.787830 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:42.787799    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:51:43.790066 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:43.790029    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:51:43.911435 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:43.911385    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:51:43.912634 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:43.912614    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:51:43.912735 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:43.912651    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:51:43.912735 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:43.912665    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:51:43.912910 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:51:43.912896    2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-5.ec2.internal\" not found" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:51:43.913077 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:51:43.913061    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" podUID="a3127caef50db0be093e96af172c433b"
Sep 18 16:51:43.922835 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:51:43.922679    2585 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d678420b68270a\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d678420b68270a  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal,UID:a3127caef50db0be093e96af172c433b,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:BackOff,Message:Back-off restarting failed container setup in pod kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b),Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:48:46.980900618 +0000 UTC m=+29.449885565,LastTimestamp:2026-09-18 16:51:43.913025554 +0000 UTC m=+206.382010500,Count:12,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:51:44.788666 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:44.788628    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:51:45.788755 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:45.788720    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:51:46.787935 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:46.787900    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:51:47.790024 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:47.789994    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:51:48.758906 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:51:48.758864    2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 18 16:51:48.787426 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:48.787393    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:51:49.273791 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:49.273756    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:51:49.274761 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:49.274741    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:51:49.274859 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:49.274777    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:51:49.274859 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:49.274790    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:51:49.274859 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:49.274819    2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:51:49.291736 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:51:49.291696    2585 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:51:49.789459 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:49.789424    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:51:49.872364 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:51:49.872321    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass"
Sep 18 16:51:49.877610 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:51:49.877582    2585 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-5.ec2.internal\" not found"
Sep 18 16:51:50.789681 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:50.789642    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:51:51.787826 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:51.787796    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:51:52.787279 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:52.787248    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:51:53.790351 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:53.790314    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:51:54.791629 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:54.791598    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:51:55.768174 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:51:55.768139    2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 18 16:51:55.789331 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:55.789301    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:51:56.291814 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:56.291776    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:51:56.292838 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:56.292818    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:51:56.292905 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:56.292850    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:51:56.292905 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:56.292861    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:51:56.292905 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:56.292890    2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:51:56.310744 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:51:56.310713    2585 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:51:56.789977 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:56.789948    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:51:57.787918 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:57.787885    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:51:57.910759 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:57.910715    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:51:57.911871 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:57.911849    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:51:57.912008 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:57.911888    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:51:57.912008 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:57.911902    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:51:57.912191 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:51:57.912175    2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-5.ec2.internal\" not found" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:51:57.912390 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:51:57.912370    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" podUID="a3127caef50db0be093e96af172c433b"
Sep 18 16:51:57.921362 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:51:57.921266    2585 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d678420b68270a\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d678420b68270a  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal,UID:a3127caef50db0be093e96af172c433b,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:BackOff,Message:Back-off restarting failed container setup in pod kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b),Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:48:46.980900618 +0000 UTC m=+29.449885565,LastTimestamp:2026-09-18 16:51:57.912337215 +0000 UTC m=+220.381322149,Count:13,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:51:57.997443 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:51:57.997408    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 18 16:51:58.788211 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:58.788182    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:51:59.788625 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:51:59.788594    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:51:59.878075 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:51:59.878027    2585 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-5.ec2.internal\" not found"
Sep 18 16:52:00.789077 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:00.789043    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:52:01.788026 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:01.787992    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:52:02.778136 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:52:02.778093    2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 18 16:52:02.791576 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:02.791547    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:52:02.978057 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:52:02.978026    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 18 16:52:03.311830 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:03.311794    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:52:03.312997 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:03.312980    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:52:03.313098 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:03.313008    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:52:03.313098 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:03.313022    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:52:03.313098 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:03.313058    2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:52:03.330627 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:52:03.330598    2585 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:52:03.445757 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:03.445707    2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 16:52:03.448794 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:52:03.448771    2585 certificate_manager.go:596] "Failed while requesting a signed certificate from the control plane" err="cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User \"system:serviceaccount:openshift-machine-config-operator:node-bootstrapper\" cannot create resource \"certificatesigningrequests\" in API group \"certificates.k8s.io\" at the cluster scope" logger="kubernetes.io/kube-apiserver-client-kubelet.UnhandledError"
Sep 18 16:52:03.787798 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:03.787767    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:52:04.788291 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:04.788261    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:52:05.788751 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:05.788713    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:52:06.789567 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:06.789529    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:52:07.788579 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:07.788544    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:52:08.788331 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:08.788298    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:52:09.798655 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:09.798622    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:52:09.798655 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:52:09.798642    2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 18 16:52:09.878744 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:52:09.878708    2585 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-5.ec2.internal\" not found"
Sep 18 16:52:10.330997 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:10.330960    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:52:10.332075 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:10.332060    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:52:10.332146 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:10.332089    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:52:10.332146 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:10.332099    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:52:10.332146 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:10.332127    2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:52:10.350483 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:52:10.350452    2585 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:52:10.787987 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:10.787962    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:52:11.280570 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:52:11.280539    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 18 16:52:11.788906 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:11.788879    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:52:12.787650 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:12.787615    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:52:12.910550 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:12.910502    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:52:12.911546 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:12.911531    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:52:12.911628 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:12.911559    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:52:12.911628 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:12.911571    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:52:12.911807 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:52:12.911794    2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-5.ec2.internal\" not found" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:52:12.922248 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:52:12.922131    2585 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d6783f402451da\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d6783f402451da  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal,UID:a3127caef50db0be093e96af172c433b,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Pulled,Message:Container image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:beacea521116b2422086941236135068766b54396a322e81fd4104d55c0282df\" already present on machine,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:48:34.98074569 +0000 UTC m=+17.449730684,LastTimestamp:2026-09-18 16:52:12.912558455 +0000 UTC m=+235.381543388,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:52:12.991776 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:52:12.991675    2585 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d6783c7e467f59\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d6783c7e467f59  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal,UID:a3127caef50db0be093e96af172c433b,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Created,Message:Created container: setup,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:48:23.138271065 +0000 UTC m=+5.607256021,LastTimestamp:2026-09-18 16:52:12.982120429 +0000 UTC m=+235.451105387,Count:6,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:52:12.998987 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:52:12.998893    2585 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d6783c7ebf6662\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d6783c7ebf6662  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal,UID:a3127caef50db0be093e96af172c433b,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Started,Message:Started container setup,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:48:23.14619453 +0000 UTC m=+5.615179485,LastTimestamp:2026-09-18 16:52:12.990510492 +0000 UTC m=+235.459495425,Count:6,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:52:13.309197 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:13.309168    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_a3127caef50db0be093e96af172c433b/setup/4.log"
Sep 18 16:52:13.309380 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:13.309213    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" event={"ID":"a3127caef50db0be093e96af172c433b","Type":"ContainerStarted","Data":"309c63642750e1d721fad53f8a33762f5ee275478f173570d4f4f5da1c946525"}
Sep 18 16:52:13.309380 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:13.309348    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:52:13.310319 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:13.310304    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:52:13.310382 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:13.310333    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:52:13.310382 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:13.310345    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:52:13.310551 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:52:13.310537    2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-5.ec2.internal\" not found" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:52:13.788560 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:13.788534    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:52:14.790909 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:14.790871    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:52:15.787981 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:15.787948    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:52:16.788146 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:16.788112    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:52:16.809291 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:52:16.809257    2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 18 16:52:17.350948 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:17.350902    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:52:17.352755 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:17.352734    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:52:17.352879 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:17.352766    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:52:17.352879 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:17.352776    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:52:17.352879 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:17.352807    2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:52:17.368784 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:52:17.368751    2585 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:52:17.788161 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:17.788134    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:52:17.911516 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:17.911472    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:52:17.912535 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:17.912514    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:52:17.912620 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:17.912552    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:52:17.912620 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:17.912563    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:52:17.912769 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:52:17.912757    2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-5.ec2.internal\" not found" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:52:18.787743 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:18.787708    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:52:19.791625 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:19.791596    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:52:19.879651 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:52:19.879614    2585 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-5.ec2.internal\" not found"
Sep 18 16:52:20.788453 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:20.788421    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:52:21.787669 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:21.787636    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:52:22.789775 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:22.789740    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:52:23.788562 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:23.788529    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:52:23.818167 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:52:23.818135    2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 18 16:52:24.328344 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:24.328318    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_a3127caef50db0be093e96af172c433b/setup/5.log"
Sep 18 16:52:24.328756 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:24.328741    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_a3127caef50db0be093e96af172c433b/setup/4.log"
Sep 18 16:52:24.328814 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:24.328775    2585 generic.go:356] "Generic (PLEG): container finished" podID="a3127caef50db0be093e96af172c433b" containerID="309c63642750e1d721fad53f8a33762f5ee275478f173570d4f4f5da1c946525" exitCode=1
Sep 18 16:52:24.328814 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:24.328805    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" event={"ID":"a3127caef50db0be093e96af172c433b","Type":"ContainerDied","Data":"309c63642750e1d721fad53f8a33762f5ee275478f173570d4f4f5da1c946525"}
Sep 18 16:52:24.328877 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:24.328848    2585 scope.go:117] "RemoveContainer" containerID="63a9509d3c692b2ec712d63129a2bd2ab30bb1290872c793380270831125e694"
Sep 18 16:52:24.328993 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:24.328978    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:52:24.330239 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:24.329820    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:52:24.330239 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:24.329853    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:52:24.330239 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:24.329867    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:52:24.330239 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:52:24.330117    2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-5.ec2.internal\" not found" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:52:24.330239 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:24.330178    2585 scope.go:117] "RemoveContainer" containerID="63a9509d3c692b2ec712d63129a2bd2ab30bb1290872c793380270831125e694"
Sep 18 16:52:24.338872 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:52:24.338841    2585 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator_a3127caef50db0be093e96af172c433b_4 in pod sandbox cf961e79c5b92a2e43a70c6a29822bf8a5a2069147989bc7b9bb0b53e33609ad from index: no such id: '63a9509d3c692b2ec712d63129a2bd2ab30bb1290872c793380270831125e694'" containerID="63a9509d3c692b2ec712d63129a2bd2ab30bb1290872c793380270831125e694"
Sep 18 16:52:24.338947 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:52:24.338892    2585 kuberuntime_container.go:962] "Unhandled Error" err="failed to remove pod init container \"setup\": rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator_a3127caef50db0be093e96af172c433b_4 in pod sandbox cf961e79c5b92a2e43a70c6a29822bf8a5a2069147989bc7b9bb0b53e33609ad from index: no such id: '63a9509d3c692b2ec712d63129a2bd2ab30bb1290872c793380270831125e694'; Skipping pod \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b)\"" logger="UnhandledError"
Sep 18 16:52:24.339054 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:52:24.339037    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" podUID="a3127caef50db0be093e96af172c433b"
Sep 18 16:52:24.350079 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:52:24.349989    2585 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d678420b68270a\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d678420b68270a  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal,UID:a3127caef50db0be093e96af172c433b,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:BackOff,Message:Back-off restarting failed container setup in pod kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b),Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:48:46.980900618 +0000 UTC m=+29.449885565,LastTimestamp:2026-09-18 16:52:24.339000788 +0000 UTC m=+246.807985721,Count:14,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:52:24.369285 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:24.369259    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:52:24.370363 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:24.370343    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:52:24.370447 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:24.370378    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:52:24.370447 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:24.370388    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:52:24.370447 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:24.370417    2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:52:24.386722 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:52:24.386693    2585 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:52:24.788710 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:24.788684    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:52:25.332012 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:25.331985    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_a3127caef50db0be093e96af172c433b/setup/5.log"
Sep 18 16:52:25.786830 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:25.786804    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:52:26.788507 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:26.788471    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:52:27.791419 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:27.791384    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:52:28.788991 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:28.788955    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:52:29.787133 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:29.787099    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:52:29.880456 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:52:29.880405    2585 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-5.ec2.internal\" not found"
Sep 18 16:52:30.789486 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:30.789450    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:52:30.827045 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:52:30.827012    2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 18 16:52:31.386855 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:31.386827    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:52:31.388498 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:31.388467    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:52:31.388498 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:31.388503    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:52:31.388663 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:31.388512    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:52:31.388663 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:31.388539    2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:52:31.410253 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:52:31.410211    2585 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:52:31.788502 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:31.788473    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:52:32.787824 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:32.787790    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:52:33.791850 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:33.791813    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:52:34.788454 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:34.788421    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:52:34.910706 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:34.910662    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:52:34.912118 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:34.912099    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:52:34.912188 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:34.912133    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:52:34.912188 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:34.912145    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:52:34.912368 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:52:34.912354    2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-5.ec2.internal\" not found" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:52:34.912518 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:52:34.912502    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" podUID="a3127caef50db0be093e96af172c433b"
Sep 18 16:52:34.922961 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:52:34.922868    2585 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d678420b68270a\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d678420b68270a  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal,UID:a3127caef50db0be093e96af172c433b,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:BackOff,Message:Back-off restarting failed container setup in pod kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b),Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:48:46.980900618 +0000 UTC m=+29.449885565,LastTimestamp:2026-09-18 16:52:34.912472307 +0000 UTC m=+257.381457253,Count:15,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:52:35.445864 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:35.445823    2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 16:52:35.448945 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:52:35.448914    2585 certificate_manager.go:596] "Failed while requesting a signed certificate from the control plane" err="cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User \"system:serviceaccount:openshift-machine-config-operator:node-bootstrapper\" cannot create resource \"certificatesigningrequests\" in API group \"certificates.k8s.io\" at the cluster scope" logger="kubernetes.io/kube-apiserver-client-kubelet.UnhandledError"
Sep 18 16:52:35.789031 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:35.788998    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:52:36.788430 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:36.788392    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:52:37.787076 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:37.787037    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:52:37.840516 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:52:37.840480    2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 18 16:52:38.411334 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:38.411282    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:52:38.412316 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:38.412287    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:52:38.412316 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:38.412322    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:52:38.412485 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:38.412332    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:52:38.412485 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:38.412362    2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:52:38.429204 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:52:38.429170    2585 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:52:38.789359 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:38.789330    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:52:39.788238 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:39.788185    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:52:39.880611 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:52:39.880562    2585 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-5.ec2.internal\" not found"
Sep 18 16:52:40.789958 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:40.789923    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:52:40.898448 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:52:40.898416    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 18 16:52:41.787584 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:41.787553    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:52:42.788303 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:42.788271    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:52:43.742595 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:52:43.742560    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass"
Sep 18 16:52:43.787083 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:43.787053    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:52:44.787041 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:44.787005    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:52:44.852855 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:52:44.852824    2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 18 16:52:45.429858 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:45.429823    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:52:45.430965 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:45.430941    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:52:45.431096 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:45.430975    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:52:45.431096 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:45.430986    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:52:45.431096 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:45.431013    2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:52:45.447628 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:52:45.447595    2585 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:52:45.788924 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:45.788891    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:52:45.910903 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:45.910868    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:52:45.911922 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:45.911905    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:52:45.912023 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:45.911937    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:52:45.912023 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:45.911950    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:52:45.912174 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:52:45.912161    2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-5.ec2.internal\" not found" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:52:45.912338 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:52:45.912322    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" podUID="a3127caef50db0be093e96af172c433b"
Sep 18 16:52:45.920986 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:52:45.920897    2585 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d678420b68270a\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d678420b68270a  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal,UID:a3127caef50db0be093e96af172c433b,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:BackOff,Message:Back-off restarting failed container setup in pod kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b),Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:48:46.980900618 +0000 UTC m=+29.449885565,LastTimestamp:2026-09-18 16:52:45.912292847 +0000 UTC m=+268.381277779,Count:16,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:52:46.787914 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:46.787880    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:52:47.789431 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:47.789397    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:52:48.769465 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:52:48.769423    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 18 16:52:48.787322 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:48.787292    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:52:49.789124 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:49.789089    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:52:49.881455 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:52:49.881397    2585 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-5.ec2.internal\" not found"
Sep 18 16:52:50.788559 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:50.788526    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:52:51.176023 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:52:51.175925    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 18 16:52:51.789996 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:51.789960    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:52:51.862306 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:52:51.862269    2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 18 16:52:52.448240 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:52.448168    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:52:52.449448 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:52.449431    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:52:52.449562 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:52.449462    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:52:52.449562 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:52.449473    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:52:52.449562 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:52.449501    2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:52:52.467456 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:52:52.467420    2585 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:52:52.788332 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:52.788297    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:52:53.787525 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:53.787487    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:52:54.789321 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:54.789293    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:52:55.792787 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:55.792755    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:52:56.787402 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:56.787369    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:52:57.791462 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:57.791430    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:52:58.787327 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:58.787294    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:52:58.871289 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:52:58.871251    2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 18 16:52:59.468181 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:59.468129    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:52:59.469184 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:59.469157    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:52:59.469331 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:59.469189    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:52:59.469331 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:59.469203    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:52:59.469331 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:59.469254    2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:52:59.489025 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:52:59.488986    2585 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:52:59.788509 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:52:59.788481    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:52:59.881773 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:52:59.881741    2585 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-5.ec2.internal\" not found"
Sep 18 16:53:00.788794 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:53:00.788761    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:53:00.910727 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:53:00.910676    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:53:00.911929 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:53:00.911909    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:53:00.912056 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:53:00.911945    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:53:00.912056 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:53:00.911959    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:53:00.912270 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:53:00.912254    2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-5.ec2.internal\" not found" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:53:00.912456 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:53:00.912435    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" podUID="a3127caef50db0be093e96af172c433b"
Sep 18 16:53:00.925378 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:53:00.925274    2585 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d678420b68270a\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d678420b68270a  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal,UID:a3127caef50db0be093e96af172c433b,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:BackOff,Message:Back-off restarting failed container setup in pod kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b),Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:48:46.980900618 +0000 UTC m=+29.449885565,LastTimestamp:2026-09-18 16:53:00.912401096 +0000 UTC m=+283.381386028,Count:17,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:53:01.787497 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:53:01.787465    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:53:01.935381 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:53:01.935338    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:53:01.936506 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:53:01.936482    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:53:01.936578 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:53:01.936525    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:53:01.936578 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:53:01.936536    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:53:02.789329 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:53:02.789299    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:53:03.788911 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:53:03.788877    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:53:04.790153 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:53:04.790117    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:53:05.787575 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:53:05.787544    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:53:05.879673 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:53:05.879641    2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 18 16:53:06.490103 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:53:06.490048    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:53:06.491050 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:53:06.491032    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:53:06.491165 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:53:06.491068    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:53:06.491165 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:53:06.491081    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:53:06.491165 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:53:06.491117    2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:53:06.508465 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:53:06.508430    2585 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:53:06.788066 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:53:06.788038    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:53:07.445449 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:53:07.445392    2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 16:53:07.448331 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:53:07.448308    2585 certificate_manager.go:596] "Failed while requesting a signed certificate from the control plane" err="cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User \"system:serviceaccount:openshift-machine-config-operator:node-bootstrapper\" cannot create resource \"certificatesigningrequests\" in API group \"certificates.k8s.io\" at the cluster scope" logger="kubernetes.io/kube-apiserver-client-kubelet.UnhandledError"
Sep 18 16:53:07.789904 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:53:07.789871    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:53:08.787641 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:53:08.787610    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:53:09.789577 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:53:09.789544    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:53:09.882553 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:53:09.882512    2585 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-5.ec2.internal\" not found"
Sep 18 16:53:10.789035 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:53:10.789000    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:53:11.787727 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:53:11.787691    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:53:12.787593 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:53:12.787560    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:53:12.892022 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:53:12.891987    2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 18 16:53:13.509340 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:53:13.509289    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:53:13.510414 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:53:13.510397    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:53:13.510509 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:53:13.510431    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:53:13.510509 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:53:13.510444    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:53:13.510509 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:53:13.510471    2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:53:13.529323 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:53:13.529291    2585 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:53:13.789415 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:53:13.789378    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:53:14.788950 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:53:14.788917    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:53:14.910827 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:53:14.910782    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:53:14.911850 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:53:14.911831    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:53:14.911986 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:53:14.911867    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:53:14.911986 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:53:14.911881    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:53:14.912164 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:53:14.912149    2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-5.ec2.internal\" not found" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:53:14.912362 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:53:14.912342    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" podUID="a3127caef50db0be093e96af172c433b"
Sep 18 16:53:14.922738 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:53:14.922634    2585 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d678420b68270a\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d678420b68270a  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal,UID:a3127caef50db0be093e96af172c433b,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:BackOff,Message:Back-off restarting failed container setup in pod kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b),Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:48:46.980900618 +0000 UTC m=+29.449885565,LastTimestamp:2026-09-18 16:53:14.912310109 +0000 UTC m=+297.381295043,Count:18,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:53:15.789347 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:53:15.789314    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:53:16.788853 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:53:16.788822    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:53:17.787609 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:53:17.787574    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:53:18.790369 ip-10-0-143-5 kubenswrapper[2585]: I0918 16:53:18.790332    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:53:19.719612 ip-10-0-143-5 kubenswrapper[2585]: E0918 16:53:19.719537    2585 transport.go:120] "No valid client certificate is found and the server is responsive, exiting." lastCertificateAvailabilityTime="2026-09-18 16:48:19.718981931 +0000 UTC m=+2.187966889" shutdownThreshold="5m0s"
Sep 18 16:53:19.729614 ip-10-0-143-5 systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
Sep 18 16:53:19.729741 ip-10-0-143-5 systemd[1]: kubelet.service: Failed with result 'exit-code'.
Sep 18 16:53:19.729977 ip-10-0-143-5 systemd[1]: kubelet.service: Consumed 3.564s CPU time.
Sep 18 16:53:29.760032 ip-10-0-143-5 systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 1.
Sep 18 16:53:29.760198 ip-10-0-143-5 systemd[1]: Stopped Kubernetes Kubelet.
Sep 18 16:53:29.760275 ip-10-0-143-5 systemd[1]: kubelet.service: Consumed 3.564s CPU time.
Sep 18 16:53:29.769631 ip-10-0-143-5 systemd[1]: Starting Kubernetes Kubelet...
Sep 18 16:53:29.832159 ip-10-0-143-5 kubenswrapper[2881]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 16:53:29.832159 ip-10-0-143-5 kubenswrapper[2881]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 18 16:53:29.832159 ip-10-0-143-5 kubenswrapper[2881]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 16:53:29.832159 ip-10-0-143-5 kubenswrapper[2881]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 18 16:53:29.832159 ip-10-0-143-5 kubenswrapper[2881]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 16:53:29.832708 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.832232    2881 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 18 16:53:29.834307 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834290    2881 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 16:53:29.834307 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834305    2881 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 16:53:29.834376 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834309    2881 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 16:53:29.834376 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834313    2881 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 16:53:29.834376 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834316    2881 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 16:53:29.834376 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834319    2881 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 16:53:29.834376 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834325    2881 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 16:53:29.834376 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834328    2881 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 16:53:29.834376 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834332    2881 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 16:53:29.834376 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834335    2881 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 16:53:29.834376 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834338    2881 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 16:53:29.834376 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834340    2881 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 16:53:29.834376 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834343    2881 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 16:53:29.834376 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834346    2881 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 16:53:29.834376 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834349    2881 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 16:53:29.834376 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834351    2881 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 16:53:29.834376 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834354    2881 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 16:53:29.834376 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834357    2881 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 16:53:29.834376 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834360    2881 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 16:53:29.834376 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834362    2881 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 16:53:29.834376 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834365    2881 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 16:53:29.834376 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834368    2881 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 16:53:29.834894 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834371    2881 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 16:53:29.834894 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834373    2881 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 16:53:29.834894 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834376    2881 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 16:53:29.834894 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834379    2881 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 16:53:29.834894 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834382    2881 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 16:53:29.834894 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834385    2881 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 16:53:29.834894 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834388    2881 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 16:53:29.834894 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834391    2881 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 16:53:29.834894 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834393    2881 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 16:53:29.834894 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834396    2881 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 16:53:29.834894 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834398    2881 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 16:53:29.834894 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834401    2881 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 16:53:29.834894 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834403    2881 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 16:53:29.834894 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834406    2881 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 16:53:29.834894 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834409    2881 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 16:53:29.834894 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834411    2881 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 16:53:29.834894 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834416    2881 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 16:53:29.834894 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834421    2881 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 16:53:29.834894 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834423    2881 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 16:53:29.834894 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834427    2881 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 16:53:29.835442 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834430    2881 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 16:53:29.835442 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834433    2881 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 16:53:29.835442 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834436    2881 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 16:53:29.835442 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834439    2881 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 16:53:29.835442 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834441    2881 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 16:53:29.835442 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834444    2881 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 16:53:29.835442 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834446    2881 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 16:53:29.835442 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834449    2881 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 16:53:29.835442 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834451    2881 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 16:53:29.835442 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834454    2881 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 16:53:29.835442 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834456    2881 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 16:53:29.835442 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834459    2881 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 16:53:29.835442 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834462    2881 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 16:53:29.835442 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834465    2881 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 16:53:29.835442 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834468    2881 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 16:53:29.835442 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834471    2881 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 16:53:29.835442 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834474    2881 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 16:53:29.835442 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834476    2881 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 16:53:29.835442 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834479    2881 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 16:53:29.835442 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834482    2881 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 16:53:29.835967 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834485    2881 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 16:53:29.835967 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834488    2881 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 16:53:29.835967 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834491    2881 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 16:53:29.835967 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834493    2881 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 16:53:29.835967 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834496    2881 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 16:53:29.835967 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834499    2881 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 16:53:29.835967 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834501    2881 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 16:53:29.835967 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834504    2881 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 16:53:29.835967 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834506    2881 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 16:53:29.835967 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834510    2881 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 16:53:29.835967 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834514    2881 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 16:53:29.835967 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834517    2881 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 16:53:29.835967 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834520    2881 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 16:53:29.835967 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834523    2881 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 16:53:29.835967 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834525    2881 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 16:53:29.835967 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834528    2881 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 16:53:29.835967 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834530    2881 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 16:53:29.835967 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834534    2881 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 16:53:29.835967 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834536    2881 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 16:53:29.836532 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834539    2881 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 16:53:29.836532 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834542    2881 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 16:53:29.836532 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834545    2881 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 16:53:29.836532 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834547    2881 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 16:53:29.836532 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834550    2881 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 16:53:29.836532 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834553    2881 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 16:53:29.836532 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834557    2881 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 16:53:29.836532 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834560    2881 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 16:53:29.836532 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834563    2881 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 16:53:29.836532 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834565    2881 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 16:53:29.836532 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834567    2881 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 16:53:29.836532 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834570    2881 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 16:53:29.836532 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834572    2881 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 16:53:29.836532 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834577    2881 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 16:53:29.836532 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834580    2881 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 16:53:29.836532 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834583    2881 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 16:53:29.836532 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834586    2881 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 16:53:29.836532 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.834589    2881 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 16:53:29.836532 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835316    2881 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 16:53:29.837019 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835325    2881 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 16:53:29.837019 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835329    2881 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 16:53:29.837019 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835332    2881 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 16:53:29.837019 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835334    2881 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 16:53:29.837019 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835338    2881 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 16:53:29.837019 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835341    2881 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 16:53:29.837019 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835344    2881 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 16:53:29.837019 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835347    2881 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 16:53:29.837019 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835351    2881 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 16:53:29.837019 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835355    2881 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 16:53:29.837019 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835359    2881 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 16:53:29.837019 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835363    2881 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 16:53:29.837019 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835366    2881 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 16:53:29.837019 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835369    2881 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 16:53:29.837019 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835371    2881 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 16:53:29.837019 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835374    2881 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 16:53:29.837019 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835377    2881 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 16:53:29.837019 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835380    2881 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 16:53:29.837019 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835383    2881 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 16:53:29.837627 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835386    2881 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 16:53:29.837627 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835389    2881 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 16:53:29.837627 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835392    2881 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 16:53:29.837627 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835394    2881 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 16:53:29.837627 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835397    2881 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 16:53:29.837627 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835400    2881 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 16:53:29.837627 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835403    2881 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 16:53:29.837627 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835406    2881 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 16:53:29.837627 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835408    2881 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 16:53:29.837627 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835411    2881 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 16:53:29.837627 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835413    2881 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 16:53:29.837627 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835416    2881 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 16:53:29.837627 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835418    2881 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 16:53:29.837627 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835421    2881 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 16:53:29.837627 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835423    2881 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 16:53:29.837627 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835426    2881 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 16:53:29.837627 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835429    2881 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 16:53:29.837627 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835431    2881 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 16:53:29.837627 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835434    2881 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 16:53:29.838126 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835436    2881 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 16:53:29.838126 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835439    2881 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 16:53:29.838126 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835442    2881 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 16:53:29.838126 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835444    2881 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 16:53:29.838126 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835447    2881 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 16:53:29.838126 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835450    2881 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 16:53:29.838126 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835453    2881 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 16:53:29.838126 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835455    2881 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 16:53:29.838126 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835458    2881 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 16:53:29.838126 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835461    2881 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 16:53:29.838126 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835463    2881 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 16:53:29.838126 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835466    2881 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 16:53:29.838126 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835470    2881 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 16:53:29.838126 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835473    2881 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 16:53:29.838126 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835476    2881 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 16:53:29.838126 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835478    2881 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 16:53:29.838126 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835482    2881 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 16:53:29.838126 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835485    2881 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 16:53:29.838126 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835488    2881 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 16:53:29.838642 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835491    2881 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 16:53:29.838642 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835494    2881 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 16:53:29.838642 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835497    2881 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 16:53:29.838642 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835500    2881 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 16:53:29.838642 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835503    2881 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 16:53:29.838642 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835506    2881 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 16:53:29.838642 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835509    2881 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 16:53:29.838642 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835512    2881 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 16:53:29.838642 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835515    2881 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 16:53:29.838642 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835518    2881 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 16:53:29.838642 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835520    2881 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 16:53:29.838642 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835523    2881 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 16:53:29.838642 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835525    2881 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 16:53:29.838642 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835528    2881 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 16:53:29.838642 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835531    2881 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 16:53:29.838642 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835534    2881 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 16:53:29.838642 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835536    2881 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 16:53:29.838642 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835539    2881 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 16:53:29.838642 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835541    2881 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 16:53:29.838642 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835544    2881 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 16:53:29.839164 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835547    2881 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 16:53:29.839164 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835549    2881 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 16:53:29.839164 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835552    2881 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 16:53:29.839164 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835554    2881 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 16:53:29.839164 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835557    2881 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 16:53:29.839164 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835560    2881 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 16:53:29.839164 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835563    2881 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 16:53:29.839164 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835566    2881 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 16:53:29.839164 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835569    2881 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 16:53:29.839164 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835571    2881 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 16:53:29.839164 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835574    2881 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 16:53:29.839164 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835576    2881 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 16:53:29.839164 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835579    2881 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 16:53:29.839164 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835582    2881 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 16:53:29.839164 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835584    2881 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 16:53:29.839164 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835587    2881 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 16:53:29.839164 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835590    2881 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 16:53:29.839164 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835592    2881 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 16:53:29.839164 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835595    2881 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 16:53:29.839164 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835597    2881 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 16:53:29.839164 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835600    2881 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 16:53:29.839734 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835979    2881 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 16:53:29.839734 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835985    2881 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 16:53:29.839734 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835988    2881 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 16:53:29.839734 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835991    2881 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 16:53:29.839734 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.835999    2881 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 16:53:29.839734 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836002    2881 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 16:53:29.839734 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836005    2881 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 16:53:29.839734 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836007    2881 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 16:53:29.839734 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836010    2881 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 16:53:29.839734 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836013    2881 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 16:53:29.839734 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836015    2881 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 16:53:29.839734 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836018    2881 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 16:53:29.839734 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836021    2881 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 16:53:29.839734 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836024    2881 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 16:53:29.839734 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836027    2881 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 16:53:29.839734 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836030    2881 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 16:53:29.839734 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836033    2881 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 16:53:29.839734 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836040    2881 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 16:53:29.839734 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836043    2881 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 16:53:29.839734 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836046    2881 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 16:53:29.840273 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836049    2881 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 16:53:29.840273 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836052    2881 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 16:53:29.840273 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836055    2881 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 16:53:29.840273 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836058    2881 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 16:53:29.840273 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836060    2881 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 16:53:29.840273 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836063    2881 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 16:53:29.840273 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836065    2881 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 16:53:29.840273 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836068    2881 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 16:53:29.840273 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836072    2881 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 16:53:29.840273 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836076    2881 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 16:53:29.840273 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836081    2881 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 16:53:29.840273 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836085    2881 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 16:53:29.840273 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836088    2881 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 16:53:29.840273 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836090    2881 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 16:53:29.840273 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836093    2881 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 16:53:29.840273 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836096    2881 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 16:53:29.840273 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836099    2881 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 16:53:29.840273 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836102    2881 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 16:53:29.840273 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836104    2881 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 16:53:29.840765 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836107    2881 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 16:53:29.840765 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836110    2881 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 16:53:29.840765 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836112    2881 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 16:53:29.840765 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836115    2881 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 16:53:29.840765 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836118    2881 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 16:53:29.840765 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836120    2881 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 16:53:29.840765 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836123    2881 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 16:53:29.840765 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836125    2881 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 16:53:29.840765 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836128    2881 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 16:53:29.840765 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836131    2881 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 16:53:29.840765 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836135    2881 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 16:53:29.840765 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836137    2881 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 16:53:29.840765 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836140    2881 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 16:53:29.840765 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836143    2881 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 16:53:29.840765 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836146    2881 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 16:53:29.840765 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836148    2881 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 16:53:29.840765 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836151    2881 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 16:53:29.840765 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836154    2881 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 16:53:29.840765 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836156    2881 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 16:53:29.840765 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836159    2881 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 16:53:29.841506 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836161    2881 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 16:53:29.841506 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836164    2881 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 16:53:29.841506 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836168    2881 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 16:53:29.841506 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836171    2881 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 16:53:29.841506 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836174    2881 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 16:53:29.841506 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836177    2881 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 16:53:29.841506 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836179    2881 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 16:53:29.841506 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836182    2881 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 16:53:29.841506 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836185    2881 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 16:53:29.841506 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836189    2881 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 16:53:29.841506 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836192    2881 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 16:53:29.841506 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836195    2881 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 16:53:29.841506 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836198    2881 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 16:53:29.841506 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836200    2881 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 16:53:29.841506 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836203    2881 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 16:53:29.841506 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836206    2881 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 16:53:29.841506 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836208    2881 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 16:53:29.841506 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836211    2881 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 16:53:29.841506 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836214    2881 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 16:53:29.841506 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836216    2881 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 16:53:29.842423 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836233    2881 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 16:53:29.842423 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836238    2881 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 16:53:29.842423 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836243    2881 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 16:53:29.842423 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836247    2881 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 16:53:29.842423 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836249    2881 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 16:53:29.842423 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836252    2881 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 16:53:29.842423 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836255    2881 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 16:53:29.842423 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836257    2881 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 16:53:29.842423 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836260    2881 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 16:53:29.842423 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836263    2881 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 16:53:29.842423 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836266    2881 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 16:53:29.842423 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836269    2881 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 16:53:29.842423 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836271    2881 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 16:53:29.842423 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836274    2881 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 16:53:29.842423 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836277    2881 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 16:53:29.842423 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836279    2881 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 16:53:29.842423 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836282    2881 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 16:53:29.842423 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836284    2881 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 16:53:29.842423 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836287    2881 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 16:53:29.842423 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.836289    2881 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 16:53:29.843388 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836361    2881 flags.go:64] FLAG: --address="0.0.0.0"
Sep 18 16:53:29.843388 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836368    2881 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 18 16:53:29.843388 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836375    2881 flags.go:64] FLAG: --anonymous-auth="true"
Sep 18 16:53:29.843388 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836379    2881 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 18 16:53:29.843388 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836385    2881 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 18 16:53:29.843388 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836388    2881 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 18 16:53:29.843388 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836393    2881 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 18 16:53:29.843388 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836397    2881 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 18 16:53:29.843388 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836401    2881 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 18 16:53:29.843388 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836404    2881 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 18 16:53:29.843388 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836407    2881 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 18 16:53:29.843388 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836410    2881 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 18 16:53:29.843388 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836413    2881 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 18 16:53:29.843388 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836417    2881 flags.go:64] FLAG: --cgroup-root=""
Sep 18 16:53:29.843388 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836421    2881 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 18 16:53:29.843388 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836425    2881 flags.go:64] FLAG: --client-ca-file=""
Sep 18 16:53:29.843388 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836428    2881 flags.go:64] FLAG: --cloud-provider="external"
Sep 18 16:53:29.843388 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836431    2881 flags.go:64] FLAG: --cluster-dns="[]"
Sep 18 16:53:29.843388 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836436    2881 flags.go:64] FLAG: --cluster-domain=""
Sep 18 16:53:29.843388 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836439    2881 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 18 16:53:29.843388 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836442    2881 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 18 16:53:29.843388 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836445    2881 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 18 16:53:29.843388 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836449    2881 flags.go:64] FLAG: --container-log-max-files="5"
Sep 18 16:53:29.843388 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836453    2881 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 18 16:53:29.844479 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836456    2881 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 18 16:53:29.844479 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836459    2881 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 18 16:53:29.844479 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836463    2881 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 18 16:53:29.844479 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836467    2881 flags.go:64] FLAG: --contention-profiling="false"
Sep 18 16:53:29.844479 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836470    2881 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 18 16:53:29.844479 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836473    2881 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 18 16:53:29.844479 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836477    2881 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 18 16:53:29.844479 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836480    2881 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 18 16:53:29.844479 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836484    2881 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 18 16:53:29.844479 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836487    2881 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 18 16:53:29.844479 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836490    2881 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 18 16:53:29.844479 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836493    2881 flags.go:64] FLAG: --enable-load-reader="false"
Sep 18 16:53:29.844479 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836497    2881 flags.go:64] FLAG: --enable-server="true"
Sep 18 16:53:29.844479 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836501    2881 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 18 16:53:29.844479 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836508    2881 flags.go:64] FLAG: --event-burst="100"
Sep 18 16:53:29.844479 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836511    2881 flags.go:64] FLAG: --event-qps="50"
Sep 18 16:53:29.844479 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836514    2881 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 18 16:53:29.844479 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836517    2881 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 18 16:53:29.844479 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836520    2881 flags.go:64] FLAG: --eviction-hard=""
Sep 18 16:53:29.844479 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836524    2881 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 18 16:53:29.844479 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836527    2881 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 18 16:53:29.844479 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836532    2881 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 18 16:53:29.844479 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836537    2881 flags.go:64] FLAG: --eviction-soft=""
Sep 18 16:53:29.844479 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836540    2881 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 18 16:53:29.844479 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836543    2881 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 18 16:53:29.845565 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836546    2881 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 18 16:53:29.845565 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836550    2881 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 18 16:53:29.845565 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836553    2881 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 18 16:53:29.845565 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836556    2881 flags.go:64] FLAG: --fail-swap-on="true"
Sep 18 16:53:29.845565 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836558    2881 flags.go:64] FLAG: --feature-gates=""
Sep 18 16:53:29.845565 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836567    2881 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 18 16:53:29.845565 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836570    2881 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 18 16:53:29.845565 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836573    2881 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 18 16:53:29.845565 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836576    2881 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 18 16:53:29.845565 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836580    2881 flags.go:64] FLAG: --healthz-port="10248"
Sep 18 16:53:29.845565 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836583    2881 flags.go:64] FLAG: --help="false"
Sep 18 16:53:29.845565 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836586    2881 flags.go:64] FLAG: --hostname-override="ip-10-0-143-5.ec2.internal"
Sep 18 16:53:29.845565 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836589    2881 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 18 16:53:29.845565 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836593    2881 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 18 16:53:29.845565 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836596    2881 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 18 16:53:29.845565 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836599    2881 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 18 16:53:29.845565 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836602    2881 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 18 16:53:29.845565 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836605    2881 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 18 16:53:29.845565 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836608    2881 flags.go:64] FLAG: --image-service-endpoint=""
Sep 18 16:53:29.845565 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836611    2881 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 18 16:53:29.845565 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836614    2881 flags.go:64] FLAG: --kube-api-burst="100"
Sep 18 16:53:29.845565 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836617    2881 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 18 16:53:29.845565 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836620    2881 flags.go:64] FLAG: --kube-api-qps="50"
Sep 18 16:53:29.845565 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836623    2881 flags.go:64] FLAG: --kube-reserved=""
Sep 18 16:53:29.846693 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836626    2881 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 18 16:53:29.846693 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836629    2881 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 18 16:53:29.846693 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836632    2881 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 18 16:53:29.846693 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836635    2881 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 18 16:53:29.846693 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836639    2881 flags.go:64] FLAG: --lock-file=""
Sep 18 16:53:29.846693 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836643    2881 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 18 16:53:29.846693 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836646    2881 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 18 16:53:29.846693 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836649    2881 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 18 16:53:29.846693 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836655    2881 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 18 16:53:29.846693 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836658    2881 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 18 16:53:29.846693 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836661    2881 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 18 16:53:29.846693 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836663    2881 flags.go:64] FLAG: --logging-format="text"
Sep 18 16:53:29.846693 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836666    2881 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 18 16:53:29.846693 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836670    2881 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 18 16:53:29.846693 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836672    2881 flags.go:64] FLAG: --manifest-url=""
Sep 18 16:53:29.846693 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836675    2881 flags.go:64] FLAG: --manifest-url-header=""
Sep 18 16:53:29.846693 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836680    2881 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 18 16:53:29.846693 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836683    2881 flags.go:64] FLAG: --max-open-files="1000000"
Sep 18 16:53:29.846693 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836687    2881 flags.go:64] FLAG: --max-pods="110"
Sep 18 16:53:29.846693 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836690    2881 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 18 16:53:29.846693 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836693    2881 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 18 16:53:29.846693 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836696    2881 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 18 16:53:29.846693 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836699    2881 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 18 16:53:29.846693 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836702    2881 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 18 16:53:29.846693 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836705    2881 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 18 16:53:29.847710 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836708    2881 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 18 16:53:29.847710 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836715    2881 flags.go:64] FLAG: --node-status-max-images="50"
Sep 18 16:53:29.847710 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836717    2881 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 18 16:53:29.847710 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836720    2881 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 18 16:53:29.847710 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836723    2881 flags.go:64] FLAG: --pod-cidr=""
Sep 18 16:53:29.847710 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836727    2881 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 18 16:53:29.847710 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836733    2881 flags.go:64] FLAG: --pod-manifest-path=""
Sep 18 16:53:29.847710 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836736    2881 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 18 16:53:29.847710 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836739    2881 flags.go:64] FLAG: --pods-per-core="0"
Sep 18 16:53:29.847710 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836742    2881 flags.go:64] FLAG: --port="10250"
Sep 18 16:53:29.847710 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836745    2881 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 18 16:53:29.847710 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836751    2881 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-011eca0bb8cdaad48"
Sep 18 16:53:29.847710 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836754    2881 flags.go:64] FLAG: --qos-reserved=""
Sep 18 16:53:29.847710 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836757    2881 flags.go:64] FLAG: --read-only-port="10255"
Sep 18 16:53:29.847710 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836760    2881 flags.go:64] FLAG: --register-node="true"
Sep 18 16:53:29.847710 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836764    2881 flags.go:64] FLAG: --register-with-taints=""
Sep 18 16:53:29.847710 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836767    2881 flags.go:64] FLAG: --registry-burst="10"
Sep 18 16:53:29.847710 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836770    2881 flags.go:64] FLAG: --registry-qps="5"
Sep 18 16:53:29.847710 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836774    2881 flags.go:64] FLAG: --reserved-cpus=""
Sep 18 16:53:29.847710 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836777    2881 flags.go:64] FLAG: --reserved-memory=""
Sep 18 16:53:29.847710 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836780    2881 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 18 16:53:29.847710 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836784    2881 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 18 16:53:29.847710 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836787    2881 flags.go:64] FLAG: --rotate-certificates="false"
Sep 18 16:53:29.847710 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836790    2881 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 18 16:53:29.847710 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836793    2881 flags.go:64] FLAG: --runonce="false"
Sep 18 16:53:29.848737 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836796    2881 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 18 16:53:29.848737 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836799    2881 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 18 16:53:29.848737 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836802    2881 flags.go:64] FLAG: --seccomp-default="false"
Sep 18 16:53:29.848737 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836805    2881 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 18 16:53:29.848737 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836807    2881 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 18 16:53:29.848737 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836811    2881 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 18 16:53:29.848737 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836814    2881 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 18 16:53:29.848737 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836817    2881 flags.go:64] FLAG: --storage-driver-password="root"
Sep 18 16:53:29.848737 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836820    2881 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 18 16:53:29.848737 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836823    2881 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 18 16:53:29.848737 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836826    2881 flags.go:64] FLAG: --storage-driver-user="root"
Sep 18 16:53:29.848737 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836829    2881 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 18 16:53:29.848737 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836832    2881 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 18 16:53:29.848737 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836836    2881 flags.go:64] FLAG: --system-cgroups=""
Sep 18 16:53:29.848737 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836839    2881 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 18 16:53:29.848737 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836845    2881 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 18 16:53:29.848737 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836848    2881 flags.go:64] FLAG: --tls-cert-file=""
Sep 18 16:53:29.848737 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836850    2881 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 18 16:53:29.848737 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836856    2881 flags.go:64] FLAG: --tls-min-version=""
Sep 18 16:53:29.848737 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836859    2881 flags.go:64] FLAG: --tls-private-key-file=""
Sep 18 16:53:29.848737 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836862    2881 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 18 16:53:29.848737 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836865    2881 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 18 16:53:29.848737 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836869    2881 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 18 16:53:29.848737 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836872    2881 flags.go:64] FLAG: --v="2"
Sep 18 16:53:29.848737 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836876    2881 flags.go:64] FLAG: --version="false"
Sep 18 16:53:29.849652 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836881    2881 flags.go:64] FLAG: --vmodule=""
Sep 18 16:53:29.849652 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836884    2881 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 18 16:53:29.849652 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.836887    2881 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 18 16:53:29.849652 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837008    2881 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 16:53:29.849652 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837012    2881 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 16:53:29.849652 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837015    2881 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 16:53:29.849652 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837019    2881 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 16:53:29.849652 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837022    2881 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 16:53:29.849652 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837025    2881 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 16:53:29.849652 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837028    2881 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 16:53:29.849652 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837031    2881 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 16:53:29.849652 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837034    2881 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 16:53:29.849652 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837037    2881 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 16:53:29.849652 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837039    2881 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 16:53:29.849652 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837042    2881 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 16:53:29.849652 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837045    2881 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 16:53:29.849652 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837047    2881 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 16:53:29.849652 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837050    2881 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 16:53:29.849652 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837053    2881 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 16:53:29.849652 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837056    2881 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 16:53:29.850184 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837060    2881 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 16:53:29.850184 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837064    2881 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 16:53:29.850184 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837067    2881 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 16:53:29.850184 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837071    2881 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 16:53:29.850184 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837075    2881 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 16:53:29.850184 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837080    2881 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 16:53:29.850184 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837083    2881 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 16:53:29.850184 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837086    2881 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 16:53:29.850184 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837089    2881 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 16:53:29.850184 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837092    2881 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 16:53:29.850184 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837095    2881 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 16:53:29.850184 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837098    2881 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 16:53:29.850184 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837101    2881 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 16:53:29.850184 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837103    2881 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 16:53:29.850184 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837106    2881 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 16:53:29.850184 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837109    2881 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 16:53:29.850184 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837111    2881 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 16:53:29.850184 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837114    2881 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 16:53:29.850184 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837117    2881 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 16:53:29.850914 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837120    2881 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 16:53:29.850914 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837122    2881 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 16:53:29.850914 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837125    2881 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 16:53:29.850914 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837128    2881 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 16:53:29.850914 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837130    2881 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 16:53:29.850914 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837133    2881 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 16:53:29.850914 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837135    2881 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 16:53:29.850914 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837138    2881 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 16:53:29.850914 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837140    2881 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 16:53:29.850914 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837143    2881 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 16:53:29.850914 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837146    2881 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 16:53:29.850914 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837148    2881 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 16:53:29.850914 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837151    2881 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 16:53:29.850914 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837153    2881 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 16:53:29.850914 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837156    2881 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 16:53:29.850914 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837159    2881 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 16:53:29.850914 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837162    2881 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 16:53:29.850914 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837164    2881 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 16:53:29.850914 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837169    2881 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 16:53:29.850914 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837172    2881 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 16:53:29.851468 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837178    2881 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 16:53:29.851468 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837181    2881 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 16:53:29.851468 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837184    2881 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 16:53:29.851468 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837187    2881 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 16:53:29.851468 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837190    2881 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 16:53:29.851468 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837193    2881 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 16:53:29.851468 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837196    2881 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 16:53:29.851468 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837199    2881 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 16:53:29.851468 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837202    2881 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 16:53:29.851468 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837205    2881 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 16:53:29.851468 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837207    2881 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 16:53:29.851468 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837211    2881 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 16:53:29.851468 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837214    2881 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 16:53:29.851468 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837216    2881 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 16:53:29.851468 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837235    2881 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 16:53:29.851468 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837239    2881 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 16:53:29.851468 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837242    2881 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 16:53:29.851468 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837245    2881 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 16:53:29.851468 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837247    2881 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 16:53:29.852066 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837250    2881 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 16:53:29.852066 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837252    2881 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 16:53:29.852066 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837255    2881 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 16:53:29.852066 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837257    2881 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 16:53:29.852066 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837260    2881 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 16:53:29.852066 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837263    2881 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 16:53:29.852066 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837266    2881 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 16:53:29.852066 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837268    2881 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 16:53:29.852066 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837271    2881 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 16:53:29.852066 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837275    2881 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 16:53:29.852066 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837278    2881 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 16:53:29.852066 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837282    2881 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 16:53:29.852066 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837285    2881 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 16:53:29.852066 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837289    2881 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 16:53:29.852066 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837291    2881 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 16:53:29.852066 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837294    2881 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 16:53:29.852066 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837297    2881 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 16:53:29.852066 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837300    2881 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 16:53:29.852066 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837302    2881 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 16:53:29.852066 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837305    2881 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 16:53:29.852704 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837308    2881 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 16:53:29.852704 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837310    2881 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 16:53:29.852704 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837313    2881 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 16:53:29.852704 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.837316    2881 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 16:53:29.852704 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.837322    2881 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 16:53:29.852704 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.843477    2881 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 18 16:53:29.852704 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.843511    2881 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 18 16:53:29.852704 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.843541    2881 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 18 16:53:29.852704 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.843556    2881 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 18 16:53:29.852704 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.843657    2881 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 16:53:29.852704 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.843665    2881 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 16:53:29.852704 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.843670    2881 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 16:53:29.852704 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.843677    2881 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 16:53:29.852704 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.843682    2881 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 16:53:29.853106 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.843689    2881 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 16:53:29.853106 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.843695    2881 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 16:53:29.853106 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.843701    2881 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 16:53:29.853106 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.843705    2881 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 16:53:29.853106 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.843710    2881 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 16:53:29.853106 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.843717    2881 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 16:53:29.853106 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.843721    2881 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 16:53:29.853106 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.843727    2881 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 16:53:29.853106 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.843732    2881 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 16:53:29.853106 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.843736    2881 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 16:53:29.853106 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.843740    2881 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 16:53:29.853106 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.843745    2881 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 16:53:29.853106 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.843759    2881 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 16:53:29.853106 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.843763    2881 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 16:53:29.853106 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.843768    2881 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 16:53:29.853106 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.843773    2881 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 16:53:29.853106 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.843779    2881 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 16:53:29.853106 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.843784    2881 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 16:53:29.853106 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.843789    2881 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 16:53:29.853106 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.843794    2881 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 16:53:29.853751 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.843798    2881 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 16:53:29.853751 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.843802    2881 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 16:53:29.853751 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.843807    2881 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 16:53:29.853751 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.843811    2881 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 16:53:29.853751 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.843815    2881 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 16:53:29.853751 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.843820    2881 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 16:53:29.853751 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.843824    2881 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 16:53:29.853751 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.843828    2881 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 16:53:29.853751 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.843833    2881 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 16:53:29.853751 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.843837    2881 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 16:53:29.853751 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.843841    2881 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 16:53:29.853751 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.843846    2881 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 16:53:29.853751 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.843850    2881 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 16:53:29.853751 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.843854    2881 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 16:53:29.853751 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.843858    2881 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 16:53:29.853751 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.843862    2881 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 16:53:29.853751 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.843867    2881 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 16:53:29.853751 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.843871    2881 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 16:53:29.853751 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.843875    2881 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 16:53:29.853751 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.843879    2881 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 16:53:29.854394 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.843885    2881 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 16:53:29.854394 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.843889    2881 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 16:53:29.854394 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.843894    2881 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 16:53:29.854394 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.843898    2881 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 16:53:29.854394 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.843902    2881 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 16:53:29.854394 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.843906    2881 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 16:53:29.854394 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.843911    2881 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 16:53:29.854394 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.843916    2881 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 16:53:29.854394 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.843921    2881 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 16:53:29.854394 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.843926    2881 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 16:53:29.854394 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.843930    2881 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 16:53:29.854394 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.843934    2881 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 16:53:29.854394 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.843939    2881 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 16:53:29.854394 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.843943    2881 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 16:53:29.854394 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.843947    2881 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 16:53:29.854394 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.843952    2881 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 16:53:29.854394 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.843956    2881 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 16:53:29.854394 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.843960    2881 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 16:53:29.854394 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.843965    2881 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 16:53:29.854960 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.843972    2881 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 16:53:29.854960 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.843978    2881 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 16:53:29.854960 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.843982    2881 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 16:53:29.854960 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.843987    2881 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 16:53:29.854960 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.843992    2881 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 16:53:29.854960 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.843996    2881 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 16:53:29.854960 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844000    2881 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 16:53:29.854960 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844004    2881 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 16:53:29.854960 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844008    2881 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 16:53:29.854960 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844013    2881 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 16:53:29.854960 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844017    2881 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 16:53:29.854960 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844022    2881 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 16:53:29.854960 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844026    2881 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 16:53:29.854960 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844030    2881 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 16:53:29.854960 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844035    2881 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 16:53:29.854960 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844040    2881 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 16:53:29.854960 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844044    2881 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 16:53:29.854960 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844048    2881 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 16:53:29.854960 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844053    2881 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 16:53:29.855531 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844058    2881 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 16:53:29.855531 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844064    2881 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 16:53:29.855531 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844068    2881 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 16:53:29.855531 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844073    2881 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 16:53:29.855531 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844077    2881 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 16:53:29.855531 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844081    2881 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 16:53:29.855531 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844085    2881 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 16:53:29.855531 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844090    2881 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 16:53:29.855531 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844094    2881 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 16:53:29.855531 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844099    2881 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 16:53:29.855531 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844103    2881 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 16:53:29.855531 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844108    2881 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 16:53:29.855531 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844112    2881 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 16:53:29.855531 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844116    2881 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 16:53:29.855531 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844122    2881 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 16:53:29.855531 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844128    2881 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 16:53:29.855958 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.844137    2881 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 16:53:29.855958 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844354    2881 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 16:53:29.855958 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844364    2881 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 16:53:29.855958 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844369    2881 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 16:53:29.855958 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844374    2881 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 16:53:29.855958 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844379    2881 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 16:53:29.855958 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844383    2881 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 16:53:29.855958 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844387    2881 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 16:53:29.855958 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844392    2881 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 16:53:29.855958 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844397    2881 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 16:53:29.855958 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844402    2881 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 16:53:29.855958 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844406    2881 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 16:53:29.855958 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844411    2881 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 16:53:29.855958 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844416    2881 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 16:53:29.855958 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844420    2881 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 16:53:29.856364 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844424    2881 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 16:53:29.856364 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844429    2881 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 16:53:29.856364 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844434    2881 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 16:53:29.856364 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844438    2881 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 16:53:29.856364 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844443    2881 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 16:53:29.856364 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844447    2881 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 16:53:29.856364 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844452    2881 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 16:53:29.856364 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844456    2881 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 16:53:29.856364 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844460    2881 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 16:53:29.856364 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844465    2881 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 16:53:29.856364 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844469    2881 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 16:53:29.856364 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844473    2881 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 16:53:29.856364 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844477    2881 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 16:53:29.856364 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844481    2881 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 16:53:29.856364 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844485    2881 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 16:53:29.856364 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844489    2881 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 16:53:29.856364 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844493    2881 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 16:53:29.856364 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844497    2881 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 16:53:29.856364 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844500    2881 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 16:53:29.856364 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844504    2881 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 16:53:29.856878 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844508    2881 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 16:53:29.856878 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844513    2881 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 16:53:29.856878 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844519    2881 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 16:53:29.856878 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844523    2881 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 16:53:29.856878 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844527    2881 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 16:53:29.856878 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844531    2881 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 16:53:29.856878 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844536    2881 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 16:53:29.856878 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844541    2881 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 16:53:29.856878 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844545    2881 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 16:53:29.856878 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844549    2881 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 16:53:29.856878 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844555    2881 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 16:53:29.856878 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844559    2881 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 16:53:29.856878 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844563    2881 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 16:53:29.856878 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844567    2881 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 16:53:29.856878 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844572    2881 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 16:53:29.856878 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844577    2881 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 16:53:29.856878 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844582    2881 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 16:53:29.856878 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844587    2881 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 16:53:29.856878 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844594    2881 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 16:53:29.857419 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844600    2881 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 16:53:29.857419 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844604    2881 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 16:53:29.857419 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844609    2881 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 16:53:29.857419 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844613    2881 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 16:53:29.857419 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844618    2881 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 16:53:29.857419 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844622    2881 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 16:53:29.857419 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844626    2881 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 16:53:29.857419 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844630    2881 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 16:53:29.857419 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844636    2881 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 16:53:29.857419 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844640    2881 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 16:53:29.857419 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844644    2881 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 16:53:29.857419 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844649    2881 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 16:53:29.857419 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844654    2881 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 16:53:29.857419 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844658    2881 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 16:53:29.857419 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844662    2881 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 16:53:29.857419 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844666    2881 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 16:53:29.857419 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844671    2881 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 16:53:29.857419 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844675    2881 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 16:53:29.857419 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844679    2881 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 16:53:29.857891 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844684    2881 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 16:53:29.857891 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844688    2881 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 16:53:29.857891 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844694    2881 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 16:53:29.857891 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844700    2881 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 16:53:29.857891 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844706    2881 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 16:53:29.857891 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844712    2881 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 16:53:29.857891 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844717    2881 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 16:53:29.857891 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844722    2881 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 16:53:29.857891 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844727    2881 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 16:53:29.857891 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844731    2881 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 16:53:29.857891 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844736    2881 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 16:53:29.857891 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844741    2881 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 16:53:29.857891 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844746    2881 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 16:53:29.857891 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844751    2881 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 16:53:29.857891 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844755    2881 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 16:53:29.857891 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844759    2881 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 16:53:29.857891 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844763    2881 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 16:53:29.857891 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844768    2881 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 16:53:29.857891 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844772    2881 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 16:53:29.858387 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844777    2881 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 16:53:29.858387 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844781    2881 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 16:53:29.858387 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844785    2881 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 16:53:29.858387 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844790    2881 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 16:53:29.858387 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844794    2881 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 16:53:29.858387 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844798    2881 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 16:53:29.858387 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844802    2881 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 16:53:29.858387 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:53:29.844807    2881 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 16:53:29.858387 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.844816    2881 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 16:53:29.858387 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.845013    2881 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 18 16:53:29.858655 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:29.845798    2881 bootstrap.go:241] "Unhandled Error" err="unable to read existing bootstrap client config from /var/lib/kubelet/kubeconfig: invalid configuration: [unable to read client-cert /var/lib/kubelet/pki/kubelet-client-current.pem for default-auth due to open /var/lib/kubelet/pki/kubelet-client-current.pem: no such file or directory, unable to read client-key /var/lib/kubelet/pki/kubelet-client-current.pem for default-auth due to open /var/lib/kubelet/pki/kubelet-client-current.pem: no such file or directory]" logger="UnhandledError"
Sep 18 16:53:29.858655 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.847296    2881 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 18 16:53:29.858655 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.847522    2881 server.go:1019] "Starting client certificate rotation"
Sep 18 16:53:29.858655 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.848546    2881 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 16:53:29.858655 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.848602    2881 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 16:53:29.858655 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.848987    2881 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 16:53:29.858655 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.851137    2881 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 16:53:29.858655 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.852827    2881 log.go:25] "Validated CRI v1 runtime API"
Sep 18 16:53:29.858655 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.854024    2881 log.go:25] "Validated CRI v1 image API"
Sep 18 16:53:29.858655 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.854846    2881 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 18 16:53:29.858655 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.855612    2881 fs.go:135] Filesystem UUIDs: map[18cba501-6d4f-4ae6-a804-e7bd5d0b9af1:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 9fa1ed9b-90e2-4524-8c3f-10f506817803:/dev/nvme0n1p3]
Sep 18 16:53:29.858951 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.855631    2881 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cf961e79c5b92a2e43a70c6a29822bf8a5a2069147989bc7b9bb0b53e33609ad/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cf961e79c5b92a2e43a70c6a29822bf8a5a2069147989bc7b9bb0b53e33609ad/userdata/shm major:0 minor:45 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dbf63fb8dec529c8aaf37fe2ae42085a132de70d014ee88ca7d9f1a65f3d237c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dbf63fb8dec529c8aaf37fe2ae42085a132de70d014ee88ca7d9f1a65f3d237c/userdata/shm major:0 minor:46 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/ebba335b8b3bfba130c6ee1195e0321b3267b5564252796a15641672e828a7f0/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/070a36ce629dcac2c991de060ae001a967d5c6602a99cb938403f8b6755a1dc3/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-55:{mountpoint:/var/lib/containers/storage/overlay/ed215eac7b694e1595a342ce2af5cf1e54629de59dede7dcf28b70f48c9beb7d/merged major:0 minor:55 fsType:overlay blockSize:0}]
Sep 18 16:53:29.860496 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.860318    2881 manager.go:216] Machine: {Timestamp:2026-09-18 16:53:29.859697157 +0000 UTC m=+0.059552441 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100663 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e9ce9752f95ac695d61d46ab181ac SystemUUID:ec2e9ce9-752f-95ac-695d-61d46ab181ac BootID:5e4568ea-458b-411b-95aa-a763cc7314be Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cf961e79c5b92a2e43a70c6a29822bf8a5a2069147989bc7b9bb0b53e33609ad/userdata/shm DeviceMajor:0 DeviceMinor:45 Capacity:67108864 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dbf63fb8dec529c8aaf37fe2ae42085a132de70d014ee88ca7d9f1a65f3d237c/userdata/shm DeviceMajor:0 DeviceMinor:46 Capacity:67108864 Type:vfs Inodes:4048398 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-55 DeviceMajor:0 DeviceMinor:55 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ce:e9:e5:d3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ce:e9:e5:d3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:de:a0:b8:7c:b0:40 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 18 16:53:29.860496 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.860491    2881 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 18 16:53:29.860624 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.860591    2881 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 18 16:53:29.860935 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.860912    2881 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 18 16:53:29.861103 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.860937    2881 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-5.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 18 16:53:29.861148 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.861114    2881 topology_manager.go:138] "Creating topology manager with none policy"
Sep 18 16:53:29.861148 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.861122    2881 container_manager_linux.go:309] "Creating device plugin manager"
Sep 18 16:53:29.861148 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.861131    2881 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 16:53:29.861148 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.861147    2881 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 16:53:29.861284 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.861275    2881 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 18 16:53:29.861511 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.861502    2881 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 16:53:29.861627 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.861619    2881 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 18 16:53:29.861659 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.861649    2881 kubelet.go:486] "Attempting to sync node with API server"
Sep 18 16:53:29.861690 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.861660    2881 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 18 16:53:29.861690 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.861674    2881 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 18 16:53:29.861690 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.861686    2881 kubelet.go:398] "Adding apiserver pod source"
Sep 18 16:53:29.861795 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.861696    2881 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 18 16:53:29.862128 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.862109    2881 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Sep 18 16:53:29.862128 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.862123    2881 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 16:53:29.862309 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.862187    2881 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Sep 18 16:53:29.862309 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.862193    2881 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 16:53:29.862820 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.862799    2881 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 18 16:53:29.864714 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.864499    2881 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 18 16:53:29.864714 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.864548    2881 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 18 16:53:29.864714 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.864678    2881 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 18 16:53:29.864714 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.864694    2881 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 18 16:53:29.864714 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.864703    2881 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 18 16:53:29.864714 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.864712    2881 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 18 16:53:29.864714 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.864724    2881 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 18 16:53:29.865058 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.864739    2881 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 18 16:53:29.865058 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.864748    2881 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 18 16:53:29.865058 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.864757    2881 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 18 16:53:29.865058 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.864768    2881 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 18 16:53:29.865058 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.864776    2881 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 18 16:53:29.865058 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.864789    2881 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 18 16:53:29.865058 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.864798    2881 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 18 16:53:29.865381 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.865125    2881 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 18 16:53:29.865381 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.865153    2881 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 18 16:53:29.868942 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:29.868912    2881 certificate_manager.go:596] "Failed while requesting a signed certificate from the control plane" err="cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User \"system:serviceaccount:openshift-machine-config-operator:node-bootstrapper\" cannot create resource \"certificatesigningrequests\" in API group \"certificates.k8s.io\" at the cluster scope" logger="kubernetes.io/kube-apiserver-client-kubelet.UnhandledError"
Sep 18 16:53:29.869640 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.869619    2881 server.go:1268] "Started kubelet"
Sep 18 16:53:29.870643 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.869701    2881 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 18 16:53:29.870643 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.869791    2881 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 18 16:53:29.870643 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.869864    2881 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 18 16:53:29.870643 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.870198    2881 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 18 16:53:29.870643 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:29.870396    2881 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 18 16:53:29.870643 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:29.870469    2881 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 18 16:53:29.870643 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.870495    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:53:29.871032 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.870984    2881 server.go:310] "Adding debug handlers to kubelet server"
Sep 18 16:53:29.870767 ip-10-0-143-5 systemd[1]: Started Kubernetes Kubelet.
Sep 18 16:53:29.871307 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.871144    2881 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 18 16:53:29.871307 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.871279    2881 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 18 16:53:29.871623 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.871425    2881 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 18 16:53:29.871623 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.871438    2881 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 18 16:53:29.871623 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.871465    2881 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 18 16:53:29.871623 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:29.871575    2881 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-5.ec2.internal\" not found"
Sep 18 16:53:29.871623 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.871587    2881 reconstruct.go:97] "Volume reconstruction finished"
Sep 18 16:53:29.871623 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.871597    2881 reconciler.go:29] "Reconciler: start to sync state"
Sep 18 16:53:29.872636 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.872620    2881 factory.go:153] Registering CRI-O factory
Sep 18 16:53:29.872725 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.872640    2881 factory.go:223] Registration of the crio container factory successfully
Sep 18 16:53:29.872725 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.872693    2881 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 18 16:53:29.872725 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.872702    2881 factory.go:55] Registering systemd factory
Sep 18 16:53:29.872725 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.872710    2881 factory.go:223] Registration of the systemd container factory successfully
Sep 18 16:53:29.872725 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.872726    2881 factory.go:103] Registering Raw factory
Sep 18 16:53:29.872959 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.872742    2881 manager.go:1196] Started watching for new ooms in manager
Sep 18 16:53:29.873161 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:29.873032    2881 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-5.ec2.internal.18d67883e8e2de28  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-5.ec2.internal,UID:ip-10-0-143-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:53:29.869581864 +0000 UTC m=+0.069437164,LastTimestamp:2026-09-18 16:53:29.869581864 +0000 UTC m=+0.069437164,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:53:29.873375 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.873195    2881 manager.go:349] Starting recovery of all containers
Sep 18 16:53:29.873717 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.873695    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_a3127caef50db0be093e96af172c433b/setup/5.log"
Sep 18 16:53:29.873784 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:29.873750    2881 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 18 16:53:29.873955 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.873937    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_a3127caef50db0be093e96af172c433b/setup/5.log"
Sep 18 16:53:29.874769 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:29.874743    2881 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 18 16:53:29.874863 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:29.874767    2881 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 18 16:53:29.885974 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.885917    2881 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 18 16:53:29.887135 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.887107    2881 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 18 16:53:29.887135 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.887136    2881 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 18 16:53:29.887357 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.887160    2881 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 18 16:53:29.887357 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:29.887205    2881 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Sep 18 16:53:29.888400 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.888377    2881 manager.go:354] Recovery completed
Sep 18 16:53:29.890101 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:29.890068    2881 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass"
Sep 18 16:53:29.892374 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.892353    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_a3127caef50db0be093e96af172c433b/setup/5.log"
Sep 18 16:53:29.892468 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.892386    2881 generic.go:356] "Generic (PLEG): container finished" podID="a3127caef50db0be093e96af172c433b" containerID="309c63642750e1d721fad53f8a33762f5ee275478f173570d4f4f5da1c946525" exitCode=1
Sep 18 16:53:29.894291 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.894277    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:53:29.895975 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.895959    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:53:29.896058 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.895992    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:53:29.896058 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.896009    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:53:29.896674 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.896659    2881 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 18 16:53:29.896674 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.896671    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:53:29.896784 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.896672    2881 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 18 16:53:29.896784 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.896742    2881 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 16:53:29.896939 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.896925    2881 state_mem.go:88] "Updated default CPUSet" cpuSet=""
Sep 18 16:53:29.896992 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.896941    2881 state_mem.go:96] "Updated CPUSet assignments" assignments={}
Sep 18 16:53:29.896992 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.896964    2881 state_checkpoint.go:136] "State checkpoint: restored state from checkpoint"
Sep 18 16:53:29.896992 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.896972    2881 state_checkpoint.go:137] "State checkpoint: defaultCPUSet" defaultCpuSet=""
Sep 18 16:53:29.896992 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.896981    2881 policy_none.go:49] "None policy: Start"
Sep 18 16:53:29.897171 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.896994    2881 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 18 16:53:29.897171 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.897009    2881 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 18 16:53:29.897171 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.897144    2881 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Sep 18 16:53:29.897171 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.897154    2881 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Sep 18 16:53:29.897171 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.897163    2881 policy_none.go:47] "Start"
Sep 18 16:53:29.898096 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.898079    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:53:29.898174 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.898104    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:53:29.898174 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.898114    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:53:29.901322 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.901305    2881 manager.go:337] "Starting Device Plugin manager"
Sep 18 16:53:29.901399 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:29.901347    2881 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 18 16:53:29.901399 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.901360    2881 server.go:87] "Starting device plugin registration server"
Sep 18 16:53:29.901681 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.901666    2881 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 18 16:53:29.901752 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.901702    2881 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 18 16:53:29.901848 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.901829    2881 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 18 16:53:29.902374 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.901953    2881 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 18 16:53:29.902374 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.901966    2881 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 18 16:53:29.902811 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:29.902717    2881 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 18 16:53:29.902811 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:29.902794    2881 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-5.ec2.internal\" not found"
Sep 18 16:53:29.905966 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:29.905890    2881 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-5.ec2.internal.18d67883ea759a2a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-5.ec2.internal,UID:ip-10-0-143-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-5.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:53:29.895975466 +0000 UTC m=+0.095830748,LastTimestamp:2026-09-18 16:53:29.895975466 +0000 UTC m=+0.095830748,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:53:29.910603 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:29.910520    2881 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-5.ec2.internal.18d67883ea75f6aa  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-5.ec2.internal,UID:ip-10-0-143-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-5.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:53:29.895999146 +0000 UTC m=+0.095854424,LastTimestamp:2026-09-18 16:53:29.895999146 +0000 UTC m=+0.095854424,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:53:29.913993 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:29.913906    2881 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-5.ec2.internal.18d67883ea7631c6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-5.ec2.internal,UID:ip-10-0-143-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-143-5.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:53:29.896014278 +0000 UTC m=+0.095869556,LastTimestamp:2026-09-18 16:53:29.896014278 +0000 UTC m=+0.095869556,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:53:29.918961 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:29.918892    2881 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-5.ec2.internal.18d67883ea759a2a\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-5.ec2.internal.18d67883ea759a2a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-5.ec2.internal,UID:ip-10-0-143-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-5.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:53:29.895975466 +0000 UTC m=+0.095830748,LastTimestamp:2026-09-18 16:53:29.898094195 +0000 UTC m=+0.097949472,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:53:29.923067 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:29.922996    2881 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-5.ec2.internal.18d67883ea75f6aa\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-5.ec2.internal.18d67883ea75f6aa  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-5.ec2.internal,UID:ip-10-0-143-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-5.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:53:29.895999146 +0000 UTC m=+0.095854424,LastTimestamp:2026-09-18 16:53:29.898109122 +0000 UTC m=+0.097964399,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:53:29.928562 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:29.928496    2881 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-5.ec2.internal.18d67883ea7631c6\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-5.ec2.internal.18d67883ea7631c6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-5.ec2.internal,UID:ip-10-0-143-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-143-5.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:53:29.896014278 +0000 UTC m=+0.095869556,LastTimestamp:2026-09-18 16:53:29.898118166 +0000 UTC m=+0.097973444,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:53:29.931775 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:29.931695    2881 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-5.ec2.internal.18d67883eaf65947  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-5.ec2.internal,UID:ip-10-0-143-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:53:29.904412999 +0000 UTC m=+0.104268268,LastTimestamp:2026-09-18 16:53:29.904412999 +0000 UTC m=+0.104268268,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:53:29.988349 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.988295    2881 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-5.ec2.internal"]
Sep 18 16:53:29.988482 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.988447    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:53:29.989559 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.989537    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:53:29.989669 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.989576    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:53:29.989669 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.989587    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:53:29.989752 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.989741    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:53:29.989978 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.989959    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:53:29.990619 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.990600    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:53:29.990730 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.990635    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:53:29.990730 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.990646    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:53:29.990730 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.990646    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:53:29.990730 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.990668    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:53:29.990730 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.990680    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:53:29.990877 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:29.990796    2881 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-5.ec2.internal\" not found" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:53:29.990877 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.990796    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" event={"ID":"a3127caef50db0be093e96af172c433b","Type":"ContainerStarted","Data":"cf961e79c5b92a2e43a70c6a29822bf8a5a2069147989bc7b9bb0b53e33609ad"}
Sep 18 16:53:29.990978 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.990900    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:53:29.991605 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.991590    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:53:29.991667 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.991619    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:53:29.991667 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:29.991633    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:53:29.991795 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:29.991782    2881 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-5.ec2.internal\" not found" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:53:29.992080 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:29.992003    2881 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-5.ec2.internal.18d67883ea759a2a\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-5.ec2.internal.18d67883ea759a2a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-5.ec2.internal,UID:ip-10-0-143-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-5.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:53:29.895975466 +0000 UTC m=+0.095830748,LastTimestamp:2026-09-18 16:53:29.989555961 +0000 UTC m=+0.189411238,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:53:30.000050 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:29.999964    2881 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-5.ec2.internal.18d67883ea75f6aa\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-5.ec2.internal.18d67883ea75f6aa  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-5.ec2.internal,UID:ip-10-0-143-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-5.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:53:29.895999146 +0000 UTC m=+0.095854424,LastTimestamp:2026-09-18 16:53:29.989581438 +0000 UTC m=+0.189436715,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:53:30.002735 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:30.002719    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:53:30.003590 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:30.003571    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:53:30.003672 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:30.003604    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:53:30.003672 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:30.003614    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:53:30.003672 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:30.003637    2881 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:53:30.008402 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:30.008278    2881 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-5.ec2.internal.18d67883ea7631c6\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-5.ec2.internal.18d67883ea7631c6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-5.ec2.internal,UID:ip-10-0-143-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-143-5.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:53:29.896014278 +0000 UTC m=+0.095869556,LastTimestamp:2026-09-18 16:53:29.989590419 +0000 UTC m=+0.189445696,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:53:30.012238 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:30.012097    2881 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-5.ec2.internal.18d67883ea759a2a\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-5.ec2.internal.18d67883ea759a2a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-5.ec2.internal,UID:ip-10-0-143-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-5.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:53:29.895975466 +0000 UTC m=+0.095830748,LastTimestamp:2026-09-18 16:53:29.990618503 +0000 UTC m=+0.190473785,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:53:30.012238 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:30.012203    2881 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:53:30.017935 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:30.017869    2881 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-5.ec2.internal.18d67883ea75f6aa\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-5.ec2.internal.18d67883ea75f6aa  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-5.ec2.internal,UID:ip-10-0-143-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-5.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:53:29.895999146 +0000 UTC m=+0.095854424,LastTimestamp:2026-09-18 16:53:29.990640777 +0000 UTC m=+0.190496054,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:53:30.020795 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:30.020731    2881 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-5.ec2.internal.18d67883ea7631c6\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-5.ec2.internal.18d67883ea7631c6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-5.ec2.internal,UID:ip-10-0-143-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-143-5.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:53:29.896014278 +0000 UTC m=+0.095869556,LastTimestamp:2026-09-18 16:53:29.990650592 +0000 UTC m=+0.190505868,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:53:30.027760 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:30.027685    2881 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-5.ec2.internal.18d67883ea759a2a\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-5.ec2.internal.18d67883ea759a2a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-5.ec2.internal,UID:ip-10-0-143-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-5.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:53:29.895975466 +0000 UTC m=+0.095830748,LastTimestamp:2026-09-18 16:53:29.990659429 +0000 UTC m=+0.190514714,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:53:30.036899 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:30.036822    2881 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-5.ec2.internal.18d67883ea75f6aa\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-5.ec2.internal.18d67883ea75f6aa  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-5.ec2.internal,UID:ip-10-0-143-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-5.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:53:29.895999146 +0000 UTC m=+0.095854424,LastTimestamp:2026-09-18 16:53:29.990673104 +0000 UTC m=+0.190528381,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:53:30.046379 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:30.046297    2881 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-5.ec2.internal.18d67883ea7631c6\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-5.ec2.internal.18d67883ea7631c6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-5.ec2.internal,UID:ip-10-0-143-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-143-5.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:53:29.896014278 +0000 UTC m=+0.095869556,LastTimestamp:2026-09-18 16:53:29.990684286 +0000 UTC m=+0.190539563,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:53:30.053783 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:30.053704    2881 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-5.ec2.internal.18d67883ea759a2a\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-5.ec2.internal.18d67883ea759a2a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-5.ec2.internal,UID:ip-10-0-143-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-5.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:53:29.895975466 +0000 UTC m=+0.095830748,LastTimestamp:2026-09-18 16:53:29.991604134 +0000 UTC m=+0.191459411,Count:6,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:53:30.061965 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:30.061889    2881 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-5.ec2.internal.18d67883ea75f6aa\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-5.ec2.internal.18d67883ea75f6aa  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-5.ec2.internal,UID:ip-10-0-143-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-5.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:53:29.895999146 +0000 UTC m=+0.095854424,LastTimestamp:2026-09-18 16:53:29.991625979 +0000 UTC m=+0.191481259,Count:6,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:53:30.073116 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:30.073034    2881 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-5.ec2.internal.18d67883ea7631c6\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-5.ec2.internal.18d67883ea7631c6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-5.ec2.internal,UID:ip-10-0-143-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-143-5.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:53:29.896014278 +0000 UTC m=+0.095869556,LastTimestamp:2026-09-18 16:53:29.991639285 +0000 UTC m=+0.191494564,Count:6,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:53:30.082972 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:30.082945    2881 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="400ms"
Sep 18 16:53:30.083093 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:30.082932    2881 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-5.ec2.internal.18d67883ea759a2a\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-5.ec2.internal.18d67883ea759a2a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-5.ec2.internal,UID:ip-10-0-143-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-5.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:53:29.895975466 +0000 UTC m=+0.095830748,LastTimestamp:2026-09-18 16:53:30.003588197 +0000 UTC m=+0.203443476,Count:7,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:53:30.085928 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:30.085853    2881 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-5.ec2.internal.18d67883ea75f6aa\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-5.ec2.internal.18d67883ea75f6aa  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-5.ec2.internal,UID:ip-10-0-143-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-5.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:53:29.895999146 +0000 UTC m=+0.095854424,LastTimestamp:2026-09-18 16:53:30.003608782 +0000 UTC m=+0.203464060,Count:7,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:53:30.092098 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:30.092022    2881 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-5.ec2.internal.18d67883ea7631c6\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-5.ec2.internal.18d67883ea7631c6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-5.ec2.internal,UID:ip-10-0-143-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-143-5.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:53:29.896014278 +0000 UTC m=+0.095869556,LastTimestamp:2026-09-18 16:53:30.003617938 +0000 UTC m=+0.203473214,Count:7,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:53:30.172859 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:30.172823    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/640796da303be256712f04b36adf26a6-config\") pod \"kube-apiserver-proxy-ip-10-0-143-5.ec2.internal\" (UID: \"640796da303be256712f04b36adf26a6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-5.ec2.internal"
Sep 18 16:53:30.172997 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:30.172862    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a3127caef50db0be093e96af172c433b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal\" (UID: \"a3127caef50db0be093e96af172c433b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal"
Sep 18 16:53:30.172997 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:30.172883    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a3127caef50db0be093e96af172c433b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal\" (UID: \"a3127caef50db0be093e96af172c433b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal"
Sep 18 16:53:30.212966 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:30.212923    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:53:30.213992 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:30.213973    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:53:30.214079 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:30.214007    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:53:30.214079 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:30.214017    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:53:30.214079 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:30.214047    2881 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:53:30.216552 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:30.216472    2881 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-5.ec2.internal.18d67883ea759a2a\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-5.ec2.internal.18d67883ea759a2a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-5.ec2.internal,UID:ip-10-0-143-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-5.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:53:29.895975466 +0000 UTC m=+0.095830748,LastTimestamp:2026-09-18 16:53:30.213989872 +0000 UTC m=+0.413845149,Count:8,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:53:30.224295 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:30.224263    2881 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:53:30.224379 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:30.224293    2881 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-5.ec2.internal.18d67883ea75f6aa\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-5.ec2.internal.18d67883ea75f6aa  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-5.ec2.internal,UID:ip-10-0-143-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-5.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:53:29.895999146 +0000 UTC m=+0.095854424,LastTimestamp:2026-09-18 16:53:30.214011654 +0000 UTC m=+0.413866930,Count:8,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:53:30.273767 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:30.273691    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a3127caef50db0be093e96af172c433b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal\" (UID: \"a3127caef50db0be093e96af172c433b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal"
Sep 18 16:53:30.273767 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:30.273730    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a3127caef50db0be093e96af172c433b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal\" (UID: \"a3127caef50db0be093e96af172c433b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal"
Sep 18 16:53:30.273767 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:30.273752    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/640796da303be256712f04b36adf26a6-config\") pod \"kube-apiserver-proxy-ip-10-0-143-5.ec2.internal\" (UID: \"640796da303be256712f04b36adf26a6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-5.ec2.internal"
Sep 18 16:53:30.273956 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:30.273797    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/640796da303be256712f04b36adf26a6-config\") pod \"kube-apiserver-proxy-ip-10-0-143-5.ec2.internal\" (UID: \"640796da303be256712f04b36adf26a6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-5.ec2.internal"
Sep 18 16:53:30.273956 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:30.273809    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a3127caef50db0be093e96af172c433b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal\" (UID: \"a3127caef50db0be093e96af172c433b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal"
Sep 18 16:53:30.273956 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:30.273809    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a3127caef50db0be093e96af172c433b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal\" (UID: \"a3127caef50db0be093e96af172c433b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal"
Sep 18 16:53:30.304252 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:30.304121    2881 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d67884021cefef  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal,UID:a3127caef50db0be093e96af172c433b,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Pulled,Message:Container image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:beacea521116b2422086941236135068766b54396a322e81fd4104d55c0282df\" already present on machine,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:53:30.292817903 +0000 UTC m=+0.492673167,LastTimestamp:2026-09-18 16:53:30.292817903 +0000 UTC m=+0.492673167,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:53:30.378574 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:30.378494    2881 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d678840697120b  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal,UID:a3127caef50db0be093e96af172c433b,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Created,Message:Created container: setup,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:53:30.367930891 +0000 UTC m=+0.567786155,LastTimestamp:2026-09-18 16:53:30.367930891 +0000 UTC m=+0.567786155,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:53:30.386578 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:30.386498    2881 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d678840720bdf3  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal,UID:a3127caef50db0be093e96af172c433b,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Started,Message:Started container setup,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:53:30.376953331 +0000 UTC m=+0.576808595,LastTimestamp:2026-09-18 16:53:30.376953331 +0000 UTC m=+0.576808595,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:53:30.494281 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:30.494245    2881 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="800ms"
Sep 18 16:53:30.624898 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:30.624809    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:53:30.626000 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:30.625979    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:53:30.626103 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:30.626010    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:53:30.626103 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:30.626020    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:53:30.626103 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:30.626055    2881 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:53:30.641398 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:30.641364    2881 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:53:30.716710 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:30.716669    2881 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 18 16:53:30.724602 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:30.724571    2881 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass"
Sep 18 16:53:30.879807 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:30.879709    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:53:30.895794 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:30.895768    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_a3127caef50db0be093e96af172c433b/setup/5.log"
Sep 18 16:53:30.895949 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:30.895847    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" event={"ID":"a3127caef50db0be093e96af172c433b","Type":"ContainerStarted","Data":"2e4f5d43f7943dbbc64187d1a752a30556a1602742238256d12206b98fbe52e1"}
Sep 18 16:53:30.895949 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:30.895895    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:53:30.896737 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:30.896721    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:53:30.896798 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:30.896759    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:53:30.896798 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:30.896769    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:53:30.896982 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:30.896970    2881 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-5.ec2.internal\" not found" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:53:30.925404 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:30.925374    2881 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 18 16:53:31.017874 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:31.017824    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:53:31.019882 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:31.019856    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:53:31.020004 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:31.019895    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:53:31.020004 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:31.019909    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:53:31.303713 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:31.303670    2881 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="1.6s"
Sep 18 16:53:31.429944 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:31.429906    2881 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 18 16:53:31.442019 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:31.441980    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:53:31.443038 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:31.443019    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:53:31.443177 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:31.443053    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:53:31.443177 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:31.443067    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:53:31.443177 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:31.443107    2881 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:53:31.458754 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:31.458712    2881 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:53:31.881455 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:31.881424    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:53:31.897832 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:31.897804    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:53:31.898680 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:31.898663    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:53:31.898743 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:31.898694    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:53:31.898743 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:31.898705    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:53:31.898934 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:31.898921    2881 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-5.ec2.internal\" not found" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:53:32.010524 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:32.010492    2881 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 16:53:32.012970 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:32.012944    2881 certificate_manager.go:596] "Failed while requesting a signed certificate from the control plane" err="cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User \"system:serviceaccount:openshift-machine-config-operator:node-bootstrapper\" cannot create resource \"certificatesigningrequests\" in API group \"certificates.k8s.io\" at the cluster scope" logger="kubernetes.io/kube-apiserver-client-kubelet.UnhandledError"
Sep 18 16:53:32.558624 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:32.558588    2881 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 18 16:53:32.690828 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:32.690794    2881 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass"
Sep 18 16:53:32.880960 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:32.880884    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:53:32.912816 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:32.912783    2881 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="3.2s"
Sep 18 16:53:33.059337 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:33.059293    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:53:33.060389 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:33.060363    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:53:33.060389 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:33.060396    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:53:33.060554 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:33.060406    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:53:33.060554 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:33.060439    2881 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:53:33.077924 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:33.077890    2881 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:53:33.196206 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:33.196126    2881 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 18 16:53:33.199193 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:33.199175    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:53:33.200214 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:33.200199    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:53:33.200291 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:33.200251    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:53:33.200291 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:33.200263    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:53:33.527680 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:33.527647    2881 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 18 16:53:33.882462 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:33.882364    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:53:34.880407 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:34.880376    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:53:35.880659 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:35.880626    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:53:36.123469 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:36.123435    2881 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="6.4s"
Sep 18 16:53:36.278035 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:36.278001    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:53:36.278925 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:36.278906    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:53:36.279028 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:36.278938    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:53:36.279028 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:36.278951    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:53:36.279028 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:36.278980    2881 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:53:36.283410 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:36.283374    2881 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 18 16:53:36.289232 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:36.289202    2881 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:53:36.341476 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:36.341432    2881 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 16:53:36.344018 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:36.343994    2881 certificate_manager.go:596] "Failed while requesting a signed certificate from the control plane" err="cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User \"system:serviceaccount:openshift-machine-config-operator:node-bootstrapper\" cannot create resource \"certificatesigningrequests\" in API group \"certificates.k8s.io\" at the cluster scope" logger="kubernetes.io/kube-apiserver-client-kubelet.UnhandledError"
Sep 18 16:53:36.633117 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:36.633037    2881 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass"
Sep 18 16:53:36.880968 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:36.880936    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:53:37.801524 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:37.801474    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:53:37.802599 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:37.802578    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:53:37.802708 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:37.802609    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:53:37.802708 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:37.802620    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:53:37.879492 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:37.879455    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:53:38.308561 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:38.308526    2881 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 18 16:53:38.563915 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:38.563825    2881 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 18 16:53:38.882330 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:38.882244    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:53:39.881449 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:39.881414    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:53:39.903296 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:39.903265    2881 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-5.ec2.internal\" not found"
Sep 18 16:53:40.883947 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:40.883909    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:53:41.879739 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:41.879550    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:53:41.914360 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:41.914335    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_a3127caef50db0be093e96af172c433b/setup/6.log"
Sep 18 16:53:41.914769 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:41.914755    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_a3127caef50db0be093e96af172c433b/setup/5.log"
Sep 18 16:53:41.914807 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:41.914789    2881 generic.go:356] "Generic (PLEG): container finished" podID="a3127caef50db0be093e96af172c433b" containerID="2e4f5d43f7943dbbc64187d1a752a30556a1602742238256d12206b98fbe52e1" exitCode=1
Sep 18 16:53:41.914837 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:41.914817    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" event={"ID":"a3127caef50db0be093e96af172c433b","Type":"ContainerDied","Data":"2e4f5d43f7943dbbc64187d1a752a30556a1602742238256d12206b98fbe52e1"}
Sep 18 16:53:41.914932 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:41.914906    2881 scope.go:117] "RemoveContainer" containerID="309c63642750e1d721fad53f8a33762f5ee275478f173570d4f4f5da1c946525"
Sep 18 16:53:41.914980 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:41.914968    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:53:41.917817 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:41.917800    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:53:41.917911 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:41.917829    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:53:41.917911 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:41.917839    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:53:41.918050 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:41.918034    2881 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-5.ec2.internal\" not found" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:53:41.918099 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:41.918088    2881 scope.go:117] "RemoveContainer" containerID="309c63642750e1d721fad53f8a33762f5ee275478f173570d4f4f5da1c946525"
Sep 18 16:53:41.927898 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:41.927851    2881 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator_a3127caef50db0be093e96af172c433b_5 in pod sandbox cf961e79c5b92a2e43a70c6a29822bf8a5a2069147989bc7b9bb0b53e33609ad from index: no such id: '309c63642750e1d721fad53f8a33762f5ee275478f173570d4f4f5da1c946525'" containerID="309c63642750e1d721fad53f8a33762f5ee275478f173570d4f4f5da1c946525"
Sep 18 16:53:41.927982 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:41.927915    2881 kuberuntime_container.go:962] "Unhandled Error" err="failed to remove pod init container \"setup\": rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator_a3127caef50db0be093e96af172c433b_5 in pod sandbox cf961e79c5b92a2e43a70c6a29822bf8a5a2069147989bc7b9bb0b53e33609ad from index: no such id: '309c63642750e1d721fad53f8a33762f5ee275478f173570d4f4f5da1c946525'; Skipping pod \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b)\"" logger="UnhandledError"
Sep 18 16:53:41.928090 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:41.928070    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" podUID="a3127caef50db0be093e96af172c433b"
Sep 18 16:53:41.937040 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:41.936953    2881 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d67886b7a03b55  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal,UID:a3127caef50db0be093e96af172c433b,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:BackOff,Message:Back-off restarting failed container setup in pod kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b),Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:53:41.928033109 +0000 UTC m=+12.127888386,LastTimestamp:2026-09-18 16:53:41.928033109 +0000 UTC m=+12.127888386,Count:1,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:53:42.539838 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:42.539801    2881 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 18 16:53:42.689612 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:42.689568    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:53:42.690744 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:42.690727    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:53:42.690811 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:42.690759    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:53:42.690811 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:42.690785    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:53:42.690870 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:42.690813    2881 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:53:42.709689 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:42.709662    2881 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:53:42.881840 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:42.881771    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:53:42.919739 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:42.919711    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_a3127caef50db0be093e96af172c433b/setup/6.log"
Sep 18 16:53:43.116986 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:43.116953    2881 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass"
Sep 18 16:53:43.885464 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:43.885431    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:53:44.451660 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:44.451611    2881 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 16:53:44.455177 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:44.455149    2881 certificate_manager.go:596] "Failed while requesting a signed certificate from the control plane" err="cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User \"system:serviceaccount:openshift-machine-config-operator:node-bootstrapper\" cannot create resource \"certificatesigningrequests\" in API group \"certificates.k8s.io\" at the cluster scope" logger="kubernetes.io/kube-apiserver-client-kubelet.UnhandledError"
Sep 18 16:53:44.880529 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:44.880496    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:53:45.320548 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:45.320516    2881 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 18 16:53:45.630329 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:45.630244    2881 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 18 16:53:45.879408 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:45.879372    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:53:46.376323 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:46.376284    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:53:46.377435 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:46.377409    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:53:46.377540 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:46.377446    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:53:46.377540 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:46.377457    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:53:46.411920 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:46.411888    2881 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 18 16:53:46.878457 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:46.878428    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:53:47.879169 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:47.879133    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:53:48.881212 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:48.881185    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:53:49.549076 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:49.549041    2881 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 18 16:53:49.710016 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:49.709970    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:53:49.711112 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:49.711095    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:53:49.711253 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:49.711127    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:53:49.711253 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:49.711149    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:53:49.711253 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:49.711176    2881 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:53:49.730075 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:49.730050    2881 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:53:49.879111 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:49.879029    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:53:49.904459 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:49.904429    2881 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-5.ec2.internal\" not found"
Sep 18 16:53:50.880506 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:50.880474    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:53:51.882282 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:51.882251    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:53:52.880331 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:52.880296    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:53:53.880535 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:53.880506    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:53:54.880639 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:54.880606    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:53:54.887994 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:54.887970    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:53:54.890533 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:54.890513    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:53:54.890604 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:54.890552    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:53:54.890604 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:54.890563    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:53:54.890760 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:54.890747    2881 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-5.ec2.internal\" not found" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:53:54.899654 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:54.899542    2881 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d67884021cefef\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d67884021cefef  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal,UID:a3127caef50db0be093e96af172c433b,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Pulled,Message:Container image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:beacea521116b2422086941236135068766b54396a322e81fd4104d55c0282df\" already present on machine,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:53:30.292817903 +0000 UTC m=+0.492673167,LastTimestamp:2026-09-18 16:53:54.891515389 +0000 UTC m=+25.091370658,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:53:54.970573 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:54.970487    2881 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d678840697120b\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d678840697120b  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal,UID:a3127caef50db0be093e96af172c433b,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Created,Message:Created container: setup,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:53:30.367930891 +0000 UTC m=+0.567786155,LastTimestamp:2026-09-18 16:53:54.961435634 +0000 UTC m=+25.161290909,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:53:54.981266 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:54.981158    2881 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d678840720bdf3\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d678840720bdf3  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal,UID:a3127caef50db0be093e96af172c433b,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Started,Message:Started container setup,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:53:30.376953331 +0000 UTC m=+0.576808595,LastTimestamp:2026-09-18 16:53:54.97010208 +0000 UTC m=+25.169957368,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:53:55.882290 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:55.882259    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:53:55.940634 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:55.940605    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_a3127caef50db0be093e96af172c433b/setup/6.log"
Sep 18 16:53:55.940778 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:55.940658    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" event={"ID":"a3127caef50db0be093e96af172c433b","Type":"ContainerStarted","Data":"e77fec755dd2b0687de3bc8b43eacd97b75f99e6e3471aca954719e40a1ea4a7"}
Sep 18 16:53:55.940813 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:55.940779    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:53:55.943719 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:55.941719    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:53:55.943719 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:55.941995    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:53:55.943719 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:55.942026    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:53:55.943719 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:55.942418    2881 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-5.ec2.internal\" not found" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:53:56.557916 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:56.557875    2881 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 18 16:53:56.730687 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:56.730650    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:53:56.731759 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:56.731739    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:53:56.731849 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:56.731783    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:53:56.731849 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:56.731798    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:53:56.731849 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:56.731826    2881 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:53:56.754337 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:56.754306    2881 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:53:56.879099 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:56.879019    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:53:57.882364 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:57.882336    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:53:58.880064 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:58.880035    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:53:59.879366 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:53:59.879333    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:53:59.904952 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:53:59.904918    2881 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-5.ec2.internal\" not found"
Sep 18 16:54:00.780533 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:54:00.780496    2881 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 18 16:54:00.878927 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:00.878892    2881 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 16:54:00.879140 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:00.879123    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:54:00.881165 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:54:00.881142    2881 certificate_manager.go:596] "Failed while requesting a signed certificate from the control plane" err="cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User \"system:serviceaccount:openshift-machine-config-operator:node-bootstrapper\" cannot create resource \"certificatesigningrequests\" in API group \"certificates.k8s.io\" at the cluster scope" logger="kubernetes.io/kube-apiserver-client-kubelet.UnhandledError"
Sep 18 16:54:00.881490 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:54:00.881170    2881 certificate_manager.go:461] "Reached backoff limit, still unable to rotate certs" err="timed out waiting for the condition" logger="kubernetes.io/kube-apiserver-client-kubelet.UnhandledError"
Sep 18 16:54:01.879123 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:01.879091    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:54:02.881279 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:02.881245    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:54:03.012286 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:03.012255    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:54:03.013353 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:03.013334    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:54:03.013456 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:03.013370    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:54:03.013456 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:03.013387    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:54:03.566758 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:54:03.566726    2881 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 18 16:54:03.754809 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:03.754766    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:54:03.755911 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:03.755883    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:54:03.756043 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:03.755925    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:54:03.756043 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:03.755937    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:54:03.756043 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:03.755969    2881 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:54:03.773364 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:54:03.773334    2881 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:54:03.879820 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:03.879729    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:54:04.882875 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:04.882844    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:54:05.879066 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:05.879037    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:54:06.342666 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:54:06.342635    2881 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 18 16:54:06.882390 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:06.882359    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:54:06.958550 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:06.958526    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_a3127caef50db0be093e96af172c433b/setup/7.log"
Sep 18 16:54:06.958928 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:06.958910    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_a3127caef50db0be093e96af172c433b/setup/6.log"
Sep 18 16:54:06.959024 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:06.958950    2881 generic.go:356] "Generic (PLEG): container finished" podID="a3127caef50db0be093e96af172c433b" containerID="e77fec755dd2b0687de3bc8b43eacd97b75f99e6e3471aca954719e40a1ea4a7" exitCode=1
Sep 18 16:54:06.959024 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:06.958992    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" event={"ID":"a3127caef50db0be093e96af172c433b","Type":"ContainerDied","Data":"e77fec755dd2b0687de3bc8b43eacd97b75f99e6e3471aca954719e40a1ea4a7"}
Sep 18 16:54:06.959132 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:06.959092    2881 scope.go:117] "RemoveContainer" containerID="2e4f5d43f7943dbbc64187d1a752a30556a1602742238256d12206b98fbe52e1"
Sep 18 16:54:06.959181 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:06.959159    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:54:06.960399 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:06.960062    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:54:06.960399 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:06.960090    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:54:06.960399 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:06.960099    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:54:06.960399 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:54:06.960343    2881 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-5.ec2.internal\" not found" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:54:06.960399 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:06.960394    2881 scope.go:117] "RemoveContainer" containerID="2e4f5d43f7943dbbc64187d1a752a30556a1602742238256d12206b98fbe52e1"
Sep 18 16:54:06.969072 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:54:06.969041    2881 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator_a3127caef50db0be093e96af172c433b_6 in pod sandbox cf961e79c5b92a2e43a70c6a29822bf8a5a2069147989bc7b9bb0b53e33609ad from index: no such id: '2e4f5d43f7943dbbc64187d1a752a30556a1602742238256d12206b98fbe52e1'" containerID="2e4f5d43f7943dbbc64187d1a752a30556a1602742238256d12206b98fbe52e1"
Sep 18 16:54:06.969161 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:54:06.969091    2881 kuberuntime_container.go:962] "Unhandled Error" err="failed to remove pod init container \"setup\": rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator_a3127caef50db0be093e96af172c433b_6 in pod sandbox cf961e79c5b92a2e43a70c6a29822bf8a5a2069147989bc7b9bb0b53e33609ad from index: no such id: '2e4f5d43f7943dbbc64187d1a752a30556a1602742238256d12206b98fbe52e1'; Skipping pod \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b)\"" logger="UnhandledError"
Sep 18 16:54:06.969276 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:54:06.969258    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" podUID="a3127caef50db0be093e96af172c433b"
Sep 18 16:54:06.977829 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:54:06.977733    2881 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d67886b7a03b55\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d67886b7a03b55  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal,UID:a3127caef50db0be093e96af172c433b,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:BackOff,Message:Back-off restarting failed container setup in pod kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b),Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:53:41.928033109 +0000 UTC m=+12.127888386,LastTimestamp:2026-09-18 16:54:06.969205264 +0000 UTC m=+37.169060527,Count:2,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:54:07.218021 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:54:07.217933    2881 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass"
Sep 18 16:54:07.881033 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:07.880996    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:54:07.963462 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:07.963432    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_a3127caef50db0be093e96af172c433b/setup/7.log"
Sep 18 16:54:08.880060 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:08.880029    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:54:09.881260 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:09.881207    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:54:09.905689 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:54:09.905636    2881 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-5.ec2.internal\" not found"
Sep 18 16:54:10.506448 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:54:10.506409    2881 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 18 16:54:10.576237 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:54:10.576197    2881 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 18 16:54:10.774096 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:10.774063    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:54:10.775186 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:10.775168    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:54:10.775294 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:10.775204    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:54:10.775294 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:10.775214    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:54:10.775294 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:10.775259    2881 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:54:10.793662 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:54:10.793623    2881 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:54:10.879835 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:10.879802    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:54:11.880504 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:11.880464    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:54:12.881306 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:12.881277    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:54:13.880925 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:13.880895    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:54:14.881263 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:14.881211    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:54:15.880936 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:15.880905    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:54:16.880349 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:16.880319    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:54:17.585557 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:54:17.585517    2881 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 18 16:54:17.793931 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:17.793892    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:54:17.795049 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:17.795031    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:54:17.795139 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:17.795065    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:54:17.795139 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:17.795076    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:54:17.795139 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:17.795108    2881 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:54:17.812626 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:54:17.812594    2881 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:54:17.880955 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:17.880881    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:54:18.884582 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:18.884549    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:54:18.887878 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:18.887858    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:54:18.888823 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:18.888803    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:54:18.888993 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:18.888834    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:54:18.888993 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:18.888844    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:54:18.889072 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:54:18.889042    2881 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-5.ec2.internal\" not found" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:54:18.889203 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:54:18.889187    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" podUID="a3127caef50db0be093e96af172c433b"
Sep 18 16:54:18.898527 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:54:18.898440    2881 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d67886b7a03b55\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d67886b7a03b55  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal,UID:a3127caef50db0be093e96af172c433b,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:BackOff,Message:Back-off restarting failed container setup in pod kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b),Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:53:41.928033109 +0000 UTC m=+12.127888386,LastTimestamp:2026-09-18 16:54:18.889158318 +0000 UTC m=+49.089013595,Count:3,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:54:19.879585 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:19.879548    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:54:19.906051 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:54:19.906016    2881 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-5.ec2.internal\" not found"
Sep 18 16:54:20.879996 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:20.879957    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:54:21.880240 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:21.880185    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:54:22.880769 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:22.880739    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:54:23.885966 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:23.885936    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:54:24.594708 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:54:24.594673    2881 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 18 16:54:24.813655 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:24.813613    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:54:24.815530 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:24.815509    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:54:24.815640 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:24.815541    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:54:24.815640 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:24.815552    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:54:24.815640 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:24.815580    2881 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:54:24.833412 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:54:24.833377    2881 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:54:24.881004 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:24.880947    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:54:25.879878 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:25.879848    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:54:26.881615 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:26.881580    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:54:27.879366 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:27.879329    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:54:28.878511 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:28.878474    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:54:29.880063 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:29.880026    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:54:29.887383 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:29.887355    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:54:29.890492 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:29.890465    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:54:29.890604 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:29.890509    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:54:29.890604 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:29.890522    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:54:29.890764 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:54:29.890752    2881 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-5.ec2.internal\" not found" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:54:29.904370 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:54:29.904256    2881 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d67884021cefef\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d67884021cefef  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal,UID:a3127caef50db0be093e96af172c433b,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Pulled,Message:Container image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:beacea521116b2422086941236135068766b54396a322e81fd4104d55c0282df\" already present on machine,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:53:30.292817903 +0000 UTC m=+0.492673167,LastTimestamp:2026-09-18 16:54:29.891582579 +0000 UTC m=+60.091437866,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:54:29.906424 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:54:29.906398    2881 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-5.ec2.internal\" not found"
Sep 18 16:54:29.978653 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:54:29.978562    2881 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d678840697120b\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d678840697120b  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal,UID:a3127caef50db0be093e96af172c433b,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Created,Message:Created container: setup,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:53:30.367930891 +0000 UTC m=+0.567786155,LastTimestamp:2026-09-18 16:54:29.96813856 +0000 UTC m=+60.167993823,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:54:29.987325 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:54:29.987231    2881 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d678840720bdf3\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d678840720bdf3  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal,UID:a3127caef50db0be093e96af172c433b,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Started,Message:Started container setup,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:53:30.376953331 +0000 UTC m=+0.576808595,LastTimestamp:2026-09-18 16:54:29.978434374 +0000 UTC m=+60.178289662,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:54:29.998126 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:29.998101    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_a3127caef50db0be093e96af172c433b/setup/7.log"
Sep 18 16:54:29.998301 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:29.998149    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" event={"ID":"a3127caef50db0be093e96af172c433b","Type":"ContainerStarted","Data":"3a91cbee892450ba29615791afe17dfe13a763d0d639b116527651e7239e1f49"}
Sep 18 16:54:29.998347 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:29.998331    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:54:29.999140 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:29.999122    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:54:29.999258 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:29.999151    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:54:29.999258 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:29.999161    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:54:29.999387 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:54:29.999373    2881 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-5.ec2.internal\" not found" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:54:30.879838 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:30.879801    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:54:31.606713 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:54:31.606680    2881 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 18 16:54:31.834508 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:31.834472    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:54:31.836475 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:31.836443    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:54:31.836475 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:31.836477    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:54:31.836694 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:31.836487    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:54:31.836694 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:31.836517    2881 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:54:31.857641 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:54:31.857561    2881 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:54:31.886716 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:31.886683    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:54:32.880108 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:32.880077    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:54:32.883278 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:32.883258    2881 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 16:54:32.887034 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:54:32.887007    2881 certificate_manager.go:596] "Failed while requesting a signed certificate from the control plane" err="cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User \"system:serviceaccount:openshift-machine-config-operator:node-bootstrapper\" cannot create resource \"certificatesigningrequests\" in API group \"certificates.k8s.io\" at the cluster scope" logger="kubernetes.io/kube-apiserver-client-kubelet.UnhandledError"
Sep 18 16:54:33.883301 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:33.883268    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:54:34.881611 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:34.881579    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:54:35.880062 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:35.880028    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:54:36.880027 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:36.879995    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:54:37.881128 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:37.881097    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:54:38.264112 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:38.264080    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:54:38.266022 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:38.266001    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:54:38.266131 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:38.266034    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:54:38.266131 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:38.266045    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:54:38.616803 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:54:38.616721    2881 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 18 16:54:38.857707 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:38.857675    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:54:38.859649 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:38.859625    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:54:38.859746 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:38.859664    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:54:38.859746 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:38.859674    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:54:38.859746 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:38.859704    2881 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:54:38.879803 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:38.879715    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:54:38.879803 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:54:38.879768    2881 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:54:39.347428 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:54:39.347389    2881 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 18 16:54:39.883334 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:39.883301    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:54:39.906825 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:54:39.906791    2881 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-5.ec2.internal\" not found"
Sep 18 16:54:40.862907 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:54:40.862868    2881 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 18 16:54:40.879350 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:40.879320    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:54:41.018453 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:41.018425    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_a3127caef50db0be093e96af172c433b/setup/8.log"
Sep 18 16:54:41.018836 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:41.018817    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_a3127caef50db0be093e96af172c433b/setup/7.log"
Sep 18 16:54:41.018901 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:41.018858    2881 generic.go:356] "Generic (PLEG): container finished" podID="a3127caef50db0be093e96af172c433b" containerID="3a91cbee892450ba29615791afe17dfe13a763d0d639b116527651e7239e1f49" exitCode=1
Sep 18 16:54:41.018901 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:41.018886    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" event={"ID":"a3127caef50db0be093e96af172c433b","Type":"ContainerDied","Data":"3a91cbee892450ba29615791afe17dfe13a763d0d639b116527651e7239e1f49"}
Sep 18 16:54:41.018965 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:41.018937    2881 scope.go:117] "RemoveContainer" containerID="e77fec755dd2b0687de3bc8b43eacd97b75f99e6e3471aca954719e40a1ea4a7"
Sep 18 16:54:41.019035 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:41.019022    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:54:41.019858 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:41.019836    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:54:41.019984 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:41.019870    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:54:41.019984 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:41.019881    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:54:41.020387 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:54:41.020109    2881 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-5.ec2.internal\" not found" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:54:41.020387 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:41.020172    2881 scope.go:117] "RemoveContainer" containerID="e77fec755dd2b0687de3bc8b43eacd97b75f99e6e3471aca954719e40a1ea4a7"
Sep 18 16:54:41.046052 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:54:41.046012    2881 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator_a3127caef50db0be093e96af172c433b_7 in pod sandbox cf961e79c5b92a2e43a70c6a29822bf8a5a2069147989bc7b9bb0b53e33609ad from index: no such id: 'e77fec755dd2b0687de3bc8b43eacd97b75f99e6e3471aca954719e40a1ea4a7'" containerID="e77fec755dd2b0687de3bc8b43eacd97b75f99e6e3471aca954719e40a1ea4a7"
Sep 18 16:54:41.046184 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:54:41.046077    2881 kuberuntime_container.go:962] "Unhandled Error" err="failed to remove pod init container \"setup\": rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator_a3127caef50db0be093e96af172c433b_7 in pod sandbox cf961e79c5b92a2e43a70c6a29822bf8a5a2069147989bc7b9bb0b53e33609ad from index: no such id: 'e77fec755dd2b0687de3bc8b43eacd97b75f99e6e3471aca954719e40a1ea4a7'; Skipping pod \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b)\"" logger="UnhandledError"
Sep 18 16:54:41.046316 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:54:41.046293    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 40s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" podUID="a3127caef50db0be093e96af172c433b"
Sep 18 16:54:41.055716 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:54:41.055627    2881 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d67886b7a03b55\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d67886b7a03b55  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal,UID:a3127caef50db0be093e96af172c433b,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:BackOff,Message:Back-off restarting failed container setup in pod kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b),Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:53:41.928033109 +0000 UTC m=+12.127888386,LastTimestamp:2026-09-18 16:54:41.04624701 +0000 UTC m=+71.246102295,Count:4,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:54:41.879478 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:41.879441    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:54:42.022162 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:42.022135    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_a3127caef50db0be093e96af172c433b/setup/8.log"
Sep 18 16:54:42.880025 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:42.879996    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:54:43.880034 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:43.880003    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:54:44.878574 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:44.878540    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:54:45.629485 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:54:45.629453    2881 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 18 16:54:45.880213 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:45.880138    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:54:45.881156 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:45.881134    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:54:45.881305 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:45.881173    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:54:45.881305 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:45.881187    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:54:45.881305 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:45.881213    2881 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:54:45.882402 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:45.882376    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:54:45.897987 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:54:45.897944    2881 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:54:46.882297 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:46.882261    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:54:47.881270 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:47.881236    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:54:48.882048 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:48.882015    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:54:49.880874 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:49.880844    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:54:49.907325 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:54:49.907297    2881 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-5.ec2.internal\" not found"
Sep 18 16:54:50.880019 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:50.879986    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:54:50.888282 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:50.888254    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:54:50.898670 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:50.898645    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:54:50.898771 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:50.898680    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:54:50.898771 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:50.898690    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:54:50.898905 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:54:50.898892    2881 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-5.ec2.internal\" not found" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:54:51.882009 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:51.881972    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:54:52.393557 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:54:52.393517    2881 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass"
Sep 18 16:54:52.641349 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:54:52.641310    2881 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 18 16:54:52.881374 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:52.881341    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:54:52.887677 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:52.887660    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:54:52.888689 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:52.888668    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:54:52.888813 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:52.888706    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:54:52.888813 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:52.888721    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:54:52.888944 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:54:52.888931    2881 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-5.ec2.internal\" not found" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:54:52.889096 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:54:52.889081    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 40s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" podUID="a3127caef50db0be093e96af172c433b"
Sep 18 16:54:52.898065 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:52.898039    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:54:52.899052 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:52.899033    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:54:52.899163 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:52.899069    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:54:52.899163 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:52.899086    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:54:52.899163 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:52.899121    2881 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:54:52.905001 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:54:52.904904    2881 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d67886b7a03b55\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d67886b7a03b55  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal,UID:a3127caef50db0be093e96af172c433b,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:BackOff,Message:Back-off restarting failed container setup in pod kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b),Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:53:41.928033109 +0000 UTC m=+12.127888386,LastTimestamp:2026-09-18 16:54:52.889051296 +0000 UTC m=+83.088906574,Count:5,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:54:52.907935 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:54:52.907913    2881 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:54:53.873852 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:53.873819    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:54:54.882194 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:54.882154    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:54:55.880570 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:55.880536    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:54:56.105498 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:54:56.105458    2881 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 18 16:54:56.881175 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:56.881139    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:54:57.881102 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:57.881071    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:54:58.879111 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:58.879073    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:54:59.658135 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:54:59.658093    2881 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 18 16:54:59.880240 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:59.880198    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:54:59.907776 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:54:59.907742    2881 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-5.ec2.internal\" not found"
Sep 18 16:54:59.908835 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:59.908782    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:54:59.909969 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:59.909951    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:54:59.910084 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:59.909984    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:54:59.910084 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:59.909998    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:54:59.910084 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:54:59.910032    2881 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:54:59.928641 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:54:59.928606    2881 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:55:00.880271 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:00.880217    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:55:01.879740 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:01.879704    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:55:02.881591 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:02.881554    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:55:03.879109 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:03.879070    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:55:03.887416 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:03.887388    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:55:03.889071 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:03.889049    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:55:03.889186 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:03.889087    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:55:03.889186 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:03.889101    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:55:03.889400 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:55:03.889385    2881 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-5.ec2.internal\" not found" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:55:03.889586 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:55:03.889564    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 40s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" podUID="a3127caef50db0be093e96af172c433b"
Sep 18 16:55:03.899449 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:55:03.899348    2881 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d67886b7a03b55\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d67886b7a03b55  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal,UID:a3127caef50db0be093e96af172c433b,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:BackOff,Message:Back-off restarting failed container setup in pod kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b),Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:53:41.928033109 +0000 UTC m=+12.127888386,LastTimestamp:2026-09-18 16:55:03.889526258 +0000 UTC m=+94.089381536,Count:6,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:55:04.882652 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:04.882610    2881 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 16:55:04.884728 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:04.884711    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:55:04.884883 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:55:04.884860    2881 certificate_manager.go:596] "Failed while requesting a signed certificate from the control plane" err="cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User \"system:serviceaccount:openshift-machine-config-operator:node-bootstrapper\" cannot create resource \"certificatesigningrequests\" in API group \"certificates.k8s.io\" at the cluster scope" logger="kubernetes.io/kube-apiserver-client-kubelet.UnhandledError"
Sep 18 16:55:05.879602 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:05.879562    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:55:06.667028 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:55:06.666983    2881 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 18 16:55:06.885390 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:06.885355    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:55:06.929002 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:06.928926    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:55:06.930006 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:06.929981    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:55:06.930141 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:06.930018    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:55:06.930141 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:06.930035    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:55:06.930141 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:06.930068    2881 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:55:06.947120 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:55:06.947088    2881 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:55:07.880721 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:07.880690    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:55:08.879767 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:08.879733    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:55:09.882217 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:09.882183    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:55:09.908733 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:55:09.908698    2881 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-5.ec2.internal\" not found"
Sep 18 16:55:10.880997 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:10.880962    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:55:11.880036 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:11.880000    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:55:12.881866 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:12.881828    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:55:13.677605 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:55:13.677570    2881 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 18 16:55:13.880690 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:13.880657    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:55:13.947295 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:13.947194    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:55:13.948308 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:13.948289    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:55:13.948408 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:13.948321    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:55:13.948408 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:13.948332    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:55:13.948408 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:13.948360    2881 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:55:13.966625 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:55:13.966590    2881 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:55:14.880055 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:14.880022    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:55:14.888421 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:14.888392    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:55:14.890432 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:14.890414    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:55:14.890613 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:14.890448    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:55:14.890613 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:14.890461    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:55:14.890756 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:55:14.890740    2881 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-5.ec2.internal\" not found" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:55:14.890925 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:55:14.890903    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 40s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" podUID="a3127caef50db0be093e96af172c433b"
Sep 18 16:55:14.900143 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:55:14.900052    2881 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d67886b7a03b55\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d67886b7a03b55  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal,UID:a3127caef50db0be093e96af172c433b,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:BackOff,Message:Back-off restarting failed container setup in pod kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b),Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:53:41.928033109 +0000 UTC m=+12.127888386,LastTimestamp:2026-09-18 16:55:14.890873501 +0000 UTC m=+105.090728777,Count:7,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:55:15.277572 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:55:15.277534    2881 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 18 16:55:15.881774 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:15.881739    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:55:16.879294 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:16.879261    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:55:17.880796 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:17.880763    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:55:18.881835 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:18.881799    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:55:19.880121 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:19.880083    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:55:19.909592 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:55:19.909543    2881 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-5.ec2.internal\" not found"
Sep 18 16:55:20.688678 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:55:20.688644    2881 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 18 16:55:20.881370 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:20.881335    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:55:20.967321 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:20.967216    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:55:20.968405 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:20.968386    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:55:20.968519 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:20.968424    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:55:20.968519 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:20.968436    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:55:20.968519 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:20.968467    2881 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:55:20.989823 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:55:20.989790    2881 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:55:21.882370 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:21.882337    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:55:22.884364 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:22.884325    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:55:23.882445 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:23.882414    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:55:24.880193 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:24.880161    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:55:25.878681 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:25.878652    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:55:26.881973 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:26.881940    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:55:26.888238 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:26.888198    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:55:26.889248 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:26.889214    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:55:26.889349 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:26.889266    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:55:26.889349 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:26.889281    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:55:26.889594 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:55:26.889579    2881 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-5.ec2.internal\" not found" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:55:26.900754 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:55:26.900645    2881 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d67884021cefef\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d67884021cefef  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal,UID:a3127caef50db0be093e96af172c433b,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Pulled,Message:Container image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:beacea521116b2422086941236135068766b54396a322e81fd4104d55c0282df\" already present on machine,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:53:30.292817903 +0000 UTC m=+0.492673167,LastTimestamp:2026-09-18 16:55:26.890359671 +0000 UTC m=+117.090214934,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:55:26.975925 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:55:26.975837    2881 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d678840697120b\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d678840697120b  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal,UID:a3127caef50db0be093e96af172c433b,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Created,Message:Created container: setup,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:53:30.367930891 +0000 UTC m=+0.567786155,LastTimestamp:2026-09-18 16:55:26.964693805 +0000 UTC m=+117.164549089,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:55:26.986156 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:55:26.986064    2881 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d678840720bdf3\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d678840720bdf3  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal,UID:a3127caef50db0be093e96af172c433b,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Started,Message:Started container setup,Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:53:30.376953331 +0000 UTC m=+0.576808595,LastTimestamp:2026-09-18 16:55:26.973834194 +0000 UTC m=+117.173689477,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:55:27.094342 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:27.094317    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_a3127caef50db0be093e96af172c433b/setup/8.log"
Sep 18 16:55:27.094511 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:27.094365    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" event={"ID":"a3127caef50db0be093e96af172c433b","Type":"ContainerStarted","Data":"fc83a8ddcd0cde882b13de30a95fbc2bad9c2a12b526a3ef148eeb1b0265a883"}
Sep 18 16:55:27.094511 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:27.094482    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:55:27.095364 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:27.095346    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:55:27.095428 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:27.095377    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:55:27.095428 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:27.095388    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:55:27.095578 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:55:27.095566    2881 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-5.ec2.internal\" not found" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:55:27.571314 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:55:27.571276    2881 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass"
Sep 18 16:55:27.644938 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:55:27.644904    2881 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 18 16:55:27.699870 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:55:27.699832    2881 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 18 16:55:27.880854 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:27.880768    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:55:27.989911 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:27.989872    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:55:27.990954 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:27.990934    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:55:27.991075 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:27.990968    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:55:27.991075 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:27.990979    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:55:27.991075 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:27.991007    2881 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:55:28.010515 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:55:28.010478    2881 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:55:28.881280 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:28.881245    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:55:29.881505 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:29.881474    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:55:29.910002 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:55:29.909962    2881 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-5.ec2.internal\" not found"
Sep 18 16:55:30.882067 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:30.882031    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:55:31.541211 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:55:31.541174    2881 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 18 16:55:31.882899 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:31.882817    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:55:32.878831 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:32.878795    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:55:33.883327 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:33.883297    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:55:34.709452 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:55:34.709413    2881 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 18 16:55:34.882079 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:34.882043    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:55:35.011086 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:35.011053    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:55:35.012982 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:35.012961    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:55:35.013089 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:35.012997    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:55:35.013089 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:35.013010    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:55:35.013089 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:35.013040    2881 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:55:35.031476 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:55:35.031444    2881 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:55:35.879973 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:35.879942    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:55:36.882442 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:36.882408    2881 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 16:55:36.883284 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:36.883263    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:55:36.884980 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:55:36.884961    2881 certificate_manager.go:596] "Failed while requesting a signed certificate from the control plane" err="cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User \"system:serviceaccount:openshift-machine-config-operator:node-bootstrapper\" cannot create resource \"certificatesigningrequests\" in API group \"certificates.k8s.io\" at the cluster scope" logger="kubernetes.io/kube-apiserver-client-kubelet.UnhandledError"
Sep 18 16:55:37.879961 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:37.879923    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:55:38.113122 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:38.113094    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_a3127caef50db0be093e96af172c433b/setup/9.log"
Sep 18 16:55:38.113587 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:38.113547    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_a3127caef50db0be093e96af172c433b/setup/8.log"
Sep 18 16:55:38.113629 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:38.113582    2881 generic.go:356] "Generic (PLEG): container finished" podID="a3127caef50db0be093e96af172c433b" containerID="fc83a8ddcd0cde882b13de30a95fbc2bad9c2a12b526a3ef148eeb1b0265a883" exitCode=1
Sep 18 16:55:38.113629 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:38.113614    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" event={"ID":"a3127caef50db0be093e96af172c433b","Type":"ContainerDied","Data":"fc83a8ddcd0cde882b13de30a95fbc2bad9c2a12b526a3ef148eeb1b0265a883"}
Sep 18 16:55:38.113717 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:38.113703    2881 scope.go:117] "RemoveContainer" containerID="3a91cbee892450ba29615791afe17dfe13a763d0d639b116527651e7239e1f49"
Sep 18 16:55:38.113811 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:38.113794    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:55:38.115092 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:38.115076    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:55:38.115160 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:38.115108    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:55:38.115160 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:38.115124    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:55:38.115401 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:55:38.115384    2881 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-5.ec2.internal\" not found" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:55:38.115462 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:38.115452    2881 scope.go:117] "RemoveContainer" containerID="3a91cbee892450ba29615791afe17dfe13a763d0d639b116527651e7239e1f49"
Sep 18 16:55:38.123791 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:55:38.123753    2881 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator_a3127caef50db0be093e96af172c433b_8 in pod sandbox cf961e79c5b92a2e43a70c6a29822bf8a5a2069147989bc7b9bb0b53e33609ad from index: no such id: '3a91cbee892450ba29615791afe17dfe13a763d0d639b116527651e7239e1f49'" containerID="3a91cbee892450ba29615791afe17dfe13a763d0d639b116527651e7239e1f49"
Sep 18 16:55:38.123928 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:55:38.123816    2881 kuberuntime_container.go:962] "Unhandled Error" err="failed to remove pod init container \"setup\": rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator_a3127caef50db0be093e96af172c433b_8 in pod sandbox cf961e79c5b92a2e43a70c6a29822bf8a5a2069147989bc7b9bb0b53e33609ad from index: no such id: '3a91cbee892450ba29615791afe17dfe13a763d0d639b116527651e7239e1f49'; Skipping pod \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b)\"" logger="UnhandledError"
Sep 18 16:55:38.124028 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:55:38.124007    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" podUID="a3127caef50db0be093e96af172c433b"
Sep 18 16:55:38.133760 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:55:38.133622    2881 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d67886b7a03b55\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d67886b7a03b55  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal,UID:a3127caef50db0be093e96af172c433b,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:BackOff,Message:Back-off restarting failed container setup in pod kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b),Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:53:41.928033109 +0000 UTC m=+12.127888386,LastTimestamp:2026-09-18 16:55:38.123956153 +0000 UTC m=+128.323811452,Count:8,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:55:38.881421 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:38.881382    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:55:39.118006 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:39.117978    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_a3127caef50db0be093e96af172c433b/setup/9.log"
Sep 18 16:55:39.881078 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:39.881047    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:55:39.910579 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:55:39.910547    2881 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-5.ec2.internal\" not found"
Sep 18 16:55:40.879412 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:40.879377    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:55:41.720997 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:55:41.720952    2881 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 18 16:55:41.880132 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:41.880100    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:55:42.031825 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:42.031796    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:55:42.032900 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:42.032879    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:55:42.033003 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:42.032922    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:55:42.033003 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:42.032936    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:55:42.033003 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:42.032965    2881 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:55:42.051374 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:55:42.051344    2881 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:55:42.880447 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:42.880419    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:55:43.885418 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:43.885381    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:55:44.881613 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:44.881582    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:55:45.879976 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:45.879941    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:55:46.881663 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:46.881627    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:55:47.885468 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:47.885430    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:55:48.733873 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:55:48.733838    2881 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 18 16:55:48.879398 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:48.879368    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:55:49.051994 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:49.051958    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:55:49.053034 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:49.053014    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:55:49.053128 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:49.053051    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:55:49.053128 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:49.053062    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:55:49.053128 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:49.053087    2881 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:55:49.072350 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:55:49.072313    2881 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:55:49.880606 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:49.880574    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:55:49.887910 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:49.887878    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:55:49.888930 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:49.888906    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:55:49.889048 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:49.888943    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:55:49.889048 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:49.888956    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:55:49.889140 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:55:49.889128    2881 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-5.ec2.internal\" not found" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:55:49.889311 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:55:49.889294    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" podUID="a3127caef50db0be093e96af172c433b"
Sep 18 16:55:49.900236 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:55:49.900135    2881 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d67886b7a03b55\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d67886b7a03b55  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal,UID:a3127caef50db0be093e96af172c433b,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:BackOff,Message:Back-off restarting failed container setup in pod kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b),Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:53:41.928033109 +0000 UTC m=+12.127888386,LastTimestamp:2026-09-18 16:55:49.889266086 +0000 UTC m=+140.089121363,Count:9,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:55:49.911327 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:55:49.911296    2881 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-5.ec2.internal\" not found"
Sep 18 16:55:50.695026 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:50.694978    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:55:50.696117 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:50.696100    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:55:50.696196 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:50.696139    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:55:50.696196 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:50.696152    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:55:50.884575 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:50.884544    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:55:51.888184 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:51.888150    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:55:52.880715 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:52.880684    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:55:53.886486 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:53.886455    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:55:54.881078 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:54.881046    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:55:55.744580 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:55:55.744549    2881 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 18 16:55:55.880820 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:55.880785    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:55:56.072922 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:56.072895    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:55:56.073954 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:56.073926    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:55:56.074092 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:56.073962    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:55:56.074092 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:56.073974    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:55:56.074092 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:56.074005    2881 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:55:56.094211 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:55:56.094173    2881 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:55:56.894732 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:56.894704    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:55:57.880196 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:57.880166    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:55:58.882580 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:58.882553    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:55:59.881434 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:55:59.881402    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:55:59.911946 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:55:59.911913    2881 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-5.ec2.internal\" not found"
Sep 18 16:56:00.882629 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:00.882602    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:56:01.881174 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:01.881140    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:56:02.758064 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:56:02.758033    2881 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 18 16:56:02.880420 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:02.880394    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:56:03.094713 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:03.094672    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:56:03.095734 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:03.095715    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:56:03.095840 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:03.095746    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:56:03.095840 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:03.095756    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:56:03.095840 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:03.095792    2881 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:56:03.113538 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:56:03.113504    2881 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:56:03.887382 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:03.887333    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:56:03.889623 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:03.889597    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:56:03.891083 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:03.891062    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:56:03.891219 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:03.891094    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:56:03.891219 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:03.891106    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:56:03.891364 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:56:03.891344    2881 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-5.ec2.internal\" not found" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:56:03.891500 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:56:03.891482    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" podUID="a3127caef50db0be093e96af172c433b"
Sep 18 16:56:03.902737 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:56:03.902642    2881 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d67886b7a03b55\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d67886b7a03b55  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal,UID:a3127caef50db0be093e96af172c433b,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:BackOff,Message:Back-off restarting failed container setup in pod kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b),Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:53:41.928033109 +0000 UTC m=+12.127888386,LastTimestamp:2026-09-18 16:56:03.891457139 +0000 UTC m=+154.091312416,Count:10,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:56:04.883908 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:04.883880    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:56:05.880716 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:05.880675    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:56:06.880010 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:06.879976    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:56:07.880060 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:07.880027    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:56:07.976790 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:56:07.976754    2881 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 18 16:56:08.881341 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:08.881307    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:56:08.882361 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:08.882334    2881 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 16:56:08.884998 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:56:08.884973    2881 certificate_manager.go:596] "Failed while requesting a signed certificate from the control plane" err="cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User \"system:serviceaccount:openshift-machine-config-operator:node-bootstrapper\" cannot create resource \"certificatesigningrequests\" in API group \"certificates.k8s.io\" at the cluster scope" logger="kubernetes.io/kube-apiserver-client-kubelet.UnhandledError"
Sep 18 16:56:09.769022 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:56:09.768983    2881 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 18 16:56:09.880100 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:09.880066    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:56:09.912986 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:56:09.912951    2881 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-5.ec2.internal\" not found"
Sep 18 16:56:10.114551 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:10.114463    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:56:10.116367 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:10.116347    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:56:10.116512 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:10.116383    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:56:10.116512 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:10.116398    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:56:10.116512 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:10.116437    2881 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:56:10.156724 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:56:10.156683    2881 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:56:10.883793 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:10.883752    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:56:11.881001 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:11.880971    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:56:12.542087 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:56:12.542051    2881 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 18 16:56:12.884888 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:12.884802    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:56:13.879703 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:13.879674    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:56:14.879378 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:14.879344    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:56:15.881015 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:15.880977    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:56:15.888246 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:15.888211    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:56:15.889216 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:15.889196    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:56:15.889350 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:15.889244    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:56:15.889350 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:15.889257    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:56:15.889476 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:56:15.889463    2881 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-5.ec2.internal\" not found" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:56:16.788314 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:56:16.788278    2881 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 18 16:56:16.880038 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:16.880014    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:56:17.157015 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:17.156925    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:56:17.158061 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:17.158044    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:56:17.158141 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:17.158076    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:56:17.158141 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:17.158087    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:56:17.158141 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:17.158122    2881 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:56:17.179290 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:56:17.179259    2881 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:56:17.880373 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:17.880336    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:56:17.887608 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:17.887578    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:56:17.888591 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:17.888573    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:56:17.888689 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:17.888610    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:56:17.888689 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:17.888624    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:56:17.888908 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:56:17.888892    2881 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-5.ec2.internal\" not found" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:56:17.889086 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:56:17.889066    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" podUID="a3127caef50db0be093e96af172c433b"
Sep 18 16:56:17.899355 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:56:17.899258    2881 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d67886b7a03b55\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d67886b7a03b55  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal,UID:a3127caef50db0be093e96af172c433b,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:BackOff,Message:Back-off restarting failed container setup in pod kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b),Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:53:41.928033109 +0000 UTC m=+12.127888386,LastTimestamp:2026-09-18 16:56:17.889034027 +0000 UTC m=+168.088889303,Count:11,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:56:18.880645 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:18.880608    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:56:19.880436 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:19.880405    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:56:19.913091 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:56:19.913044    2881 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-5.ec2.internal\" not found"
Sep 18 16:56:20.893088 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:20.893055    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:56:21.250276 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:56:21.250174    2881 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass"
Sep 18 16:56:21.881563 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:21.881530    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:56:22.880847 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:22.880815    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:56:23.800764 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:56:23.800725    2881 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 18 16:56:23.880822 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:23.880794    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:56:24.179677 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:24.179577    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:56:24.180994 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:24.180977    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:56:24.181069 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:24.181009    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:56:24.181069 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:24.181022    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:56:24.181069 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:24.181056    2881 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:56:24.198501 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:56:24.198468    2881 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:56:24.881795 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:24.881763    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:56:25.886309 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:25.886276    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:56:26.886024 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:26.885992    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:56:27.887260 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:27.887211    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:56:28.056066 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:56:28.056031    2881 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 18 16:56:28.878992 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:28.878963    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:56:29.880915 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:29.880887    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:56:29.913545 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:56:29.913512    2881 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-5.ec2.internal\" not found"
Sep 18 16:56:30.813172 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:56:30.813132    2881 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 18 16:56:30.884598 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:30.884564    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:56:31.198730 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:31.198635    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:56:31.199675 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:31.199656    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:56:31.199819 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:31.199691    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:56:31.199819 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:31.199706    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:56:31.199819 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:31.199743    2881 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:56:31.227200 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:56:31.227166    2881 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:56:31.879458 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:31.879422    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:56:32.881033 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:32.881000    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:56:32.888267 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:32.888246    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:56:32.894819 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:32.894799    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:56:32.894819 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:32.894832    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:56:32.894967 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:32.894842    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:56:32.895081 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:56:32.895067    2881 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-5.ec2.internal\" not found" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:56:32.895251 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:56:32.895216    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" podUID="a3127caef50db0be093e96af172c433b"
Sep 18 16:56:32.906945 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:56:32.906851    2881 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d67886b7a03b55\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d67886b7a03b55  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal,UID:a3127caef50db0be093e96af172c433b,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:BackOff,Message:Back-off restarting failed container setup in pod kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b),Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:53:41.928033109 +0000 UTC m=+12.127888386,LastTimestamp:2026-09-18 16:56:32.895187683 +0000 UTC m=+183.095042960,Count:12,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:56:33.882842 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:33.882808    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:56:34.880109 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:34.880074    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:56:35.880190 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:35.880157    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:56:36.880543 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:36.880508    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:56:37.826980 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:56:37.826945    2881 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 18 16:56:37.882434 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:37.882399    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:56:38.227913 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:38.227811    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:56:38.229785 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:38.229760    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:56:38.229939 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:38.229802    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:56:38.229939 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:38.229817    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:56:38.229939 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:38.229851    2881 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:56:38.256684 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:56:38.256650    2881 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:56:38.881132 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:38.881101    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:56:39.882891 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:39.882860    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:56:39.914269 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:56:39.914239    2881 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-5.ec2.internal\" not found"
Sep 18 16:56:40.882341 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:40.882304    2881 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 16:56:40.882561 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:40.882420    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:56:40.896872 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:40.896844    2881 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 16:56:41.085107 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:41.085079    2881 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4dzwj"
Sep 18 16:56:41.883693 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:41.883663    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:56:42.888540 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:42.888510    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:56:43.882579 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:43.882551    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:56:44.844374 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:56:44.844329    2881 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 18 16:56:44.883040 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:44.883005    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:56:45.257209 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:45.257109    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:56:45.258198 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:45.258177    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:56:45.258376 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:45.258213    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:56:45.258376 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:45.258248    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:56:45.258376 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:45.258286    2881 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:56:45.282988 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:56:45.282947    2881 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:56:45.880112 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:45.880074    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:56:45.887614 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:45.887589    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:56:45.890318 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:45.890286    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:56:45.890318 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:45.890318    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:56:45.890480 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:45.890329    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:56:45.890579 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:56:45.890566    2881 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-5.ec2.internal\" not found" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:56:45.890738 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:56:45.890719    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" podUID="a3127caef50db0be093e96af172c433b"
Sep 18 16:56:45.899929 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:56:45.899841    2881 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d67886b7a03b55\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal.18d67886b7a03b55  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal,UID:a3127caef50db0be093e96af172c433b,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:BackOff,Message:Back-off restarting failed container setup in pod kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b),Source:EventSource{Component:kubelet,Host:ip-10-0-143-5.ec2.internal,},FirstTimestamp:2026-09-18 16:53:41.928033109 +0000 UTC m=+12.127888386,LastTimestamp:2026-09-18 16:56:45.890688797 +0000 UTC m=+196.090544073,Count:13,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-5.ec2.internal,}"
Sep 18 16:56:46.882818 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:46.882786    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:56:47.884861 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:47.884832    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:56:48.880805 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:48.880772    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:56:49.890952 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:49.890923    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:56:49.915105 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:56:49.915074    2881 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-5.ec2.internal\" not found"
Sep 18 16:56:50.880895 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:50.880865    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:56:51.855149 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:56:51.855117    2881 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 18 16:56:51.883640 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:51.883610    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:56:52.283259 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:52.283217    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:56:52.286381 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:52.286353    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:56:52.286541 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:52.286396    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:56:52.286541 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:52.286410    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:56:52.286541 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:52.286443    2881 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:56:52.308863 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:56:52.308826    2881 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:56:52.881609 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:52.881584    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:56:53.878916 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:53.878878    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:56:54.883169 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:54.883136    2881 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:56:55.563551 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:55.563519    2881 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4dzwj"
Sep 18 16:56:55.848762 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:55.848678    2881 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 18 16:56:55.910833 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:55.910804    2881 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-143-5.ec2.internal" not found
Sep 18 16:56:55.944904 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:55.944875    2881 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-143-5.ec2.internal" not found
Sep 18 16:56:56.030176 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:56.030149    2881 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-143-5.ec2.internal" not found
Sep 18 16:56:56.314006 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:56.313979    2881 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-143-5.ec2.internal" not found
Sep 18 16:56:56.314006 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:56:56.314007    2881 csi_plugin.go:399] Failed to initialize CSINode: error updating CSINode annotation: timed out waiting for the condition; caused by: nodes "ip-10-0-143-5.ec2.internal" not found
Sep 18 16:56:56.369050 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:56.369027    2881 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-143-5.ec2.internal" not found
Sep 18 16:56:56.404909 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:56.404885    2881 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-143-5.ec2.internal" not found
Sep 18 16:56:56.477937 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:56.477910    2881 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-143-5.ec2.internal" not found
Sep 18 16:56:56.566285 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:56.566190    2881 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-17 16:51:55 +0000 UTC" deadline="2028-05-12 13:40:38.082172033 +0000 UTC"
Sep 18 16:56:56.566285 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:56.566245    2881 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14444h43m41.515929845s"
Sep 18 16:56:56.739667 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:56.739638    2881 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-143-5.ec2.internal" not found
Sep 18 16:56:56.739667 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:56:56.739664    2881 csi_plugin.go:399] Failed to initialize CSINode: error updating CSINode annotation: timed out waiting for the condition; caused by: nodes "ip-10-0-143-5.ec2.internal" not found
Sep 18 16:56:56.996544 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:56.996472    2881 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-143-5.ec2.internal" not found
Sep 18 16:56:57.016640 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:57.016618    2881 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-143-5.ec2.internal" not found
Sep 18 16:56:57.091885 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:57.091857    2881 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-143-5.ec2.internal" not found
Sep 18 16:56:57.373720 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:57.373694    2881 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-143-5.ec2.internal" not found
Sep 18 16:56:57.373720 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:56:57.373718    2881 csi_plugin.go:399] Failed to initialize CSINode: error updating CSINode annotation: timed out waiting for the condition; caused by: nodes "ip-10-0-143-5.ec2.internal" not found
Sep 18 16:56:58.870683 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:56:58.870644    2881 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-143-5.ec2.internal\" not found" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:56:58.888162 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:58.888139    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:56:58.890538 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:58.890520    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:56:58.890643 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:58.890550    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:56:58.890643 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:58.890560    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:56:58.890802 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:56:58.890788    2881 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-5.ec2.internal\" not found" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:56:59.244202 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:59.244119    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_a3127caef50db0be093e96af172c433b/setup/9.log"
Sep 18 16:56:59.244202 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:59.244164    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" event={"ID":"a3127caef50db0be093e96af172c433b","Type":"ContainerStarted","Data":"5a1a260f7fcf9e054488a21120f0bc2e2d4a1984e7dfdda5af9badd275584b95"}
Sep 18 16:56:59.244415 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:59.244297    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:56:59.245195 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:59.245176    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:56:59.245273 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:59.245203    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:56:59.245273 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:59.245215    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:56:59.245429 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:56:59.245417    2881 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-5.ec2.internal\" not found" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:56:59.309801 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:59.309771    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:56:59.310807 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:59.310792    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:56:59.310871 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:59.310823    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:56:59.310871 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:59.310833    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:56:59.310871 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:59.310858    2881 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:56:59.320022 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:56:59.320001    2881 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:56:59.320087 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:56:59.320027    2881 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-5.ec2.internal\": node \"ip-10-0-143-5.ec2.internal\" not found"
Sep 18 16:56:59.340861 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:56:59.340827    2881 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-5.ec2.internal\" not found"
Sep 18 16:56:59.441200 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:56:59.441168    2881 kubelet_node_status.go:448] "Node not becoming ready in time after startup"
Sep 18 16:56:59.915487 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:56:59.915450    2881 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-5.ec2.internal\" not found"
Sep 18 16:56:59.944765 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:56:59.944720    2881 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?"
Sep 18 16:57:04.946291 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:04.946253    2881 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?"
Sep 18 16:57:07.308124 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:07.308088    2881 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 16:57:07.965740 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:07.965712    2881 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 16:57:09.356662 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:09.356615    2881 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-5.ec2.internal\": node \"ip-10-0-143-5.ec2.internal\" not found"
Sep 18 16:57:09.916301 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:09.916264    2881 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-5.ec2.internal\" not found"
Sep 18 16:57:09.947151 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:09.947112    2881 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?"
Sep 18 16:57:10.263870 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:10.263844    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_a3127caef50db0be093e96af172c433b/setup/10.log"
Sep 18 16:57:10.264261 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:10.264245    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_a3127caef50db0be093e96af172c433b/setup/9.log"
Sep 18 16:57:10.264327 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:10.264280    2881 generic.go:356] "Generic (PLEG): container finished" podID="a3127caef50db0be093e96af172c433b" containerID="5a1a260f7fcf9e054488a21120f0bc2e2d4a1984e7dfdda5af9badd275584b95" exitCode=1
Sep 18 16:57:10.264327 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:10.264308    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" event={"ID":"a3127caef50db0be093e96af172c433b","Type":"ContainerDied","Data":"5a1a260f7fcf9e054488a21120f0bc2e2d4a1984e7dfdda5af9badd275584b95"}
Sep 18 16:57:10.264393 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:10.264351    2881 scope.go:117] "RemoveContainer" containerID="fc83a8ddcd0cde882b13de30a95fbc2bad9c2a12b526a3ef148eeb1b0265a883"
Sep 18 16:57:10.264501 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:10.264487    2881 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:57:10.269160 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:10.269142    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:57:10.269281 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:10.269171    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:57:10.269281 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:10.269181    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:57:10.269396 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:10.269383    2881 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-5.ec2.internal\" not found" node="ip-10-0-143-5.ec2.internal"
Sep 18 16:57:10.269444 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:10.269434    2881 scope.go:117] "RemoveContainer" containerID="fc83a8ddcd0cde882b13de30a95fbc2bad9c2a12b526a3ef148eeb1b0265a883"
Sep 18 16:57:10.277239 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:10.277192    2881 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator_a3127caef50db0be093e96af172c433b_9 in pod sandbox cf961e79c5b92a2e43a70c6a29822bf8a5a2069147989bc7b9bb0b53e33609ad from index: no such id: 'fc83a8ddcd0cde882b13de30a95fbc2bad9c2a12b526a3ef148eeb1b0265a883'" containerID="fc83a8ddcd0cde882b13de30a95fbc2bad9c2a12b526a3ef148eeb1b0265a883"
Sep 18 16:57:10.277325 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:10.277264    2881 kuberuntime_container.go:962] "Unhandled Error" err="failed to remove pod init container \"setup\": rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator_a3127caef50db0be093e96af172c433b_9 in pod sandbox cf961e79c5b92a2e43a70c6a29822bf8a5a2069147989bc7b9bb0b53e33609ad from index: no such id: 'fc83a8ddcd0cde882b13de30a95fbc2bad9c2a12b526a3ef148eeb1b0265a883'; Skipping pod \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b)\"" logger="UnhandledError"
Sep 18 16:57:10.277446 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:10.277428    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" podUID="a3127caef50db0be093e96af172c433b"
Sep 18 16:57:11.267783 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:11.267748    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_a3127caef50db0be093e96af172c433b/setup/10.log"
Sep 18 16:57:12.226052 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:12.226015    2881 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 16:57:14.950204 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:14.950160    2881 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?"
Sep 18 16:57:19.516925 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:19.516872    2881 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-5.ec2.internal\": node \"ip-10-0-143-5.ec2.internal\" not found"
Sep 18 16:57:19.916704 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:19.916604    2881 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-5.ec2.internal\" not found"
Sep 18 16:57:19.951258 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:19.951205    2881 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?"
Sep 18 16:57:23.984385 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:23.984343    2881 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 16:57:24.009514 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:24.009483    2881 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 18 16:57:24.024182 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:24.024151    2881 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 16:57:24.056888 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:24.056859    2881 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-nb7r6"
Sep 18 16:57:24.069279 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:24.069243    2881 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-nb7r6"
Sep 18 16:57:24.072153 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:24.072134    2881 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal"
Sep 18 16:57:24.091132 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:24.091105    2881 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 16:57:24.092827 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:24.092810    2881 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-5.ec2.internal"
Sep 18 16:57:24.202115 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:24.202084    2881 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 16:57:24.887814 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:24.887785    2881 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal"
Sep 18 16:57:24.900805 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:24.900785    2881 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 16:57:24.900898 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:24.900839    2881 kubelet.go:3250] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal"
Sep 18 16:57:24.901050 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:24.901030    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" podUID="a3127caef50db0be093e96af172c433b"
Sep 18 16:57:24.951849 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:24.951819    2881 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?"
Sep 18 16:57:24.985178 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:24.985149    2881 apiserver.go:52] "Watching apiserver"
Sep 18 16:57:24.992169 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:24.992145    2881 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 18 16:57:24.992515 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:24.992494    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-ktwhm","openshift-image-registry/node-ca-sghwn","openshift-multus/multus-additional-cni-plugins-n4nk2","openshift-multus/network-metrics-daemon-lpt5n","openshift-network-diagnostics/network-check-target-pc44x","openshift-network-operator/iptables-alerter-mzj44","kube-system/global-pull-secret-syncer-fx8nf","kube-system/kube-apiserver-proxy-ip-10-0-143-5.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal","openshift-multus/multus-8rtmt","openshift-ovn-kubernetes/ovnkube-node-97wk9","kube-system/konnectivity-agent-g7d7f","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwddj"]
Sep 18 16:57:24.994940 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:24.994923    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fx8nf"
Sep 18 16:57:24.995005 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:24.994991    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fx8nf" podUID="a4df48ac-dd04-4504-b828-66d7e2e986e1"
Sep 18 16:57:24.997090 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:24.997073    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-n4nk2"
Sep 18 16:57:24.999204 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:24.999182    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpt5n"
Sep 18 16:57:24.999355 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:24.999260    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lpt5n" podUID="89faf7f5-d633-4219-b3fc-323f8c23b599"
Sep 18 16:57:24.999631 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:24.999610    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 18 16:57:24.999889 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:24.999862    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bjmf8\""
Sep 18 16:57:24.999889 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:24.999879    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 18 16:57:25.000074 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:24.999915    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 18 16:57:25.000074 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:24.999971    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 18 16:57:25.000074 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:24.999983    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 18 16:57:25.001534 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.001512    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ktwhm"
Sep 18 16:57:25.003750 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.003730    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8rtmt"
Sep 18 16:57:25.003904 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.003868    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 18 16:57:25.004023 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.003925    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 18 16:57:25.004023 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.003870    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7q7k4\""
Sep 18 16:57:25.006145 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.006124    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-pxctw\""
Sep 18 16:57:25.006261 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.006148    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-97wk9"
Sep 18 16:57:25.006322 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.006282    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 18 16:57:25.008574 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.008558    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-g7d7f"
Sep 18 16:57:25.010831 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.010813    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 18 16:57:25.010953 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.010877    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 18 16:57:25.010953 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.010936    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwddj"
Sep 18 16:57:25.011124 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.010953    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-l249t\""
Sep 18 16:57:25.013113 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.013097    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-pvsd6\""
Sep 18 16:57:25.013290 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.013273    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pc44x"
Sep 18 16:57:25.013365 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.013330    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 18 16:57:25.013365 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:25.013347    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pc44x" podUID="6f463717-a487-485b-aa4b-7772d35b0d22"
Sep 18 16:57:25.013486 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.013335    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 18 16:57:25.013575 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.013559    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 18 16:57:25.015745 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.015727    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-sghwn"
Sep 18 16:57:25.017951 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.017930    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 18 16:57:25.018068 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.017933    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-8pbqv\""
Sep 18 16:57:25.018128 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.018068    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mzj44"
Sep 18 16:57:25.018197 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.018181    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 18 16:57:25.018345 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.018303    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 18 16:57:25.020633 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.020367    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nq4f2\""
Sep 18 16:57:25.020633 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.020566    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 18 16:57:25.020633 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.020653    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 18 16:57:25.020633 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.020873    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 18 16:57:25.021893 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.021875    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-97wk9"
Sep 18 16:57:25.072191 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.072151    2881 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 16:52:24 +0000 UTC" deadline="2028-03-27 06:49:57.193460775 +0000 UTC"
Sep 18 16:57:25.072191 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.072184    2881 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13333h52m32.121280405s"
Sep 18 16:57:25.072539 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.072526    2881 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 18 16:57:25.135956 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.135894    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-5.ec2.internal" podStartSLOduration=1.135877292 podStartE2EDuration="1.135877292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:57:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:57:25.135816338 +0000 UTC m=+235.335671623" watchObservedRunningTime="2026-09-18 16:57:25.135877292 +0000 UTC m=+235.335732586"
Sep 18 16:57:25.141446 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.141375    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b6ec1dac-2d47-4156-bb74-b7055bedbc2b-hostroot\") pod \"multus-8rtmt\" (UID: \"b6ec1dac-2d47-4156-bb74-b7055bedbc2b\") " pod="openshift-multus/multus-8rtmt"
Sep 18 16:57:25.141446 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.141405    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kc9qn\" (UniqueName: \"kubernetes.io/projected/7295391c-ebd9-4f37-9f70-a86b0bab3fcf-kube-api-access-kc9qn\") pod \"node-ca-sghwn\" (UID: \"7295391c-ebd9-4f37-9f70-a86b0bab3fcf\") " pod="openshift-image-registry/node-ca-sghwn"
Sep 18 16:57:25.141446 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.141422    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fb327355-e5c6-48fd-abf3-24fa50a8e0b5-agent-certs\") pod \"konnectivity-agent-g7d7f\" (UID: \"fb327355-e5c6-48fd-abf3-24fa50a8e0b5\") " pod="kube-system/konnectivity-agent-g7d7f"
Sep 18 16:57:25.141446 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.141438    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b2r6\" (UniqueName: \"kubernetes.io/projected/d019d1b3-e6ee-4b27-9fc1-a6675650d42a-kube-api-access-8b2r6\") pod \"multus-additional-cni-plugins-n4nk2\" (UID: \"d019d1b3-e6ee-4b27-9fc1-a6675650d42a\") " pod="openshift-multus/multus-additional-cni-plugins-n4nk2"
Sep 18 16:57:25.141665 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.141482    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437-etc-sysctl-conf\") pod \"tuned-ktwhm\" (UID: \"05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437\") " pod="openshift-cluster-node-tuning-operator/tuned-ktwhm"
Sep 18 16:57:25.141665 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.141519    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437-var-lib-kubelet\") pod \"tuned-ktwhm\" (UID: \"05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437\") " pod="openshift-cluster-node-tuning-operator/tuned-ktwhm"
Sep 18 16:57:25.141665 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.141536    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437-etc-sysconfig\") pod \"tuned-ktwhm\" (UID: \"05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437\") " pod="openshift-cluster-node-tuning-operator/tuned-ktwhm"
Sep 18 16:57:25.141665 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.141551    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7cc9ec4e-c391-465a-b3e2-8478fa3940fb-registration-dir\") pod \"aws-ebs-csi-driver-node-bwddj\" (UID: \"7cc9ec4e-c391-465a-b3e2-8478fa3940fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwddj"
Sep 18 16:57:25.141665 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.141573    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fb327355-e5c6-48fd-abf3-24fa50a8e0b5-konnectivity-ca\") pod \"konnectivity-agent-g7d7f\" (UID: \"fb327355-e5c6-48fd-abf3-24fa50a8e0b5\") " pod="kube-system/konnectivity-agent-g7d7f"
Sep 18 16:57:25.141665 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.141590    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6ec1dac-2d47-4156-bb74-b7055bedbc2b-system-cni-dir\") pod \"multus-8rtmt\" (UID: \"b6ec1dac-2d47-4156-bb74-b7055bedbc2b\") " pod="openshift-multus/multus-8rtmt"
Sep 18 16:57:25.141665 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.141616    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6ec1dac-2d47-4156-bb74-b7055bedbc2b-multus-cni-dir\") pod \"multus-8rtmt\" (UID: \"b6ec1dac-2d47-4156-bb74-b7055bedbc2b\") " pod="openshift-multus/multus-8rtmt"
Sep 18 16:57:25.141665 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.141637    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b6ec1dac-2d47-4156-bb74-b7055bedbc2b-host-run-netns\") pod \"multus-8rtmt\" (UID: \"b6ec1dac-2d47-4156-bb74-b7055bedbc2b\") " pod="openshift-multus/multus-8rtmt"
Sep 18 16:57:25.141665 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.141652    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d019d1b3-e6ee-4b27-9fc1-a6675650d42a-cni-binary-copy\") pod \"multus-additional-cni-plugins-n4nk2\" (UID: \"d019d1b3-e6ee-4b27-9fc1-a6675650d42a\") " pod="openshift-multus/multus-additional-cni-plugins-n4nk2"
Sep 18 16:57:25.141665 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.141667    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437-etc-sysctl-d\") pod \"tuned-ktwhm\" (UID: \"05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437\") " pod="openshift-cluster-node-tuning-operator/tuned-ktwhm"
Sep 18 16:57:25.142122 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.141704    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437-run\") pod \"tuned-ktwhm\" (UID: \"05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437\") " pod="openshift-cluster-node-tuning-operator/tuned-ktwhm"
Sep 18 16:57:25.142122 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.141743    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b6ec1dac-2d47-4156-bb74-b7055bedbc2b-host-var-lib-cni-multus\") pod \"multus-8rtmt\" (UID: \"b6ec1dac-2d47-4156-bb74-b7055bedbc2b\") " pod="openshift-multus/multus-8rtmt"
Sep 18 16:57:25.142122 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.141770    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b6ec1dac-2d47-4156-bb74-b7055bedbc2b-multus-conf-dir\") pod \"multus-8rtmt\" (UID: \"b6ec1dac-2d47-4156-bb74-b7055bedbc2b\") " pod="openshift-multus/multus-8rtmt"
Sep 18 16:57:25.142122 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.141795    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7cc9ec4e-c391-465a-b3e2-8478fa3940fb-sys-fs\") pod \"aws-ebs-csi-driver-node-bwddj\" (UID: \"7cc9ec4e-c391-465a-b3e2-8478fa3940fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwddj"
Sep 18 16:57:25.142122 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.141820    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d019d1b3-e6ee-4b27-9fc1-a6675650d42a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n4nk2\" (UID: \"d019d1b3-e6ee-4b27-9fc1-a6675650d42a\") " pod="openshift-multus/multus-additional-cni-plugins-n4nk2"
Sep 18 16:57:25.142122 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.141847    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhsfc\" (UniqueName: \"kubernetes.io/projected/6f463717-a487-485b-aa4b-7772d35b0d22-kube-api-access-rhsfc\") pod \"network-check-target-pc44x\" (UID: \"6f463717-a487-485b-aa4b-7772d35b0d22\") " pod="openshift-network-diagnostics/network-check-target-pc44x"
Sep 18 16:57:25.142122 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.141869    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d019d1b3-e6ee-4b27-9fc1-a6675650d42a-cnibin\") pod \"multus-additional-cni-plugins-n4nk2\" (UID: \"d019d1b3-e6ee-4b27-9fc1-a6675650d42a\") " pod="openshift-multus/multus-additional-cni-plugins-n4nk2"
Sep 18 16:57:25.142122 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.141899    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7cc9ec4e-c391-465a-b3e2-8478fa3940fb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bwddj\" (UID: \"7cc9ec4e-c391-465a-b3e2-8478fa3940fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwddj"
Sep 18 16:57:25.142122 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.141927    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437-etc-tuned\") pod \"tuned-ktwhm\" (UID: \"05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437\") " pod="openshift-cluster-node-tuning-operator/tuned-ktwhm"
Sep 18 16:57:25.142122 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.141949    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b6ec1dac-2d47-4156-bb74-b7055bedbc2b-multus-socket-dir-parent\") pod \"multus-8rtmt\" (UID: \"b6ec1dac-2d47-4156-bb74-b7055bedbc2b\") " pod="openshift-multus/multus-8rtmt"
Sep 18 16:57:25.142122 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.141965    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7295391c-ebd9-4f37-9f70-a86b0bab3fcf-host\") pod \"node-ca-sghwn\" (UID: \"7295391c-ebd9-4f37-9f70-a86b0bab3fcf\") " pod="openshift-image-registry/node-ca-sghwn"
Sep 18 16:57:25.142122 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.141980    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klxbq\" (UniqueName: \"kubernetes.io/projected/e1eb4986-e287-4a38-980c-3275bd8e2770-kube-api-access-klxbq\") pod \"iptables-alerter-mzj44\" (UID: \"e1eb4986-e287-4a38-980c-3275bd8e2770\") " pod="openshift-network-operator/iptables-alerter-mzj44"
Sep 18 16:57:25.142122 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.141994    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7cc9ec4e-c391-465a-b3e2-8478fa3940fb-device-dir\") pod \"aws-ebs-csi-driver-node-bwddj\" (UID: \"7cc9ec4e-c391-465a-b3e2-8478fa3940fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwddj"
Sep 18 16:57:25.142122 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.142008    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b6ec1dac-2d47-4156-bb74-b7055bedbc2b-host-var-lib-cni-bin\") pod \"multus-8rtmt\" (UID: \"b6ec1dac-2d47-4156-bb74-b7055bedbc2b\") " pod="openshift-multus/multus-8rtmt"
Sep 18 16:57:25.142122 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.142031    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7295391c-ebd9-4f37-9f70-a86b0bab3fcf-serviceca\") pod \"node-ca-sghwn\" (UID: \"7295391c-ebd9-4f37-9f70-a86b0bab3fcf\") " pod="openshift-image-registry/node-ca-sghwn"
Sep 18 16:57:25.142122 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.142053    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e1eb4986-e287-4a38-980c-3275bd8e2770-iptables-alerter-script\") pod \"iptables-alerter-mzj44\" (UID: \"e1eb4986-e287-4a38-980c-3275bd8e2770\") " pod="openshift-network-operator/iptables-alerter-mzj44"
Sep 18 16:57:25.142727 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.142089    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d019d1b3-e6ee-4b27-9fc1-a6675650d42a-system-cni-dir\") pod \"multus-additional-cni-plugins-n4nk2\" (UID: \"d019d1b3-e6ee-4b27-9fc1-a6675650d42a\") " pod="openshift-multus/multus-additional-cni-plugins-n4nk2"
Sep 18 16:57:25.142727 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.142111    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47rhc\" (UniqueName: \"kubernetes.io/projected/89faf7f5-d633-4219-b3fc-323f8c23b599-kube-api-access-47rhc\") pod \"network-metrics-daemon-lpt5n\" (UID: \"89faf7f5-d633-4219-b3fc-323f8c23b599\") " pod="openshift-multus/network-metrics-daemon-lpt5n"
Sep 18 16:57:25.142727 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.142132    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437-etc-kubernetes\") pod \"tuned-ktwhm\" (UID: \"05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437\") " pod="openshift-cluster-node-tuning-operator/tuned-ktwhm"
Sep 18 16:57:25.142727 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.142147    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a4df48ac-dd04-4504-b828-66d7e2e986e1-dbus\") pod \"global-pull-secret-syncer-fx8nf\" (UID: \"a4df48ac-dd04-4504-b828-66d7e2e986e1\") " pod="kube-system/global-pull-secret-syncer-fx8nf"
Sep 18 16:57:25.142727 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.142176    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6ec1dac-2d47-4156-bb74-b7055bedbc2b-host-var-lib-kubelet\") pod \"multus-8rtmt\" (UID: \"b6ec1dac-2d47-4156-bb74-b7055bedbc2b\") " pod="openshift-multus/multus-8rtmt"
Sep 18 16:57:25.142727 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.142196    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7cc9ec4e-c391-465a-b3e2-8478fa3940fb-etc-selinux\") pod \"aws-ebs-csi-driver-node-bwddj\" (UID: \"7cc9ec4e-c391-465a-b3e2-8478fa3940fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwddj"
Sep 18 16:57:25.142727 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.142211    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437-sys\") pod \"tuned-ktwhm\" (UID: \"05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437\") " pod="openshift-cluster-node-tuning-operator/tuned-ktwhm"
Sep 18 16:57:25.142727 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.142255    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b6ec1dac-2d47-4156-bb74-b7055bedbc2b-host-run-multus-certs\") pod \"multus-8rtmt\" (UID: \"b6ec1dac-2d47-4156-bb74-b7055bedbc2b\") " pod="openshift-multus/multus-8rtmt"
Sep 18 16:57:25.142727 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.142284    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d019d1b3-e6ee-4b27-9fc1-a6675650d42a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n4nk2\" (UID: \"d019d1b3-e6ee-4b27-9fc1-a6675650d42a\") " pod="openshift-multus/multus-additional-cni-plugins-n4nk2"
Sep 18 16:57:25.142727 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.142307    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zkw7\" (UniqueName: \"kubernetes.io/projected/05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437-kube-api-access-7zkw7\") pod \"tuned-ktwhm\" (UID: \"05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437\") " pod="openshift-cluster-node-tuning-operator/tuned-ktwhm"
Sep 18 16:57:25.142727 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.142329    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437-etc-systemd\") pod \"tuned-ktwhm\" (UID: \"05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437\") " pod="openshift-cluster-node-tuning-operator/tuned-ktwhm"
Sep 18 16:57:25.142727 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.142375    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hmqr\" (UniqueName: \"kubernetes.io/projected/b6ec1dac-2d47-4156-bb74-b7055bedbc2b-kube-api-access-4hmqr\") pod \"multus-8rtmt\" (UID: \"b6ec1dac-2d47-4156-bb74-b7055bedbc2b\") " pod="openshift-multus/multus-8rtmt"
Sep 18 16:57:25.142727 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.142400    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a4df48ac-dd04-4504-b828-66d7e2e986e1-original-pull-secret\") pod \"global-pull-secret-syncer-fx8nf\" (UID: \"a4df48ac-dd04-4504-b828-66d7e2e986e1\") " pod="kube-system/global-pull-secret-syncer-fx8nf"
Sep 18 16:57:25.142727 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.142422    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437-host\") pod \"tuned-ktwhm\" (UID: \"05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437\") " pod="openshift-cluster-node-tuning-operator/tuned-ktwhm"
Sep 18 16:57:25.142727 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.142446    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b6ec1dac-2d47-4156-bb74-b7055bedbc2b-etc-kubernetes\") pod \"multus-8rtmt\" (UID: \"b6ec1dac-2d47-4156-bb74-b7055bedbc2b\") " pod="openshift-multus/multus-8rtmt"
Sep 18 16:57:25.142727 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.142472    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a4df48ac-dd04-4504-b828-66d7e2e986e1-kubelet-config\") pod \"global-pull-secret-syncer-fx8nf\" (UID: \"a4df48ac-dd04-4504-b828-66d7e2e986e1\") " pod="kube-system/global-pull-secret-syncer-fx8nf"
Sep 18 16:57:25.143187 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.142487    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e1eb4986-e287-4a38-980c-3275bd8e2770-host-slash\") pod \"iptables-alerter-mzj44\" (UID: \"e1eb4986-e287-4a38-980c-3275bd8e2770\") " pod="openshift-network-operator/iptables-alerter-mzj44"
Sep 18 16:57:25.143187 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.142502    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d019d1b3-e6ee-4b27-9fc1-a6675650d42a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n4nk2\" (UID: \"d019d1b3-e6ee-4b27-9fc1-a6675650d42a\") " pod="openshift-multus/multus-additional-cni-plugins-n4nk2"
Sep 18 16:57:25.143187 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.142525    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437-etc-modprobe-d\") pod \"tuned-ktwhm\" (UID: \"05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437\") " pod="openshift-cluster-node-tuning-operator/tuned-ktwhm"
Sep 18 16:57:25.143187 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.142544    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437-lib-modules\") pod \"tuned-ktwhm\" (UID: \"05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437\") " pod="openshift-cluster-node-tuning-operator/tuned-ktwhm"
Sep 18 16:57:25.143187 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.142558    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7cc9ec4e-c391-465a-b3e2-8478fa3940fb-socket-dir\") pod \"aws-ebs-csi-driver-node-bwddj\" (UID: \"7cc9ec4e-c391-465a-b3e2-8478fa3940fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwddj"
Sep 18 16:57:25.143187 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.142574    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpdk8\" (UniqueName: \"kubernetes.io/projected/7cc9ec4e-c391-465a-b3e2-8478fa3940fb-kube-api-access-dpdk8\") pod \"aws-ebs-csi-driver-node-bwddj\" (UID: \"7cc9ec4e-c391-465a-b3e2-8478fa3940fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwddj"
Sep 18 16:57:25.143187 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.142597    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89faf7f5-d633-4219-b3fc-323f8c23b599-metrics-certs\") pod \"network-metrics-daemon-lpt5n\" (UID: \"89faf7f5-d633-4219-b3fc-323f8c23b599\") " pod="openshift-multus/network-metrics-daemon-lpt5n"
Sep 18 16:57:25.143187 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.142611    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b6ec1dac-2d47-4156-bb74-b7055bedbc2b-cni-binary-copy\") pod \"multus-8rtmt\" (UID: \"b6ec1dac-2d47-4156-bb74-b7055bedbc2b\") " pod="openshift-multus/multus-8rtmt"
Sep 18 16:57:25.143187 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.142627    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b6ec1dac-2d47-4156-bb74-b7055bedbc2b-host-run-k8s-cni-cncf-io\") pod \"multus-8rtmt\" (UID: \"b6ec1dac-2d47-4156-bb74-b7055bedbc2b\") " pod="openshift-multus/multus-8rtmt"
Sep 18 16:57:25.143187 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.142640    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437-tmp\") pod \"tuned-ktwhm\" (UID: \"05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437\") " pod="openshift-cluster-node-tuning-operator/tuned-ktwhm"
Sep 18 16:57:25.143187 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.142663    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b6ec1dac-2d47-4156-bb74-b7055bedbc2b-cnibin\") pod \"multus-8rtmt\" (UID: \"b6ec1dac-2d47-4156-bb74-b7055bedbc2b\") " pod="openshift-multus/multus-8rtmt"
Sep 18 16:57:25.143187 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.142684    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b6ec1dac-2d47-4156-bb74-b7055bedbc2b-multus-daemon-config\") pod \"multus-8rtmt\" (UID: \"b6ec1dac-2d47-4156-bb74-b7055bedbc2b\") " pod="openshift-multus/multus-8rtmt"
Sep 18 16:57:25.243154 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.243113    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klxbq\" (UniqueName: \"kubernetes.io/projected/e1eb4986-e287-4a38-980c-3275bd8e2770-kube-api-access-klxbq\") pod \"iptables-alerter-mzj44\" (UID: \"e1eb4986-e287-4a38-980c-3275bd8e2770\") " pod="openshift-network-operator/iptables-alerter-mzj44"
Sep 18 16:57:25.243154 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.243154    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7cc9ec4e-c391-465a-b3e2-8478fa3940fb-device-dir\") pod \"aws-ebs-csi-driver-node-bwddj\" (UID: \"7cc9ec4e-c391-465a-b3e2-8478fa3940fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwddj"
Sep 18 16:57:25.243441 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.243172    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b6ec1dac-2d47-4156-bb74-b7055bedbc2b-host-var-lib-cni-bin\") pod \"multus-8rtmt\" (UID: \"b6ec1dac-2d47-4156-bb74-b7055bedbc2b\") " pod="openshift-multus/multus-8rtmt"
Sep 18 16:57:25.243441 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.243195    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7295391c-ebd9-4f37-9f70-a86b0bab3fcf-serviceca\") pod \"node-ca-sghwn\" (UID: \"7295391c-ebd9-4f37-9f70-a86b0bab3fcf\") " pod="openshift-image-registry/node-ca-sghwn"
Sep 18 16:57:25.243441 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.243264    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b6ec1dac-2d47-4156-bb74-b7055bedbc2b-host-var-lib-cni-bin\") pod \"multus-8rtmt\" (UID: \"b6ec1dac-2d47-4156-bb74-b7055bedbc2b\") " pod="openshift-multus/multus-8rtmt"
Sep 18 16:57:25.243441 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.243277    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7cc9ec4e-c391-465a-b3e2-8478fa3940fb-device-dir\") pod \"aws-ebs-csi-driver-node-bwddj\" (UID: \"7cc9ec4e-c391-465a-b3e2-8478fa3940fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwddj"
Sep 18 16:57:25.243441 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.243302    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e1eb4986-e287-4a38-980c-3275bd8e2770-iptables-alerter-script\") pod \"iptables-alerter-mzj44\" (UID: \"e1eb4986-e287-4a38-980c-3275bd8e2770\") " pod="openshift-network-operator/iptables-alerter-mzj44"
Sep 18 16:57:25.243441 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.243329    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d019d1b3-e6ee-4b27-9fc1-a6675650d42a-system-cni-dir\") pod \"multus-additional-cni-plugins-n4nk2\" (UID: \"d019d1b3-e6ee-4b27-9fc1-a6675650d42a\") " pod="openshift-multus/multus-additional-cni-plugins-n4nk2"
Sep 18 16:57:25.243441 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.243353    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47rhc\" (UniqueName: \"kubernetes.io/projected/89faf7f5-d633-4219-b3fc-323f8c23b599-kube-api-access-47rhc\") pod \"network-metrics-daemon-lpt5n\" (UID: \"89faf7f5-d633-4219-b3fc-323f8c23b599\") " pod="openshift-multus/network-metrics-daemon-lpt5n"
Sep 18 16:57:25.243441 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.243370    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437-etc-kubernetes\") pod \"tuned-ktwhm\" (UID: \"05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437\") " pod="openshift-cluster-node-tuning-operator/tuned-ktwhm"
Sep 18 16:57:25.243441 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.243392    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d019d1b3-e6ee-4b27-9fc1-a6675650d42a-system-cni-dir\") pod \"multus-additional-cni-plugins-n4nk2\" (UID: \"d019d1b3-e6ee-4b27-9fc1-a6675650d42a\") " pod="openshift-multus/multus-additional-cni-plugins-n4nk2"
Sep 18 16:57:25.243441 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.243413    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a4df48ac-dd04-4504-b828-66d7e2e986e1-dbus\") pod \"global-pull-secret-syncer-fx8nf\" (UID: \"a4df48ac-dd04-4504-b828-66d7e2e986e1\") " pod="kube-system/global-pull-secret-syncer-fx8nf"
Sep 18 16:57:25.243441 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.243436    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437-etc-kubernetes\") pod \"tuned-ktwhm\" (UID: \"05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437\") " pod="openshift-cluster-node-tuning-operator/tuned-ktwhm"
Sep 18 16:57:25.243441 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.243443    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6ec1dac-2d47-4156-bb74-b7055bedbc2b-host-var-lib-kubelet\") pod \"multus-8rtmt\" (UID: \"b6ec1dac-2d47-4156-bb74-b7055bedbc2b\") " pod="openshift-multus/multus-8rtmt"
Sep 18 16:57:25.244022 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.243468    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7cc9ec4e-c391-465a-b3e2-8478fa3940fb-etc-selinux\") pod \"aws-ebs-csi-driver-node-bwddj\" (UID: \"7cc9ec4e-c391-465a-b3e2-8478fa3940fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwddj"
Sep 18 16:57:25.244022 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.243493    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437-sys\") pod \"tuned-ktwhm\" (UID: \"05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437\") " pod="openshift-cluster-node-tuning-operator/tuned-ktwhm"
Sep 18 16:57:25.244022 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.243516    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b6ec1dac-2d47-4156-bb74-b7055bedbc2b-host-run-multus-certs\") pod \"multus-8rtmt\" (UID: \"b6ec1dac-2d47-4156-bb74-b7055bedbc2b\") " pod="openshift-multus/multus-8rtmt"
Sep 18 16:57:25.244022 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.243540    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6ec1dac-2d47-4156-bb74-b7055bedbc2b-host-var-lib-kubelet\") pod \"multus-8rtmt\" (UID: \"b6ec1dac-2d47-4156-bb74-b7055bedbc2b\") " pod="openshift-multus/multus-8rtmt"
Sep 18 16:57:25.244022 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.243559    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437-sys\") pod \"tuned-ktwhm\" (UID: \"05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437\") " pod="openshift-cluster-node-tuning-operator/tuned-ktwhm"
Sep 18 16:57:25.244022 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.243579    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b6ec1dac-2d47-4156-bb74-b7055bedbc2b-host-run-multus-certs\") pod \"multus-8rtmt\" (UID: \"b6ec1dac-2d47-4156-bb74-b7055bedbc2b\") " pod="openshift-multus/multus-8rtmt"
Sep 18 16:57:25.244022 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.243586    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a4df48ac-dd04-4504-b828-66d7e2e986e1-dbus\") pod \"global-pull-secret-syncer-fx8nf\" (UID: \"a4df48ac-dd04-4504-b828-66d7e2e986e1\") " pod="kube-system/global-pull-secret-syncer-fx8nf"
Sep 18 16:57:25.244022 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.243542    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d019d1b3-e6ee-4b27-9fc1-a6675650d42a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n4nk2\" (UID: \"d019d1b3-e6ee-4b27-9fc1-a6675650d42a\") " pod="openshift-multus/multus-additional-cni-plugins-n4nk2"
Sep 18 16:57:25.244022 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.243617    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zkw7\" (UniqueName: \"kubernetes.io/projected/05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437-kube-api-access-7zkw7\") pod \"tuned-ktwhm\" (UID: \"05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437\") " pod="openshift-cluster-node-tuning-operator/tuned-ktwhm"
Sep 18 16:57:25.244022 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.243620    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7cc9ec4e-c391-465a-b3e2-8478fa3940fb-etc-selinux\") pod \"aws-ebs-csi-driver-node-bwddj\" (UID: \"7cc9ec4e-c391-465a-b3e2-8478fa3940fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwddj"
Sep 18 16:57:25.244022 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.243641    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437-etc-systemd\") pod \"tuned-ktwhm\" (UID: \"05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437\") " pod="openshift-cluster-node-tuning-operator/tuned-ktwhm"
Sep 18 16:57:25.244022 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.243665    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hmqr\" (UniqueName: \"kubernetes.io/projected/b6ec1dac-2d47-4156-bb74-b7055bedbc2b-kube-api-access-4hmqr\") pod \"multus-8rtmt\" (UID: \"b6ec1dac-2d47-4156-bb74-b7055bedbc2b\") " pod="openshift-multus/multus-8rtmt"
Sep 18 16:57:25.244022 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.243688    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a4df48ac-dd04-4504-b828-66d7e2e986e1-original-pull-secret\") pod \"global-pull-secret-syncer-fx8nf\" (UID: \"a4df48ac-dd04-4504-b828-66d7e2e986e1\") " pod="kube-system/global-pull-secret-syncer-fx8nf"
Sep 18 16:57:25.244022 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.243701    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d019d1b3-e6ee-4b27-9fc1-a6675650d42a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n4nk2\" (UID: \"d019d1b3-e6ee-4b27-9fc1-a6675650d42a\") " pod="openshift-multus/multus-additional-cni-plugins-n4nk2"
Sep 18 16:57:25.244022 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.243711    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437-host\") pod \"tuned-ktwhm\" (UID: \"05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437\") " pod="openshift-cluster-node-tuning-operator/tuned-ktwhm"
Sep 18 16:57:25.244022 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.243732    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437-etc-systemd\") pod \"tuned-ktwhm\" (UID: \"05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437\") " pod="openshift-cluster-node-tuning-operator/tuned-ktwhm"
Sep 18 16:57:25.244022 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.243746    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b6ec1dac-2d47-4156-bb74-b7055bedbc2b-etc-kubernetes\") pod \"multus-8rtmt\" (UID: \"b6ec1dac-2d47-4156-bb74-b7055bedbc2b\") " pod="openshift-multus/multus-8rtmt"
Sep 18 16:57:25.244022 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.243749    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437-host\") pod \"tuned-ktwhm\" (UID: \"05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437\") " pod="openshift-cluster-node-tuning-operator/tuned-ktwhm"
Sep 18 16:57:25.244901 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.243733    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7295391c-ebd9-4f37-9f70-a86b0bab3fcf-serviceca\") pod \"node-ca-sghwn\" (UID: \"7295391c-ebd9-4f37-9f70-a86b0bab3fcf\") " pod="openshift-image-registry/node-ca-sghwn"
Sep 18 16:57:25.244901 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.243790    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a4df48ac-dd04-4504-b828-66d7e2e986e1-kubelet-config\") pod \"global-pull-secret-syncer-fx8nf\" (UID: \"a4df48ac-dd04-4504-b828-66d7e2e986e1\") " pod="kube-system/global-pull-secret-syncer-fx8nf"
Sep 18 16:57:25.244901 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.243808    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b6ec1dac-2d47-4156-bb74-b7055bedbc2b-etc-kubernetes\") pod \"multus-8rtmt\" (UID: \"b6ec1dac-2d47-4156-bb74-b7055bedbc2b\") " pod="openshift-multus/multus-8rtmt"
Sep 18 16:57:25.244901 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.243815    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e1eb4986-e287-4a38-980c-3275bd8e2770-host-slash\") pod \"iptables-alerter-mzj44\" (UID: \"e1eb4986-e287-4a38-980c-3275bd8e2770\") " pod="openshift-network-operator/iptables-alerter-mzj44"
Sep 18 16:57:25.244901 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.243840    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e1eb4986-e287-4a38-980c-3275bd8e2770-host-slash\") pod \"iptables-alerter-mzj44\" (UID: \"e1eb4986-e287-4a38-980c-3275bd8e2770\") " pod="openshift-network-operator/iptables-alerter-mzj44"
Sep 18 16:57:25.244901 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:25.243839    2881 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 16:57:25.244901 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.243860    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a4df48ac-dd04-4504-b828-66d7e2e986e1-kubelet-config\") pod \"global-pull-secret-syncer-fx8nf\" (UID: \"a4df48ac-dd04-4504-b828-66d7e2e986e1\") " pod="kube-system/global-pull-secret-syncer-fx8nf"
Sep 18 16:57:25.244901 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.243864    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d019d1b3-e6ee-4b27-9fc1-a6675650d42a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n4nk2\" (UID: \"d019d1b3-e6ee-4b27-9fc1-a6675650d42a\") " pod="openshift-multus/multus-additional-cni-plugins-n4nk2"
Sep 18 16:57:25.244901 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:25.243902    2881 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4df48ac-dd04-4504-b828-66d7e2e986e1-original-pull-secret podName:a4df48ac-dd04-4504-b828-66d7e2e986e1 nodeName:}" failed. No retries permitted until 2026-09-18 16:57:25.743888095 +0000 UTC m=+235.943743363 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a4df48ac-dd04-4504-b828-66d7e2e986e1-original-pull-secret") pod "global-pull-secret-syncer-fx8nf" (UID: "a4df48ac-dd04-4504-b828-66d7e2e986e1") : object "kube-system"/"original-pull-secret" not registered
Sep 18 16:57:25.244901 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.243916    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e1eb4986-e287-4a38-980c-3275bd8e2770-iptables-alerter-script\") pod \"iptables-alerter-mzj44\" (UID: \"e1eb4986-e287-4a38-980c-3275bd8e2770\") " pod="openshift-network-operator/iptables-alerter-mzj44"
Sep 18 16:57:25.244901 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.243926    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437-etc-modprobe-d\") pod \"tuned-ktwhm\" (UID: \"05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437\") " pod="openshift-cluster-node-tuning-operator/tuned-ktwhm"
Sep 18 16:57:25.244901 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.243980    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437-lib-modules\") pod \"tuned-ktwhm\" (UID: \"05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437\") " pod="openshift-cluster-node-tuning-operator/tuned-ktwhm"
Sep 18 16:57:25.244901 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.244011    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7cc9ec4e-c391-465a-b3e2-8478fa3940fb-socket-dir\") pod \"aws-ebs-csi-driver-node-bwddj\" (UID: \"7cc9ec4e-c391-465a-b3e2-8478fa3940fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwddj"
Sep 18 16:57:25.244901 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.244038    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpdk8\" (UniqueName: \"kubernetes.io/projected/7cc9ec4e-c391-465a-b3e2-8478fa3940fb-kube-api-access-dpdk8\") pod \"aws-ebs-csi-driver-node-bwddj\" (UID: \"7cc9ec4e-c391-465a-b3e2-8478fa3940fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwddj"
Sep 18 16:57:25.244901 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.244047    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437-etc-modprobe-d\") pod \"tuned-ktwhm\" (UID: \"05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437\") " pod="openshift-cluster-node-tuning-operator/tuned-ktwhm"
Sep 18 16:57:25.244901 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.244061    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89faf7f5-d633-4219-b3fc-323f8c23b599-metrics-certs\") pod \"network-metrics-daemon-lpt5n\" (UID: \"89faf7f5-d633-4219-b3fc-323f8c23b599\") " pod="openshift-multus/network-metrics-daemon-lpt5n"
Sep 18 16:57:25.244901 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.244078    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b6ec1dac-2d47-4156-bb74-b7055bedbc2b-cni-binary-copy\") pod \"multus-8rtmt\" (UID: \"b6ec1dac-2d47-4156-bb74-b7055bedbc2b\") " pod="openshift-multus/multus-8rtmt"
Sep 18 16:57:25.245644 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.244097    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b6ec1dac-2d47-4156-bb74-b7055bedbc2b-host-run-k8s-cni-cncf-io\") pod \"multus-8rtmt\" (UID: \"b6ec1dac-2d47-4156-bb74-b7055bedbc2b\") " pod="openshift-multus/multus-8rtmt"
Sep 18 16:57:25.245644 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.244096    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437-lib-modules\") pod \"tuned-ktwhm\" (UID: \"05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437\") " pod="openshift-cluster-node-tuning-operator/tuned-ktwhm"
Sep 18 16:57:25.245644 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.244133    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b6ec1dac-2d47-4156-bb74-b7055bedbc2b-host-run-k8s-cni-cncf-io\") pod \"multus-8rtmt\" (UID: \"b6ec1dac-2d47-4156-bb74-b7055bedbc2b\") " pod="openshift-multus/multus-8rtmt"
Sep 18 16:57:25.245644 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:25.244151    2881 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:57:25.245644 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.244162    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437-tmp\") pod \"tuned-ktwhm\" (UID: \"05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437\") " pod="openshift-cluster-node-tuning-operator/tuned-ktwhm"
Sep 18 16:57:25.245644 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:25.244196    2881 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89faf7f5-d633-4219-b3fc-323f8c23b599-metrics-certs podName:89faf7f5-d633-4219-b3fc-323f8c23b599 nodeName:}" failed. No retries permitted until 2026-09-18 16:57:25.744182243 +0000 UTC m=+235.944037517 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/89faf7f5-d633-4219-b3fc-323f8c23b599-metrics-certs") pod "network-metrics-daemon-lpt5n" (UID: "89faf7f5-d633-4219-b3fc-323f8c23b599") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:57:25.245644 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.244214    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b6ec1dac-2d47-4156-bb74-b7055bedbc2b-cnibin\") pod \"multus-8rtmt\" (UID: \"b6ec1dac-2d47-4156-bb74-b7055bedbc2b\") " pod="openshift-multus/multus-8rtmt"
Sep 18 16:57:25.245644 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.244259    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b6ec1dac-2d47-4156-bb74-b7055bedbc2b-multus-daemon-config\") pod \"multus-8rtmt\" (UID: \"b6ec1dac-2d47-4156-bb74-b7055bedbc2b\") " pod="openshift-multus/multus-8rtmt"
Sep 18 16:57:25.245644 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.244271    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7cc9ec4e-c391-465a-b3e2-8478fa3940fb-socket-dir\") pod \"aws-ebs-csi-driver-node-bwddj\" (UID: \"7cc9ec4e-c391-465a-b3e2-8478fa3940fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwddj"
Sep 18 16:57:25.245644 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.244285    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b6ec1dac-2d47-4156-bb74-b7055bedbc2b-hostroot\") pod \"multus-8rtmt\" (UID: \"b6ec1dac-2d47-4156-bb74-b7055bedbc2b\") " pod="openshift-multus/multus-8rtmt"
Sep 18 16:57:25.245644 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.244311    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc9qn\" (UniqueName: \"kubernetes.io/projected/7295391c-ebd9-4f37-9f70-a86b0bab3fcf-kube-api-access-kc9qn\") pod \"node-ca-sghwn\" (UID: \"7295391c-ebd9-4f37-9f70-a86b0bab3fcf\") " pod="openshift-image-registry/node-ca-sghwn"
Sep 18 16:57:25.245644 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.244314    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b6ec1dac-2d47-4156-bb74-b7055bedbc2b-cnibin\") pod \"multus-8rtmt\" (UID: \"b6ec1dac-2d47-4156-bb74-b7055bedbc2b\") " pod="openshift-multus/multus-8rtmt"
Sep 18 16:57:25.245644 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.244338    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fb327355-e5c6-48fd-abf3-24fa50a8e0b5-agent-certs\") pod \"konnectivity-agent-g7d7f\" (UID: \"fb327355-e5c6-48fd-abf3-24fa50a8e0b5\") " pod="kube-system/konnectivity-agent-g7d7f"
Sep 18 16:57:25.245644 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.244348    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b6ec1dac-2d47-4156-bb74-b7055bedbc2b-hostroot\") pod \"multus-8rtmt\" (UID: \"b6ec1dac-2d47-4156-bb74-b7055bedbc2b\") " pod="openshift-multus/multus-8rtmt"
Sep 18 16:57:25.245644 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.244364    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b2r6\" (UniqueName: \"kubernetes.io/projected/d019d1b3-e6ee-4b27-9fc1-a6675650d42a-kube-api-access-8b2r6\") pod \"multus-additional-cni-plugins-n4nk2\" (UID: \"d019d1b3-e6ee-4b27-9fc1-a6675650d42a\") " pod="openshift-multus/multus-additional-cni-plugins-n4nk2"
Sep 18 16:57:25.245644 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.244389    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437-etc-sysctl-conf\") pod \"tuned-ktwhm\" (UID: \"05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437\") " pod="openshift-cluster-node-tuning-operator/tuned-ktwhm"
Sep 18 16:57:25.245644 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.244413    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437-var-lib-kubelet\") pod \"tuned-ktwhm\" (UID: \"05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437\") " pod="openshift-cluster-node-tuning-operator/tuned-ktwhm"
Sep 18 16:57:25.246182 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.244436    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437-etc-sysconfig\") pod \"tuned-ktwhm\" (UID: \"05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437\") " pod="openshift-cluster-node-tuning-operator/tuned-ktwhm"
Sep 18 16:57:25.246182 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.244464    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7cc9ec4e-c391-465a-b3e2-8478fa3940fb-registration-dir\") pod \"aws-ebs-csi-driver-node-bwddj\" (UID: \"7cc9ec4e-c391-465a-b3e2-8478fa3940fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwddj"
Sep 18 16:57:25.246182 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.244488    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fb327355-e5c6-48fd-abf3-24fa50a8e0b5-konnectivity-ca\") pod \"konnectivity-agent-g7d7f\" (UID: \"fb327355-e5c6-48fd-abf3-24fa50a8e0b5\") " pod="kube-system/konnectivity-agent-g7d7f"
Sep 18 16:57:25.246182 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.244492    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d019d1b3-e6ee-4b27-9fc1-a6675650d42a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n4nk2\" (UID: \"d019d1b3-e6ee-4b27-9fc1-a6675650d42a\") " pod="openshift-multus/multus-additional-cni-plugins-n4nk2"
Sep 18 16:57:25.246182 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.244505    2881 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 18 16:57:25.246182 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.244540    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437-var-lib-kubelet\") pod \"tuned-ktwhm\" (UID: \"05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437\") " pod="openshift-cluster-node-tuning-operator/tuned-ktwhm"
Sep 18 16:57:25.246182 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.244558    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6ec1dac-2d47-4156-bb74-b7055bedbc2b-system-cni-dir\") pod \"multus-8rtmt\" (UID: \"b6ec1dac-2d47-4156-bb74-b7055bedbc2b\") " pod="openshift-multus/multus-8rtmt"
Sep 18 16:57:25.246182 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.244605    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437-etc-sysconfig\") pod \"tuned-ktwhm\" (UID: \"05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437\") " pod="openshift-cluster-node-tuning-operator/tuned-ktwhm"
Sep 18 16:57:25.246182 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.244651    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7cc9ec4e-c391-465a-b3e2-8478fa3940fb-registration-dir\") pod \"aws-ebs-csi-driver-node-bwddj\" (UID: \"7cc9ec4e-c391-465a-b3e2-8478fa3940fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwddj"
Sep 18 16:57:25.246182 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.244657    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437-etc-sysctl-conf\") pod \"tuned-ktwhm\" (UID: \"05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437\") " pod="openshift-cluster-node-tuning-operator/tuned-ktwhm"
Sep 18 16:57:25.246182 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.244511    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6ec1dac-2d47-4156-bb74-b7055bedbc2b-system-cni-dir\") pod \"multus-8rtmt\" (UID: \"b6ec1dac-2d47-4156-bb74-b7055bedbc2b\") " pod="openshift-multus/multus-8rtmt"
Sep 18 16:57:25.246182 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.244699    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6ec1dac-2d47-4156-bb74-b7055bedbc2b-multus-cni-dir\") pod \"multus-8rtmt\" (UID: \"b6ec1dac-2d47-4156-bb74-b7055bedbc2b\") " pod="openshift-multus/multus-8rtmt"
Sep 18 16:57:25.246182 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.244727    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b6ec1dac-2d47-4156-bb74-b7055bedbc2b-multus-daemon-config\") pod \"multus-8rtmt\" (UID: \"b6ec1dac-2d47-4156-bb74-b7055bedbc2b\") " pod="openshift-multus/multus-8rtmt"
Sep 18 16:57:25.246182 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.244742    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b6ec1dac-2d47-4156-bb74-b7055bedbc2b-host-run-netns\") pod \"multus-8rtmt\" (UID: \"b6ec1dac-2d47-4156-bb74-b7055bedbc2b\") " pod="openshift-multus/multus-8rtmt"
Sep 18 16:57:25.246182 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.244774    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d019d1b3-e6ee-4b27-9fc1-a6675650d42a-cni-binary-copy\") pod \"multus-additional-cni-plugins-n4nk2\" (UID: \"d019d1b3-e6ee-4b27-9fc1-a6675650d42a\") " pod="openshift-multus/multus-additional-cni-plugins-n4nk2"
Sep 18 16:57:25.246182 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.244777    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6ec1dac-2d47-4156-bb74-b7055bedbc2b-multus-cni-dir\") pod \"multus-8rtmt\" (UID: \"b6ec1dac-2d47-4156-bb74-b7055bedbc2b\") " pod="openshift-multus/multus-8rtmt"
Sep 18 16:57:25.246182 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.244813    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437-etc-sysctl-d\") pod \"tuned-ktwhm\" (UID: \"05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437\") " pod="openshift-cluster-node-tuning-operator/tuned-ktwhm"
Sep 18 16:57:25.246182 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.244839    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437-run\") pod \"tuned-ktwhm\" (UID: \"05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437\") " pod="openshift-cluster-node-tuning-operator/tuned-ktwhm"
Sep 18 16:57:25.246710 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.244867    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b6ec1dac-2d47-4156-bb74-b7055bedbc2b-host-var-lib-cni-multus\") pod \"multus-8rtmt\" (UID: \"b6ec1dac-2d47-4156-bb74-b7055bedbc2b\") " pod="openshift-multus/multus-8rtmt"
Sep 18 16:57:25.246710 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.244871    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b6ec1dac-2d47-4156-bb74-b7055bedbc2b-host-run-netns\") pod \"multus-8rtmt\" (UID: \"b6ec1dac-2d47-4156-bb74-b7055bedbc2b\") " pod="openshift-multus/multus-8rtmt"
Sep 18 16:57:25.246710 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.244892    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b6ec1dac-2d47-4156-bb74-b7055bedbc2b-multus-conf-dir\") pod \"multus-8rtmt\" (UID: \"b6ec1dac-2d47-4156-bb74-b7055bedbc2b\") " pod="openshift-multus/multus-8rtmt"
Sep 18 16:57:25.246710 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.244917    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7cc9ec4e-c391-465a-b3e2-8478fa3940fb-sys-fs\") pod \"aws-ebs-csi-driver-node-bwddj\" (UID: \"7cc9ec4e-c391-465a-b3e2-8478fa3940fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwddj"
Sep 18 16:57:25.246710 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.244941    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d019d1b3-e6ee-4b27-9fc1-a6675650d42a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n4nk2\" (UID: \"d019d1b3-e6ee-4b27-9fc1-a6675650d42a\") " pod="openshift-multus/multus-additional-cni-plugins-n4nk2"
Sep 18 16:57:25.246710 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.244946    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437-run\") pod \"tuned-ktwhm\" (UID: \"05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437\") " pod="openshift-cluster-node-tuning-operator/tuned-ktwhm"
Sep 18 16:57:25.246710 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.244968    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhsfc\" (UniqueName: \"kubernetes.io/projected/6f463717-a487-485b-aa4b-7772d35b0d22-kube-api-access-rhsfc\") pod \"network-check-target-pc44x\" (UID: \"6f463717-a487-485b-aa4b-7772d35b0d22\") " pod="openshift-network-diagnostics/network-check-target-pc44x"
Sep 18 16:57:25.246710 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.244992    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d019d1b3-e6ee-4b27-9fc1-a6675650d42a-cnibin\") pod \"multus-additional-cni-plugins-n4nk2\" (UID: \"d019d1b3-e6ee-4b27-9fc1-a6675650d42a\") " pod="openshift-multus/multus-additional-cni-plugins-n4nk2"
Sep 18 16:57:25.246710 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.245016    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7cc9ec4e-c391-465a-b3e2-8478fa3940fb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bwddj\" (UID: \"7cc9ec4e-c391-465a-b3e2-8478fa3940fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwddj"
Sep 18 16:57:25.246710 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.245039    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437-etc-tuned\") pod \"tuned-ktwhm\" (UID: \"05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437\") " pod="openshift-cluster-node-tuning-operator/tuned-ktwhm"
Sep 18 16:57:25.246710 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.245054    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437-etc-sysctl-d\") pod \"tuned-ktwhm\" (UID: \"05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437\") " pod="openshift-cluster-node-tuning-operator/tuned-ktwhm"
Sep 18 16:57:25.246710 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.245062    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b6ec1dac-2d47-4156-bb74-b7055bedbc2b-cni-binary-copy\") pod \"multus-8rtmt\" (UID: \"b6ec1dac-2d47-4156-bb74-b7055bedbc2b\") " pod="openshift-multus/multus-8rtmt"
Sep 18 16:57:25.246710 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.245130    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7cc9ec4e-c391-465a-b3e2-8478fa3940fb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bwddj\" (UID: \"7cc9ec4e-c391-465a-b3e2-8478fa3940fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwddj"
Sep 18 16:57:25.246710 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.245163    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fb327355-e5c6-48fd-abf3-24fa50a8e0b5-konnectivity-ca\") pod \"konnectivity-agent-g7d7f\" (UID: \"fb327355-e5c6-48fd-abf3-24fa50a8e0b5\") " pod="kube-system/konnectivity-agent-g7d7f"
Sep 18 16:57:25.246710 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.245064    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b6ec1dac-2d47-4156-bb74-b7055bedbc2b-multus-socket-dir-parent\") pod \"multus-8rtmt\" (UID: \"b6ec1dac-2d47-4156-bb74-b7055bedbc2b\") " pod="openshift-multus/multus-8rtmt"
Sep 18 16:57:25.246710 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.245206    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d019d1b3-e6ee-4b27-9fc1-a6675650d42a-cni-binary-copy\") pod \"multus-additional-cni-plugins-n4nk2\" (UID: \"d019d1b3-e6ee-4b27-9fc1-a6675650d42a\") " pod="openshift-multus/multus-additional-cni-plugins-n4nk2"
Sep 18 16:57:25.246710 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.245213    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7295391c-ebd9-4f37-9f70-a86b0bab3fcf-host\") pod \"node-ca-sghwn\" (UID: \"7295391c-ebd9-4f37-9f70-a86b0bab3fcf\") " pod="openshift-image-registry/node-ca-sghwn"
Sep 18 16:57:25.247174 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.245305    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7295391c-ebd9-4f37-9f70-a86b0bab3fcf-host\") pod \"node-ca-sghwn\" (UID: \"7295391c-ebd9-4f37-9f70-a86b0bab3fcf\") " pod="openshift-image-registry/node-ca-sghwn"
Sep 18 16:57:25.247174 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.245342    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b6ec1dac-2d47-4156-bb74-b7055bedbc2b-host-var-lib-cni-multus\") pod \"multus-8rtmt\" (UID: \"b6ec1dac-2d47-4156-bb74-b7055bedbc2b\") " pod="openshift-multus/multus-8rtmt"
Sep 18 16:57:25.247174 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.245347    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b6ec1dac-2d47-4156-bb74-b7055bedbc2b-multus-conf-dir\") pod \"multus-8rtmt\" (UID: \"b6ec1dac-2d47-4156-bb74-b7055bedbc2b\") " pod="openshift-multus/multus-8rtmt"
Sep 18 16:57:25.247174 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.245306    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7cc9ec4e-c391-465a-b3e2-8478fa3940fb-sys-fs\") pod \"aws-ebs-csi-driver-node-bwddj\" (UID: \"7cc9ec4e-c391-465a-b3e2-8478fa3940fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwddj"
Sep 18 16:57:25.247174 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.245362    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d019d1b3-e6ee-4b27-9fc1-a6675650d42a-cnibin\") pod \"multus-additional-cni-plugins-n4nk2\" (UID: \"d019d1b3-e6ee-4b27-9fc1-a6675650d42a\") " pod="openshift-multus/multus-additional-cni-plugins-n4nk2"
Sep 18 16:57:25.247174 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.245395    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b6ec1dac-2d47-4156-bb74-b7055bedbc2b-multus-socket-dir-parent\") pod \"multus-8rtmt\" (UID: \"b6ec1dac-2d47-4156-bb74-b7055bedbc2b\") " pod="openshift-multus/multus-8rtmt"
Sep 18 16:57:25.247174 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.245503    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d019d1b3-e6ee-4b27-9fc1-a6675650d42a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n4nk2\" (UID: \"d019d1b3-e6ee-4b27-9fc1-a6675650d42a\") " pod="openshift-multus/multus-additional-cni-plugins-n4nk2"
Sep 18 16:57:25.247747 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.247726    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437-etc-tuned\") pod \"tuned-ktwhm\" (UID: \"05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437\") " pod="openshift-cluster-node-tuning-operator/tuned-ktwhm"
Sep 18 16:57:25.247831 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.247791    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437-tmp\") pod \"tuned-ktwhm\" (UID: \"05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437\") " pod="openshift-cluster-node-tuning-operator/tuned-ktwhm"
Sep 18 16:57:25.248056 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.248039    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fb327355-e5c6-48fd-abf3-24fa50a8e0b5-agent-certs\") pod \"konnectivity-agent-g7d7f\" (UID: \"fb327355-e5c6-48fd-abf3-24fa50a8e0b5\") " pod="kube-system/konnectivity-agent-g7d7f"
Sep 18 16:57:25.254497 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.254468    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hmqr\" (UniqueName: \"kubernetes.io/projected/b6ec1dac-2d47-4156-bb74-b7055bedbc2b-kube-api-access-4hmqr\") pod \"multus-8rtmt\" (UID: \"b6ec1dac-2d47-4156-bb74-b7055bedbc2b\") " pod="openshift-multus/multus-8rtmt"
Sep 18 16:57:25.254674 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.254574    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klxbq\" (UniqueName: \"kubernetes.io/projected/e1eb4986-e287-4a38-980c-3275bd8e2770-kube-api-access-klxbq\") pod \"iptables-alerter-mzj44\" (UID: \"e1eb4986-e287-4a38-980c-3275bd8e2770\") " pod="openshift-network-operator/iptables-alerter-mzj44"
Sep 18 16:57:25.254792 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.254770    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zkw7\" (UniqueName: \"kubernetes.io/projected/05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437-kube-api-access-7zkw7\") pod \"tuned-ktwhm\" (UID: \"05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437\") " pod="openshift-cluster-node-tuning-operator/tuned-ktwhm"
Sep 18 16:57:25.255011 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.254995    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47rhc\" (UniqueName: \"kubernetes.io/projected/89faf7f5-d633-4219-b3fc-323f8c23b599-kube-api-access-47rhc\") pod \"network-metrics-daemon-lpt5n\" (UID: \"89faf7f5-d633-4219-b3fc-323f8c23b599\") " pod="openshift-multus/network-metrics-daemon-lpt5n"
Sep 18 16:57:25.255689 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:25.255672    2881 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 16:57:25.255689 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:25.255690    2881 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 16:57:25.255841 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:25.255704    2881 projected.go:196] Error preparing data for projected volume kube-api-access-rhsfc for pod openshift-network-diagnostics/network-check-target-pc44x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:57:25.255841 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:25.255781    2881 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f463717-a487-485b-aa4b-7772d35b0d22-kube-api-access-rhsfc podName:6f463717-a487-485b-aa4b-7772d35b0d22 nodeName:}" failed. No retries permitted until 2026-09-18 16:57:25.755752997 +0000 UTC m=+235.955608267 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-rhsfc" (UniqueName: "kubernetes.io/projected/6f463717-a487-485b-aa4b-7772d35b0d22-kube-api-access-rhsfc") pod "network-check-target-pc44x" (UID: "6f463717-a487-485b-aa4b-7772d35b0d22") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:57:25.257768 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.257746    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc9qn\" (UniqueName: \"kubernetes.io/projected/7295391c-ebd9-4f37-9f70-a86b0bab3fcf-kube-api-access-kc9qn\") pod \"node-ca-sghwn\" (UID: \"7295391c-ebd9-4f37-9f70-a86b0bab3fcf\") " pod="openshift-image-registry/node-ca-sghwn"
Sep 18 16:57:25.257884 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.257782    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b2r6\" (UniqueName: \"kubernetes.io/projected/d019d1b3-e6ee-4b27-9fc1-a6675650d42a-kube-api-access-8b2r6\") pod \"multus-additional-cni-plugins-n4nk2\" (UID: \"d019d1b3-e6ee-4b27-9fc1-a6675650d42a\") " pod="openshift-multus/multus-additional-cni-plugins-n4nk2"
Sep 18 16:57:25.260745 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.260721    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpdk8\" (UniqueName: \"kubernetes.io/projected/7cc9ec4e-c391-465a-b3e2-8478fa3940fb-kube-api-access-dpdk8\") pod \"aws-ebs-csi-driver-node-bwddj\" (UID: \"7cc9ec4e-c391-465a-b3e2-8478fa3940fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwddj"
Sep 18 16:57:25.289505 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.289468    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-97wk9"
Sep 18 16:57:25.289978 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:25.289949    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" podUID="a3127caef50db0be093e96af172c433b"
Sep 18 16:57:25.306128 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.306096    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-n4nk2"
Sep 18 16:57:25.312627 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.312611    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ktwhm"
Sep 18 16:57:25.318275 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.318199    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8rtmt"
Sep 18 16:57:25.321254 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.321197    2881 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 16:57:25.324280 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:57:25.324246    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05b58fce_4a3d_47c1_b9cd_fe9dbc9bf437.slice/crio-0365412ec13d8659b13a3e4a1cbfe125cf732b314119bf222a884e1b4528d266 WatchSource:0}: Error finding container 0365412ec13d8659b13a3e4a1cbfe125cf732b314119bf222a884e1b4528d266: Status 404 returned error can't find the container with id 0365412ec13d8659b13a3e4a1cbfe125cf732b314119bf222a884e1b4528d266
Sep 18 16:57:25.329519 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.329494    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-g7d7f"
Sep 18 16:57:25.336280 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.336257    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwddj"
Sep 18 16:57:25.338325 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:57:25.338301    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb327355_e5c6_48fd_abf3_24fa50a8e0b5.slice/crio-0b00c4bc5354293e14ba49ef8b700698a766f3dba2f390e6a3d81a4faf788853 WatchSource:0}: Error finding container 0b00c4bc5354293e14ba49ef8b700698a766f3dba2f390e6a3d81a4faf788853: Status 404 returned error can't find the container with id 0b00c4bc5354293e14ba49ef8b700698a766f3dba2f390e6a3d81a4faf788853
Sep 18 16:57:25.341831 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.341804    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-sghwn"
Sep 18 16:57:25.343046 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.343023    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-97wk9"]
Sep 18 16:57:25.344967 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:57:25.344792    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cc9ec4e_c391_465a_b3e2_8478fa3940fb.slice/crio-330c8da92f25fc0404208296a3b060693435347d92a665fb7757eb30669fadfc WatchSource:0}: Error finding container 330c8da92f25fc0404208296a3b060693435347d92a665fb7757eb30669fadfc: Status 404 returned error can't find the container with id 330c8da92f25fc0404208296a3b060693435347d92a665fb7757eb30669fadfc
Sep 18 16:57:25.347029 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.347006    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mzj44"
Sep 18 16:57:25.350682 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:57:25.350656    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7295391c_ebd9_4f37_9f70_a86b0bab3fcf.slice/crio-fc2d30247d299fe4883e81434eae301608b289e1ce58fe6256d8ae3d2b96c0d0 WatchSource:0}: Error finding container fc2d30247d299fe4883e81434eae301608b289e1ce58fe6256d8ae3d2b96c0d0: Status 404 returned error can't find the container with id fc2d30247d299fe4883e81434eae301608b289e1ce58fe6256d8ae3d2b96c0d0
Sep 18 16:57:25.353535 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.353513    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-7dglz"]
Sep 18 16:57:25.356776 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:57:25.356746    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1eb4986_e287_4a38_980c_3275bd8e2770.slice/crio-7540a1080c9797168621a4fa16a380f92b099881ebdb81b06f02718f72162572 WatchSource:0}: Error finding container 7540a1080c9797168621a4fa16a380f92b099881ebdb81b06f02718f72162572: Status 404 returned error can't find the container with id 7540a1080c9797168621a4fa16a380f92b099881ebdb81b06f02718f72162572
Sep 18 16:57:25.356869 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.356855    2881 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-97wk9"]
Sep 18 16:57:25.356986 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.356970    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7dglz"
Sep 18 16:57:25.360065 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.360042    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 18 16:57:25.360151 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.360089    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 18 16:57:25.360207 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.360158    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 18 16:57:25.360604 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.360431    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 18 16:57:25.360604 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.360508    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 18 16:57:25.361313 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.361294    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-brbn9\""
Sep 18 16:57:25.362030 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.362015    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 18 16:57:25.445805 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.445706    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a03f2d01-68c8-4498-927e-c8722e1829b9-etc-openvswitch\") pod \"ovnkube-node-7dglz\" (UID: \"a03f2d01-68c8-4498-927e-c8722e1829b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7dglz"
Sep 18 16:57:25.445805 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.445745    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a03f2d01-68c8-4498-927e-c8722e1829b9-ovnkube-config\") pod \"ovnkube-node-7dglz\" (UID: \"a03f2d01-68c8-4498-927e-c8722e1829b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7dglz"
Sep 18 16:57:25.445805 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.445777    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a03f2d01-68c8-4498-927e-c8722e1829b9-run-systemd\") pod \"ovnkube-node-7dglz\" (UID: \"a03f2d01-68c8-4498-927e-c8722e1829b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7dglz"
Sep 18 16:57:25.445805 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.445793    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a03f2d01-68c8-4498-927e-c8722e1829b9-host-run-ovn-kubernetes\") pod \"ovnkube-node-7dglz\" (UID: \"a03f2d01-68c8-4498-927e-c8722e1829b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7dglz"
Sep 18 16:57:25.445805 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.445810    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a03f2d01-68c8-4498-927e-c8722e1829b9-ovn-node-metrics-cert\") pod \"ovnkube-node-7dglz\" (UID: \"a03f2d01-68c8-4498-927e-c8722e1829b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7dglz"
Sep 18 16:57:25.446126 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.445832    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a03f2d01-68c8-4498-927e-c8722e1829b9-env-overrides\") pod \"ovnkube-node-7dglz\" (UID: \"a03f2d01-68c8-4498-927e-c8722e1829b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7dglz"
Sep 18 16:57:25.446126 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.445854    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a03f2d01-68c8-4498-927e-c8722e1829b9-var-lib-openvswitch\") pod \"ovnkube-node-7dglz\" (UID: \"a03f2d01-68c8-4498-927e-c8722e1829b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7dglz"
Sep 18 16:57:25.446126 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.445879    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a03f2d01-68c8-4498-927e-c8722e1829b9-host-slash\") pod \"ovnkube-node-7dglz\" (UID: \"a03f2d01-68c8-4498-927e-c8722e1829b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7dglz"
Sep 18 16:57:25.446126 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.445937    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a03f2d01-68c8-4498-927e-c8722e1829b9-systemd-units\") pod \"ovnkube-node-7dglz\" (UID: \"a03f2d01-68c8-4498-927e-c8722e1829b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7dglz"
Sep 18 16:57:25.446126 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.445987    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a03f2d01-68c8-4498-927e-c8722e1829b9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7dglz\" (UID: \"a03f2d01-68c8-4498-927e-c8722e1829b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7dglz"
Sep 18 16:57:25.446126 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.446016    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a03f2d01-68c8-4498-927e-c8722e1829b9-ovnkube-script-lib\") pod \"ovnkube-node-7dglz\" (UID: \"a03f2d01-68c8-4498-927e-c8722e1829b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7dglz"
Sep 18 16:57:25.446126 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.446033    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6xtj\" (UniqueName: \"kubernetes.io/projected/a03f2d01-68c8-4498-927e-c8722e1829b9-kube-api-access-m6xtj\") pod \"ovnkube-node-7dglz\" (UID: \"a03f2d01-68c8-4498-927e-c8722e1829b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7dglz"
Sep 18 16:57:25.446126 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.446091    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a03f2d01-68c8-4498-927e-c8722e1829b9-host-run-netns\") pod \"ovnkube-node-7dglz\" (UID: \"a03f2d01-68c8-4498-927e-c8722e1829b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7dglz"
Sep 18 16:57:25.446126 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.446120    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a03f2d01-68c8-4498-927e-c8722e1829b9-run-ovn\") pod \"ovnkube-node-7dglz\" (UID: \"a03f2d01-68c8-4498-927e-c8722e1829b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7dglz"
Sep 18 16:57:25.446436 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.446144    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a03f2d01-68c8-4498-927e-c8722e1829b9-host-cni-netd\") pod \"ovnkube-node-7dglz\" (UID: \"a03f2d01-68c8-4498-927e-c8722e1829b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7dglz"
Sep 18 16:57:25.446436 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.446169    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a03f2d01-68c8-4498-927e-c8722e1829b9-run-openvswitch\") pod \"ovnkube-node-7dglz\" (UID: \"a03f2d01-68c8-4498-927e-c8722e1829b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7dglz"
Sep 18 16:57:25.446436 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.446186    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a03f2d01-68c8-4498-927e-c8722e1829b9-host-kubelet\") pod \"ovnkube-node-7dglz\" (UID: \"a03f2d01-68c8-4498-927e-c8722e1829b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7dglz"
Sep 18 16:57:25.446436 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.446202    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a03f2d01-68c8-4498-927e-c8722e1829b9-node-log\") pod \"ovnkube-node-7dglz\" (UID: \"a03f2d01-68c8-4498-927e-c8722e1829b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7dglz"
Sep 18 16:57:25.446436 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.446217    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a03f2d01-68c8-4498-927e-c8722e1829b9-host-cni-bin\") pod \"ovnkube-node-7dglz\" (UID: \"a03f2d01-68c8-4498-927e-c8722e1829b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7dglz"
Sep 18 16:57:25.446436 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.446257    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a03f2d01-68c8-4498-927e-c8722e1829b9-log-socket\") pod \"ovnkube-node-7dglz\" (UID: \"a03f2d01-68c8-4498-927e-c8722e1829b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7dglz"
Sep 18 16:57:25.547584 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.547533    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a03f2d01-68c8-4498-927e-c8722e1829b9-log-socket\") pod \"ovnkube-node-7dglz\" (UID: \"a03f2d01-68c8-4498-927e-c8722e1829b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7dglz"
Sep 18 16:57:25.547794 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.547582    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a03f2d01-68c8-4498-927e-c8722e1829b9-etc-openvswitch\") pod \"ovnkube-node-7dglz\" (UID: \"a03f2d01-68c8-4498-927e-c8722e1829b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7dglz"
Sep 18 16:57:25.547794 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.547635    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a03f2d01-68c8-4498-927e-c8722e1829b9-etc-openvswitch\") pod \"ovnkube-node-7dglz\" (UID: \"a03f2d01-68c8-4498-927e-c8722e1829b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7dglz"
Sep 18 16:57:25.547794 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.547645    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a03f2d01-68c8-4498-927e-c8722e1829b9-log-socket\") pod \"ovnkube-node-7dglz\" (UID: \"a03f2d01-68c8-4498-927e-c8722e1829b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7dglz"
Sep 18 16:57:25.547794 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.547676    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a03f2d01-68c8-4498-927e-c8722e1829b9-ovnkube-config\") pod \"ovnkube-node-7dglz\" (UID: \"a03f2d01-68c8-4498-927e-c8722e1829b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7dglz"
Sep 18 16:57:25.547794 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.547708    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a03f2d01-68c8-4498-927e-c8722e1829b9-run-systemd\") pod \"ovnkube-node-7dglz\" (UID: \"a03f2d01-68c8-4498-927e-c8722e1829b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7dglz"
Sep 18 16:57:25.547794 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.547728    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a03f2d01-68c8-4498-927e-c8722e1829b9-host-run-ovn-kubernetes\") pod \"ovnkube-node-7dglz\" (UID: \"a03f2d01-68c8-4498-927e-c8722e1829b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7dglz"
Sep 18 16:57:25.547794 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.547743    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a03f2d01-68c8-4498-927e-c8722e1829b9-ovn-node-metrics-cert\") pod \"ovnkube-node-7dglz\" (UID: \"a03f2d01-68c8-4498-927e-c8722e1829b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7dglz"
Sep 18 16:57:25.547794 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.547759    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a03f2d01-68c8-4498-927e-c8722e1829b9-env-overrides\") pod \"ovnkube-node-7dglz\" (UID: \"a03f2d01-68c8-4498-927e-c8722e1829b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7dglz"
Sep 18 16:57:25.547794 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.547764    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a03f2d01-68c8-4498-927e-c8722e1829b9-run-systemd\") pod \"ovnkube-node-7dglz\" (UID: \"a03f2d01-68c8-4498-927e-c8722e1829b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7dglz"
Sep 18 16:57:25.547794 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.547775    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a03f2d01-68c8-4498-927e-c8722e1829b9-host-run-ovn-kubernetes\") pod \"ovnkube-node-7dglz\" (UID: \"a03f2d01-68c8-4498-927e-c8722e1829b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7dglz"
Sep 18 16:57:25.547794 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.547787    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a03f2d01-68c8-4498-927e-c8722e1829b9-var-lib-openvswitch\") pod \"ovnkube-node-7dglz\" (UID: \"a03f2d01-68c8-4498-927e-c8722e1829b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7dglz"
Sep 18 16:57:25.548317 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.547815    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a03f2d01-68c8-4498-927e-c8722e1829b9-host-slash\") pod \"ovnkube-node-7dglz\" (UID: \"a03f2d01-68c8-4498-927e-c8722e1829b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7dglz"
Sep 18 16:57:25.548317 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.547832    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a03f2d01-68c8-4498-927e-c8722e1829b9-systemd-units\") pod \"ovnkube-node-7dglz\" (UID: \"a03f2d01-68c8-4498-927e-c8722e1829b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7dglz"
Sep 18 16:57:25.548317 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.547866    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a03f2d01-68c8-4498-927e-c8722e1829b9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7dglz\" (UID: \"a03f2d01-68c8-4498-927e-c8722e1829b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7dglz"
Sep 18 16:57:25.548317 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.547873    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a03f2d01-68c8-4498-927e-c8722e1829b9-var-lib-openvswitch\") pod \"ovnkube-node-7dglz\" (UID: \"a03f2d01-68c8-4498-927e-c8722e1829b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7dglz"
Sep 18 16:57:25.548317 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.547883    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a03f2d01-68c8-4498-927e-c8722e1829b9-host-slash\") pod \"ovnkube-node-7dglz\" (UID: \"a03f2d01-68c8-4498-927e-c8722e1829b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7dglz"
Sep 18 16:57:25.548317 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.547883    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a03f2d01-68c8-4498-927e-c8722e1829b9-ovnkube-script-lib\") pod \"ovnkube-node-7dglz\" (UID: \"a03f2d01-68c8-4498-927e-c8722e1829b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7dglz"
Sep 18 16:57:25.548317 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.547932    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a03f2d01-68c8-4498-927e-c8722e1829b9-systemd-units\") pod \"ovnkube-node-7dglz\" (UID: \"a03f2d01-68c8-4498-927e-c8722e1829b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7dglz"
Sep 18 16:57:25.548317 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.547943    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a03f2d01-68c8-4498-927e-c8722e1829b9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7dglz\" (UID: \"a03f2d01-68c8-4498-927e-c8722e1829b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7dglz"
Sep 18 16:57:25.548317 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.547994    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6xtj\" (UniqueName: \"kubernetes.io/projected/a03f2d01-68c8-4498-927e-c8722e1829b9-kube-api-access-m6xtj\") pod \"ovnkube-node-7dglz\" (UID: \"a03f2d01-68c8-4498-927e-c8722e1829b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7dglz"
Sep 18 16:57:25.548317 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.548184    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a03f2d01-68c8-4498-927e-c8722e1829b9-host-run-netns\") pod \"ovnkube-node-7dglz\" (UID: \"a03f2d01-68c8-4498-927e-c8722e1829b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7dglz"
Sep 18 16:57:25.548317 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.548246    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a03f2d01-68c8-4498-927e-c8722e1829b9-run-ovn\") pod \"ovnkube-node-7dglz\" (UID: \"a03f2d01-68c8-4498-927e-c8722e1829b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7dglz"
Sep 18 16:57:25.548317 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.548271    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a03f2d01-68c8-4498-927e-c8722e1829b9-env-overrides\") pod \"ovnkube-node-7dglz\" (UID: \"a03f2d01-68c8-4498-927e-c8722e1829b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7dglz"
Sep 18 16:57:25.548317 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.548281    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a03f2d01-68c8-4498-927e-c8722e1829b9-host-cni-netd\") pod \"ovnkube-node-7dglz\" (UID: \"a03f2d01-68c8-4498-927e-c8722e1829b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7dglz"
Sep 18 16:57:25.548317 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.548284    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a03f2d01-68c8-4498-927e-c8722e1829b9-host-run-netns\") pod \"ovnkube-node-7dglz\" (UID: \"a03f2d01-68c8-4498-927e-c8722e1829b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7dglz"
Sep 18 16:57:25.548317 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.548321    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a03f2d01-68c8-4498-927e-c8722e1829b9-run-openvswitch\") pod \"ovnkube-node-7dglz\" (UID: \"a03f2d01-68c8-4498-927e-c8722e1829b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7dglz"
Sep 18 16:57:25.548317 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.548328    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a03f2d01-68c8-4498-927e-c8722e1829b9-run-ovn\") pod \"ovnkube-node-7dglz\" (UID: \"a03f2d01-68c8-4498-927e-c8722e1829b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7dglz"
Sep 18 16:57:25.548840 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.548350    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a03f2d01-68c8-4498-927e-c8722e1829b9-host-kubelet\") pod \"ovnkube-node-7dglz\" (UID: \"a03f2d01-68c8-4498-927e-c8722e1829b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7dglz"
Sep 18 16:57:25.548840 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.548352    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a03f2d01-68c8-4498-927e-c8722e1829b9-ovnkube-config\") pod \"ovnkube-node-7dglz\" (UID: \"a03f2d01-68c8-4498-927e-c8722e1829b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7dglz"
Sep 18 16:57:25.548840 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.548377    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a03f2d01-68c8-4498-927e-c8722e1829b9-node-log\") pod \"ovnkube-node-7dglz\" (UID: \"a03f2d01-68c8-4498-927e-c8722e1829b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7dglz"
Sep 18 16:57:25.548840 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.548373    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a03f2d01-68c8-4498-927e-c8722e1829b9-host-cni-netd\") pod \"ovnkube-node-7dglz\" (UID: \"a03f2d01-68c8-4498-927e-c8722e1829b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7dglz"
Sep 18 16:57:25.548840 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.548400    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a03f2d01-68c8-4498-927e-c8722e1829b9-host-cni-bin\") pod \"ovnkube-node-7dglz\" (UID: \"a03f2d01-68c8-4498-927e-c8722e1829b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7dglz"
Sep 18 16:57:25.548840 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.548414    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a03f2d01-68c8-4498-927e-c8722e1829b9-host-kubelet\") pod \"ovnkube-node-7dglz\" (UID: \"a03f2d01-68c8-4498-927e-c8722e1829b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7dglz"
Sep 18 16:57:25.548840 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.548429    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a03f2d01-68c8-4498-927e-c8722e1829b9-node-log\") pod \"ovnkube-node-7dglz\" (UID: \"a03f2d01-68c8-4498-927e-c8722e1829b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7dglz"
Sep 18 16:57:25.548840 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.548433    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a03f2d01-68c8-4498-927e-c8722e1829b9-run-openvswitch\") pod \"ovnkube-node-7dglz\" (UID: \"a03f2d01-68c8-4498-927e-c8722e1829b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7dglz"
Sep 18 16:57:25.548840 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.548466    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a03f2d01-68c8-4498-927e-c8722e1829b9-host-cni-bin\") pod \"ovnkube-node-7dglz\" (UID: \"a03f2d01-68c8-4498-927e-c8722e1829b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7dglz"
Sep 18 16:57:25.548840 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.548464    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a03f2d01-68c8-4498-927e-c8722e1829b9-ovnkube-script-lib\") pod \"ovnkube-node-7dglz\" (UID: \"a03f2d01-68c8-4498-927e-c8722e1829b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7dglz"
Sep 18 16:57:25.550931 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.550913    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a03f2d01-68c8-4498-927e-c8722e1829b9-ovn-node-metrics-cert\") pod \"ovnkube-node-7dglz\" (UID: \"a03f2d01-68c8-4498-927e-c8722e1829b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7dglz"
Sep 18 16:57:25.558209 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.558187    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6xtj\" (UniqueName: \"kubernetes.io/projected/a03f2d01-68c8-4498-927e-c8722e1829b9-kube-api-access-m6xtj\") pod \"ovnkube-node-7dglz\" (UID: \"a03f2d01-68c8-4498-927e-c8722e1829b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7dglz"
Sep 18 16:57:25.667260 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.667208    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7dglz"
Sep 18 16:57:25.677607 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:57:25.677557    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda03f2d01_68c8_4498_927e_c8722e1829b9.slice/crio-ff90843354827183ff37ca2c262bfc20c804c952ba1b23ea552d0ffad337ffa6 WatchSource:0}: Error finding container ff90843354827183ff37ca2c262bfc20c804c952ba1b23ea552d0ffad337ffa6: Status 404 returned error can't find the container with id ff90843354827183ff37ca2c262bfc20c804c952ba1b23ea552d0ffad337ffa6
Sep 18 16:57:25.749660 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.749549    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89faf7f5-d633-4219-b3fc-323f8c23b599-metrics-certs\") pod \"network-metrics-daemon-lpt5n\" (UID: \"89faf7f5-d633-4219-b3fc-323f8c23b599\") " pod="openshift-multus/network-metrics-daemon-lpt5n"
Sep 18 16:57:25.749660 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.749644    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a4df48ac-dd04-4504-b828-66d7e2e986e1-original-pull-secret\") pod \"global-pull-secret-syncer-fx8nf\" (UID: \"a4df48ac-dd04-4504-b828-66d7e2e986e1\") " pod="kube-system/global-pull-secret-syncer-fx8nf"
Sep 18 16:57:25.749903 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:25.749776    2881 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 16:57:25.749903 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:25.749847    2881 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4df48ac-dd04-4504-b828-66d7e2e986e1-original-pull-secret podName:a4df48ac-dd04-4504-b828-66d7e2e986e1 nodeName:}" failed. No retries permitted until 2026-09-18 16:57:26.749828061 +0000 UTC m=+236.949683338 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a4df48ac-dd04-4504-b828-66d7e2e986e1-original-pull-secret") pod "global-pull-secret-syncer-fx8nf" (UID: "a4df48ac-dd04-4504-b828-66d7e2e986e1") : object "kube-system"/"original-pull-secret" not registered
Sep 18 16:57:25.750303 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:25.750262    2881 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:57:25.750482 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:25.750351    2881 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89faf7f5-d633-4219-b3fc-323f8c23b599-metrics-certs podName:89faf7f5-d633-4219-b3fc-323f8c23b599 nodeName:}" failed. No retries permitted until 2026-09-18 16:57:26.750331081 +0000 UTC m=+236.950186361 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/89faf7f5-d633-4219-b3fc-323f8c23b599-metrics-certs") pod "network-metrics-daemon-lpt5n" (UID: "89faf7f5-d633-4219-b3fc-323f8c23b599") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:57:25.849995 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:25.849951    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhsfc\" (UniqueName: \"kubernetes.io/projected/6f463717-a487-485b-aa4b-7772d35b0d22-kube-api-access-rhsfc\") pod \"network-check-target-pc44x\" (UID: \"6f463717-a487-485b-aa4b-7772d35b0d22\") " pod="openshift-network-diagnostics/network-check-target-pc44x"
Sep 18 16:57:25.850331 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:25.850192    2881 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 16:57:25.850331 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:25.850214    2881 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 16:57:25.850331 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:25.850241    2881 projected.go:196] Error preparing data for projected volume kube-api-access-rhsfc for pod openshift-network-diagnostics/network-check-target-pc44x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:57:25.850331 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:25.850308    2881 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f463717-a487-485b-aa4b-7772d35b0d22-kube-api-access-rhsfc podName:6f463717-a487-485b-aa4b-7772d35b0d22 nodeName:}" failed. No retries permitted until 2026-09-18 16:57:26.850288884 +0000 UTC m=+237.050144170 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-rhsfc" (UniqueName: "kubernetes.io/projected/6f463717-a487-485b-aa4b-7772d35b0d22-kube-api-access-rhsfc") pod "network-check-target-pc44x" (UID: "6f463717-a487-485b-aa4b-7772d35b0d22") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:57:26.073164 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:26.073090    2881 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 16:52:24 +0000 UTC" deadline="2028-02-21 14:21:22.332969155 +0000 UTC"
Sep 18 16:57:26.073164 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:26.073161    2881 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12501h23m56.259812707s"
Sep 18 16:57:26.298082 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:26.297918    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7dglz" event={"ID":"a03f2d01-68c8-4498-927e-c8722e1829b9","Type":"ContainerStarted","Data":"ff90843354827183ff37ca2c262bfc20c804c952ba1b23ea552d0ffad337ffa6"}
Sep 18 16:57:26.301140 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:26.301037    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-sghwn" event={"ID":"7295391c-ebd9-4f37-9f70-a86b0bab3fcf","Type":"ContainerStarted","Data":"fc2d30247d299fe4883e81434eae301608b289e1ce58fe6256d8ae3d2b96c0d0"}
Sep 18 16:57:26.309772 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:26.309735    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-g7d7f" event={"ID":"fb327355-e5c6-48fd-abf3-24fa50a8e0b5","Type":"ContainerStarted","Data":"0b00c4bc5354293e14ba49ef8b700698a766f3dba2f390e6a3d81a4faf788853"}
Sep 18 16:57:26.313814 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:26.313771    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ktwhm" event={"ID":"05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437","Type":"ContainerStarted","Data":"0365412ec13d8659b13a3e4a1cbfe125cf732b314119bf222a884e1b4528d266"}
Sep 18 16:57:26.318015 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:26.317964    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n4nk2" event={"ID":"d019d1b3-e6ee-4b27-9fc1-a6675650d42a","Type":"ContainerStarted","Data":"f43cd5cbd8fa20fc008ceb7367c98296b06d5701feaeb582cebbf37b475958ee"}
Sep 18 16:57:26.325612 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:26.325446    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mzj44" event={"ID":"e1eb4986-e287-4a38-980c-3275bd8e2770","Type":"ContainerStarted","Data":"7540a1080c9797168621a4fa16a380f92b099881ebdb81b06f02718f72162572"}
Sep 18 16:57:26.329373 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:26.329266    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwddj" event={"ID":"7cc9ec4e-c391-465a-b3e2-8478fa3940fb","Type":"ContainerStarted","Data":"330c8da92f25fc0404208296a3b060693435347d92a665fb7757eb30669fadfc"}
Sep 18 16:57:26.333664 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:26.333591    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8rtmt" event={"ID":"b6ec1dac-2d47-4156-bb74-b7055bedbc2b","Type":"ContainerStarted","Data":"380d38ecfc4419da5cad2897b15914e2efad9a28a909021c22058aa69734d2f8"}
Sep 18 16:57:26.758673 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:26.757831    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a4df48ac-dd04-4504-b828-66d7e2e986e1-original-pull-secret\") pod \"global-pull-secret-syncer-fx8nf\" (UID: \"a4df48ac-dd04-4504-b828-66d7e2e986e1\") " pod="kube-system/global-pull-secret-syncer-fx8nf"
Sep 18 16:57:26.758673 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:26.757894    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89faf7f5-d633-4219-b3fc-323f8c23b599-metrics-certs\") pod \"network-metrics-daemon-lpt5n\" (UID: \"89faf7f5-d633-4219-b3fc-323f8c23b599\") " pod="openshift-multus/network-metrics-daemon-lpt5n"
Sep 18 16:57:26.758673 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:26.758071    2881 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:57:26.758673 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:26.758133    2881 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89faf7f5-d633-4219-b3fc-323f8c23b599-metrics-certs podName:89faf7f5-d633-4219-b3fc-323f8c23b599 nodeName:}" failed. No retries permitted until 2026-09-18 16:57:28.758114874 +0000 UTC m=+238.957970169 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/89faf7f5-d633-4219-b3fc-323f8c23b599-metrics-certs") pod "network-metrics-daemon-lpt5n" (UID: "89faf7f5-d633-4219-b3fc-323f8c23b599") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:57:26.758673 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:26.758199    2881 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 16:57:26.758673 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:26.758253    2881 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4df48ac-dd04-4504-b828-66d7e2e986e1-original-pull-secret podName:a4df48ac-dd04-4504-b828-66d7e2e986e1 nodeName:}" failed. No retries permitted until 2026-09-18 16:57:28.758241682 +0000 UTC m=+238.958096948 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a4df48ac-dd04-4504-b828-66d7e2e986e1-original-pull-secret") pod "global-pull-secret-syncer-fx8nf" (UID: "a4df48ac-dd04-4504-b828-66d7e2e986e1") : object "kube-system"/"original-pull-secret" not registered
Sep 18 16:57:26.859927 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:26.859154    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhsfc\" (UniqueName: \"kubernetes.io/projected/6f463717-a487-485b-aa4b-7772d35b0d22-kube-api-access-rhsfc\") pod \"network-check-target-pc44x\" (UID: \"6f463717-a487-485b-aa4b-7772d35b0d22\") " pod="openshift-network-diagnostics/network-check-target-pc44x"
Sep 18 16:57:26.859927 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:26.859416    2881 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 16:57:26.859927 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:26.859443    2881 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 16:57:26.859927 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:26.859478    2881 projected.go:196] Error preparing data for projected volume kube-api-access-rhsfc for pod openshift-network-diagnostics/network-check-target-pc44x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:57:26.859927 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:26.859539    2881 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f463717-a487-485b-aa4b-7772d35b0d22-kube-api-access-rhsfc podName:6f463717-a487-485b-aa4b-7772d35b0d22 nodeName:}" failed. No retries permitted until 2026-09-18 16:57:28.859519464 +0000 UTC m=+239.059374734 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-rhsfc" (UniqueName: "kubernetes.io/projected/6f463717-a487-485b-aa4b-7772d35b0d22-kube-api-access-rhsfc") pod "network-check-target-pc44x" (UID: "6f463717-a487-485b-aa4b-7772d35b0d22") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:57:26.888329 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:26.887448    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pc44x"
Sep 18 16:57:26.888329 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:26.887589    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pc44x" podUID="6f463717-a487-485b-aa4b-7772d35b0d22"
Sep 18 16:57:26.888329 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:26.888011    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpt5n"
Sep 18 16:57:26.888329 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:26.888119    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lpt5n" podUID="89faf7f5-d633-4219-b3fc-323f8c23b599"
Sep 18 16:57:26.888329 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:26.888181    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fx8nf"
Sep 18 16:57:26.888329 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:26.888270    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fx8nf" podUID="a4df48ac-dd04-4504-b828-66d7e2e986e1"
Sep 18 16:57:28.774320 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:28.774262    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a4df48ac-dd04-4504-b828-66d7e2e986e1-original-pull-secret\") pod \"global-pull-secret-syncer-fx8nf\" (UID: \"a4df48ac-dd04-4504-b828-66d7e2e986e1\") " pod="kube-system/global-pull-secret-syncer-fx8nf"
Sep 18 16:57:28.774826 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:28.774332    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89faf7f5-d633-4219-b3fc-323f8c23b599-metrics-certs\") pod \"network-metrics-daemon-lpt5n\" (UID: \"89faf7f5-d633-4219-b3fc-323f8c23b599\") " pod="openshift-multus/network-metrics-daemon-lpt5n"
Sep 18 16:57:28.774826 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:28.774473    2881 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:57:28.774826 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:28.774536    2881 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 16:57:28.774826 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:28.774544    2881 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89faf7f5-d633-4219-b3fc-323f8c23b599-metrics-certs podName:89faf7f5-d633-4219-b3fc-323f8c23b599 nodeName:}" failed. No retries permitted until 2026-09-18 16:57:32.774523749 +0000 UTC m=+242.974379028 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/89faf7f5-d633-4219-b3fc-323f8c23b599-metrics-certs") pod "network-metrics-daemon-lpt5n" (UID: "89faf7f5-d633-4219-b3fc-323f8c23b599") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:57:28.774826 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:28.774646    2881 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4df48ac-dd04-4504-b828-66d7e2e986e1-original-pull-secret podName:a4df48ac-dd04-4504-b828-66d7e2e986e1 nodeName:}" failed. No retries permitted until 2026-09-18 16:57:32.774628122 +0000 UTC m=+242.974483390 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a4df48ac-dd04-4504-b828-66d7e2e986e1-original-pull-secret") pod "global-pull-secret-syncer-fx8nf" (UID: "a4df48ac-dd04-4504-b828-66d7e2e986e1") : object "kube-system"/"original-pull-secret" not registered
Sep 18 16:57:28.875765 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:28.875718    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhsfc\" (UniqueName: \"kubernetes.io/projected/6f463717-a487-485b-aa4b-7772d35b0d22-kube-api-access-rhsfc\") pod \"network-check-target-pc44x\" (UID: \"6f463717-a487-485b-aa4b-7772d35b0d22\") " pod="openshift-network-diagnostics/network-check-target-pc44x"
Sep 18 16:57:28.875988 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:28.875917    2881 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 16:57:28.875988 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:28.875942    2881 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 16:57:28.875988 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:28.875957    2881 projected.go:196] Error preparing data for projected volume kube-api-access-rhsfc for pod openshift-network-diagnostics/network-check-target-pc44x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:57:28.876167 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:28.876027    2881 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f463717-a487-485b-aa4b-7772d35b0d22-kube-api-access-rhsfc podName:6f463717-a487-485b-aa4b-7772d35b0d22 nodeName:}" failed. No retries permitted until 2026-09-18 16:57:32.876007789 +0000 UTC m=+243.075863058 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-rhsfc" (UniqueName: "kubernetes.io/projected/6f463717-a487-485b-aa4b-7772d35b0d22-kube-api-access-rhsfc") pod "network-check-target-pc44x" (UID: "6f463717-a487-485b-aa4b-7772d35b0d22") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:57:28.888204 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:28.888167    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpt5n"
Sep 18 16:57:28.888434 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:28.888342    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lpt5n" podUID="89faf7f5-d633-4219-b3fc-323f8c23b599"
Sep 18 16:57:28.888434 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:28.888182    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pc44x"
Sep 18 16:57:28.888551 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:28.888439    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pc44x" podUID="6f463717-a487-485b-aa4b-7772d35b0d22"
Sep 18 16:57:28.888551 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:28.888182    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fx8nf"
Sep 18 16:57:28.888551 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:28.888520    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fx8nf" podUID="a4df48ac-dd04-4504-b828-66d7e2e986e1"
Sep 18 16:57:29.953388 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:29.953338    2881 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?"
Sep 18 16:57:30.888009 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:30.887970    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpt5n"
Sep 18 16:57:30.888285 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:30.888141    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lpt5n" podUID="89faf7f5-d633-4219-b3fc-323f8c23b599"
Sep 18 16:57:30.888547 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:30.888515    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pc44x"
Sep 18 16:57:30.888673 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:30.888516    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fx8nf"
Sep 18 16:57:30.888673 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:30.888648    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pc44x" podUID="6f463717-a487-485b-aa4b-7772d35b0d22"
Sep 18 16:57:30.888776 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:30.888731    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fx8nf" podUID="a4df48ac-dd04-4504-b828-66d7e2e986e1"
Sep 18 16:57:32.808336 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:32.807982    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a4df48ac-dd04-4504-b828-66d7e2e986e1-original-pull-secret\") pod \"global-pull-secret-syncer-fx8nf\" (UID: \"a4df48ac-dd04-4504-b828-66d7e2e986e1\") " pod="kube-system/global-pull-secret-syncer-fx8nf"
Sep 18 16:57:32.808336 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:32.808048    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89faf7f5-d633-4219-b3fc-323f8c23b599-metrics-certs\") pod \"network-metrics-daemon-lpt5n\" (UID: \"89faf7f5-d633-4219-b3fc-323f8c23b599\") " pod="openshift-multus/network-metrics-daemon-lpt5n"
Sep 18 16:57:32.808336 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:32.808182    2881 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 16:57:32.808336 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:32.808205    2881 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:57:32.808336 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:32.808279    2881 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89faf7f5-d633-4219-b3fc-323f8c23b599-metrics-certs podName:89faf7f5-d633-4219-b3fc-323f8c23b599 nodeName:}" failed. No retries permitted until 2026-09-18 16:57:40.8082589 +0000 UTC m=+251.008114169 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/89faf7f5-d633-4219-b3fc-323f8c23b599-metrics-certs") pod "network-metrics-daemon-lpt5n" (UID: "89faf7f5-d633-4219-b3fc-323f8c23b599") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:57:32.808336 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:32.808299    2881 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4df48ac-dd04-4504-b828-66d7e2e986e1-original-pull-secret podName:a4df48ac-dd04-4504-b828-66d7e2e986e1 nodeName:}" failed. No retries permitted until 2026-09-18 16:57:40.808289398 +0000 UTC m=+251.008144667 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a4df48ac-dd04-4504-b828-66d7e2e986e1-original-pull-secret") pod "global-pull-secret-syncer-fx8nf" (UID: "a4df48ac-dd04-4504-b828-66d7e2e986e1") : object "kube-system"/"original-pull-secret" not registered
Sep 18 16:57:32.887986 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:32.887935    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pc44x"
Sep 18 16:57:32.887986 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:32.887975    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpt5n"
Sep 18 16:57:32.888269 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:32.888096    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pc44x" podUID="6f463717-a487-485b-aa4b-7772d35b0d22"
Sep 18 16:57:32.888337 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:32.888288    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fx8nf"
Sep 18 16:57:32.888428 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:32.888406    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lpt5n" podUID="89faf7f5-d633-4219-b3fc-323f8c23b599"
Sep 18 16:57:32.888512 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:32.888491    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fx8nf" podUID="a4df48ac-dd04-4504-b828-66d7e2e986e1"
Sep 18 16:57:32.909016 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:32.908942    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhsfc\" (UniqueName: \"kubernetes.io/projected/6f463717-a487-485b-aa4b-7772d35b0d22-kube-api-access-rhsfc\") pod \"network-check-target-pc44x\" (UID: \"6f463717-a487-485b-aa4b-7772d35b0d22\") " pod="openshift-network-diagnostics/network-check-target-pc44x"
Sep 18 16:57:32.909210 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:32.909106    2881 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 16:57:32.909210 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:32.909126    2881 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 16:57:32.909210 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:32.909138    2881 projected.go:196] Error preparing data for projected volume kube-api-access-rhsfc for pod openshift-network-diagnostics/network-check-target-pc44x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:57:32.909210 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:32.909201    2881 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f463717-a487-485b-aa4b-7772d35b0d22-kube-api-access-rhsfc podName:6f463717-a487-485b-aa4b-7772d35b0d22 nodeName:}" failed. No retries permitted until 2026-09-18 16:57:40.909181883 +0000 UTC m=+251.109037153 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-rhsfc" (UniqueName: "kubernetes.io/projected/6f463717-a487-485b-aa4b-7772d35b0d22-kube-api-access-rhsfc") pod "network-check-target-pc44x" (UID: "6f463717-a487-485b-aa4b-7772d35b0d22") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:57:34.888101 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:34.888055    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pc44x"
Sep 18 16:57:34.888101 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:34.888084    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fx8nf"
Sep 18 16:57:34.888101 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:34.888104    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpt5n"
Sep 18 16:57:34.891337 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:34.888713    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pc44x" podUID="6f463717-a487-485b-aa4b-7772d35b0d22"
Sep 18 16:57:34.891337 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:34.888724    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fx8nf" podUID="a4df48ac-dd04-4504-b828-66d7e2e986e1"
Sep 18 16:57:34.891337 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:34.888849    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lpt5n" podUID="89faf7f5-d633-4219-b3fc-323f8c23b599"
Sep 18 16:57:34.955072 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:34.955030    2881 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?"
Sep 18 16:57:36.888337 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:36.888294    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fx8nf"
Sep 18 16:57:36.888337 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:36.888323    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpt5n"
Sep 18 16:57:36.888827 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:36.888294    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pc44x"
Sep 18 16:57:36.888827 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:36.888439    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fx8nf" podUID="a4df48ac-dd04-4504-b828-66d7e2e986e1"
Sep 18 16:57:36.888827 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:36.888574    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pc44x" podUID="6f463717-a487-485b-aa4b-7772d35b0d22"
Sep 18 16:57:36.888827 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:36.888664    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lpt5n" podUID="89faf7f5-d633-4219-b3fc-323f8c23b599"
Sep 18 16:57:38.887701 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:38.887367    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fx8nf"
Sep 18 16:57:38.887701 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:38.887665    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pc44x"
Sep 18 16:57:38.888250 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:38.887750    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fx8nf" podUID="a4df48ac-dd04-4504-b828-66d7e2e986e1"
Sep 18 16:57:38.888250 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:38.887814    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pc44x" podUID="6f463717-a487-485b-aa4b-7772d35b0d22"
Sep 18 16:57:38.888250 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:38.887867    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpt5n"
Sep 18 16:57:38.888250 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:38.887949    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lpt5n" podUID="89faf7f5-d633-4219-b3fc-323f8c23b599"
Sep 18 16:57:38.888250 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:38.887958    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" podUID="a3127caef50db0be093e96af172c433b"
Sep 18 16:57:39.749766 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:39.749725    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-nmhbw"]
Sep 18 16:57:39.754633 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:39.754606    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nmhbw"
Sep 18 16:57:39.757322 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:39.757201    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 18 16:57:39.757322 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:39.757242    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 18 16:57:39.757322 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:39.757248    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-w66jd\""
Sep 18 16:57:39.760048 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:39.760023    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8925ea6c-2280-428c-bb7f-1dc4e8bab99b-tmp-dir\") pod \"node-resolver-nmhbw\" (UID: \"8925ea6c-2280-428c-bb7f-1dc4e8bab99b\") " pod="openshift-dns/node-resolver-nmhbw"
Sep 18 16:57:39.760201 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:39.760097    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcw28\" (UniqueName: \"kubernetes.io/projected/8925ea6c-2280-428c-bb7f-1dc4e8bab99b-kube-api-access-xcw28\") pod \"node-resolver-nmhbw\" (UID: \"8925ea6c-2280-428c-bb7f-1dc4e8bab99b\") " pod="openshift-dns/node-resolver-nmhbw"
Sep 18 16:57:39.760201 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:39.760189    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8925ea6c-2280-428c-bb7f-1dc4e8bab99b-hosts-file\") pod \"node-resolver-nmhbw\" (UID: \"8925ea6c-2280-428c-bb7f-1dc4e8bab99b\") " pod="openshift-dns/node-resolver-nmhbw"
Sep 18 16:57:39.860971 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:39.860935    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcw28\" (UniqueName: \"kubernetes.io/projected/8925ea6c-2280-428c-bb7f-1dc4e8bab99b-kube-api-access-xcw28\") pod \"node-resolver-nmhbw\" (UID: \"8925ea6c-2280-428c-bb7f-1dc4e8bab99b\") " pod="openshift-dns/node-resolver-nmhbw"
Sep 18 16:57:39.861329 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:39.861003    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8925ea6c-2280-428c-bb7f-1dc4e8bab99b-hosts-file\") pod \"node-resolver-nmhbw\" (UID: \"8925ea6c-2280-428c-bb7f-1dc4e8bab99b\") " pod="openshift-dns/node-resolver-nmhbw"
Sep 18 16:57:39.861329 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:39.861029    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8925ea6c-2280-428c-bb7f-1dc4e8bab99b-tmp-dir\") pod \"node-resolver-nmhbw\" (UID: \"8925ea6c-2280-428c-bb7f-1dc4e8bab99b\") " pod="openshift-dns/node-resolver-nmhbw"
Sep 18 16:57:39.861329 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:39.861236    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8925ea6c-2280-428c-bb7f-1dc4e8bab99b-hosts-file\") pod \"node-resolver-nmhbw\" (UID: \"8925ea6c-2280-428c-bb7f-1dc4e8bab99b\") " pod="openshift-dns/node-resolver-nmhbw"
Sep 18 16:57:39.861497 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:39.861421    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8925ea6c-2280-428c-bb7f-1dc4e8bab99b-tmp-dir\") pod \"node-resolver-nmhbw\" (UID: \"8925ea6c-2280-428c-bb7f-1dc4e8bab99b\") " pod="openshift-dns/node-resolver-nmhbw"
Sep 18 16:57:39.879564 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:39.879529    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcw28\" (UniqueName: \"kubernetes.io/projected/8925ea6c-2280-428c-bb7f-1dc4e8bab99b-kube-api-access-xcw28\") pod \"node-resolver-nmhbw\" (UID: \"8925ea6c-2280-428c-bb7f-1dc4e8bab99b\") " pod="openshift-dns/node-resolver-nmhbw"
Sep 18 16:57:39.956386 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:39.956344    2881 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?"
Sep 18 16:57:40.065300 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:40.065255    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nmhbw"
Sep 18 16:57:40.868668 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:40.868631    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a4df48ac-dd04-4504-b828-66d7e2e986e1-original-pull-secret\") pod \"global-pull-secret-syncer-fx8nf\" (UID: \"a4df48ac-dd04-4504-b828-66d7e2e986e1\") " pod="kube-system/global-pull-secret-syncer-fx8nf"
Sep 18 16:57:40.868668 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:40.868676    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89faf7f5-d633-4219-b3fc-323f8c23b599-metrics-certs\") pod \"network-metrics-daemon-lpt5n\" (UID: \"89faf7f5-d633-4219-b3fc-323f8c23b599\") " pod="openshift-multus/network-metrics-daemon-lpt5n"
Sep 18 16:57:40.868898 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:40.868785    2881 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:57:40.868898 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:40.868786    2881 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 16:57:40.868898 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:40.868842    2881 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89faf7f5-d633-4219-b3fc-323f8c23b599-metrics-certs podName:89faf7f5-d633-4219-b3fc-323f8c23b599 nodeName:}" failed. No retries permitted until 2026-09-18 16:57:56.868827465 +0000 UTC m=+267.068682730 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/89faf7f5-d633-4219-b3fc-323f8c23b599-metrics-certs") pod "network-metrics-daemon-lpt5n" (UID: "89faf7f5-d633-4219-b3fc-323f8c23b599") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:57:40.868898 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:40.868856    2881 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4df48ac-dd04-4504-b828-66d7e2e986e1-original-pull-secret podName:a4df48ac-dd04-4504-b828-66d7e2e986e1 nodeName:}" failed. No retries permitted until 2026-09-18 16:57:56.868849959 +0000 UTC m=+267.068705222 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a4df48ac-dd04-4504-b828-66d7e2e986e1-original-pull-secret") pod "global-pull-secret-syncer-fx8nf" (UID: "a4df48ac-dd04-4504-b828-66d7e2e986e1") : object "kube-system"/"original-pull-secret" not registered
Sep 18 16:57:40.887914 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:40.887875    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fx8nf"
Sep 18 16:57:40.888108 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:40.887875    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pc44x"
Sep 18 16:57:40.888108 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:40.888025    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fx8nf" podUID="a4df48ac-dd04-4504-b828-66d7e2e986e1"
Sep 18 16:57:40.888108 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:40.887875    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpt5n"
Sep 18 16:57:40.888108 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:40.888073    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pc44x" podUID="6f463717-a487-485b-aa4b-7772d35b0d22"
Sep 18 16:57:40.888276 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:40.888171    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lpt5n" podUID="89faf7f5-d633-4219-b3fc-323f8c23b599"
Sep 18 16:57:40.969824 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:40.969784    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhsfc\" (UniqueName: \"kubernetes.io/projected/6f463717-a487-485b-aa4b-7772d35b0d22-kube-api-access-rhsfc\") pod \"network-check-target-pc44x\" (UID: \"6f463717-a487-485b-aa4b-7772d35b0d22\") " pod="openshift-network-diagnostics/network-check-target-pc44x"
Sep 18 16:57:40.970324 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:40.970028    2881 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 16:57:40.970324 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:40.970055    2881 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 16:57:40.970324 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:40.970069    2881 projected.go:196] Error preparing data for projected volume kube-api-access-rhsfc for pod openshift-network-diagnostics/network-check-target-pc44x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:57:40.970324 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:40.970168    2881 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f463717-a487-485b-aa4b-7772d35b0d22-kube-api-access-rhsfc podName:6f463717-a487-485b-aa4b-7772d35b0d22 nodeName:}" failed. No retries permitted until 2026-09-18 16:57:56.970140079 +0000 UTC m=+267.169995354 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-rhsfc" (UniqueName: "kubernetes.io/projected/6f463717-a487-485b-aa4b-7772d35b0d22-kube-api-access-rhsfc") pod "network-check-target-pc44x" (UID: "6f463717-a487-485b-aa4b-7772d35b0d22") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:57:42.123545 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:57:42.123504    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8925ea6c_2280_428c_bb7f_1dc4e8bab99b.slice/crio-788a98dc7de0a4aa96a8b0a68ba42218f424402cfe38652606dfa2abf40a4bd7 WatchSource:0}: Error finding container 788a98dc7de0a4aa96a8b0a68ba42218f424402cfe38652606dfa2abf40a4bd7: Status 404 returned error can't find the container with id 788a98dc7de0a4aa96a8b0a68ba42218f424402cfe38652606dfa2abf40a4bd7
Sep 18 16:57:42.377246 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:42.376919    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-sghwn" event={"ID":"7295391c-ebd9-4f37-9f70-a86b0bab3fcf","Type":"ContainerStarted","Data":"efd1932d3d3f3edaef347e0460aed472c33f8a770882ff814b9c92cc753768f9"}
Sep 18 16:57:42.380312 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:42.380280    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-g7d7f" event={"ID":"fb327355-e5c6-48fd-abf3-24fa50a8e0b5","Type":"ContainerStarted","Data":"95a681c51ce325e4abb16f9191a07464a3555adfb0551e43bf229517b98c72da"}
Sep 18 16:57:42.383598 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:42.383521    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ktwhm" event={"ID":"05b58fce-4a3d-47c1-b9cd-fe9dbc9bf437","Type":"ContainerStarted","Data":"79e2ef40a3401c058820768553d2956f70a11b113ac097b80a9848f0b1df7161"}
Sep 18 16:57:42.385810 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:42.385773    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n4nk2" event={"ID":"d019d1b3-e6ee-4b27-9fc1-a6675650d42a","Type":"ContainerStarted","Data":"6de0f775b53ded296b01e51184fcf50f79ec02ffffc8b523647725d888941359"}
Sep 18 16:57:42.387984 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:42.387948    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwddj" event={"ID":"7cc9ec4e-c391-465a-b3e2-8478fa3940fb","Type":"ContainerStarted","Data":"360d9b527f56607e9c6c2ab64b1b932291761c239406588941a8b9d62abe36e2"}
Sep 18 16:57:42.390057 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:42.390005    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8rtmt" event={"ID":"b6ec1dac-2d47-4156-bb74-b7055bedbc2b","Type":"ContainerStarted","Data":"2ffc494783af035c0c52bc7dc0732f55128538bc2a5c140540dfdbac58276732"}
Sep 18 16:57:42.392794 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:42.392248    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-sghwn" podStartSLOduration=22.627721135 podStartE2EDuration="39.392204144s" totalImagesPullingTime="16.764483009s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:57:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:57:42.391813819 +0000 UTC m=+252.591669106" watchObservedRunningTime="2026-09-18 16:57:42.392204144 +0000 UTC m=+252.592059433"
Sep 18 16:57:42.395663 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:42.395362    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nmhbw" event={"ID":"8925ea6c-2280-428c-bb7f-1dc4e8bab99b","Type":"ContainerStarted","Data":"788a98dc7de0a4aa96a8b0a68ba42218f424402cfe38652606dfa2abf40a4bd7"}
Sep 18 16:57:42.399657 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:42.399562    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7dglz" event={"ID":"a03f2d01-68c8-4498-927e-c8722e1829b9","Type":"ContainerStarted","Data":"92643f1353b757bee28b1e1e85191d2e2bf9699e98089631e02922f8debd003f"}
Sep 18 16:57:42.429932 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:42.429862    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-8rtmt" podStartSLOduration=26.601476273 podStartE2EDuration="43.429837469s" totalImagesPullingTime="16.828361196s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:56:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:57:42.428786157 +0000 UTC m=+252.628641470" watchObservedRunningTime="2026-09-18 16:57:42.429837469 +0000 UTC m=+252.629692757"
Sep 18 16:57:42.458099 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:42.457938    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-ktwhm" podStartSLOduration=26.666177047 podStartE2EDuration="43.457916512s" totalImagesPullingTime="16.791739465s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:56:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:57:42.457819553 +0000 UTC m=+252.657674851" watchObservedRunningTime="2026-09-18 16:57:42.457916512 +0000 UTC m=+252.657771800"
Sep 18 16:57:42.458713 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:42.458683    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-g7d7f" podStartSLOduration=26.680695575 podStartE2EDuration="43.4586729s" totalImagesPullingTime="16.777977325s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:56:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:57:42.441439257 +0000 UTC m=+252.641294545" watchObservedRunningTime="2026-09-18 16:57:42.4586729 +0000 UTC m=+252.658528252"
Sep 18 16:57:42.474640 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:42.474586    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-nmhbw" podStartSLOduration=3.474570177 podStartE2EDuration="3.474570177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:57:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:57:42.474503495 +0000 UTC m=+252.674358782" watchObservedRunningTime="2026-09-18 16:57:42.474570177 +0000 UTC m=+252.674425462"
Sep 18 16:57:42.887828 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:42.887784    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fx8nf"
Sep 18 16:57:42.887828 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:42.887820    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pc44x"
Sep 18 16:57:42.888074 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:42.887935    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fx8nf" podUID="a4df48ac-dd04-4504-b828-66d7e2e986e1"
Sep 18 16:57:42.888074 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:42.887995    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pc44x" podUID="6f463717-a487-485b-aa4b-7772d35b0d22"
Sep 18 16:57:42.888074 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:42.888010    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpt5n"
Sep 18 16:57:42.888196 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:42.888103    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lpt5n" podUID="89faf7f5-d633-4219-b3fc-323f8c23b599"
Sep 18 16:57:43.313180 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:43.312995    2881 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 18 16:57:43.403963 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:43.403923    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nmhbw" event={"ID":"8925ea6c-2280-428c-bb7f-1dc4e8bab99b","Type":"ContainerStarted","Data":"6a44f676cee0da44ea9acdfbd6a04757b134dba9cb1e6da7747097e26bbc4078"}
Sep 18 16:57:43.406971 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:43.406938    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7dglz" event={"ID":"a03f2d01-68c8-4498-927e-c8722e1829b9","Type":"ContainerStarted","Data":"94882638f1207bb1ed3cc84aab88d9a10779c472a0dcd06d47076086ca31785d"}
Sep 18 16:57:43.407160 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:43.407020    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7dglz" event={"ID":"a03f2d01-68c8-4498-927e-c8722e1829b9","Type":"ContainerStarted","Data":"6be771ba6d806d941453ed163bba3cac1203d2e535451484a2ad476265be840c"}
Sep 18 16:57:43.407160 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:43.407067    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7dglz" event={"ID":"a03f2d01-68c8-4498-927e-c8722e1829b9","Type":"ContainerStarted","Data":"c119efef2f227f5f43b3074576c07367c290a732ebfa3c1d6ed1a67e22a800b7"}
Sep 18 16:57:43.407160 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:43.407099    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7dglz" event={"ID":"a03f2d01-68c8-4498-927e-c8722e1829b9","Type":"ContainerStarted","Data":"1764b05cbb29cb92e64e3f0a533fbdbe53201336e74e351ce47894e20dd35e49"}
Sep 18 16:57:43.407160 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:43.407140    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7dglz" event={"ID":"a03f2d01-68c8-4498-927e-c8722e1829b9","Type":"ContainerStarted","Data":"9d68f1e2322ea4eb7d379f8ed037682e26b0a6b7f0cea3eb36a63366a6671979"}
Sep 18 16:57:43.408393 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:43.408366    2881 generic.go:356] "Generic (PLEG): container finished" podID="d019d1b3-e6ee-4b27-9fc1-a6675650d42a" containerID="6de0f775b53ded296b01e51184fcf50f79ec02ffffc8b523647725d888941359" exitCode=0
Sep 18 16:57:43.408486 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:43.408450    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n4nk2" event={"ID":"d019d1b3-e6ee-4b27-9fc1-a6675650d42a","Type":"ContainerDied","Data":"6de0f775b53ded296b01e51184fcf50f79ec02ffffc8b523647725d888941359"}
Sep 18 16:57:43.410474 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:43.410453    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwddj" event={"ID":"7cc9ec4e-c391-465a-b3e2-8478fa3940fb","Type":"ContainerStarted","Data":"97559d34d940e504b7672da8fe675402a50a0d58bbf2249d82e905ee2a44e6f6"}
Sep 18 16:57:44.045196 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:44.045005    2881 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-18T16:57:43.313168225Z","UUID":"07481eee-288d-47c5-8c43-86e095741e83","Handler":null,"Name":"","Endpoint":""}
Sep 18 16:57:44.047164 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:44.047136    2881 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 18 16:57:44.047394 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:44.047179    2881 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 18 16:57:44.414474 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:44.414376    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mzj44" event={"ID":"e1eb4986-e287-4a38-980c-3275bd8e2770","Type":"ContainerStarted","Data":"76d4a973e5f94679b806d50e51cfc12b2dc2c15d7023a6836e269ece05a8064f"}
Sep 18 16:57:44.416821 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:44.416784    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwddj" event={"ID":"7cc9ec4e-c391-465a-b3e2-8478fa3940fb","Type":"ContainerStarted","Data":"035f135952026be6872cec7ee18d76d1f12626633aa4dac70f2d8365abf0f425"}
Sep 18 16:57:44.429575 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:44.429523    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-mzj44" podStartSLOduration=23.670193028 podStartE2EDuration="40.429506442s" totalImagesPullingTime="16.759313414s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:57:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:57:44.429017211 +0000 UTC m=+254.628872497" watchObservedRunningTime="2026-09-18 16:57:44.429506442 +0000 UTC m=+254.629361728"
Sep 18 16:57:44.445065 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:44.445001    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwddj" podStartSLOduration=27.007601863 podStartE2EDuration="45.444981218s" totalImagesPullingTime="18.437379355s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:56:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:57:44.444886411 +0000 UTC m=+254.644741697" watchObservedRunningTime="2026-09-18 16:57:44.444981218 +0000 UTC m=+254.644836505"
Sep 18 16:57:44.887504 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:44.887458    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpt5n"
Sep 18 16:57:44.887704 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:44.887461    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fx8nf"
Sep 18 16:57:44.887704 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:44.887611    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lpt5n" podUID="89faf7f5-d633-4219-b3fc-323f8c23b599"
Sep 18 16:57:44.887704 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:44.887473    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pc44x"
Sep 18 16:57:44.887704 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:44.887678    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fx8nf" podUID="a4df48ac-dd04-4504-b828-66d7e2e986e1"
Sep 18 16:57:44.887917 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:44.887748    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pc44x" podUID="6f463717-a487-485b-aa4b-7772d35b0d22"
Sep 18 16:57:44.957337 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:44.957295    2881 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?"
Sep 18 16:57:45.330648 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:45.330599    2881 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-g7d7f"
Sep 18 16:57:45.331308 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:45.331275    2881 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-g7d7f"
Sep 18 16:57:45.423885 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:45.423810    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7dglz" event={"ID":"a03f2d01-68c8-4498-927e-c8722e1829b9","Type":"ContainerStarted","Data":"8a0fc097210e0690e5e9fc7af5a7ba75170776de3d8baa279d9ffbf44419c544"}
Sep 18 16:57:45.424541 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:45.424333    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-g7d7f"
Sep 18 16:57:45.424691 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:45.424675    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-g7d7f"
Sep 18 16:57:46.888408 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:46.888368    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pc44x"
Sep 18 16:57:46.889047 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:46.888510    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pc44x" podUID="6f463717-a487-485b-aa4b-7772d35b0d22"
Sep 18 16:57:46.889047 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:46.888811    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fx8nf"
Sep 18 16:57:46.889047 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:46.888829    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpt5n"
Sep 18 16:57:46.889047 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:46.888948    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fx8nf" podUID="a4df48ac-dd04-4504-b828-66d7e2e986e1"
Sep 18 16:57:46.889241 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:46.889050    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lpt5n" podUID="89faf7f5-d633-4219-b3fc-323f8c23b599"
Sep 18 16:57:48.434732 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:48.434518    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7dglz" event={"ID":"a03f2d01-68c8-4498-927e-c8722e1829b9","Type":"ContainerStarted","Data":"86cc449c50d1b50b299dfdab800d1af00511e9386561e27c424a5a84fb943332"}
Sep 18 16:57:48.435392 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:48.434795    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7dglz"
Sep 18 16:57:48.436746 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:48.436718    2881 generic.go:356] "Generic (PLEG): container finished" podID="d019d1b3-e6ee-4b27-9fc1-a6675650d42a" containerID="dc035aed921d4401e0adbd17856ae5fb2bec6966922a0d6b98cdd778d1cae423" exitCode=0
Sep 18 16:57:48.436860 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:48.436765    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n4nk2" event={"ID":"d019d1b3-e6ee-4b27-9fc1-a6675650d42a","Type":"ContainerDied","Data":"dc035aed921d4401e0adbd17856ae5fb2bec6966922a0d6b98cdd778d1cae423"}
Sep 18 16:57:48.451859 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:48.451834    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7dglz"
Sep 18 16:57:48.489419 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:48.489352    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-7dglz" podStartSLOduration=6.911363753 podStartE2EDuration="23.489330408s" totalImagesPullingTime="16.577966655s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:57:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:57:48.478528671 +0000 UTC m=+258.678383956" watchObservedRunningTime="2026-09-18 16:57:48.489330408 +0000 UTC m=+258.689185697"
Sep 18 16:57:48.887837 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:48.887800    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pc44x"
Sep 18 16:57:48.888031 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:48.887903    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pc44x" podUID="6f463717-a487-485b-aa4b-7772d35b0d22"
Sep 18 16:57:48.888031 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:48.887806    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fx8nf"
Sep 18 16:57:48.888031 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:48.887930    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpt5n"
Sep 18 16:57:48.888141 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:48.888025    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fx8nf" podUID="a4df48ac-dd04-4504-b828-66d7e2e986e1"
Sep 18 16:57:48.888141 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:48.888097    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lpt5n" podUID="89faf7f5-d633-4219-b3fc-323f8c23b599"
Sep 18 16:57:49.441098 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:49.441064    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7dglz"
Sep 18 16:57:49.441665 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:49.441301    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7dglz"
Sep 18 16:57:49.460895 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:49.460867    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7dglz"
Sep 18 16:57:49.711455 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:49.711187    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lpt5n"]
Sep 18 16:57:49.711629 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:49.711495    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpt5n"
Sep 18 16:57:49.711688 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:49.711621    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lpt5n" podUID="89faf7f5-d633-4219-b3fc-323f8c23b599"
Sep 18 16:57:49.713885 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:49.713857    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pc44x"]
Sep 18 16:57:49.714051 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:49.713971    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pc44x"
Sep 18 16:57:49.714122 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:49.714106    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pc44x" podUID="6f463717-a487-485b-aa4b-7772d35b0d22"
Sep 18 16:57:49.714457 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:49.714433    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fx8nf"]
Sep 18 16:57:49.714552 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:49.714513    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fx8nf"
Sep 18 16:57:49.714607 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:49.714588    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fx8nf" podUID="a4df48ac-dd04-4504-b828-66d7e2e986e1"
Sep 18 16:57:49.957757 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:49.957713    2881 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?"
Sep 18 16:57:50.445690 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:50.445656    2881 generic.go:356] "Generic (PLEG): container finished" podID="d019d1b3-e6ee-4b27-9fc1-a6675650d42a" containerID="c7ff46dc1a7fe47db99c6361c1ae27275e1c913a5ec6933458dd750955545e53" exitCode=0
Sep 18 16:57:50.446268 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:50.445737    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n4nk2" event={"ID":"d019d1b3-e6ee-4b27-9fc1-a6675650d42a","Type":"ContainerDied","Data":"c7ff46dc1a7fe47db99c6361c1ae27275e1c913a5ec6933458dd750955545e53"}
Sep 18 16:57:50.887857 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:50.887821    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fx8nf"
Sep 18 16:57:50.887857 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:50.887848    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpt5n"
Sep 18 16:57:50.888076 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:50.887831    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pc44x"
Sep 18 16:57:50.888076 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:50.887944    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fx8nf" podUID="a4df48ac-dd04-4504-b828-66d7e2e986e1"
Sep 18 16:57:50.888076 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:50.888060    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pc44x" podUID="6f463717-a487-485b-aa4b-7772d35b0d22"
Sep 18 16:57:50.888322 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:50.888293    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lpt5n" podUID="89faf7f5-d633-4219-b3fc-323f8c23b599"
Sep 18 16:57:50.888518 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:50.888499    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" podUID="a3127caef50db0be093e96af172c433b"
Sep 18 16:57:51.040426 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:51.040400    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nmhbw_8925ea6c-2280-428c-bb7f-1dc4e8bab99b/dns-node-resolver/0.log"
Sep 18 16:57:51.454721 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:51.454682    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n4nk2" event={"ID":"d019d1b3-e6ee-4b27-9fc1-a6675650d42a","Type":"ContainerStarted","Data":"14b0dc38ae7abdf2981bc9f51055a2f73ca3b579e521551d98f1765f6cb34190"}
Sep 18 16:57:51.812750 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:51.812723    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-sghwn_7295391c-ebd9-4f37-9f70-a86b0bab3fcf/node-ca/0.log"
Sep 18 16:57:52.458906 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:52.458870    2881 generic.go:356] "Generic (PLEG): container finished" podID="d019d1b3-e6ee-4b27-9fc1-a6675650d42a" containerID="14b0dc38ae7abdf2981bc9f51055a2f73ca3b579e521551d98f1765f6cb34190" exitCode=0
Sep 18 16:57:52.459437 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:52.458918    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n4nk2" event={"ID":"d019d1b3-e6ee-4b27-9fc1-a6675650d42a","Type":"ContainerDied","Data":"14b0dc38ae7abdf2981bc9f51055a2f73ca3b579e521551d98f1765f6cb34190"}
Sep 18 16:57:52.888381 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:52.888340    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fx8nf"
Sep 18 16:57:52.888679 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:52.888361    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpt5n"
Sep 18 16:57:52.888679 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:52.888508    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fx8nf" podUID="a4df48ac-dd04-4504-b828-66d7e2e986e1"
Sep 18 16:57:52.888679 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:52.888359    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pc44x"
Sep 18 16:57:52.888679 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:52.888578    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lpt5n" podUID="89faf7f5-d633-4219-b3fc-323f8c23b599"
Sep 18 16:57:52.888679 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:52.888630    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pc44x" podUID="6f463717-a487-485b-aa4b-7772d35b0d22"
Sep 18 16:57:54.888124 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:54.888081    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pc44x"
Sep 18 16:57:54.888573 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:54.888081    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpt5n"
Sep 18 16:57:54.888573 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:54.888198    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pc44x" podUID="6f463717-a487-485b-aa4b-7772d35b0d22"
Sep 18 16:57:54.888573 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:54.888295    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lpt5n" podUID="89faf7f5-d633-4219-b3fc-323f8c23b599"
Sep 18 16:57:54.888573 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:54.888081    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fx8nf"
Sep 18 16:57:54.888573 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:54.888369    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fx8nf" podUID="a4df48ac-dd04-4504-b828-66d7e2e986e1"
Sep 18 16:57:54.958953 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:54.958901    2881 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?"
Sep 18 16:57:56.888322 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:56.888116    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fx8nf"
Sep 18 16:57:56.888692 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:56.888139    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpt5n"
Sep 18 16:57:56.888692 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:56.888417    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fx8nf" podUID="a4df48ac-dd04-4504-b828-66d7e2e986e1"
Sep 18 16:57:56.888692 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:56.888142    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pc44x"
Sep 18 16:57:56.888692 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:56.888514    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lpt5n" podUID="89faf7f5-d633-4219-b3fc-323f8c23b599"
Sep 18 16:57:56.888692 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:56.888551    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pc44x" podUID="6f463717-a487-485b-aa4b-7772d35b0d22"
Sep 18 16:57:56.892352 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:56.892323    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a4df48ac-dd04-4504-b828-66d7e2e986e1-original-pull-secret\") pod \"global-pull-secret-syncer-fx8nf\" (UID: \"a4df48ac-dd04-4504-b828-66d7e2e986e1\") " pod="kube-system/global-pull-secret-syncer-fx8nf"
Sep 18 16:57:56.892466 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:56.892368    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89faf7f5-d633-4219-b3fc-323f8c23b599-metrics-certs\") pod \"network-metrics-daemon-lpt5n\" (UID: \"89faf7f5-d633-4219-b3fc-323f8c23b599\") " pod="openshift-multus/network-metrics-daemon-lpt5n"
Sep 18 16:57:56.892517 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:56.892492    2881 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:57:56.892561 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:56.892551    2881 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89faf7f5-d633-4219-b3fc-323f8c23b599-metrics-certs podName:89faf7f5-d633-4219-b3fc-323f8c23b599 nodeName:}" failed. No retries permitted until 2026-09-18 16:58:28.89253651 +0000 UTC m=+299.092391773 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/89faf7f5-d633-4219-b3fc-323f8c23b599-metrics-certs") pod "network-metrics-daemon-lpt5n" (UID: "89faf7f5-d633-4219-b3fc-323f8c23b599") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:57:56.892672 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:56.892492    2881 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 16:57:56.892672 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:56.892596    2881 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4df48ac-dd04-4504-b828-66d7e2e986e1-original-pull-secret podName:a4df48ac-dd04-4504-b828-66d7e2e986e1 nodeName:}" failed. No retries permitted until 2026-09-18 16:58:28.892587483 +0000 UTC m=+299.092442747 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a4df48ac-dd04-4504-b828-66d7e2e986e1-original-pull-secret") pod "global-pull-secret-syncer-fx8nf" (UID: "a4df48ac-dd04-4504-b828-66d7e2e986e1") : object "kube-system"/"original-pull-secret" not registered
Sep 18 16:57:56.993021 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:56.992967    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhsfc\" (UniqueName: \"kubernetes.io/projected/6f463717-a487-485b-aa4b-7772d35b0d22-kube-api-access-rhsfc\") pod \"network-check-target-pc44x\" (UID: \"6f463717-a487-485b-aa4b-7772d35b0d22\") " pod="openshift-network-diagnostics/network-check-target-pc44x"
Sep 18 16:57:56.993242 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:56.993136    2881 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 16:57:56.993242 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:56.993159    2881 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 16:57:56.993242 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:56.993170    2881 projected.go:196] Error preparing data for projected volume kube-api-access-rhsfc for pod openshift-network-diagnostics/network-check-target-pc44x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:57:56.993436 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:56.993248    2881 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f463717-a487-485b-aa4b-7772d35b0d22-kube-api-access-rhsfc podName:6f463717-a487-485b-aa4b-7772d35b0d22 nodeName:}" failed. No retries permitted until 2026-09-18 16:58:28.993233866 +0000 UTC m=+299.193089143 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-rhsfc" (UniqueName: "kubernetes.io/projected/6f463717-a487-485b-aa4b-7772d35b0d22-kube-api-access-rhsfc") pod "network-check-target-pc44x" (UID: "6f463717-a487-485b-aa4b-7772d35b0d22") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:57:57.474395 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:57.474356    2881 generic.go:356] "Generic (PLEG): container finished" podID="d019d1b3-e6ee-4b27-9fc1-a6675650d42a" containerID="ee3fdf869514baccf6d2cf8941b7c0f792af30b1f90c8250b25eac3986e1e815" exitCode=0
Sep 18 16:57:57.474563 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:57.474408    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n4nk2" event={"ID":"d019d1b3-e6ee-4b27-9fc1-a6675650d42a","Type":"ContainerDied","Data":"ee3fdf869514baccf6d2cf8941b7c0f792af30b1f90c8250b25eac3986e1e815"}
Sep 18 16:57:58.479947 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:58.479914    2881 generic.go:356] "Generic (PLEG): container finished" podID="d019d1b3-e6ee-4b27-9fc1-a6675650d42a" containerID="a6621bbe5ded189c78401e239a2b577d56fb8d18a4175b7f4a1510b14d316b41" exitCode=0
Sep 18 16:57:58.480412 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:58.479966    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n4nk2" event={"ID":"d019d1b3-e6ee-4b27-9fc1-a6675650d42a","Type":"ContainerDied","Data":"a6621bbe5ded189c78401e239a2b577d56fb8d18a4175b7f4a1510b14d316b41"}
Sep 18 16:57:58.887990 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:58.887944    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fx8nf"
Sep 18 16:57:58.888186 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:58.887999    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpt5n"
Sep 18 16:57:58.888186 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:58.887952    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pc44x"
Sep 18 16:57:58.888186 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:58.888082    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fx8nf" podUID="a4df48ac-dd04-4504-b828-66d7e2e986e1"
Sep 18 16:57:58.888186 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:58.888125    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pc44x" podUID="6f463717-a487-485b-aa4b-7772d35b0d22"
Sep 18 16:57:58.888365 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:58.888209    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lpt5n" podUID="89faf7f5-d633-4219-b3fc-323f8c23b599"
Sep 18 16:57:59.485384 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:59.485351    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n4nk2" event={"ID":"d019d1b3-e6ee-4b27-9fc1-a6675650d42a","Type":"ContainerStarted","Data":"b070e4be5e30ab816f413c6900b0c11e9928e47efd6fe6ac2f29c4a99742fd85"}
Sep 18 16:57:59.505169 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:57:59.505112    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-n4nk2" podStartSLOduration=33.219134679 podStartE2EDuration="1m0.505095782s" totalImagesPullingTime="27.285961103s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:56:59 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:57:59.504650091 +0000 UTC m=+269.704505376" watchObservedRunningTime="2026-09-18 16:57:59.505095782 +0000 UTC m=+269.704951067"
Sep 18 16:57:59.960095 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:57:59.960059    2881 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?"
Sep 18 16:58:00.888199 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:00.888157    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpt5n"
Sep 18 16:58:00.888199 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:00.888193    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fx8nf"
Sep 18 16:58:00.888674 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:00.888289    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pc44x"
Sep 18 16:58:00.888674 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:58:00.888303    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lpt5n" podUID="89faf7f5-d633-4219-b3fc-323f8c23b599"
Sep 18 16:58:00.888674 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:58:00.888360    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pc44x" podUID="6f463717-a487-485b-aa4b-7772d35b0d22"
Sep 18 16:58:00.888674 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:58:00.888417    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fx8nf" podUID="a4df48ac-dd04-4504-b828-66d7e2e986e1"
Sep 18 16:58:02.888131 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:02.888087    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pc44x"
Sep 18 16:58:02.888131 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:02.888114    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpt5n"
Sep 18 16:58:02.888614 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:02.888097    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fx8nf"
Sep 18 16:58:02.888614 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:58:02.888202    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pc44x" podUID="6f463717-a487-485b-aa4b-7772d35b0d22"
Sep 18 16:58:02.888614 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:58:02.888308    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fx8nf" podUID="a4df48ac-dd04-4504-b828-66d7e2e986e1"
Sep 18 16:58:02.888614 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:58:02.888368    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lpt5n" podUID="89faf7f5-d633-4219-b3fc-323f8c23b599"
Sep 18 16:58:04.887627 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:04.887585    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fx8nf"
Sep 18 16:58:04.887627 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:04.887592    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpt5n"
Sep 18 16:58:04.887627 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:04.887597    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pc44x"
Sep 18 16:58:04.888160 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:58:04.887715    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fx8nf" podUID="a4df48ac-dd04-4504-b828-66d7e2e986e1"
Sep 18 16:58:04.888160 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:58:04.887796    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lpt5n" podUID="89faf7f5-d633-4219-b3fc-323f8c23b599"
Sep 18 16:58:04.888160 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:58:04.887976    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pc44x" podUID="6f463717-a487-485b-aa4b-7772d35b0d22"
Sep 18 16:58:04.888292 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:58:04.888196    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" podUID="a3127caef50db0be093e96af172c433b"
Sep 18 16:58:04.961125 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:58:04.961087    2881 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?"
Sep 18 16:58:06.887821 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:06.887777    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fx8nf"
Sep 18 16:58:06.888374 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:06.887777    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpt5n"
Sep 18 16:58:06.888374 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:58:06.887917    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fx8nf" podUID="a4df48ac-dd04-4504-b828-66d7e2e986e1"
Sep 18 16:58:06.888374 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:58:06.888006    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lpt5n" podUID="89faf7f5-d633-4219-b3fc-323f8c23b599"
Sep 18 16:58:06.888374 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:06.887777    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pc44x"
Sep 18 16:58:06.888374 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:58:06.888074    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pc44x" podUID="6f463717-a487-485b-aa4b-7772d35b0d22"
Sep 18 16:58:08.888375 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:08.888326    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpt5n"
Sep 18 16:58:08.888867 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:08.888325    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fx8nf"
Sep 18 16:58:08.888867 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:58:08.888459    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lpt5n" podUID="89faf7f5-d633-4219-b3fc-323f8c23b599"
Sep 18 16:58:08.888867 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:58:08.888520    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fx8nf" podUID="a4df48ac-dd04-4504-b828-66d7e2e986e1"
Sep 18 16:58:08.888867 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:08.888325    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pc44x"
Sep 18 16:58:08.888867 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:58:08.888624    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pc44x" podUID="6f463717-a487-485b-aa4b-7772d35b0d22"
Sep 18 16:58:09.962630 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:58:09.962586    2881 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?"
Sep 18 16:58:10.887894 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:10.887837    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpt5n"
Sep 18 16:58:10.887894 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:10.887873    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pc44x"
Sep 18 16:58:10.888175 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:10.887860    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fx8nf"
Sep 18 16:58:10.888175 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:58:10.887972    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lpt5n" podUID="89faf7f5-d633-4219-b3fc-323f8c23b599"
Sep 18 16:58:10.888175 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:58:10.888043    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fx8nf" podUID="a4df48ac-dd04-4504-b828-66d7e2e986e1"
Sep 18 16:58:10.888175 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:58:10.888090    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pc44x" podUID="6f463717-a487-485b-aa4b-7772d35b0d22"
Sep 18 16:58:12.887619 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:12.887582    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpt5n"
Sep 18 16:58:12.887619 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:12.887605    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pc44x"
Sep 18 16:58:12.888096 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:58:12.887701    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lpt5n" podUID="89faf7f5-d633-4219-b3fc-323f8c23b599"
Sep 18 16:58:12.888096 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:12.887725    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fx8nf"
Sep 18 16:58:12.888096 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:58:12.887801    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fx8nf" podUID="a4df48ac-dd04-4504-b828-66d7e2e986e1"
Sep 18 16:58:12.888096 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:58:12.887872    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pc44x" podUID="6f463717-a487-485b-aa4b-7772d35b0d22"
Sep 18 16:58:14.887756 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:14.887711    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fx8nf"
Sep 18 16:58:14.888259 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:14.887711    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpt5n"
Sep 18 16:58:14.888259 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:58:14.887836    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fx8nf" podUID="a4df48ac-dd04-4504-b828-66d7e2e986e1"
Sep 18 16:58:14.888259 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:14.887711    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pc44x"
Sep 18 16:58:14.888259 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:58:14.887942    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lpt5n" podUID="89faf7f5-d633-4219-b3fc-323f8c23b599"
Sep 18 16:58:14.888259 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:58:14.887969    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pc44x" podUID="6f463717-a487-485b-aa4b-7772d35b0d22"
Sep 18 16:58:16.887361 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:16.887318    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpt5n"
Sep 18 16:58:16.887839 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:16.887318    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fx8nf"
Sep 18 16:58:16.887839 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:16.887318    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pc44x"
Sep 18 16:58:16.890306 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:16.890279    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 16:58:16.890453 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:16.890279    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 16:58:16.891409 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:16.891389    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-plhlv\""
Sep 18 16:58:16.891495 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:16.891395    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 16:58:16.891495 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:16.891452    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-d4kk4\""
Sep 18 16:58:16.891495 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:16.891452    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 16:58:17.888048 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:58:17.888005    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" podUID="a3127caef50db0be093e96af172c433b"
Sep 18 16:58:20.790089 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:20.790060    2881 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-5.ec2.internal" event="NodeReady"
Sep 18 16:58:20.849883 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:20.849849    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-b66z2"]
Sep 18 16:58:20.887433 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:20.887393    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-kjcm9"]
Sep 18 16:58:20.887614 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:20.887515    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b66z2"
Sep 18 16:58:20.890241 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:20.890203    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 18 16:58:20.890241 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:20.890202    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-hn8sq\""
Sep 18 16:58:20.890458 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:20.890246    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 18 16:58:20.890458 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:20.890202    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 18 16:58:20.898407 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:20.898381    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b66z2"]
Sep 18 16:58:20.898533 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:20.898413    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kjcm9"]
Sep 18 16:58:20.898533 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:20.898471    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kjcm9"
Sep 18 16:58:20.901134 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:20.901108    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 18 16:58:20.901134 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:20.901108    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 18 16:58:20.901325 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:20.901210    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bm6wt\""
Sep 18 16:58:20.901325 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:20.901318    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 18 16:58:20.901434 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:20.901423    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 18 16:58:20.921046 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:20.921017    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5bf8875fb5-z4njj"]
Sep 18 16:58:20.937957 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:20.937925    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bf8875fb5-z4njj"]
Sep 18 16:58:20.938113 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:20.938017    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bf8875fb5-z4njj"
Sep 18 16:58:20.940837 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:20.940810    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 18 16:58:20.941022 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:20.940886    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 18 16:58:20.941022 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:20.940966    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 18 16:58:20.941199 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:20.941180    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 18 16:58:20.941316 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:20.941295    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 18 16:58:20.941432 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:20.941419    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-9s7t7\""
Sep 18 16:58:20.941615 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:20.941596    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 18 16:58:20.941738 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:20.941664    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 18 16:58:20.945307 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:20.945283    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 18 16:58:20.945461 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:20.945444    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k78kb\" (UniqueName: \"kubernetes.io/projected/7b5b1755-3465-43cd-b0d1-750ef9bce496-kube-api-access-k78kb\") pod \"ingress-canary-b66z2\" (UID: \"7b5b1755-3465-43cd-b0d1-750ef9bce496\") " pod="openshift-ingress-canary/ingress-canary-b66z2"
Sep 18 16:58:20.945519 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:20.945491    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7b5b1755-3465-43cd-b0d1-750ef9bce496-cert\") pod \"ingress-canary-b66z2\" (UID: \"7b5b1755-3465-43cd-b0d1-750ef9bce496\") " pod="openshift-ingress-canary/ingress-canary-b66z2"
Sep 18 16:58:20.950613 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:20.950586    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-c8l26"]
Sep 18 16:58:20.963976 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:20.963947    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-c8l26"]
Sep 18 16:58:20.964132 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:20.964069    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-c8l26"
Sep 18 16:58:20.966836 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:20.966814    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 18 16:58:20.966966 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:20.966840    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-kgcdr\""
Sep 18 16:58:20.966966 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:20.966951    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 18 16:58:21.045968 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:21.045868    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47f2a06a-1509-44ba-a107-56212158df3d-tmp-dir\") pod \"dns-default-c8l26\" (UID: \"47f2a06a-1509-44ba-a107-56212158df3d\") " pod="openshift-dns/dns-default-c8l26"
Sep 18 16:58:21.045968 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:21.045938    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124-console-oauth-config\") pod \"console-5bf8875fb5-z4njj\" (UID: \"44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124\") " pod="openshift-console/console-5bf8875fb5-z4njj"
Sep 18 16:58:21.045968 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:21.045965    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/47f2a06a-1509-44ba-a107-56212158df3d-metrics-tls\") pod \"dns-default-c8l26\" (UID: \"47f2a06a-1509-44ba-a107-56212158df3d\") " pod="openshift-dns/dns-default-c8l26"
Sep 18 16:58:21.046280 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:21.045993    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqz6r\" (UniqueName: \"kubernetes.io/projected/b7fad33b-7ced-4268-8aab-5e98a1831c7a-kube-api-access-sqz6r\") pod \"insights-runtime-extractor-kjcm9\" (UID: \"b7fad33b-7ced-4268-8aab-5e98a1831c7a\") " pod="openshift-insights/insights-runtime-extractor-kjcm9"
Sep 18 16:58:21.046280 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:21.046027    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k78kb\" (UniqueName: \"kubernetes.io/projected/7b5b1755-3465-43cd-b0d1-750ef9bce496-kube-api-access-k78kb\") pod \"ingress-canary-b66z2\" (UID: \"7b5b1755-3465-43cd-b0d1-750ef9bce496\") " pod="openshift-ingress-canary/ingress-canary-b66z2"
Sep 18 16:58:21.046280 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:21.046052    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124-oauth-serving-cert\") pod \"console-5bf8875fb5-z4njj\" (UID: \"44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124\") " pod="openshift-console/console-5bf8875fb5-z4njj"
Sep 18 16:58:21.046280 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:21.046071    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/47f2a06a-1509-44ba-a107-56212158df3d-config-volume\") pod \"dns-default-c8l26\" (UID: \"47f2a06a-1509-44ba-a107-56212158df3d\") " pod="openshift-dns/dns-default-c8l26"
Sep 18 16:58:21.046607 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:21.046585    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7b5b1755-3465-43cd-b0d1-750ef9bce496-cert\") pod \"ingress-canary-b66z2\" (UID: \"7b5b1755-3465-43cd-b0d1-750ef9bce496\") " pod="openshift-ingress-canary/ingress-canary-b66z2"
Sep 18 16:58:21.046675 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:21.046634    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b7fad33b-7ced-4268-8aab-5e98a1831c7a-crio-socket\") pod \"insights-runtime-extractor-kjcm9\" (UID: \"b7fad33b-7ced-4268-8aab-5e98a1831c7a\") " pod="openshift-insights/insights-runtime-extractor-kjcm9"
Sep 18 16:58:21.046827 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:21.046812    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124-trusted-ca-bundle\") pod \"console-5bf8875fb5-z4njj\" (UID: \"44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124\") " pod="openshift-console/console-5bf8875fb5-z4njj"
Sep 18 16:58:21.046890 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:21.046867    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b7fad33b-7ced-4268-8aab-5e98a1831c7a-data-volume\") pod \"insights-runtime-extractor-kjcm9\" (UID: \"b7fad33b-7ced-4268-8aab-5e98a1831c7a\") " pod="openshift-insights/insights-runtime-extractor-kjcm9"
Sep 18 16:58:21.046946 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:21.046905    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7fad33b-7ced-4268-8aab-5e98a1831c7a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kjcm9\" (UID: \"b7fad33b-7ced-4268-8aab-5e98a1831c7a\") " pod="openshift-insights/insights-runtime-extractor-kjcm9"
Sep 18 16:58:21.046946 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:21.046939    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124-console-config\") pod \"console-5bf8875fb5-z4njj\" (UID: \"44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124\") " pod="openshift-console/console-5bf8875fb5-z4njj"
Sep 18 16:58:21.047050 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:21.046973    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87lpz\" (UniqueName: \"kubernetes.io/projected/44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124-kube-api-access-87lpz\") pod \"console-5bf8875fb5-z4njj\" (UID: \"44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124\") " pod="openshift-console/console-5bf8875fb5-z4njj"
Sep 18 16:58:21.047050 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:21.047000    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lt8z\" (UniqueName: \"kubernetes.io/projected/47f2a06a-1509-44ba-a107-56212158df3d-kube-api-access-4lt8z\") pod \"dns-default-c8l26\" (UID: \"47f2a06a-1509-44ba-a107-56212158df3d\") " pod="openshift-dns/dns-default-c8l26"
Sep 18 16:58:21.047050 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:21.047036    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124-service-ca\") pod \"console-5bf8875fb5-z4njj\" (UID: \"44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124\") " pod="openshift-console/console-5bf8875fb5-z4njj"
Sep 18 16:58:21.047200 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:21.047077    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7fad33b-7ced-4268-8aab-5e98a1831c7a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kjcm9\" (UID: \"b7fad33b-7ced-4268-8aab-5e98a1831c7a\") " pod="openshift-insights/insights-runtime-extractor-kjcm9"
Sep 18 16:58:21.047200 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:21.047116    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124-console-serving-cert\") pod \"console-5bf8875fb5-z4njj\" (UID: \"44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124\") " pod="openshift-console/console-5bf8875fb5-z4njj"
Sep 18 16:58:21.051936 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:21.051903    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7b5b1755-3465-43cd-b0d1-750ef9bce496-cert\") pod \"ingress-canary-b66z2\" (UID: \"7b5b1755-3465-43cd-b0d1-750ef9bce496\") " pod="openshift-ingress-canary/ingress-canary-b66z2"
Sep 18 16:58:21.058728 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:21.058701    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k78kb\" (UniqueName: \"kubernetes.io/projected/7b5b1755-3465-43cd-b0d1-750ef9bce496-kube-api-access-k78kb\") pod \"ingress-canary-b66z2\" (UID: \"7b5b1755-3465-43cd-b0d1-750ef9bce496\") " pod="openshift-ingress-canary/ingress-canary-b66z2"
Sep 18 16:58:21.147920 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:21.147879    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqz6r\" (UniqueName: \"kubernetes.io/projected/b7fad33b-7ced-4268-8aab-5e98a1831c7a-kube-api-access-sqz6r\") pod \"insights-runtime-extractor-kjcm9\" (UID: \"b7fad33b-7ced-4268-8aab-5e98a1831c7a\") " pod="openshift-insights/insights-runtime-extractor-kjcm9"
Sep 18 16:58:21.147920 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:21.147924    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124-oauth-serving-cert\") pod \"console-5bf8875fb5-z4njj\" (UID: \"44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124\") " pod="openshift-console/console-5bf8875fb5-z4njj"
Sep 18 16:58:21.148204 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:21.147941    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/47f2a06a-1509-44ba-a107-56212158df3d-config-volume\") pod \"dns-default-c8l26\" (UID: \"47f2a06a-1509-44ba-a107-56212158df3d\") " pod="openshift-dns/dns-default-c8l26"
Sep 18 16:58:21.148204 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:21.147980    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b7fad33b-7ced-4268-8aab-5e98a1831c7a-crio-socket\") pod \"insights-runtime-extractor-kjcm9\" (UID: \"b7fad33b-7ced-4268-8aab-5e98a1831c7a\") " pod="openshift-insights/insights-runtime-extractor-kjcm9"
Sep 18 16:58:21.148204 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:21.148002    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124-trusted-ca-bundle\") pod \"console-5bf8875fb5-z4njj\" (UID: \"44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124\") " pod="openshift-console/console-5bf8875fb5-z4njj"
Sep 18 16:58:21.148204 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:21.148082    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b7fad33b-7ced-4268-8aab-5e98a1831c7a-data-volume\") pod \"insights-runtime-extractor-kjcm9\" (UID: \"b7fad33b-7ced-4268-8aab-5e98a1831c7a\") " pod="openshift-insights/insights-runtime-extractor-kjcm9"
Sep 18 16:58:21.148204 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:21.148110    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b7fad33b-7ced-4268-8aab-5e98a1831c7a-crio-socket\") pod \"insights-runtime-extractor-kjcm9\" (UID: \"b7fad33b-7ced-4268-8aab-5e98a1831c7a\") " pod="openshift-insights/insights-runtime-extractor-kjcm9"
Sep 18 16:58:21.148204 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:21.148129    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7fad33b-7ced-4268-8aab-5e98a1831c7a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kjcm9\" (UID: \"b7fad33b-7ced-4268-8aab-5e98a1831c7a\") " pod="openshift-insights/insights-runtime-extractor-kjcm9"
Sep 18 16:58:21.148505 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:21.148295    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124-console-config\") pod \"console-5bf8875fb5-z4njj\" (UID: \"44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124\") " pod="openshift-console/console-5bf8875fb5-z4njj"
Sep 18 16:58:21.148548 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:21.148506    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b7fad33b-7ced-4268-8aab-5e98a1831c7a-data-volume\") pod \"insights-runtime-extractor-kjcm9\" (UID: \"b7fad33b-7ced-4268-8aab-5e98a1831c7a\") " pod="openshift-insights/insights-runtime-extractor-kjcm9"
Sep 18 16:58:21.148548 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:21.148525    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87lpz\" (UniqueName: \"kubernetes.io/projected/44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124-kube-api-access-87lpz\") pod \"console-5bf8875fb5-z4njj\" (UID: \"44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124\") " pod="openshift-console/console-5bf8875fb5-z4njj"
Sep 18 16:58:21.148637 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:21.148560    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lt8z\" (UniqueName: \"kubernetes.io/projected/47f2a06a-1509-44ba-a107-56212158df3d-kube-api-access-4lt8z\") pod \"dns-default-c8l26\" (UID: \"47f2a06a-1509-44ba-a107-56212158df3d\") " pod="openshift-dns/dns-default-c8l26"
Sep 18 16:58:21.148637 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:21.148590    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124-service-ca\") pod \"console-5bf8875fb5-z4njj\" (UID: \"44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124\") " pod="openshift-console/console-5bf8875fb5-z4njj"
Sep 18 16:58:21.148637 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:21.148621    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7fad33b-7ced-4268-8aab-5e98a1831c7a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kjcm9\" (UID: \"b7fad33b-7ced-4268-8aab-5e98a1831c7a\") " pod="openshift-insights/insights-runtime-extractor-kjcm9"
Sep 18 16:58:21.148867 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:21.148653    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124-console-serving-cert\") pod \"console-5bf8875fb5-z4njj\" (UID: \"44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124\") " pod="openshift-console/console-5bf8875fb5-z4njj"
Sep 18 16:58:21.148867 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:21.148684    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47f2a06a-1509-44ba-a107-56212158df3d-tmp-dir\") pod \"dns-default-c8l26\" (UID: \"47f2a06a-1509-44ba-a107-56212158df3d\") " pod="openshift-dns/dns-default-c8l26"
Sep 18 16:58:21.148867 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:21.148746    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124-console-oauth-config\") pod \"console-5bf8875fb5-z4njj\" (UID: \"44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124\") " pod="openshift-console/console-5bf8875fb5-z4njj"
Sep 18 16:58:21.148867 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:21.148776    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/47f2a06a-1509-44ba-a107-56212158df3d-metrics-tls\") pod \"dns-default-c8l26\" (UID: \"47f2a06a-1509-44ba-a107-56212158df3d\") " pod="openshift-dns/dns-default-c8l26"
Sep 18 16:58:21.148867 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:21.148815    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7fad33b-7ced-4268-8aab-5e98a1831c7a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kjcm9\" (UID: \"b7fad33b-7ced-4268-8aab-5e98a1831c7a\") " pod="openshift-insights/insights-runtime-extractor-kjcm9"
Sep 18 16:58:21.149128 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:21.148910    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124-oauth-serving-cert\") pod \"console-5bf8875fb5-z4njj\" (UID: \"44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124\") " pod="openshift-console/console-5bf8875fb5-z4njj"
Sep 18 16:58:21.149128 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:21.148943    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124-console-config\") pod \"console-5bf8875fb5-z4njj\" (UID: \"44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124\") " pod="openshift-console/console-5bf8875fb5-z4njj"
Sep 18 16:58:21.149128 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:21.148998    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124-trusted-ca-bundle\") pod \"console-5bf8875fb5-z4njj\" (UID: \"44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124\") " pod="openshift-console/console-5bf8875fb5-z4njj"
Sep 18 16:58:21.149301 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:21.149186    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47f2a06a-1509-44ba-a107-56212158df3d-tmp-dir\") pod \"dns-default-c8l26\" (UID: \"47f2a06a-1509-44ba-a107-56212158df3d\") " pod="openshift-dns/dns-default-c8l26"
Sep 18 16:58:21.149395 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:21.149368    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/47f2a06a-1509-44ba-a107-56212158df3d-config-volume\") pod \"dns-default-c8l26\" (UID: \"47f2a06a-1509-44ba-a107-56212158df3d\") " pod="openshift-dns/dns-default-c8l26"
Sep 18 16:58:21.149501 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:21.149484    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124-service-ca\") pod \"console-5bf8875fb5-z4njj\" (UID: \"44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124\") " pod="openshift-console/console-5bf8875fb5-z4njj"
Sep 18 16:58:21.151103 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:21.151084    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/47f2a06a-1509-44ba-a107-56212158df3d-metrics-tls\") pod \"dns-default-c8l26\" (UID: \"47f2a06a-1509-44ba-a107-56212158df3d\") " pod="openshift-dns/dns-default-c8l26"
Sep 18 16:58:21.151320 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:21.151301    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124-console-serving-cert\") pod \"console-5bf8875fb5-z4njj\" (UID: \"44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124\") " pod="openshift-console/console-5bf8875fb5-z4njj"
Sep 18 16:58:21.151530 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:21.151507    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7fad33b-7ced-4268-8aab-5e98a1831c7a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kjcm9\" (UID: \"b7fad33b-7ced-4268-8aab-5e98a1831c7a\") " pod="openshift-insights/insights-runtime-extractor-kjcm9"
Sep 18 16:58:21.151596 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:21.151583    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124-console-oauth-config\") pod \"console-5bf8875fb5-z4njj\" (UID: \"44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124\") " pod="openshift-console/console-5bf8875fb5-z4njj"
Sep 18 16:58:21.161274 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:21.161247    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87lpz\" (UniqueName: \"kubernetes.io/projected/44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124-kube-api-access-87lpz\") pod \"console-5bf8875fb5-z4njj\" (UID: \"44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124\") " pod="openshift-console/console-5bf8875fb5-z4njj"
Sep 18 16:58:21.161392 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:21.161246    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lt8z\" (UniqueName: \"kubernetes.io/projected/47f2a06a-1509-44ba-a107-56212158df3d-kube-api-access-4lt8z\") pod \"dns-default-c8l26\" (UID: \"47f2a06a-1509-44ba-a107-56212158df3d\") " pod="openshift-dns/dns-default-c8l26"
Sep 18 16:58:21.161392 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:21.161350    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqz6r\" (UniqueName: \"kubernetes.io/projected/b7fad33b-7ced-4268-8aab-5e98a1831c7a-kube-api-access-sqz6r\") pod \"insights-runtime-extractor-kjcm9\" (UID: \"b7fad33b-7ced-4268-8aab-5e98a1831c7a\") " pod="openshift-insights/insights-runtime-extractor-kjcm9"
Sep 18 16:58:21.196298 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:21.196214    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b66z2"
Sep 18 16:58:21.206539 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:21.206521    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kjcm9"
Sep 18 16:58:21.247490 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:21.247451    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bf8875fb5-z4njj"
Sep 18 16:58:21.274056 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:21.274010    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-c8l26"
Sep 18 16:58:21.466621 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:21.466592    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7dglz"
Sep 18 16:58:21.485648 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:21.485616    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b66z2"]
Sep 18 16:58:21.488962 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:21.488925    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kjcm9"]
Sep 18 16:58:21.489379 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:58:21.489349    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b5b1755_3465_43cd_b0d1_750ef9bce496.slice/crio-f42439f3ca358eff7d677a9cbc7e15cc400468889e1b4fd12cd073d98efbb3d5 WatchSource:0}: Error finding container f42439f3ca358eff7d677a9cbc7e15cc400468889e1b4fd12cd073d98efbb3d5: Status 404 returned error can't find the container with id f42439f3ca358eff7d677a9cbc7e15cc400468889e1b4fd12cd073d98efbb3d5
Sep 18 16:58:21.491799 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:58:21.491773    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7fad33b_7ced_4268_8aab_5e98a1831c7a.slice/crio-f91e65615bd528ddb18e3f1d2d6b88c1df7d32219e69a2ff6267a23e53a749fd WatchSource:0}: Error finding container f91e65615bd528ddb18e3f1d2d6b88c1df7d32219e69a2ff6267a23e53a749fd: Status 404 returned error can't find the container with id f91e65615bd528ddb18e3f1d2d6b88c1df7d32219e69a2ff6267a23e53a749fd
Sep 18 16:58:21.533020 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:21.532987    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bf8875fb5-z4njj"]
Sep 18 16:58:21.537150 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:21.537113    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b66z2" event={"ID":"7b5b1755-3465-43cd-b0d1-750ef9bce496","Type":"ContainerStarted","Data":"f42439f3ca358eff7d677a9cbc7e15cc400468889e1b4fd12cd073d98efbb3d5"}
Sep 18 16:58:21.537293 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:58:21.537128    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44e9b3b1_f2cd_47e8_beb6_3b6a2c3e3124.slice/crio-14c52aaeae102c3cd976efe66fdda8aab7fdf9f5d437c94c66490b3c73fc0850 WatchSource:0}: Error finding container 14c52aaeae102c3cd976efe66fdda8aab7fdf9f5d437c94c66490b3c73fc0850: Status 404 returned error can't find the container with id 14c52aaeae102c3cd976efe66fdda8aab7fdf9f5d437c94c66490b3c73fc0850
Sep 18 16:58:21.538499 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:21.538475    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kjcm9" event={"ID":"b7fad33b-7ced-4268-8aab-5e98a1831c7a","Type":"ContainerStarted","Data":"f91e65615bd528ddb18e3f1d2d6b88c1df7d32219e69a2ff6267a23e53a749fd"}
Sep 18 16:58:21.550742 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:21.550710    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-c8l26"]
Sep 18 16:58:21.554278 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:58:21.554253    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47f2a06a_1509_44ba_a107_56212158df3d.slice/crio-21fb3040b945f9d7535a850b69acfcdbbb8e937d9b89b125db49f0132ba4db55 WatchSource:0}: Error finding container 21fb3040b945f9d7535a850b69acfcdbbb8e937d9b89b125db49f0132ba4db55: Status 404 returned error can't find the container with id 21fb3040b945f9d7535a850b69acfcdbbb8e937d9b89b125db49f0132ba4db55
Sep 18 16:58:22.360248 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:22.360016    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-skzgp"]
Sep 18 16:58:22.364262 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:22.363561    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-skzgp"
Sep 18 16:58:22.368201 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:22.367389    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 18 16:58:22.368201 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:22.367475    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 18 16:58:22.368201 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:22.367597    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 18 16:58:22.368201 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:22.367667    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-sgk7t\""
Sep 18 16:58:22.368201 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:22.367780    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 18 16:58:22.368201 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:22.367851    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 18 16:58:22.375069 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:22.374994    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-skzgp"]
Sep 18 16:58:22.469033 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:22.468820    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nshzx\" (UniqueName: \"kubernetes.io/projected/809dca2e-a794-48bd-a35c-eafd66d3a19b-kube-api-access-nshzx\") pod \"prometheus-operator-7bd5d56d6-skzgp\" (UID: \"809dca2e-a794-48bd-a35c-eafd66d3a19b\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-skzgp"
Sep 18 16:58:22.469033 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:22.468893    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/809dca2e-a794-48bd-a35c-eafd66d3a19b-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-skzgp\" (UID: \"809dca2e-a794-48bd-a35c-eafd66d3a19b\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-skzgp"
Sep 18 16:58:22.469033 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:22.468976    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/809dca2e-a794-48bd-a35c-eafd66d3a19b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-skzgp\" (UID: \"809dca2e-a794-48bd-a35c-eafd66d3a19b\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-skzgp"
Sep 18 16:58:22.469417 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:22.469043    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/809dca2e-a794-48bd-a35c-eafd66d3a19b-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-skzgp\" (UID: \"809dca2e-a794-48bd-a35c-eafd66d3a19b\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-skzgp"
Sep 18 16:58:22.553326 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:22.545915    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kjcm9" event={"ID":"b7fad33b-7ced-4268-8aab-5e98a1831c7a","Type":"ContainerStarted","Data":"a4df770cf46fd3af4a51542fe9a43c6537bedb88b1ece43bf6b5934ff3d806fc"}
Sep 18 16:58:22.553326 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:22.545993    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kjcm9" event={"ID":"b7fad33b-7ced-4268-8aab-5e98a1831c7a","Type":"ContainerStarted","Data":"5c2afd09a7606c61262144f15c4093572ef42e9fa6dcb1c48449a759ad3bdbbe"}
Sep 18 16:58:22.553326 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:22.549453    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c8l26" event={"ID":"47f2a06a-1509-44ba-a107-56212158df3d","Type":"ContainerStarted","Data":"21fb3040b945f9d7535a850b69acfcdbbb8e937d9b89b125db49f0132ba4db55"}
Sep 18 16:58:22.557395 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:22.554929    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bf8875fb5-z4njj" event={"ID":"44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124","Type":"ContainerStarted","Data":"14c52aaeae102c3cd976efe66fdda8aab7fdf9f5d437c94c66490b3c73fc0850"}
Sep 18 16:58:22.569634 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:22.569591    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/809dca2e-a794-48bd-a35c-eafd66d3a19b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-skzgp\" (UID: \"809dca2e-a794-48bd-a35c-eafd66d3a19b\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-skzgp"
Sep 18 16:58:22.569801 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:22.569648    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/809dca2e-a794-48bd-a35c-eafd66d3a19b-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-skzgp\" (UID: \"809dca2e-a794-48bd-a35c-eafd66d3a19b\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-skzgp"
Sep 18 16:58:22.569801 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:22.569729    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nshzx\" (UniqueName: \"kubernetes.io/projected/809dca2e-a794-48bd-a35c-eafd66d3a19b-kube-api-access-nshzx\") pod \"prometheus-operator-7bd5d56d6-skzgp\" (UID: \"809dca2e-a794-48bd-a35c-eafd66d3a19b\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-skzgp"
Sep 18 16:58:22.569801 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:22.569766    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/809dca2e-a794-48bd-a35c-eafd66d3a19b-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-skzgp\" (UID: \"809dca2e-a794-48bd-a35c-eafd66d3a19b\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-skzgp"
Sep 18 16:58:22.571834 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:22.571806    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/809dca2e-a794-48bd-a35c-eafd66d3a19b-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-skzgp\" (UID: \"809dca2e-a794-48bd-a35c-eafd66d3a19b\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-skzgp"
Sep 18 16:58:22.574336 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:22.574310    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/809dca2e-a794-48bd-a35c-eafd66d3a19b-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-skzgp\" (UID: \"809dca2e-a794-48bd-a35c-eafd66d3a19b\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-skzgp"
Sep 18 16:58:22.574764 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:22.574735    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/809dca2e-a794-48bd-a35c-eafd66d3a19b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-skzgp\" (UID: \"809dca2e-a794-48bd-a35c-eafd66d3a19b\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-skzgp"
Sep 18 16:58:22.584994 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:22.584962    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nshzx\" (UniqueName: \"kubernetes.io/projected/809dca2e-a794-48bd-a35c-eafd66d3a19b-kube-api-access-nshzx\") pod \"prometheus-operator-7bd5d56d6-skzgp\" (UID: \"809dca2e-a794-48bd-a35c-eafd66d3a19b\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-skzgp"
Sep 18 16:58:22.679801 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:22.679755    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-skzgp"
Sep 18 16:58:22.990076 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:22.989955    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-skzgp"]
Sep 18 16:58:23.001149 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:58:23.001109    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod809dca2e_a794_48bd_a35c_eafd66d3a19b.slice/crio-e2b5dc9ce0e08094c2ec1d25e86f54e6850d6281202e47d4d2247a9687409f06 WatchSource:0}: Error finding container e2b5dc9ce0e08094c2ec1d25e86f54e6850d6281202e47d4d2247a9687409f06: Status 404 returned error can't find the container with id e2b5dc9ce0e08094c2ec1d25e86f54e6850d6281202e47d4d2247a9687409f06
Sep 18 16:58:23.560572 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:23.560518    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-skzgp" event={"ID":"809dca2e-a794-48bd-a35c-eafd66d3a19b","Type":"ContainerStarted","Data":"e2b5dc9ce0e08094c2ec1d25e86f54e6850d6281202e47d4d2247a9687409f06"}
Sep 18 16:58:28.581447 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:28.581309    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bf8875fb5-z4njj" event={"ID":"44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124","Type":"ContainerStarted","Data":"e2b8212407ef3b1521331a8fa22162d9bc0f042f936ac1a98b25ff7c3172910c"}
Sep 18 16:58:28.585012 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:28.584706    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kjcm9" event={"ID":"b7fad33b-7ced-4268-8aab-5e98a1831c7a","Type":"ContainerStarted","Data":"abacaab377796ed06ca19c8eb8e08e699a514ecc330dc1a2473c60df433e8825"}
Sep 18 16:58:28.587733 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:28.587694    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c8l26" event={"ID":"47f2a06a-1509-44ba-a107-56212158df3d","Type":"ContainerStarted","Data":"b26457a9dca16c601d302b894c01351cdedabfacc548c3bbabf212586a0ffefd"}
Sep 18 16:58:28.587932 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:28.587759    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c8l26" event={"ID":"47f2a06a-1509-44ba-a107-56212158df3d","Type":"ContainerStarted","Data":"c3292e873664381c1d465ced512012e61dc01f7b7236a058e4e432207a8a0299"}
Sep 18 16:58:28.587932 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:28.587798    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-c8l26"
Sep 18 16:58:28.590949 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:28.590908    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-skzgp" event={"ID":"809dca2e-a794-48bd-a35c-eafd66d3a19b","Type":"ContainerStarted","Data":"9bb2c4be325f1d0a0cb1d19825bf30c753e89eab98e61e2d854aa32315057379"}
Sep 18 16:58:28.591116 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:28.590979    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-skzgp" event={"ID":"809dca2e-a794-48bd-a35c-eafd66d3a19b","Type":"ContainerStarted","Data":"e7d8797b77c1b22a52e07653980a0ffa6f8749b34b6ad7c3818c1a8f79bf903c"}
Sep 18 16:58:28.594020 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:28.593963    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b66z2" event={"ID":"7b5b1755-3465-43cd-b0d1-750ef9bce496","Type":"ContainerStarted","Data":"b4fe8d50eb00809c016d8d7a4e85afc6d13c157ce3a8e2cf658fc3d3aae804c0"}
Sep 18 16:58:28.604755 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:28.604703    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5bf8875fb5-z4njj" podStartSLOduration=1.845090287 podStartE2EDuration="8.604685544s" totalImagesPullingTime="6.759595257s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:58:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:58:28.60397323 +0000 UTC m=+298.803828506" watchObservedRunningTime="2026-09-18 16:58:28.604685544 +0000 UTC m=+298.804540829"
Sep 18 16:58:28.651031 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:28.650964    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-skzgp" podStartSLOduration=1.368193752 podStartE2EDuration="6.650942001s" totalImagesPullingTime="5.282748249s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:58:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:58:28.627186791 +0000 UTC m=+298.827042081" watchObservedRunningTime="2026-09-18 16:58:28.650942001 +0000 UTC m=+298.850797288"
Sep 18 16:58:28.651358 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:28.651335    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-kjcm9" podStartSLOduration=2.669210451 podStartE2EDuration="8.651328085s" totalImagesPullingTime="5.982117634s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:58:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:58:28.650557897 +0000 UTC m=+298.850413187" watchObservedRunningTime="2026-09-18 16:58:28.651328085 +0000 UTC m=+298.851183566"
Sep 18 16:58:28.672042 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:28.671984    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-b66z2" podStartSLOduration=1.8838337859999998 podStartE2EDuration="8.671967956s" totalImagesPullingTime="6.78813417s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:58:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:58:28.671570211 +0000 UTC m=+298.871425498" watchObservedRunningTime="2026-09-18 16:58:28.671967956 +0000 UTC m=+298.871823239"
Sep 18 16:58:28.698199 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:28.698139    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-c8l26" podStartSLOduration=2.22228922 podStartE2EDuration="8.698119878s" totalImagesPullingTime="6.475830658s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:58:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:58:28.697688946 +0000 UTC m=+298.897544233" watchObservedRunningTime="2026-09-18 16:58:28.698119878 +0000 UTC m=+298.897975163"
Sep 18 16:58:28.888602 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:58:28.888491    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" podUID="a3127caef50db0be093e96af172c433b"
Sep 18 16:58:28.928831 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:28.928786    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89faf7f5-d633-4219-b3fc-323f8c23b599-metrics-certs\") pod \"network-metrics-daemon-lpt5n\" (UID: \"89faf7f5-d633-4219-b3fc-323f8c23b599\") " pod="openshift-multus/network-metrics-daemon-lpt5n"
Sep 18 16:58:28.929038 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:28.928872    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a4df48ac-dd04-4504-b828-66d7e2e986e1-original-pull-secret\") pod \"global-pull-secret-syncer-fx8nf\" (UID: \"a4df48ac-dd04-4504-b828-66d7e2e986e1\") " pod="kube-system/global-pull-secret-syncer-fx8nf"
Sep 18 16:58:28.931318 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:28.931289    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89faf7f5-d633-4219-b3fc-323f8c23b599-metrics-certs\") pod \"network-metrics-daemon-lpt5n\" (UID: \"89faf7f5-d633-4219-b3fc-323f8c23b599\") " pod="openshift-multus/network-metrics-daemon-lpt5n"
Sep 18 16:58:28.931435 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:28.931318    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a4df48ac-dd04-4504-b828-66d7e2e986e1-original-pull-secret\") pod \"global-pull-secret-syncer-fx8nf\" (UID: \"a4df48ac-dd04-4504-b828-66d7e2e986e1\") " pod="kube-system/global-pull-secret-syncer-fx8nf"
Sep 18 16:58:29.029716 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:29.029660    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhsfc\" (UniqueName: \"kubernetes.io/projected/6f463717-a487-485b-aa4b-7772d35b0d22-kube-api-access-rhsfc\") pod \"network-check-target-pc44x\" (UID: \"6f463717-a487-485b-aa4b-7772d35b0d22\") " pod="openshift-network-diagnostics/network-check-target-pc44x"
Sep 18 16:58:29.032276 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:29.032251    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhsfc\" (UniqueName: \"kubernetes.io/projected/6f463717-a487-485b-aa4b-7772d35b0d22-kube-api-access-rhsfc\") pod \"network-check-target-pc44x\" (UID: \"6f463717-a487-485b-aa4b-7772d35b0d22\") " pod="openshift-network-diagnostics/network-check-target-pc44x"
Sep 18 16:58:29.198516 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:29.198418    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpt5n"
Sep 18 16:58:29.203898 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:29.203873    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fx8nf"
Sep 18 16:58:29.208709 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:29.208684    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pc44x"
Sep 18 16:58:29.476384 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:29.476302    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lpt5n"]
Sep 18 16:58:29.479290 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:29.479261    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fx8nf"]
Sep 18 16:58:29.479922 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:58:29.479895    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89faf7f5_d633_4219_b3fc_323f8c23b599.slice/crio-411f44e8315032af27ca42fcac33efa52a5391e9a6c5d846ef6cd44b5e4fe586 WatchSource:0}: Error finding container 411f44e8315032af27ca42fcac33efa52a5391e9a6c5d846ef6cd44b5e4fe586: Status 404 returned error can't find the container with id 411f44e8315032af27ca42fcac33efa52a5391e9a6c5d846ef6cd44b5e4fe586
Sep 18 16:58:29.482107 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:58:29.482081    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4df48ac_dd04_4504_b828_66d7e2e986e1.slice/crio-d08dba84d3877db7b87c75a43e52c35383aa0e3bbefa1eec8af6538c426938db WatchSource:0}: Error finding container d08dba84d3877db7b87c75a43e52c35383aa0e3bbefa1eec8af6538c426938db: Status 404 returned error can't find the container with id d08dba84d3877db7b87c75a43e52c35383aa0e3bbefa1eec8af6538c426938db
Sep 18 16:58:29.505954 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:29.505919    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pc44x"]
Sep 18 16:58:29.509706 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:58:29.509676    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f463717_a487_485b_aa4b_7772d35b0d22.slice/crio-8bd1b0ce46fa5d73c258535253e260394facb4bcfdfedd10f4dbbddeca3be2a3 WatchSource:0}: Error finding container 8bd1b0ce46fa5d73c258535253e260394facb4bcfdfedd10f4dbbddeca3be2a3: Status 404 returned error can't find the container with id 8bd1b0ce46fa5d73c258535253e260394facb4bcfdfedd10f4dbbddeca3be2a3
Sep 18 16:58:29.598109 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:29.598066    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fx8nf" event={"ID":"a4df48ac-dd04-4504-b828-66d7e2e986e1","Type":"ContainerStarted","Data":"d08dba84d3877db7b87c75a43e52c35383aa0e3bbefa1eec8af6538c426938db"}
Sep 18 16:58:29.599330 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:29.599305    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pc44x" event={"ID":"6f463717-a487-485b-aa4b-7772d35b0d22","Type":"ContainerStarted","Data":"8bd1b0ce46fa5d73c258535253e260394facb4bcfdfedd10f4dbbddeca3be2a3"}
Sep 18 16:58:29.600575 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:29.600520    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lpt5n" event={"ID":"89faf7f5-d633-4219-b3fc-323f8c23b599","Type":"ContainerStarted","Data":"411f44e8315032af27ca42fcac33efa52a5391e9a6c5d846ef6cd44b5e4fe586"}
Sep 18 16:58:29.902981 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:29.902832    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_a3127caef50db0be093e96af172c433b/setup/10.log"
Sep 18 16:58:29.920186 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:29.904479    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_a3127caef50db0be093e96af172c433b/setup/10.log"
Sep 18 16:58:29.920186 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:29.905280    2881 kubelet.go:1648] "Image garbage collection succeeded"
Sep 18 16:58:30.792273 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:30.792143    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-dn7sr"]
Sep 18 16:58:30.802456 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:30.796587    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-tblfl"]
Sep 18 16:58:30.802456 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:30.798372    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-dn7sr"
Sep 18 16:58:30.802456 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:30.799543    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-tblfl"
Sep 18 16:58:30.802456 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:30.801364    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-dn7sr"]
Sep 18 16:58:30.805050 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:30.804595    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-9tbbc\""
Sep 18 16:58:30.805587 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:30.805561    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 18 16:58:30.805877 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:30.805685    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-hnf52\""
Sep 18 16:58:30.806175 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:30.806156    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 18 16:58:30.810135 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:30.808086    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 18 16:58:30.810135 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:30.808392    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 18 16:58:30.810135 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:30.808539    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-tblfl"]
Sep 18 16:58:30.810889 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:30.810709    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 18 16:58:30.815648 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:30.815580    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-pbhzd"]
Sep 18 16:58:30.818562 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:30.818497    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pbhzd"
Sep 18 16:58:30.821307 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:30.820988    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 18 16:58:30.821660 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:30.821639    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 18 16:58:30.821753 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:30.821711    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-7xszz\""
Sep 18 16:58:30.821828 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:30.821650    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 18 16:58:30.945487 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:30.945450    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bea5387e-1e26-4be9-bd38-71c49eb4225d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pbhzd\" (UID: \"bea5387e-1e26-4be9-bd38-71c49eb4225d\") " pod="openshift-monitoring/node-exporter-pbhzd"
Sep 18 16:58:30.945685 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:30.945503    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c199118-475a-4607-ba80-9a8027f1b047-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-tblfl\" (UID: \"8c199118-475a-4607-ba80-9a8027f1b047\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-tblfl"
Sep 18 16:58:30.945685 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:30.945553    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/69a62b4d-4c8d-4dc4-81a8-35628f51c837-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-dn7sr\" (UID: \"69a62b4d-4c8d-4dc4-81a8-35628f51c837\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-dn7sr"
Sep 18 16:58:30.945685 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:30.945573    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c199118-475a-4607-ba80-9a8027f1b047-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-tblfl\" (UID: \"8c199118-475a-4607-ba80-9a8027f1b047\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-tblfl"
Sep 18 16:58:30.945685 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:30.945599    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bea5387e-1e26-4be9-bd38-71c49eb4225d-root\") pod \"node-exporter-pbhzd\" (UID: \"bea5387e-1e26-4be9-bd38-71c49eb4225d\") " pod="openshift-monitoring/node-exporter-pbhzd"
Sep 18 16:58:30.945685 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:30.945634    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/69a62b4d-4c8d-4dc4-81a8-35628f51c837-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-dn7sr\" (UID: \"69a62b4d-4c8d-4dc4-81a8-35628f51c837\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-dn7sr"
Sep 18 16:58:30.945685 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:30.945667    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8c199118-475a-4607-ba80-9a8027f1b047-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-tblfl\" (UID: \"8c199118-475a-4607-ba80-9a8027f1b047\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-tblfl"
Sep 18 16:58:30.946179 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:30.945700    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/69a62b4d-4c8d-4dc4-81a8-35628f51c837-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-dn7sr\" (UID: \"69a62b4d-4c8d-4dc4-81a8-35628f51c837\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-dn7sr"
Sep 18 16:58:30.946179 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:30.945725    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bea5387e-1e26-4be9-bd38-71c49eb4225d-node-exporter-textfile\") pod \"node-exporter-pbhzd\" (UID: \"bea5387e-1e26-4be9-bd38-71c49eb4225d\") " pod="openshift-monitoring/node-exporter-pbhzd"
Sep 18 16:58:30.946179 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:30.945770    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bea5387e-1e26-4be9-bd38-71c49eb4225d-metrics-client-ca\") pod \"node-exporter-pbhzd\" (UID: \"bea5387e-1e26-4be9-bd38-71c49eb4225d\") " pod="openshift-monitoring/node-exporter-pbhzd"
Sep 18 16:58:30.946179 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:30.945806    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8c199118-475a-4607-ba80-9a8027f1b047-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-tblfl\" (UID: \"8c199118-475a-4607-ba80-9a8027f1b047\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-tblfl"
Sep 18 16:58:30.946179 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:30.945836    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bea5387e-1e26-4be9-bd38-71c49eb4225d-sys\") pod \"node-exporter-pbhzd\" (UID: \"bea5387e-1e26-4be9-bd38-71c49eb4225d\") " pod="openshift-monitoring/node-exporter-pbhzd"
Sep 18 16:58:30.946179 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:30.945866    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bea5387e-1e26-4be9-bd38-71c49eb4225d-node-exporter-accelerators-collector-config\") pod \"node-exporter-pbhzd\" (UID: \"bea5387e-1e26-4be9-bd38-71c49eb4225d\") " pod="openshift-monitoring/node-exporter-pbhzd"
Sep 18 16:58:30.946179 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:30.945916    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lssbx\" (UniqueName: \"kubernetes.io/projected/69a62b4d-4c8d-4dc4-81a8-35628f51c837-kube-api-access-lssbx\") pod \"openshift-state-metrics-86c5bc79d6-dn7sr\" (UID: \"69a62b4d-4c8d-4dc4-81a8-35628f51c837\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-dn7sr"
Sep 18 16:58:30.946179 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:30.945953    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8c199118-475a-4607-ba80-9a8027f1b047-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-tblfl\" (UID: \"8c199118-475a-4607-ba80-9a8027f1b047\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-tblfl"
Sep 18 16:58:30.946179 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:30.945988    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bea5387e-1e26-4be9-bd38-71c49eb4225d-node-exporter-tls\") pod \"node-exporter-pbhzd\" (UID: \"bea5387e-1e26-4be9-bd38-71c49eb4225d\") " pod="openshift-monitoring/node-exporter-pbhzd"
Sep 18 16:58:30.946179 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:30.946008    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxvjv\" (UniqueName: \"kubernetes.io/projected/bea5387e-1e26-4be9-bd38-71c49eb4225d-kube-api-access-jxvjv\") pod \"node-exporter-pbhzd\" (UID: \"bea5387e-1e26-4be9-bd38-71c49eb4225d\") " pod="openshift-monitoring/node-exporter-pbhzd"
Sep 18 16:58:30.946179 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:30.946024    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bea5387e-1e26-4be9-bd38-71c49eb4225d-node-exporter-wtmp\") pod \"node-exporter-pbhzd\" (UID: \"bea5387e-1e26-4be9-bd38-71c49eb4225d\") " pod="openshift-monitoring/node-exporter-pbhzd"
Sep 18 16:58:30.946179 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:30.946052    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8tdt\" (UniqueName: \"kubernetes.io/projected/8c199118-475a-4607-ba80-9a8027f1b047-kube-api-access-q8tdt\") pod \"kube-state-metrics-84c885658c-tblfl\" (UID: \"8c199118-475a-4607-ba80-9a8027f1b047\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-tblfl"
Sep 18 16:58:31.046591 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:31.046552    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bea5387e-1e26-4be9-bd38-71c49eb4225d-node-exporter-accelerators-collector-config\") pod \"node-exporter-pbhzd\" (UID: \"bea5387e-1e26-4be9-bd38-71c49eb4225d\") " pod="openshift-monitoring/node-exporter-pbhzd"
Sep 18 16:58:31.046591 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:31.046604    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lssbx\" (UniqueName: \"kubernetes.io/projected/69a62b4d-4c8d-4dc4-81a8-35628f51c837-kube-api-access-lssbx\") pod \"openshift-state-metrics-86c5bc79d6-dn7sr\" (UID: \"69a62b4d-4c8d-4dc4-81a8-35628f51c837\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-dn7sr"
Sep 18 16:58:31.046867 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:31.046632    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8c199118-475a-4607-ba80-9a8027f1b047-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-tblfl\" (UID: \"8c199118-475a-4607-ba80-9a8027f1b047\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-tblfl"
Sep 18 16:58:31.046867 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:31.046653    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bea5387e-1e26-4be9-bd38-71c49eb4225d-node-exporter-tls\") pod \"node-exporter-pbhzd\" (UID: \"bea5387e-1e26-4be9-bd38-71c49eb4225d\") " pod="openshift-monitoring/node-exporter-pbhzd"
Sep 18 16:58:31.046867 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:31.046674    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxvjv\" (UniqueName: \"kubernetes.io/projected/bea5387e-1e26-4be9-bd38-71c49eb4225d-kube-api-access-jxvjv\") pod \"node-exporter-pbhzd\" (UID: \"bea5387e-1e26-4be9-bd38-71c49eb4225d\") " pod="openshift-monitoring/node-exporter-pbhzd"
Sep 18 16:58:31.046867 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:31.046698    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bea5387e-1e26-4be9-bd38-71c49eb4225d-node-exporter-wtmp\") pod \"node-exporter-pbhzd\" (UID: \"bea5387e-1e26-4be9-bd38-71c49eb4225d\") " pod="openshift-monitoring/node-exporter-pbhzd"
Sep 18 16:58:31.046867 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:31.046727    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8tdt\" (UniqueName: \"kubernetes.io/projected/8c199118-475a-4607-ba80-9a8027f1b047-kube-api-access-q8tdt\") pod \"kube-state-metrics-84c885658c-tblfl\" (UID: \"8c199118-475a-4607-ba80-9a8027f1b047\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-tblfl"
Sep 18 16:58:31.046867 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:31.046757    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bea5387e-1e26-4be9-bd38-71c49eb4225d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pbhzd\" (UID: \"bea5387e-1e26-4be9-bd38-71c49eb4225d\") " pod="openshift-monitoring/node-exporter-pbhzd"
Sep 18 16:58:31.046867 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:31.046782    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c199118-475a-4607-ba80-9a8027f1b047-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-tblfl\" (UID: \"8c199118-475a-4607-ba80-9a8027f1b047\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-tblfl"
Sep 18 16:58:31.046867 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:31.046809    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/69a62b4d-4c8d-4dc4-81a8-35628f51c837-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-dn7sr\" (UID: \"69a62b4d-4c8d-4dc4-81a8-35628f51c837\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-dn7sr"
Sep 18 16:58:31.046867 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:31.046836    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c199118-475a-4607-ba80-9a8027f1b047-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-tblfl\" (UID: \"8c199118-475a-4607-ba80-9a8027f1b047\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-tblfl"
Sep 18 16:58:31.046867 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:31.046860    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bea5387e-1e26-4be9-bd38-71c49eb4225d-root\") pod \"node-exporter-pbhzd\" (UID: \"bea5387e-1e26-4be9-bd38-71c49eb4225d\") " pod="openshift-monitoring/node-exporter-pbhzd"
Sep 18 16:58:31.047423 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:31.046890    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/69a62b4d-4c8d-4dc4-81a8-35628f51c837-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-dn7sr\" (UID: \"69a62b4d-4c8d-4dc4-81a8-35628f51c837\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-dn7sr"
Sep 18 16:58:31.047423 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:31.046937    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8c199118-475a-4607-ba80-9a8027f1b047-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-tblfl\" (UID: \"8c199118-475a-4607-ba80-9a8027f1b047\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-tblfl"
Sep 18 16:58:31.047423 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:31.046980    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/69a62b4d-4c8d-4dc4-81a8-35628f51c837-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-dn7sr\" (UID: \"69a62b4d-4c8d-4dc4-81a8-35628f51c837\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-dn7sr"
Sep 18 16:58:31.047423 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:31.047007    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bea5387e-1e26-4be9-bd38-71c49eb4225d-node-exporter-textfile\") pod \"node-exporter-pbhzd\" (UID: \"bea5387e-1e26-4be9-bd38-71c49eb4225d\") " pod="openshift-monitoring/node-exporter-pbhzd"
Sep 18 16:58:31.047423 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:31.047036    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bea5387e-1e26-4be9-bd38-71c49eb4225d-metrics-client-ca\") pod \"node-exporter-pbhzd\" (UID: \"bea5387e-1e26-4be9-bd38-71c49eb4225d\") " pod="openshift-monitoring/node-exporter-pbhzd"
Sep 18 16:58:31.047423 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:31.047069    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8c199118-475a-4607-ba80-9a8027f1b047-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-tblfl\" (UID: \"8c199118-475a-4607-ba80-9a8027f1b047\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-tblfl"
Sep 18 16:58:31.047423 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:31.047103    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bea5387e-1e26-4be9-bd38-71c49eb4225d-sys\") pod \"node-exporter-pbhzd\" (UID: \"bea5387e-1e26-4be9-bd38-71c49eb4225d\") " pod="openshift-monitoring/node-exporter-pbhzd"
Sep 18 16:58:31.047423 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:31.047199    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bea5387e-1e26-4be9-bd38-71c49eb4225d-sys\") pod \"node-exporter-pbhzd\" (UID: \"bea5387e-1e26-4be9-bd38-71c49eb4225d\") " pod="openshift-monitoring/node-exporter-pbhzd"
Sep 18 16:58:31.047423 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:31.047275    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bea5387e-1e26-4be9-bd38-71c49eb4225d-node-exporter-accelerators-collector-config\") pod \"node-exporter-pbhzd\" (UID: \"bea5387e-1e26-4be9-bd38-71c49eb4225d\") " pod="openshift-monitoring/node-exporter-pbhzd"
Sep 18 16:58:31.047886 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:31.047483    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bea5387e-1e26-4be9-bd38-71c49eb4225d-node-exporter-wtmp\") pod \"node-exporter-pbhzd\" (UID: \"bea5387e-1e26-4be9-bd38-71c49eb4225d\") " pod="openshift-monitoring/node-exporter-pbhzd"
Sep 18 16:58:31.047886 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:31.047565    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8c199118-475a-4607-ba80-9a8027f1b047-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-tblfl\" (UID: \"8c199118-475a-4607-ba80-9a8027f1b047\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-tblfl"
Sep 18 16:58:31.047886 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:31.047773    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bea5387e-1e26-4be9-bd38-71c49eb4225d-node-exporter-textfile\") pod \"node-exporter-pbhzd\" (UID: \"bea5387e-1e26-4be9-bd38-71c49eb4225d\") " pod="openshift-monitoring/node-exporter-pbhzd"
Sep 18 16:58:31.047886 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:58:31.047866    2881 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 18 16:58:31.048165 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:58:31.047926    2881 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69a62b4d-4c8d-4dc4-81a8-35628f51c837-openshift-state-metrics-tls podName:69a62b4d-4c8d-4dc4-81a8-35628f51c837 nodeName:}" failed. No retries permitted until 2026-09-18 16:58:31.547906506 +0000 UTC m=+301.747761776 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/69a62b4d-4c8d-4dc4-81a8-35628f51c837-openshift-state-metrics-tls") pod "openshift-state-metrics-86c5bc79d6-dn7sr" (UID: "69a62b4d-4c8d-4dc4-81a8-35628f51c837") : secret "openshift-state-metrics-tls" not found
Sep 18 16:58:31.048165 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:31.048056    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bea5387e-1e26-4be9-bd38-71c49eb4225d-metrics-client-ca\") pod \"node-exporter-pbhzd\" (UID: \"bea5387e-1e26-4be9-bd38-71c49eb4225d\") " pod="openshift-monitoring/node-exporter-pbhzd"
Sep 18 16:58:31.048835 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:31.048807    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/69a62b4d-4c8d-4dc4-81a8-35628f51c837-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-dn7sr\" (UID: \"69a62b4d-4c8d-4dc4-81a8-35628f51c837\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-dn7sr"
Sep 18 16:58:31.048835 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:31.048823    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8c199118-475a-4607-ba80-9a8027f1b047-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-tblfl\" (UID: \"8c199118-475a-4607-ba80-9a8027f1b047\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-tblfl"
Sep 18 16:58:31.048989 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:31.048861    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c199118-475a-4607-ba80-9a8027f1b047-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-tblfl\" (UID: \"8c199118-475a-4607-ba80-9a8027f1b047\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-tblfl"
Sep 18 16:58:31.048989 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:31.048895    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bea5387e-1e26-4be9-bd38-71c49eb4225d-root\") pod \"node-exporter-pbhzd\" (UID: \"bea5387e-1e26-4be9-bd38-71c49eb4225d\") " pod="openshift-monitoring/node-exporter-pbhzd"
Sep 18 16:58:31.048989 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:58:31.048924    2881 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 18 16:58:31.048989 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:58:31.048981    2881 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c199118-475a-4607-ba80-9a8027f1b047-kube-state-metrics-tls podName:8c199118-475a-4607-ba80-9a8027f1b047 nodeName:}" failed. No retries permitted until 2026-09-18 16:58:31.548964993 +0000 UTC m=+301.748820258 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/8c199118-475a-4607-ba80-9a8027f1b047-kube-state-metrics-tls") pod "kube-state-metrics-84c885658c-tblfl" (UID: "8c199118-475a-4607-ba80-9a8027f1b047") : secret "kube-state-metrics-tls" not found
Sep 18 16:58:31.050397 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:31.050294    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/69a62b4d-4c8d-4dc4-81a8-35628f51c837-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-dn7sr\" (UID: \"69a62b4d-4c8d-4dc4-81a8-35628f51c837\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-dn7sr"
Sep 18 16:58:31.050828 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:31.050792    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bea5387e-1e26-4be9-bd38-71c49eb4225d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pbhzd\" (UID: \"bea5387e-1e26-4be9-bd38-71c49eb4225d\") " pod="openshift-monitoring/node-exporter-pbhzd"
Sep 18 16:58:31.051043 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:31.051018    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8c199118-475a-4607-ba80-9a8027f1b047-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-tblfl\" (UID: \"8c199118-475a-4607-ba80-9a8027f1b047\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-tblfl"
Sep 18 16:58:31.051434 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:31.051413    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bea5387e-1e26-4be9-bd38-71c49eb4225d-node-exporter-tls\") pod \"node-exporter-pbhzd\" (UID: \"bea5387e-1e26-4be9-bd38-71c49eb4225d\") " pod="openshift-monitoring/node-exporter-pbhzd"
Sep 18 16:58:31.060703 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:31.060630    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8tdt\" (UniqueName: \"kubernetes.io/projected/8c199118-475a-4607-ba80-9a8027f1b047-kube-api-access-q8tdt\") pod \"kube-state-metrics-84c885658c-tblfl\" (UID: \"8c199118-475a-4607-ba80-9a8027f1b047\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-tblfl"
Sep 18 16:58:31.060944 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:31.060903    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lssbx\" (UniqueName: \"kubernetes.io/projected/69a62b4d-4c8d-4dc4-81a8-35628f51c837-kube-api-access-lssbx\") pod \"openshift-state-metrics-86c5bc79d6-dn7sr\" (UID: \"69a62b4d-4c8d-4dc4-81a8-35628f51c837\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-dn7sr"
Sep 18 16:58:31.062254 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:31.062213    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxvjv\" (UniqueName: \"kubernetes.io/projected/bea5387e-1e26-4be9-bd38-71c49eb4225d-kube-api-access-jxvjv\") pod \"node-exporter-pbhzd\" (UID: \"bea5387e-1e26-4be9-bd38-71c49eb4225d\") " pod="openshift-monitoring/node-exporter-pbhzd"
Sep 18 16:58:31.136820 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:31.136785    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pbhzd"
Sep 18 16:58:31.151810 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:58:31.151767    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbea5387e_1e26_4be9_bd38_71c49eb4225d.slice/crio-935de93a56123179b8015b2c64546aa8656c9573b5a278acfc725f7bd77073bf WatchSource:0}: Error finding container 935de93a56123179b8015b2c64546aa8656c9573b5a278acfc725f7bd77073bf: Status 404 returned error can't find the container with id 935de93a56123179b8015b2c64546aa8656c9573b5a278acfc725f7bd77073bf
Sep 18 16:58:31.249412 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:31.248369    2881 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5bf8875fb5-z4njj"
Sep 18 16:58:31.249412 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:31.248763    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5bf8875fb5-z4njj"
Sep 18 16:58:31.254638 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:31.254603    2881 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5bf8875fb5-z4njj"
Sep 18 16:58:31.551085 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:31.551037    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c199118-475a-4607-ba80-9a8027f1b047-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-tblfl\" (UID: \"8c199118-475a-4607-ba80-9a8027f1b047\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-tblfl"
Sep 18 16:58:31.551368 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:31.551129    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/69a62b4d-4c8d-4dc4-81a8-35628f51c837-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-dn7sr\" (UID: \"69a62b4d-4c8d-4dc4-81a8-35628f51c837\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-dn7sr"
Sep 18 16:58:31.554271 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:31.554213    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c199118-475a-4607-ba80-9a8027f1b047-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-tblfl\" (UID: \"8c199118-475a-4607-ba80-9a8027f1b047\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-tblfl"
Sep 18 16:58:31.554412 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:31.554213    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/69a62b4d-4c8d-4dc4-81a8-35628f51c837-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-dn7sr\" (UID: \"69a62b4d-4c8d-4dc4-81a8-35628f51c837\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-dn7sr"
Sep 18 16:58:31.611622 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:31.611473    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pbhzd" event={"ID":"bea5387e-1e26-4be9-bd38-71c49eb4225d","Type":"ContainerStarted","Data":"935de93a56123179b8015b2c64546aa8656c9573b5a278acfc725f7bd77073bf"}
Sep 18 16:58:31.614265 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:31.614145    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lpt5n" event={"ID":"89faf7f5-d633-4219-b3fc-323f8c23b599","Type":"ContainerStarted","Data":"07a9330cf62363a67bfb93352d80f5784a073121faae5ed332d81170acce156a"}
Sep 18 16:58:31.614265 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:31.614243    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lpt5n" event={"ID":"89faf7f5-d633-4219-b3fc-323f8c23b599","Type":"ContainerStarted","Data":"af56073d4e084bfdb64d4df53bb4ca6ae488d50ff0db84fe63d25ba6580404fb"}
Sep 18 16:58:31.620198 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:31.620144    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5bf8875fb5-z4njj"
Sep 18 16:58:31.633243 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:31.633179    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-lpt5n" podStartSLOduration=91.558093531 podStartE2EDuration="1m32.633160891s" totalImagesPullingTime="1.07506736s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:56:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:58:31.632261453 +0000 UTC m=+301.832116740" watchObservedRunningTime="2026-09-18 16:58:31.633160891 +0000 UTC m=+301.833016176"
Sep 18 16:58:31.717778 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:31.717743    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-dn7sr"
Sep 18 16:58:31.729042 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:31.728953    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-tblfl"
Sep 18 16:58:31.879560 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:31.879452    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 16:58:31.884058 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:31.884023    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:58:31.887018 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:31.886612    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 18 16:58:31.887018 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:31.886858    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 18 16:58:31.887505 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:31.887482    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 18 16:58:31.887597 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:31.887541    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-hjrwt\""
Sep 18 16:58:31.887737 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:31.887720    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 18 16:58:31.887960 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:31.887941    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 18 16:58:31.888286 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:31.888092    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 18 16:58:31.888286 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:31.888143    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 18 16:58:31.888413 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:31.888350    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 18 16:58:31.888837 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:31.888578    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 18 16:58:31.902642 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:31.902025    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 16:58:32.055986 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:32.055899    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cdcc3b80-4274-48f4-bf97-09412758b028-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cdcc3b80-4274-48f4-bf97-09412758b028\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:58:32.055986 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:32.055961    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cdcc3b80-4274-48f4-bf97-09412758b028-web-config\") pod \"alertmanager-main-0\" (UID: \"cdcc3b80-4274-48f4-bf97-09412758b028\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:58:32.056374 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:32.056008    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cdcc3b80-4274-48f4-bf97-09412758b028-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cdcc3b80-4274-48f4-bf97-09412758b028\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:58:32.056374 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:32.056067    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cdcc3b80-4274-48f4-bf97-09412758b028-config-volume\") pod \"alertmanager-main-0\" (UID: \"cdcc3b80-4274-48f4-bf97-09412758b028\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:58:32.056374 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:32.056115    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cdcc3b80-4274-48f4-bf97-09412758b028-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cdcc3b80-4274-48f4-bf97-09412758b028\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:58:32.056374 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:32.056141    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cdcc3b80-4274-48f4-bf97-09412758b028-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cdcc3b80-4274-48f4-bf97-09412758b028\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:58:32.056374 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:32.056173    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cdcc3b80-4274-48f4-bf97-09412758b028-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cdcc3b80-4274-48f4-bf97-09412758b028\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:58:32.056374 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:32.056253    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cdcc3b80-4274-48f4-bf97-09412758b028-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cdcc3b80-4274-48f4-bf97-09412758b028\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:58:32.056374 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:32.056303    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cdcc3b80-4274-48f4-bf97-09412758b028-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cdcc3b80-4274-48f4-bf97-09412758b028\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:58:32.056374 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:32.056332    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cdcc3b80-4274-48f4-bf97-09412758b028-config-out\") pod \"alertmanager-main-0\" (UID: \"cdcc3b80-4274-48f4-bf97-09412758b028\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:58:32.056374 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:32.056363    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cdcc3b80-4274-48f4-bf97-09412758b028-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cdcc3b80-4274-48f4-bf97-09412758b028\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:58:32.056894 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:32.056389    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5dtm\" (UniqueName: \"kubernetes.io/projected/cdcc3b80-4274-48f4-bf97-09412758b028-kube-api-access-f5dtm\") pod \"alertmanager-main-0\" (UID: \"cdcc3b80-4274-48f4-bf97-09412758b028\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:58:32.056894 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:32.056461    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cdcc3b80-4274-48f4-bf97-09412758b028-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cdcc3b80-4274-48f4-bf97-09412758b028\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:58:32.157680 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:32.157588    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cdcc3b80-4274-48f4-bf97-09412758b028-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cdcc3b80-4274-48f4-bf97-09412758b028\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:58:32.157680 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:32.157644    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cdcc3b80-4274-48f4-bf97-09412758b028-web-config\") pod \"alertmanager-main-0\" (UID: \"cdcc3b80-4274-48f4-bf97-09412758b028\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:58:32.157941 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:32.157686    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cdcc3b80-4274-48f4-bf97-09412758b028-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cdcc3b80-4274-48f4-bf97-09412758b028\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:58:32.157941 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:58:32.157780    2881 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 18 16:58:32.157941 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:32.157785    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cdcc3b80-4274-48f4-bf97-09412758b028-config-volume\") pod \"alertmanager-main-0\" (UID: \"cdcc3b80-4274-48f4-bf97-09412758b028\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:58:32.157941 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:58:32.157855    2881 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdcc3b80-4274-48f4-bf97-09412758b028-secret-alertmanager-main-tls podName:cdcc3b80-4274-48f4-bf97-09412758b028 nodeName:}" failed. No retries permitted until 2026-09-18 16:58:32.657838382 +0000 UTC m=+302.857693646 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/cdcc3b80-4274-48f4-bf97-09412758b028-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "cdcc3b80-4274-48f4-bf97-09412758b028") : secret "alertmanager-main-tls" not found
Sep 18 16:58:32.157941 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:32.157885    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cdcc3b80-4274-48f4-bf97-09412758b028-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cdcc3b80-4274-48f4-bf97-09412758b028\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:58:32.157941 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:32.157902    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cdcc3b80-4274-48f4-bf97-09412758b028-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cdcc3b80-4274-48f4-bf97-09412758b028\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:58:32.157941 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:32.157928    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cdcc3b80-4274-48f4-bf97-09412758b028-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cdcc3b80-4274-48f4-bf97-09412758b028\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:58:32.157941 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:32.157946    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cdcc3b80-4274-48f4-bf97-09412758b028-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cdcc3b80-4274-48f4-bf97-09412758b028\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:58:32.158420 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:32.157969    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cdcc3b80-4274-48f4-bf97-09412758b028-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cdcc3b80-4274-48f4-bf97-09412758b028\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:58:32.158420 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:32.157990    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cdcc3b80-4274-48f4-bf97-09412758b028-config-out\") pod \"alertmanager-main-0\" (UID: \"cdcc3b80-4274-48f4-bf97-09412758b028\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:58:32.158420 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:32.158013    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cdcc3b80-4274-48f4-bf97-09412758b028-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cdcc3b80-4274-48f4-bf97-09412758b028\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:58:32.158420 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:32.158030    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5dtm\" (UniqueName: \"kubernetes.io/projected/cdcc3b80-4274-48f4-bf97-09412758b028-kube-api-access-f5dtm\") pod \"alertmanager-main-0\" (UID: \"cdcc3b80-4274-48f4-bf97-09412758b028\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:58:32.158420 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:32.158052    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cdcc3b80-4274-48f4-bf97-09412758b028-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cdcc3b80-4274-48f4-bf97-09412758b028\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:58:32.158682 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:32.158460    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cdcc3b80-4274-48f4-bf97-09412758b028-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cdcc3b80-4274-48f4-bf97-09412758b028\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:58:32.158912 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:32.158886    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cdcc3b80-4274-48f4-bf97-09412758b028-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cdcc3b80-4274-48f4-bf97-09412758b028\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:58:32.160055 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:32.160003    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cdcc3b80-4274-48f4-bf97-09412758b028-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cdcc3b80-4274-48f4-bf97-09412758b028\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:58:32.161359 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:32.161335    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cdcc3b80-4274-48f4-bf97-09412758b028-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cdcc3b80-4274-48f4-bf97-09412758b028\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:58:32.161826 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:32.161784    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cdcc3b80-4274-48f4-bf97-09412758b028-web-config\") pod \"alertmanager-main-0\" (UID: \"cdcc3b80-4274-48f4-bf97-09412758b028\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:58:32.161923 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:32.161849    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cdcc3b80-4274-48f4-bf97-09412758b028-config-volume\") pod \"alertmanager-main-0\" (UID: \"cdcc3b80-4274-48f4-bf97-09412758b028\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:58:32.161990 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:32.161918    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cdcc3b80-4274-48f4-bf97-09412758b028-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cdcc3b80-4274-48f4-bf97-09412758b028\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:58:32.162051 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:32.161993    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cdcc3b80-4274-48f4-bf97-09412758b028-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cdcc3b80-4274-48f4-bf97-09412758b028\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:58:32.162145 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:32.162120    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cdcc3b80-4274-48f4-bf97-09412758b028-config-out\") pod \"alertmanager-main-0\" (UID: \"cdcc3b80-4274-48f4-bf97-09412758b028\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:58:32.163318 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:32.163293    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cdcc3b80-4274-48f4-bf97-09412758b028-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cdcc3b80-4274-48f4-bf97-09412758b028\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:58:32.163486 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:32.163468    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cdcc3b80-4274-48f4-bf97-09412758b028-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cdcc3b80-4274-48f4-bf97-09412758b028\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:58:32.170699 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:32.170660    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5dtm\" (UniqueName: \"kubernetes.io/projected/cdcc3b80-4274-48f4-bf97-09412758b028-kube-api-access-f5dtm\") pod \"alertmanager-main-0\" (UID: \"cdcc3b80-4274-48f4-bf97-09412758b028\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:58:32.662869 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:32.662822    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cdcc3b80-4274-48f4-bf97-09412758b028-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cdcc3b80-4274-48f4-bf97-09412758b028\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:58:32.665758 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:32.665724    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cdcc3b80-4274-48f4-bf97-09412758b028-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cdcc3b80-4274-48f4-bf97-09412758b028\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:58:32.801726 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:32.801683    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:58:33.784961 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:33.784920    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-96b4948d4-zsn4k"]
Sep 18 16:58:33.788857 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:33.788820    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-96b4948d4-zsn4k"
Sep 18 16:58:33.791851 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:33.791675    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-ifhfoh85l2k1\""
Sep 18 16:58:33.791851 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:33.791674    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 18 16:58:33.791851 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:33.791738    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 18 16:58:33.792137 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:33.791911    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 18 16:58:33.792137 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:33.792006    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-qv45p\""
Sep 18 16:58:33.792304 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:33.792284    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 18 16:58:33.792526 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:33.792504    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 18 16:58:33.801078 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:33.801002    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-96b4948d4-zsn4k"]
Sep 18 16:58:33.876036 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:33.875986    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/0de2e307-fa16-46a0-bc95-2a59d4fa9d44-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-96b4948d4-zsn4k\" (UID: \"0de2e307-fa16-46a0-bc95-2a59d4fa9d44\") " pod="openshift-monitoring/thanos-querier-96b4948d4-zsn4k"
Sep 18 16:58:33.876252 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:33.876042    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/0de2e307-fa16-46a0-bc95-2a59d4fa9d44-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-96b4948d4-zsn4k\" (UID: \"0de2e307-fa16-46a0-bc95-2a59d4fa9d44\") " pod="openshift-monitoring/thanos-querier-96b4948d4-zsn4k"
Sep 18 16:58:33.876252 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:33.876149    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0de2e307-fa16-46a0-bc95-2a59d4fa9d44-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-96b4948d4-zsn4k\" (UID: \"0de2e307-fa16-46a0-bc95-2a59d4fa9d44\") " pod="openshift-monitoring/thanos-querier-96b4948d4-zsn4k"
Sep 18 16:58:33.876380 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:33.876247    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/0de2e307-fa16-46a0-bc95-2a59d4fa9d44-secret-thanos-querier-tls\") pod \"thanos-querier-96b4948d4-zsn4k\" (UID: \"0de2e307-fa16-46a0-bc95-2a59d4fa9d44\") " pod="openshift-monitoring/thanos-querier-96b4948d4-zsn4k"
Sep 18 16:58:33.876380 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:33.876288    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0de2e307-fa16-46a0-bc95-2a59d4fa9d44-secret-grpc-tls\") pod \"thanos-querier-96b4948d4-zsn4k\" (UID: \"0de2e307-fa16-46a0-bc95-2a59d4fa9d44\") " pod="openshift-monitoring/thanos-querier-96b4948d4-zsn4k"
Sep 18 16:58:33.876477 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:33.876393    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0de2e307-fa16-46a0-bc95-2a59d4fa9d44-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-96b4948d4-zsn4k\" (UID: \"0de2e307-fa16-46a0-bc95-2a59d4fa9d44\") " pod="openshift-monitoring/thanos-querier-96b4948d4-zsn4k"
Sep 18 16:58:33.876477 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:33.876428    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0de2e307-fa16-46a0-bc95-2a59d4fa9d44-metrics-client-ca\") pod \"thanos-querier-96b4948d4-zsn4k\" (UID: \"0de2e307-fa16-46a0-bc95-2a59d4fa9d44\") " pod="openshift-monitoring/thanos-querier-96b4948d4-zsn4k"
Sep 18 16:58:33.876477 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:33.876452    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpzfl\" (UniqueName: \"kubernetes.io/projected/0de2e307-fa16-46a0-bc95-2a59d4fa9d44-kube-api-access-gpzfl\") pod \"thanos-querier-96b4948d4-zsn4k\" (UID: \"0de2e307-fa16-46a0-bc95-2a59d4fa9d44\") " pod="openshift-monitoring/thanos-querier-96b4948d4-zsn4k"
Sep 18 16:58:33.977451 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:33.977410    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/0de2e307-fa16-46a0-bc95-2a59d4fa9d44-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-96b4948d4-zsn4k\" (UID: \"0de2e307-fa16-46a0-bc95-2a59d4fa9d44\") " pod="openshift-monitoring/thanos-querier-96b4948d4-zsn4k"
Sep 18 16:58:33.977660 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:33.977465    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0de2e307-fa16-46a0-bc95-2a59d4fa9d44-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-96b4948d4-zsn4k\" (UID: \"0de2e307-fa16-46a0-bc95-2a59d4fa9d44\") " pod="openshift-monitoring/thanos-querier-96b4948d4-zsn4k"
Sep 18 16:58:33.977660 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:33.977516    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/0de2e307-fa16-46a0-bc95-2a59d4fa9d44-secret-thanos-querier-tls\") pod \"thanos-querier-96b4948d4-zsn4k\" (UID: \"0de2e307-fa16-46a0-bc95-2a59d4fa9d44\") " pod="openshift-monitoring/thanos-querier-96b4948d4-zsn4k"
Sep 18 16:58:33.977660 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:33.977539    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0de2e307-fa16-46a0-bc95-2a59d4fa9d44-secret-grpc-tls\") pod \"thanos-querier-96b4948d4-zsn4k\" (UID: \"0de2e307-fa16-46a0-bc95-2a59d4fa9d44\") " pod="openshift-monitoring/thanos-querier-96b4948d4-zsn4k"
Sep 18 16:58:33.977822 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:33.977667    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0de2e307-fa16-46a0-bc95-2a59d4fa9d44-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-96b4948d4-zsn4k\" (UID: \"0de2e307-fa16-46a0-bc95-2a59d4fa9d44\") " pod="openshift-monitoring/thanos-querier-96b4948d4-zsn4k"
Sep 18 16:58:33.977822 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:33.977718    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0de2e307-fa16-46a0-bc95-2a59d4fa9d44-metrics-client-ca\") pod \"thanos-querier-96b4948d4-zsn4k\" (UID: \"0de2e307-fa16-46a0-bc95-2a59d4fa9d44\") " pod="openshift-monitoring/thanos-querier-96b4948d4-zsn4k"
Sep 18 16:58:33.977822 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:33.977755    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpzfl\" (UniqueName: \"kubernetes.io/projected/0de2e307-fa16-46a0-bc95-2a59d4fa9d44-kube-api-access-gpzfl\") pod \"thanos-querier-96b4948d4-zsn4k\" (UID: \"0de2e307-fa16-46a0-bc95-2a59d4fa9d44\") " pod="openshift-monitoring/thanos-querier-96b4948d4-zsn4k"
Sep 18 16:58:33.977822 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:33.977794    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/0de2e307-fa16-46a0-bc95-2a59d4fa9d44-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-96b4948d4-zsn4k\" (UID: \"0de2e307-fa16-46a0-bc95-2a59d4fa9d44\") " pod="openshift-monitoring/thanos-querier-96b4948d4-zsn4k"
Sep 18 16:58:33.978694 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:33.978662    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0de2e307-fa16-46a0-bc95-2a59d4fa9d44-metrics-client-ca\") pod \"thanos-querier-96b4948d4-zsn4k\" (UID: \"0de2e307-fa16-46a0-bc95-2a59d4fa9d44\") " pod="openshift-monitoring/thanos-querier-96b4948d4-zsn4k"
Sep 18 16:58:33.980778 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:33.980710    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0de2e307-fa16-46a0-bc95-2a59d4fa9d44-secret-grpc-tls\") pod \"thanos-querier-96b4948d4-zsn4k\" (UID: \"0de2e307-fa16-46a0-bc95-2a59d4fa9d44\") " pod="openshift-monitoring/thanos-querier-96b4948d4-zsn4k"
Sep 18 16:58:33.980778 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:33.980762    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/0de2e307-fa16-46a0-bc95-2a59d4fa9d44-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-96b4948d4-zsn4k\" (UID: \"0de2e307-fa16-46a0-bc95-2a59d4fa9d44\") " pod="openshift-monitoring/thanos-querier-96b4948d4-zsn4k"
Sep 18 16:58:33.980984 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:33.980941    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0de2e307-fa16-46a0-bc95-2a59d4fa9d44-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-96b4948d4-zsn4k\" (UID: \"0de2e307-fa16-46a0-bc95-2a59d4fa9d44\") " pod="openshift-monitoring/thanos-querier-96b4948d4-zsn4k"
Sep 18 16:58:33.981152 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:33.981128    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0de2e307-fa16-46a0-bc95-2a59d4fa9d44-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-96b4948d4-zsn4k\" (UID: \"0de2e307-fa16-46a0-bc95-2a59d4fa9d44\") " pod="openshift-monitoring/thanos-querier-96b4948d4-zsn4k"
Sep 18 16:58:33.981287 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:33.981210    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/0de2e307-fa16-46a0-bc95-2a59d4fa9d44-secret-thanos-querier-tls\") pod \"thanos-querier-96b4948d4-zsn4k\" (UID: \"0de2e307-fa16-46a0-bc95-2a59d4fa9d44\") " pod="openshift-monitoring/thanos-querier-96b4948d4-zsn4k"
Sep 18 16:58:33.981417 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:33.981395    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/0de2e307-fa16-46a0-bc95-2a59d4fa9d44-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-96b4948d4-zsn4k\" (UID: \"0de2e307-fa16-46a0-bc95-2a59d4fa9d44\") " pod="openshift-monitoring/thanos-querier-96b4948d4-zsn4k"
Sep 18 16:58:33.987990 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:33.987964    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpzfl\" (UniqueName: \"kubernetes.io/projected/0de2e307-fa16-46a0-bc95-2a59d4fa9d44-kube-api-access-gpzfl\") pod \"thanos-querier-96b4948d4-zsn4k\" (UID: \"0de2e307-fa16-46a0-bc95-2a59d4fa9d44\") " pod="openshift-monitoring/thanos-querier-96b4948d4-zsn4k"
Sep 18 16:58:34.102121 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:34.102027    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-96b4948d4-zsn4k"
Sep 18 16:58:34.634758 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:34.634641    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pc44x" event={"ID":"6f463717-a487-485b-aa4b-7772d35b0d22","Type":"ContainerStarted","Data":"3ac83784853806cf062feef654fa2bde2b111e3efcfe09040621e1638be5a68f"}
Sep 18 16:58:34.634758 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:34.634707    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-pc44x"
Sep 18 16:58:34.636576 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:34.636548    2881 generic.go:356] "Generic (PLEG): container finished" podID="bea5387e-1e26-4be9-bd38-71c49eb4225d" containerID="14d5c90e3f127abb2125b4bc5b03038e3ebf57aa3b0fa0705ef0514784fbdd84" exitCode=0
Sep 18 16:58:34.636738 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:34.636629    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pbhzd" event={"ID":"bea5387e-1e26-4be9-bd38-71c49eb4225d","Type":"ContainerDied","Data":"14d5c90e3f127abb2125b4bc5b03038e3ebf57aa3b0fa0705ef0514784fbdd84"}
Sep 18 16:58:34.638257 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:34.638209    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fx8nf" event={"ID":"a4df48ac-dd04-4504-b828-66d7e2e986e1","Type":"ContainerStarted","Data":"b7f76225cf4cf829b239223af72fcfb6e38d3a6163c7fa827a08b55bab645175"}
Sep 18 16:58:34.654369 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:34.654208    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-pc44x" podStartSLOduration=88.790936753 podStartE2EDuration="1m33.654185794s" totalImagesPullingTime="4.863249041s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:57:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:58:34.653159386 +0000 UTC m=+304.853014674" watchObservedRunningTime="2026-09-18 16:58:34.654185794 +0000 UTC m=+304.854041081"
Sep 18 16:58:34.668764 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:34.668717    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-fx8nf" podStartSLOduration=90.763712667 podStartE2EDuration="1m35.668698434s" totalImagesPullingTime="4.904985767s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:56:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:58:34.667648996 +0000 UTC m=+304.867504283" watchObservedRunningTime="2026-09-18 16:58:34.668698434 +0000 UTC m=+304.868553719"
Sep 18 16:58:34.680614 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:34.680578    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-dn7sr"]
Sep 18 16:58:34.681892 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:34.681866    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-tblfl"]
Sep 18 16:58:34.684444 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:58:34.684416    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69a62b4d_4c8d_4dc4_81a8_35628f51c837.slice/crio-badf56ca2c00aec207fbc1032827c170307b53e91d71c219f235c569844e9fd6 WatchSource:0}: Error finding container badf56ca2c00aec207fbc1032827c170307b53e91d71c219f235c569844e9fd6: Status 404 returned error can't find the container with id badf56ca2c00aec207fbc1032827c170307b53e91d71c219f235c569844e9fd6
Sep 18 16:58:34.685723 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:58:34.685686    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c199118_475a_4607_ba80_9a8027f1b047.slice/crio-795012641c3d178761e5be7e54968172708ac20c108e0a9769414b0d7a21ee71 WatchSource:0}: Error finding container 795012641c3d178761e5be7e54968172708ac20c108e0a9769414b0d7a21ee71: Status 404 returned error can't find the container with id 795012641c3d178761e5be7e54968172708ac20c108e0a9769414b0d7a21ee71
Sep 18 16:58:34.697249 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:34.697207    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 16:58:34.701019 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:58:34.700989    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdcc3b80_4274_48f4_bf97_09412758b028.slice/crio-f1f87f64fd82a59012dc7117adc9f18211f8d07d7c56a4dae614a843e0f76150 WatchSource:0}: Error finding container f1f87f64fd82a59012dc7117adc9f18211f8d07d7c56a4dae614a843e0f76150: Status 404 returned error can't find the container with id f1f87f64fd82a59012dc7117adc9f18211f8d07d7c56a4dae614a843e0f76150
Sep 18 16:58:34.708824 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:34.708677    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-96b4948d4-zsn4k"]
Sep 18 16:58:34.714036 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:58:34.713994    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0de2e307_fa16_46a0_bc95_2a59d4fa9d44.slice/crio-0fdb02e83722d8b90b0597d5857d9955a5c63d8ec582783c30778868b2127aa4 WatchSource:0}: Error finding container 0fdb02e83722d8b90b0597d5857d9955a5c63d8ec582783c30778868b2127aa4: Status 404 returned error can't find the container with id 0fdb02e83722d8b90b0597d5857d9955a5c63d8ec582783c30778868b2127aa4
Sep 18 16:58:35.202826 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:35.202791    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-67b6f56d48-lvpbv"]
Sep 18 16:58:35.205477 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:35.205450    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-67b6f56d48-lvpbv"
Sep 18 16:58:35.209188 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:35.209157    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 18 16:58:35.211798 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:35.209266    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 18 16:58:35.211798 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:35.210126    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-l4fp6\""
Sep 18 16:58:35.212023 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:35.210174    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 18 16:58:35.212023 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:35.210265    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-c9p7taqgrce96\""
Sep 18 16:58:35.212129 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:35.210363    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 18 16:58:35.226837 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:35.226802    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-67b6f56d48-lvpbv"]
Sep 18 16:58:35.292636 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:35.292594    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0d71645f-8eff-473b-b150-d623c873cf57-secret-metrics-server-client-certs\") pod \"metrics-server-67b6f56d48-lvpbv\" (UID: \"0d71645f-8eff-473b-b150-d623c873cf57\") " pod="openshift-monitoring/metrics-server-67b6f56d48-lvpbv"
Sep 18 16:58:35.292850 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:35.292645    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d71645f-8eff-473b-b150-d623c873cf57-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-67b6f56d48-lvpbv\" (UID: \"0d71645f-8eff-473b-b150-d623c873cf57\") " pod="openshift-monitoring/metrics-server-67b6f56d48-lvpbv"
Sep 18 16:58:35.292850 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:35.292711    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjz5z\" (UniqueName: \"kubernetes.io/projected/0d71645f-8eff-473b-b150-d623c873cf57-kube-api-access-mjz5z\") pod \"metrics-server-67b6f56d48-lvpbv\" (UID: \"0d71645f-8eff-473b-b150-d623c873cf57\") " pod="openshift-monitoring/metrics-server-67b6f56d48-lvpbv"
Sep 18 16:58:35.292850 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:35.292756    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0d71645f-8eff-473b-b150-d623c873cf57-secret-metrics-server-tls\") pod \"metrics-server-67b6f56d48-lvpbv\" (UID: \"0d71645f-8eff-473b-b150-d623c873cf57\") " pod="openshift-monitoring/metrics-server-67b6f56d48-lvpbv"
Sep 18 16:58:35.292850 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:35.292829    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0d71645f-8eff-473b-b150-d623c873cf57-metrics-server-audit-profiles\") pod \"metrics-server-67b6f56d48-lvpbv\" (UID: \"0d71645f-8eff-473b-b150-d623c873cf57\") " pod="openshift-monitoring/metrics-server-67b6f56d48-lvpbv"
Sep 18 16:58:35.293063 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:35.292866    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0d71645f-8eff-473b-b150-d623c873cf57-client-ca-bundle\") pod \"metrics-server-67b6f56d48-lvpbv\" (UID: \"0d71645f-8eff-473b-b150-d623c873cf57\") " pod="openshift-monitoring/metrics-server-67b6f56d48-lvpbv"
Sep 18 16:58:35.293063 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:35.292890    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0d71645f-8eff-473b-b150-d623c873cf57-audit-log\") pod \"metrics-server-67b6f56d48-lvpbv\" (UID: \"0d71645f-8eff-473b-b150-d623c873cf57\") " pod="openshift-monitoring/metrics-server-67b6f56d48-lvpbv"
Sep 18 16:58:35.393795 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:35.393741    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0d71645f-8eff-473b-b150-d623c873cf57-metrics-server-audit-profiles\") pod \"metrics-server-67b6f56d48-lvpbv\" (UID: \"0d71645f-8eff-473b-b150-d623c873cf57\") " pod="openshift-monitoring/metrics-server-67b6f56d48-lvpbv"
Sep 18 16:58:35.394012 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:35.393805    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0d71645f-8eff-473b-b150-d623c873cf57-client-ca-bundle\") pod \"metrics-server-67b6f56d48-lvpbv\" (UID: \"0d71645f-8eff-473b-b150-d623c873cf57\") " pod="openshift-monitoring/metrics-server-67b6f56d48-lvpbv"
Sep 18 16:58:35.394012 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:35.393845    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0d71645f-8eff-473b-b150-d623c873cf57-audit-log\") pod \"metrics-server-67b6f56d48-lvpbv\" (UID: \"0d71645f-8eff-473b-b150-d623c873cf57\") " pod="openshift-monitoring/metrics-server-67b6f56d48-lvpbv"
Sep 18 16:58:35.394012 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:35.393901    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0d71645f-8eff-473b-b150-d623c873cf57-secret-metrics-server-client-certs\") pod \"metrics-server-67b6f56d48-lvpbv\" (UID: \"0d71645f-8eff-473b-b150-d623c873cf57\") " pod="openshift-monitoring/metrics-server-67b6f56d48-lvpbv"
Sep 18 16:58:35.394012 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:35.393928    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d71645f-8eff-473b-b150-d623c873cf57-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-67b6f56d48-lvpbv\" (UID: \"0d71645f-8eff-473b-b150-d623c873cf57\") " pod="openshift-monitoring/metrics-server-67b6f56d48-lvpbv"
Sep 18 16:58:35.394012 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:35.393960    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjz5z\" (UniqueName: \"kubernetes.io/projected/0d71645f-8eff-473b-b150-d623c873cf57-kube-api-access-mjz5z\") pod \"metrics-server-67b6f56d48-lvpbv\" (UID: \"0d71645f-8eff-473b-b150-d623c873cf57\") " pod="openshift-monitoring/metrics-server-67b6f56d48-lvpbv"
Sep 18 16:58:35.394012 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:35.393999    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0d71645f-8eff-473b-b150-d623c873cf57-secret-metrics-server-tls\") pod \"metrics-server-67b6f56d48-lvpbv\" (UID: \"0d71645f-8eff-473b-b150-d623c873cf57\") " pod="openshift-monitoring/metrics-server-67b6f56d48-lvpbv"
Sep 18 16:58:35.395273 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:35.395180    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0d71645f-8eff-473b-b150-d623c873cf57-audit-log\") pod \"metrics-server-67b6f56d48-lvpbv\" (UID: \"0d71645f-8eff-473b-b150-d623c873cf57\") " pod="openshift-monitoring/metrics-server-67b6f56d48-lvpbv"
Sep 18 16:58:35.395450 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:35.395424    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d71645f-8eff-473b-b150-d623c873cf57-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-67b6f56d48-lvpbv\" (UID: \"0d71645f-8eff-473b-b150-d623c873cf57\") " pod="openshift-monitoring/metrics-server-67b6f56d48-lvpbv"
Sep 18 16:58:35.395522 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:35.395443    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0d71645f-8eff-473b-b150-d623c873cf57-metrics-server-audit-profiles\") pod \"metrics-server-67b6f56d48-lvpbv\" (UID: \"0d71645f-8eff-473b-b150-d623c873cf57\") " pod="openshift-monitoring/metrics-server-67b6f56d48-lvpbv"
Sep 18 16:58:35.398049 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:35.397573    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0d71645f-8eff-473b-b150-d623c873cf57-secret-metrics-server-tls\") pod \"metrics-server-67b6f56d48-lvpbv\" (UID: \"0d71645f-8eff-473b-b150-d623c873cf57\") " pod="openshift-monitoring/metrics-server-67b6f56d48-lvpbv"
Sep 18 16:58:35.398049 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:35.398009    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0d71645f-8eff-473b-b150-d623c873cf57-client-ca-bundle\") pod \"metrics-server-67b6f56d48-lvpbv\" (UID: \"0d71645f-8eff-473b-b150-d623c873cf57\") " pod="openshift-monitoring/metrics-server-67b6f56d48-lvpbv"
Sep 18 16:58:35.399643 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:35.399595    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0d71645f-8eff-473b-b150-d623c873cf57-secret-metrics-server-client-certs\") pod \"metrics-server-67b6f56d48-lvpbv\" (UID: \"0d71645f-8eff-473b-b150-d623c873cf57\") " pod="openshift-monitoring/metrics-server-67b6f56d48-lvpbv"
Sep 18 16:58:35.404308 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:35.404276    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjz5z\" (UniqueName: \"kubernetes.io/projected/0d71645f-8eff-473b-b150-d623c873cf57-kube-api-access-mjz5z\") pod \"metrics-server-67b6f56d48-lvpbv\" (UID: \"0d71645f-8eff-473b-b150-d623c873cf57\") " pod="openshift-monitoring/metrics-server-67b6f56d48-lvpbv"
Sep 18 16:58:35.522189 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:35.521895    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-67b6f56d48-lvpbv"
Sep 18 16:58:35.548090 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:35.548037    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-j8k2q"]
Sep 18 16:58:35.551555 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:35.551480    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-j8k2q"
Sep 18 16:58:35.554613 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:35.554125    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 18 16:58:35.554613 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:35.554444    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-cpvkj\""
Sep 18 16:58:35.563102 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:35.561861    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-j8k2q"]
Sep 18 16:58:35.649710 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:35.649594    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cdcc3b80-4274-48f4-bf97-09412758b028","Type":"ContainerStarted","Data":"f1f87f64fd82a59012dc7117adc9f18211f8d07d7c56a4dae614a843e0f76150"}
Sep 18 16:58:35.656641 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:35.656604    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pbhzd" event={"ID":"bea5387e-1e26-4be9-bd38-71c49eb4225d","Type":"ContainerStarted","Data":"09fd2f23e9ecd77cea1c25cfabad577a624c657467677628a77ecefa7bae9b0d"}
Sep 18 16:58:35.656837 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:35.656672    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pbhzd" event={"ID":"bea5387e-1e26-4be9-bd38-71c49eb4225d","Type":"ContainerStarted","Data":"b05b1ecf436a1b65ec1ec0402ae11dfc23f2120c0ed96c02a00e4160a8898b4f"}
Sep 18 16:58:35.658897 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:35.658867    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-96b4948d4-zsn4k" event={"ID":"0de2e307-fa16-46a0-bc95-2a59d4fa9d44","Type":"ContainerStarted","Data":"0fdb02e83722d8b90b0597d5857d9955a5c63d8ec582783c30778868b2127aa4"}
Sep 18 16:58:35.664179 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:35.664085    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-dn7sr" event={"ID":"69a62b4d-4c8d-4dc4-81a8-35628f51c837","Type":"ContainerStarted","Data":"628f6d5d4e4e81cd25588cba037af0fd10dc79e014c677c361cad66e2b126f97"}
Sep 18 16:58:35.664179 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:35.664154    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-dn7sr" event={"ID":"69a62b4d-4c8d-4dc4-81a8-35628f51c837","Type":"ContainerStarted","Data":"9ec58601c9953fbc3f977a31ef88559a7d1e5737daaf779b3c5e991814170e65"}
Sep 18 16:58:35.664454 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:35.664186    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-dn7sr" event={"ID":"69a62b4d-4c8d-4dc4-81a8-35628f51c837","Type":"ContainerStarted","Data":"badf56ca2c00aec207fbc1032827c170307b53e91d71c219f235c569844e9fd6"}
Sep 18 16:58:35.666070 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:35.665962    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-tblfl" event={"ID":"8c199118-475a-4607-ba80-9a8027f1b047","Type":"ContainerStarted","Data":"795012641c3d178761e5be7e54968172708ac20c108e0a9769414b0d7a21ee71"}
Sep 18 16:58:35.697176 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:35.697129    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzt5k\" (UniqueName: \"kubernetes.io/projected/62a6075e-7f67-4a49-80f6-a8f59132bb8a-kube-api-access-zzt5k\") pod \"monitoring-plugin-5984459ff8-j8k2q\" (UID: \"62a6075e-7f67-4a49-80f6-a8f59132bb8a\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-j8k2q"
Sep 18 16:58:35.697345 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:35.697245    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/62a6075e-7f67-4a49-80f6-a8f59132bb8a-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-j8k2q\" (UID: \"62a6075e-7f67-4a49-80f6-a8f59132bb8a\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-j8k2q"
Sep 18 16:58:35.799306 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:35.798572    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzt5k\" (UniqueName: \"kubernetes.io/projected/62a6075e-7f67-4a49-80f6-a8f59132bb8a-kube-api-access-zzt5k\") pod \"monitoring-plugin-5984459ff8-j8k2q\" (UID: \"62a6075e-7f67-4a49-80f6-a8f59132bb8a\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-j8k2q"
Sep 18 16:58:35.799306 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:35.798719    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/62a6075e-7f67-4a49-80f6-a8f59132bb8a-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-j8k2q\" (UID: \"62a6075e-7f67-4a49-80f6-a8f59132bb8a\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-j8k2q"
Sep 18 16:58:35.800208 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:58:35.799929    2881 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 18 16:58:35.800208 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:58:35.800003    2881 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62a6075e-7f67-4a49-80f6-a8f59132bb8a-monitoring-plugin-cert podName:62a6075e-7f67-4a49-80f6-a8f59132bb8a nodeName:}" failed. No retries permitted until 2026-09-18 16:58:36.299981335 +0000 UTC m=+306.499836607 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/62a6075e-7f67-4a49-80f6-a8f59132bb8a-monitoring-plugin-cert") pod "monitoring-plugin-5984459ff8-j8k2q" (UID: "62a6075e-7f67-4a49-80f6-a8f59132bb8a") : secret "monitoring-plugin-cert" not found
Sep 18 16:58:35.814048 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:35.813981    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzt5k\" (UniqueName: \"kubernetes.io/projected/62a6075e-7f67-4a49-80f6-a8f59132bb8a-kube-api-access-zzt5k\") pod \"monitoring-plugin-5984459ff8-j8k2q\" (UID: \"62a6075e-7f67-4a49-80f6-a8f59132bb8a\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-j8k2q"
Sep 18 16:58:36.304943 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:36.304900    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/62a6075e-7f67-4a49-80f6-a8f59132bb8a-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-j8k2q\" (UID: \"62a6075e-7f67-4a49-80f6-a8f59132bb8a\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-j8k2q"
Sep 18 16:58:36.307655 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:36.307621    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/62a6075e-7f67-4a49-80f6-a8f59132bb8a-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-j8k2q\" (UID: \"62a6075e-7f67-4a49-80f6-a8f59132bb8a\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-j8k2q"
Sep 18 16:58:36.469519 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:36.469492    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-j8k2q"
Sep 18 16:58:36.552992 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:36.552935    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-pbhzd" podStartSLOduration=3.252670922 podStartE2EDuration="6.552917959s" totalImagesPullingTime="3.221004112s" totalInitContainerRuntime="79.242925ms" isStatefulPod=true podCreationTimestamp="2026-09-18 16:58:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:58:35.677729108 +0000 UTC m=+305.877584389" watchObservedRunningTime="2026-09-18 16:58:36.552917959 +0000 UTC m=+306.752773288"
Sep 18 16:58:36.553693 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:36.553667    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-789fdf96f9-9vphn"]
Sep 18 16:58:36.557433 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:36.557370    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-789fdf96f9-9vphn"
Sep 18 16:58:36.568858 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:36.568823    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-789fdf96f9-9vphn"]
Sep 18 16:58:36.709600 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:36.709549    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ade8157b-6ddf-4e19-9129-11e6177db425-service-ca\") pod \"console-789fdf96f9-9vphn\" (UID: \"ade8157b-6ddf-4e19-9129-11e6177db425\") " pod="openshift-console/console-789fdf96f9-9vphn"
Sep 18 16:58:36.709823 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:36.709611    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45wr9\" (UniqueName: \"kubernetes.io/projected/ade8157b-6ddf-4e19-9129-11e6177db425-kube-api-access-45wr9\") pod \"console-789fdf96f9-9vphn\" (UID: \"ade8157b-6ddf-4e19-9129-11e6177db425\") " pod="openshift-console/console-789fdf96f9-9vphn"
Sep 18 16:58:36.709823 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:36.709651    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ade8157b-6ddf-4e19-9129-11e6177db425-console-oauth-config\") pod \"console-789fdf96f9-9vphn\" (UID: \"ade8157b-6ddf-4e19-9129-11e6177db425\") " pod="openshift-console/console-789fdf96f9-9vphn"
Sep 18 16:58:36.709823 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:36.709694    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ade8157b-6ddf-4e19-9129-11e6177db425-console-serving-cert\") pod \"console-789fdf96f9-9vphn\" (UID: \"ade8157b-6ddf-4e19-9129-11e6177db425\") " pod="openshift-console/console-789fdf96f9-9vphn"
Sep 18 16:58:36.709985 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:36.709807    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ade8157b-6ddf-4e19-9129-11e6177db425-console-config\") pod \"console-789fdf96f9-9vphn\" (UID: \"ade8157b-6ddf-4e19-9129-11e6177db425\") " pod="openshift-console/console-789fdf96f9-9vphn"
Sep 18 16:58:36.709985 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:36.709869    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ade8157b-6ddf-4e19-9129-11e6177db425-oauth-serving-cert\") pod \"console-789fdf96f9-9vphn\" (UID: \"ade8157b-6ddf-4e19-9129-11e6177db425\") " pod="openshift-console/console-789fdf96f9-9vphn"
Sep 18 16:58:36.709985 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:36.709923    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ade8157b-6ddf-4e19-9129-11e6177db425-trusted-ca-bundle\") pod \"console-789fdf96f9-9vphn\" (UID: \"ade8157b-6ddf-4e19-9129-11e6177db425\") " pod="openshift-console/console-789fdf96f9-9vphn"
Sep 18 16:58:36.811549 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:36.811453    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ade8157b-6ddf-4e19-9129-11e6177db425-service-ca\") pod \"console-789fdf96f9-9vphn\" (UID: \"ade8157b-6ddf-4e19-9129-11e6177db425\") " pod="openshift-console/console-789fdf96f9-9vphn"
Sep 18 16:58:36.811549 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:36.811512    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45wr9\" (UniqueName: \"kubernetes.io/projected/ade8157b-6ddf-4e19-9129-11e6177db425-kube-api-access-45wr9\") pod \"console-789fdf96f9-9vphn\" (UID: \"ade8157b-6ddf-4e19-9129-11e6177db425\") " pod="openshift-console/console-789fdf96f9-9vphn"
Sep 18 16:58:36.811799 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:36.811556    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ade8157b-6ddf-4e19-9129-11e6177db425-console-oauth-config\") pod \"console-789fdf96f9-9vphn\" (UID: \"ade8157b-6ddf-4e19-9129-11e6177db425\") " pod="openshift-console/console-789fdf96f9-9vphn"
Sep 18 16:58:36.811799 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:36.811633    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ade8157b-6ddf-4e19-9129-11e6177db425-console-serving-cert\") pod \"console-789fdf96f9-9vphn\" (UID: \"ade8157b-6ddf-4e19-9129-11e6177db425\") " pod="openshift-console/console-789fdf96f9-9vphn"
Sep 18 16:58:36.811799 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:36.811735    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ade8157b-6ddf-4e19-9129-11e6177db425-console-config\") pod \"console-789fdf96f9-9vphn\" (UID: \"ade8157b-6ddf-4e19-9129-11e6177db425\") " pod="openshift-console/console-789fdf96f9-9vphn"
Sep 18 16:58:36.811952 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:36.811784    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ade8157b-6ddf-4e19-9129-11e6177db425-oauth-serving-cert\") pod \"console-789fdf96f9-9vphn\" (UID: \"ade8157b-6ddf-4e19-9129-11e6177db425\") " pod="openshift-console/console-789fdf96f9-9vphn"
Sep 18 16:58:36.811952 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:36.811854    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ade8157b-6ddf-4e19-9129-11e6177db425-trusted-ca-bundle\") pod \"console-789fdf96f9-9vphn\" (UID: \"ade8157b-6ddf-4e19-9129-11e6177db425\") " pod="openshift-console/console-789fdf96f9-9vphn"
Sep 18 16:58:36.812442 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:36.812381    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ade8157b-6ddf-4e19-9129-11e6177db425-service-ca\") pod \"console-789fdf96f9-9vphn\" (UID: \"ade8157b-6ddf-4e19-9129-11e6177db425\") " pod="openshift-console/console-789fdf96f9-9vphn"
Sep 18 16:58:36.812784 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:36.812758    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ade8157b-6ddf-4e19-9129-11e6177db425-console-config\") pod \"console-789fdf96f9-9vphn\" (UID: \"ade8157b-6ddf-4e19-9129-11e6177db425\") " pod="openshift-console/console-789fdf96f9-9vphn"
Sep 18 16:58:36.812919 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:36.812883    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ade8157b-6ddf-4e19-9129-11e6177db425-trusted-ca-bundle\") pod \"console-789fdf96f9-9vphn\" (UID: \"ade8157b-6ddf-4e19-9129-11e6177db425\") " pod="openshift-console/console-789fdf96f9-9vphn"
Sep 18 16:58:36.812981 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:36.812935    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ade8157b-6ddf-4e19-9129-11e6177db425-oauth-serving-cert\") pod \"console-789fdf96f9-9vphn\" (UID: \"ade8157b-6ddf-4e19-9129-11e6177db425\") " pod="openshift-console/console-789fdf96f9-9vphn"
Sep 18 16:58:36.814966 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:36.814823    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ade8157b-6ddf-4e19-9129-11e6177db425-console-oauth-config\") pod \"console-789fdf96f9-9vphn\" (UID: \"ade8157b-6ddf-4e19-9129-11e6177db425\") " pod="openshift-console/console-789fdf96f9-9vphn"
Sep 18 16:58:36.815203 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:36.815166    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ade8157b-6ddf-4e19-9129-11e6177db425-console-serving-cert\") pod \"console-789fdf96f9-9vphn\" (UID: \"ade8157b-6ddf-4e19-9129-11e6177db425\") " pod="openshift-console/console-789fdf96f9-9vphn"
Sep 18 16:58:36.821252 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:36.821197    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45wr9\" (UniqueName: \"kubernetes.io/projected/ade8157b-6ddf-4e19-9129-11e6177db425-kube-api-access-45wr9\") pod \"console-789fdf96f9-9vphn\" (UID: \"ade8157b-6ddf-4e19-9129-11e6177db425\") " pod="openshift-console/console-789fdf96f9-9vphn"
Sep 18 16:58:36.870403 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:36.870319    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-789fdf96f9-9vphn"
Sep 18 16:58:37.099828 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.099130    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 16:58:37.106148 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.106114    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:58:37.109300 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.109266    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 18 16:58:37.110463 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.110282    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 18 16:58:37.110463 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.110357    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 18 16:58:37.120451 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.113808    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-b19nhunhjj893\""
Sep 18 16:58:37.120451 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.115440    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 18 16:58:37.120451 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.115671    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 18 16:58:37.120451 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.115918    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 18 16:58:37.120451 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.116095    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 18 16:58:37.120451 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.116286    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-dj4zs\""
Sep 18 16:58:37.120451 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.116487    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 18 16:58:37.120451 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.116789    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 18 16:58:37.120451 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.116977    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 18 16:58:37.120451 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.117561    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 18 16:58:37.120451 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.117756    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 18 16:58:37.140295 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.134191    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 16:58:37.147058 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.147024    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 18 16:58:37.159084 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.150967    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 18 16:58:37.216348 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.216307    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/465d3c10-1648-4252-87c0-9b27987d8de4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"465d3c10-1648-4252-87c0-9b27987d8de4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:58:37.216348 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.216376    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/465d3c10-1648-4252-87c0-9b27987d8de4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"465d3c10-1648-4252-87c0-9b27987d8de4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:58:37.216348 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.216403    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/465d3c10-1648-4252-87c0-9b27987d8de4-config\") pod \"prometheus-k8s-0\" (UID: \"465d3c10-1648-4252-87c0-9b27987d8de4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:58:37.216348 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.216482    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/465d3c10-1648-4252-87c0-9b27987d8de4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"465d3c10-1648-4252-87c0-9b27987d8de4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:58:37.216348 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.216519    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/465d3c10-1648-4252-87c0-9b27987d8de4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"465d3c10-1648-4252-87c0-9b27987d8de4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:58:37.216348 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.216547    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/465d3c10-1648-4252-87c0-9b27987d8de4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"465d3c10-1648-4252-87c0-9b27987d8de4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:58:37.216348 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.216575    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/465d3c10-1648-4252-87c0-9b27987d8de4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"465d3c10-1648-4252-87c0-9b27987d8de4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:58:37.216348 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.216632    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/465d3c10-1648-4252-87c0-9b27987d8de4-web-config\") pod \"prometheus-k8s-0\" (UID: \"465d3c10-1648-4252-87c0-9b27987d8de4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:58:37.216348 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.216655    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/465d3c10-1648-4252-87c0-9b27987d8de4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"465d3c10-1648-4252-87c0-9b27987d8de4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:58:37.216348 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.216684    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/465d3c10-1648-4252-87c0-9b27987d8de4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"465d3c10-1648-4252-87c0-9b27987d8de4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:58:37.216348 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.216707    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/465d3c10-1648-4252-87c0-9b27987d8de4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"465d3c10-1648-4252-87c0-9b27987d8de4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:58:37.216348 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.216733    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/465d3c10-1648-4252-87c0-9b27987d8de4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"465d3c10-1648-4252-87c0-9b27987d8de4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:58:37.216348 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.216755    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/465d3c10-1648-4252-87c0-9b27987d8de4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"465d3c10-1648-4252-87c0-9b27987d8de4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:58:37.216348 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.216779    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7248m\" (UniqueName: \"kubernetes.io/projected/465d3c10-1648-4252-87c0-9b27987d8de4-kube-api-access-7248m\") pod \"prometheus-k8s-0\" (UID: \"465d3c10-1648-4252-87c0-9b27987d8de4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:58:37.216348 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.216811    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/465d3c10-1648-4252-87c0-9b27987d8de4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"465d3c10-1648-4252-87c0-9b27987d8de4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:58:37.216348 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.216865    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/465d3c10-1648-4252-87c0-9b27987d8de4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"465d3c10-1648-4252-87c0-9b27987d8de4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:58:37.218192 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.216911    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/465d3c10-1648-4252-87c0-9b27987d8de4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"465d3c10-1648-4252-87c0-9b27987d8de4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:58:37.218192 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.216947    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/465d3c10-1648-4252-87c0-9b27987d8de4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"465d3c10-1648-4252-87c0-9b27987d8de4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:58:37.218192 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.216990    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/465d3c10-1648-4252-87c0-9b27987d8de4-config-out\") pod \"prometheus-k8s-0\" (UID: \"465d3c10-1648-4252-87c0-9b27987d8de4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:58:37.218192 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.217012    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/465d3c10-1648-4252-87c0-9b27987d8de4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"465d3c10-1648-4252-87c0-9b27987d8de4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:58:37.318118 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.318037    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/465d3c10-1648-4252-87c0-9b27987d8de4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"465d3c10-1648-4252-87c0-9b27987d8de4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:58:37.318118 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.318082    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/465d3c10-1648-4252-87c0-9b27987d8de4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"465d3c10-1648-4252-87c0-9b27987d8de4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:58:37.318118 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.318113    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/465d3c10-1648-4252-87c0-9b27987d8de4-config-out\") pod \"prometheus-k8s-0\" (UID: \"465d3c10-1648-4252-87c0-9b27987d8de4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:58:37.318711 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.318128    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/465d3c10-1648-4252-87c0-9b27987d8de4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"465d3c10-1648-4252-87c0-9b27987d8de4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:58:37.318711 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.318156    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/465d3c10-1648-4252-87c0-9b27987d8de4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"465d3c10-1648-4252-87c0-9b27987d8de4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:58:37.318711 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.318199    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/465d3c10-1648-4252-87c0-9b27987d8de4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"465d3c10-1648-4252-87c0-9b27987d8de4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:58:37.318711 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.318249    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/465d3c10-1648-4252-87c0-9b27987d8de4-config\") pod \"prometheus-k8s-0\" (UID: \"465d3c10-1648-4252-87c0-9b27987d8de4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:58:37.318711 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.318289    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/465d3c10-1648-4252-87c0-9b27987d8de4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"465d3c10-1648-4252-87c0-9b27987d8de4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:58:37.318711 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.318311    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/465d3c10-1648-4252-87c0-9b27987d8de4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"465d3c10-1648-4252-87c0-9b27987d8de4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:58:37.318711 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.318336    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/465d3c10-1648-4252-87c0-9b27987d8de4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"465d3c10-1648-4252-87c0-9b27987d8de4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:58:37.318711 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.318364    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/465d3c10-1648-4252-87c0-9b27987d8de4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"465d3c10-1648-4252-87c0-9b27987d8de4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:58:37.318711 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.318420    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/465d3c10-1648-4252-87c0-9b27987d8de4-web-config\") pod \"prometheus-k8s-0\" (UID: \"465d3c10-1648-4252-87c0-9b27987d8de4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:58:37.318711 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.318442    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/465d3c10-1648-4252-87c0-9b27987d8de4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"465d3c10-1648-4252-87c0-9b27987d8de4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:58:37.318711 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.318469    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/465d3c10-1648-4252-87c0-9b27987d8de4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"465d3c10-1648-4252-87c0-9b27987d8de4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:58:37.318711 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.318493    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/465d3c10-1648-4252-87c0-9b27987d8de4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"465d3c10-1648-4252-87c0-9b27987d8de4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:58:37.318711 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.318516    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/465d3c10-1648-4252-87c0-9b27987d8de4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"465d3c10-1648-4252-87c0-9b27987d8de4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:58:37.318711 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.318538    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/465d3c10-1648-4252-87c0-9b27987d8de4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"465d3c10-1648-4252-87c0-9b27987d8de4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:58:37.318711 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.318564    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7248m\" (UniqueName: \"kubernetes.io/projected/465d3c10-1648-4252-87c0-9b27987d8de4-kube-api-access-7248m\") pod \"prometheus-k8s-0\" (UID: \"465d3c10-1648-4252-87c0-9b27987d8de4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:58:37.318711 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.318617    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/465d3c10-1648-4252-87c0-9b27987d8de4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"465d3c10-1648-4252-87c0-9b27987d8de4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:58:37.318711 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.318654    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/465d3c10-1648-4252-87c0-9b27987d8de4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"465d3c10-1648-4252-87c0-9b27987d8de4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:58:37.319558 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.318888    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/465d3c10-1648-4252-87c0-9b27987d8de4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"465d3c10-1648-4252-87c0-9b27987d8de4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:58:37.319558 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.319041    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/465d3c10-1648-4252-87c0-9b27987d8de4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"465d3c10-1648-4252-87c0-9b27987d8de4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:58:37.319558 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.319193    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/465d3c10-1648-4252-87c0-9b27987d8de4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"465d3c10-1648-4252-87c0-9b27987d8de4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:58:37.319718 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.319683    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/465d3c10-1648-4252-87c0-9b27987d8de4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"465d3c10-1648-4252-87c0-9b27987d8de4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:58:37.322440 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.320938    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/465d3c10-1648-4252-87c0-9b27987d8de4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"465d3c10-1648-4252-87c0-9b27987d8de4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:58:37.322440 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.322052    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/465d3c10-1648-4252-87c0-9b27987d8de4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"465d3c10-1648-4252-87c0-9b27987d8de4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:58:37.323325 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.322807    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/465d3c10-1648-4252-87c0-9b27987d8de4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"465d3c10-1648-4252-87c0-9b27987d8de4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:58:37.324161 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.323646    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/465d3c10-1648-4252-87c0-9b27987d8de4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"465d3c10-1648-4252-87c0-9b27987d8de4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:58:37.324161 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.324102    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/465d3c10-1648-4252-87c0-9b27987d8de4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"465d3c10-1648-4252-87c0-9b27987d8de4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:58:37.325142 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.324777    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/465d3c10-1648-4252-87c0-9b27987d8de4-web-config\") pod \"prometheus-k8s-0\" (UID: \"465d3c10-1648-4252-87c0-9b27987d8de4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:58:37.325273 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.325161    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/465d3c10-1648-4252-87c0-9b27987d8de4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"465d3c10-1648-4252-87c0-9b27987d8de4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:58:37.325478 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.325355    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/465d3c10-1648-4252-87c0-9b27987d8de4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"465d3c10-1648-4252-87c0-9b27987d8de4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:58:37.325797 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.325668    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/465d3c10-1648-4252-87c0-9b27987d8de4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"465d3c10-1648-4252-87c0-9b27987d8de4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:58:37.326102 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.326066    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/465d3c10-1648-4252-87c0-9b27987d8de4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"465d3c10-1648-4252-87c0-9b27987d8de4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:58:37.326359 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.326333    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/465d3c10-1648-4252-87c0-9b27987d8de4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"465d3c10-1648-4252-87c0-9b27987d8de4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:58:37.327020 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.326772    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/465d3c10-1648-4252-87c0-9b27987d8de4-config\") pod \"prometheus-k8s-0\" (UID: \"465d3c10-1648-4252-87c0-9b27987d8de4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:58:37.327129 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.327082    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/465d3c10-1648-4252-87c0-9b27987d8de4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"465d3c10-1648-4252-87c0-9b27987d8de4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:58:37.327189 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.327165    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/465d3c10-1648-4252-87c0-9b27987d8de4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"465d3c10-1648-4252-87c0-9b27987d8de4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:58:37.327381 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.327307    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/465d3c10-1648-4252-87c0-9b27987d8de4-config-out\") pod \"prometheus-k8s-0\" (UID: \"465d3c10-1648-4252-87c0-9b27987d8de4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:58:37.339580 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.339539    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7248m\" (UniqueName: \"kubernetes.io/projected/465d3c10-1648-4252-87c0-9b27987d8de4-kube-api-access-7248m\") pod \"prometheus-k8s-0\" (UID: \"465d3c10-1648-4252-87c0-9b27987d8de4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:58:37.379822 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.379561    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-67b6f56d48-lvpbv"]
Sep 18 16:58:37.394058 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.392558    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-j8k2q"]
Sep 18 16:58:37.395558 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.395531    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-789fdf96f9-9vphn"]
Sep 18 16:58:37.471173 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.470822    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:58:37.677805 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.677765    2881 generic.go:356] "Generic (PLEG): container finished" podID="cdcc3b80-4274-48f4-bf97-09412758b028" containerID="5056d7b6fd3c12099973e25a5392669f654423823bb3610d8fc9c1b6b4f633a4" exitCode=0
Sep 18 16:58:37.678021 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.677859    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cdcc3b80-4274-48f4-bf97-09412758b028","Type":"ContainerDied","Data":"5056d7b6fd3c12099973e25a5392669f654423823bb3610d8fc9c1b6b4f633a4"}
Sep 18 16:58:37.681399 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.681061    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-96b4948d4-zsn4k" event={"ID":"0de2e307-fa16-46a0-bc95-2a59d4fa9d44","Type":"ContainerStarted","Data":"9f7613b8bee8cbf157a89b18b7601aa80d80fad8be48f90a9f58f8ecd29db4ee"}
Sep 18 16:58:37.681399 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.681143    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-96b4948d4-zsn4k" event={"ID":"0de2e307-fa16-46a0-bc95-2a59d4fa9d44","Type":"ContainerStarted","Data":"bf0b3e836fe632b37d72d26111b5ff68c4ab98e629b39f0b73757207c0379154"}
Sep 18 16:58:37.681399 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.681183    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-96b4948d4-zsn4k" event={"ID":"0de2e307-fa16-46a0-bc95-2a59d4fa9d44","Type":"ContainerStarted","Data":"73a8c885156c72bbcb14b2815e77a509b50b3a75ccc924278113692f0bc13c81"}
Sep 18 16:58:37.685343 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.685257    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-789fdf96f9-9vphn" event={"ID":"ade8157b-6ddf-4e19-9129-11e6177db425","Type":"ContainerStarted","Data":"c8bdfe8065a5b19df10ed0c12a6bf45d267b00d3ff6a6b8526b82f4e343dcee0"}
Sep 18 16:58:37.685343 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.685315    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-789fdf96f9-9vphn" event={"ID":"ade8157b-6ddf-4e19-9129-11e6177db425","Type":"ContainerStarted","Data":"e8b7054e9db9a68636f76e8dcabd2ab420c46424bb01c8c0ae9a081bdd4e41e4"}
Sep 18 16:58:37.686720 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.686691    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-j8k2q" event={"ID":"62a6075e-7f67-4a49-80f6-a8f59132bb8a","Type":"ContainerStarted","Data":"da734f34e9f2bc51785efbb75e5c3419fb2078ef29c10546f074f6e8dc5ced95"}
Sep 18 16:58:37.688097 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.687982    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-67b6f56d48-lvpbv" event={"ID":"0d71645f-8eff-473b-b150-d623c873cf57","Type":"ContainerStarted","Data":"5d2f6f08dbdde5f36f1a3ce3cfeb5d986d4bff2ec6d6acc8703fa18a0f1653ab"}
Sep 18 16:58:37.692936 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.692897    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-dn7sr" event={"ID":"69a62b4d-4c8d-4dc4-81a8-35628f51c837","Type":"ContainerStarted","Data":"9f3b09ae1a2c6af7c75cf8b24e15690f302034ba95439f12cc59e216416de3b2"}
Sep 18 16:58:37.695414 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.695383    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-tblfl" event={"ID":"8c199118-475a-4607-ba80-9a8027f1b047","Type":"ContainerStarted","Data":"eadbe4dfd9ec88fc1c8e7197d8b0868abb15c6c5018e859ded0a094d92d07f1b"}
Sep 18 16:58:37.695556 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.695442    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-tblfl" event={"ID":"8c199118-475a-4607-ba80-9a8027f1b047","Type":"ContainerStarted","Data":"67622b8334ddaf33f6d3606a97b56ffba1be821025650b65b31152bebc651e4f"}
Sep 18 16:58:37.695556 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.695477    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-tblfl" event={"ID":"8c199118-475a-4607-ba80-9a8027f1b047","Type":"ContainerStarted","Data":"d95eff51a9db0675463470a8c06cf9ff68ff56d30d8bb9a00385b4f05e1159f5"}
Sep 18 16:58:37.724475 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.724406    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-84c885658c-tblfl" podStartSLOduration=5.350085975 podStartE2EDuration="7.724382445s" totalImagesPullingTime="2.37429647s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:58:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:58:37.722953219 +0000 UTC m=+307.922808507" watchObservedRunningTime="2026-09-18 16:58:37.724382445 +0000 UTC m=+307.924237731"
Sep 18 16:58:37.742093 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.742027    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-dn7sr" podStartSLOduration=5.511900413 podStartE2EDuration="7.742003621s" totalImagesPullingTime="2.230103208s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:58:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:58:37.740298334 +0000 UTC m=+307.940153624" watchObservedRunningTime="2026-09-18 16:58:37.742003621 +0000 UTC m=+307.941858908"
Sep 18 16:58:37.761964 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.761900    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-789fdf96f9-9vphn" podStartSLOduration=1.761879654 podStartE2EDuration="1.761879654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:58:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:58:37.761399458 +0000 UTC m=+307.961254751" watchObservedRunningTime="2026-09-18 16:58:37.761879654 +0000 UTC m=+307.961734940"
Sep 18 16:58:37.801631 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:37.801605    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 16:58:37.803669 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:58:37.803640    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod465d3c10_1648_4252_87c0_9b27987d8de4.slice/crio-e22c3d63bb5ab0c0a2c14ab5d070a1420d24f6ec1b0ae16c13369c7c4eebb04e WatchSource:0}: Error finding container e22c3d63bb5ab0c0a2c14ab5d070a1420d24f6ec1b0ae16c13369c7c4eebb04e: Status 404 returned error can't find the container with id e22c3d63bb5ab0c0a2c14ab5d070a1420d24f6ec1b0ae16c13369c7c4eebb04e
Sep 18 16:58:38.603993 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:38.603724    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-c8l26"
Sep 18 16:58:38.702946 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:38.702716    2881 generic.go:356] "Generic (PLEG): container finished" podID="465d3c10-1648-4252-87c0-9b27987d8de4" containerID="d19aaaa3cb5076efa7701213a33440d300a23856117de91debd003b20572bbd9" exitCode=0
Sep 18 16:58:38.702946 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:38.702796    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"465d3c10-1648-4252-87c0-9b27987d8de4","Type":"ContainerDied","Data":"d19aaaa3cb5076efa7701213a33440d300a23856117de91debd003b20572bbd9"}
Sep 18 16:58:38.702946 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:38.702883    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"465d3c10-1648-4252-87c0-9b27987d8de4","Type":"ContainerStarted","Data":"e22c3d63bb5ab0c0a2c14ab5d070a1420d24f6ec1b0ae16c13369c7c4eebb04e"}
Sep 18 16:58:40.719213 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:40.719120    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cdcc3b80-4274-48f4-bf97-09412758b028","Type":"ContainerStarted","Data":"b79fe3e67efe7ac4b03a8f3d49ca83ad9ce2bdf1f7ec22690fa25417ae9ca2f0"}
Sep 18 16:58:40.719677 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:40.719259    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cdcc3b80-4274-48f4-bf97-09412758b028","Type":"ContainerStarted","Data":"38b073a43938b5b39c5377d3db5f137cb185ded58b08baddca2ea8359d03d924"}
Sep 18 16:58:40.719677 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:40.719309    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cdcc3b80-4274-48f4-bf97-09412758b028","Type":"ContainerStarted","Data":"eeb12d37525c657f4dfeadb4eb0daf215733ee95c5bfcb5265a0204a7a936ebe"}
Sep 18 16:58:40.719677 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:40.719349    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cdcc3b80-4274-48f4-bf97-09412758b028","Type":"ContainerStarted","Data":"02e00b963777047fbe616afa7b800f53cba468ac217909ac2ff401ad126623f5"}
Sep 18 16:58:40.719677 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:40.719390    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cdcc3b80-4274-48f4-bf97-09412758b028","Type":"ContainerStarted","Data":"48c93ecfb6ada7b36087a7dac069f6f80290096b673cb2454a3b9425062ad7b3"}
Sep 18 16:58:40.719677 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:40.719429    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cdcc3b80-4274-48f4-bf97-09412758b028","Type":"ContainerStarted","Data":"188fbe696b16bb157ccc65f0439c9b4fee879c9c862122a37cb8ffe4b2adace4"}
Sep 18 16:58:40.722876 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:40.722838    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-96b4948d4-zsn4k" event={"ID":"0de2e307-fa16-46a0-bc95-2a59d4fa9d44","Type":"ContainerStarted","Data":"c298cffc404e65125953169c650237322f448c96109346d793d2ac8585606f32"}
Sep 18 16:58:40.723023 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:40.722923    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-96b4948d4-zsn4k" event={"ID":"0de2e307-fa16-46a0-bc95-2a59d4fa9d44","Type":"ContainerStarted","Data":"06b97d2dfcd437df3a6478f026dea6d4de27da0736c728f807e4a72a5a9d9be4"}
Sep 18 16:58:40.723023 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:40.722975    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-96b4948d4-zsn4k" event={"ID":"0de2e307-fa16-46a0-bc95-2a59d4fa9d44","Type":"ContainerStarted","Data":"bf2fea1ec1a0081b584fcd70a08f9f02d7468c94d0385346b9683be9fccc275c"}
Sep 18 16:58:40.723154 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:40.723023    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-96b4948d4-zsn4k"
Sep 18 16:58:40.724688 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:40.724657    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-j8k2q" event={"ID":"62a6075e-7f67-4a49-80f6-a8f59132bb8a","Type":"ContainerStarted","Data":"fa4f416b24a8964b194a00ec72a43001a93e990ae65cddf6a22aec2380a08216"}
Sep 18 16:58:40.724827 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:40.724798    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-j8k2q"
Sep 18 16:58:40.726826 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:40.726760    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-67b6f56d48-lvpbv" event={"ID":"0d71645f-8eff-473b-b150-d623c873cf57","Type":"ContainerStarted","Data":"e278e3f2897df4ba23a59c169ac37591ca16e8e0e4fb3a9cfb4612b322d4a2d0"}
Sep 18 16:58:40.730644 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:40.730616    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-j8k2q"
Sep 18 16:58:40.752568 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:40.752498    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=5.112822121 podStartE2EDuration="9.752473573s" totalImagesPullingTime="4.616579703s" totalInitContainerRuntime="23.071749ms" isStatefulPod=false podCreationTimestamp="2026-09-18 16:58:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:58:40.749453786 +0000 UTC m=+310.949309074" watchObservedRunningTime="2026-09-18 16:58:40.752473573 +0000 UTC m=+310.952328861"
Sep 18 16:58:40.768478 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:40.768408    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5984459ff8-j8k2q" podStartSLOduration=3.228947373 podStartE2EDuration="5.76838402s" totalImagesPullingTime="2.539436647s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:58:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:58:40.767468968 +0000 UTC m=+310.967324257" watchObservedRunningTime="2026-09-18 16:58:40.76838402 +0000 UTC m=+310.968239305"
Sep 18 16:58:40.817070 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:40.817014    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-67b6f56d48-lvpbv" podStartSLOduration=3.271091599 podStartE2EDuration="5.81699475s" totalImagesPullingTime="2.545903151s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:58:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:58:40.816214114 +0000 UTC m=+311.016069394" watchObservedRunningTime="2026-09-18 16:58:40.81699475 +0000 UTC m=+311.016850037"
Sep 18 16:58:40.817503 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:40.817462    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-96b4948d4-zsn4k" podStartSLOduration=3.055628235 podStartE2EDuration="7.817450748s" totalImagesPullingTime="4.761822513s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:58:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:58:40.798040948 +0000 UTC m=+310.997896235" watchObservedRunningTime="2026-09-18 16:58:40.817450748 +0000 UTC m=+311.017306064"
Sep 18 16:58:42.741762 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:42.741722    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"465d3c10-1648-4252-87c0-9b27987d8de4","Type":"ContainerStarted","Data":"a49c37291ce93bd5de3f208f600b95bbca99d8cae254b5ff1833a01c5a4b2646"}
Sep 18 16:58:42.742111 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:42.741800    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"465d3c10-1648-4252-87c0-9b27987d8de4","Type":"ContainerStarted","Data":"0ebafaa474387c9a7fef5552fb1e5b1d54d44a85127373eb65fb2360cf13a1e2"}
Sep 18 16:58:42.742111 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:42.741830    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"465d3c10-1648-4252-87c0-9b27987d8de4","Type":"ContainerStarted","Data":"fc2abe1cb63ceb33e86831178232ba7ddf8dbb328b87144c6f4812963a78e47c"}
Sep 18 16:58:42.742111 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:42.741854    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"465d3c10-1648-4252-87c0-9b27987d8de4","Type":"ContainerStarted","Data":"29163999ddfa8e96ec418c888ccc79dc9ba5d18fdb2b5395ef3452b897dba494"}
Sep 18 16:58:42.888558 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:58:42.888519    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" podUID="a3127caef50db0be093e96af172c433b"
Sep 18 16:58:43.750540 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:43.750503    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"465d3c10-1648-4252-87c0-9b27987d8de4","Type":"ContainerStarted","Data":"9520d38fa59bb3078139b3dc71354f9e0ea33c0501d060a2be97f1cdc660f408"}
Sep 18 16:58:43.750955 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:43.750580    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"465d3c10-1648-4252-87c0-9b27987d8de4","Type":"ContainerStarted","Data":"f8d57ccce3e35ee634d3d9ac56c409947fba315cb55d3bbf2ecaa7ccb9a29854"}
Sep 18 16:58:43.785064 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:43.785009    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.053046495 podStartE2EDuration="6.784991362s" totalImagesPullingTime="3.701685737s" totalInitContainerRuntime="30.25913ms" isStatefulPod=false podCreationTimestamp="2026-09-18 16:58:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:58:43.78271765 +0000 UTC m=+313.982572939" watchObservedRunningTime="2026-09-18 16:58:43.784991362 +0000 UTC m=+313.984846647"
Sep 18 16:58:46.739519 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:46.739488    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-96b4948d4-zsn4k"
Sep 18 16:58:46.871439 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:46.871398    2881 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-789fdf96f9-9vphn"
Sep 18 16:58:46.871646 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:46.871469    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-789fdf96f9-9vphn"
Sep 18 16:58:46.876829 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:46.876800    2881 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-789fdf96f9-9vphn"
Sep 18 16:58:47.471848 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:47.471810    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:58:47.776653 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:47.776623    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-789fdf96f9-9vphn"
Sep 18 16:58:47.837158 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:47.837118    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5bf8875fb5-z4njj"]
Sep 18 16:58:54.888309 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:58:54.888258    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" podUID="a3127caef50db0be093e96af172c433b"
Sep 18 16:58:55.522325 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:55.522286    2881 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-67b6f56d48-lvpbv"
Sep 18 16:58:55.522521 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:58:55.522356    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-67b6f56d48-lvpbv"
Sep 18 16:59:05.669545 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:59:05.669508    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-pc44x"
Sep 18 16:59:06.888105 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:59:06.888064    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" podUID="a3127caef50db0be093e96af172c433b"
Sep 18 16:59:12.864953 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:59:12.864887    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5bf8875fb5-z4njj" podUID="44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124" containerName="console" containerID="cri-o://e2b8212407ef3b1521331a8fa22162d9bc0f042f936ac1a98b25ff7c3172910c" gracePeriod=15
Sep 18 16:59:13.131787 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:59:13.131693    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bf8875fb5-z4njj"
Sep 18 16:59:13.271395 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:59:13.271342    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124-trusted-ca-bundle\") pod \"44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124\" (UID: \"44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124\") "
Sep 18 16:59:13.271597 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:59:13.271409    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124-oauth-serving-cert\") pod \"44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124\" (UID: \"44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124\") "
Sep 18 16:59:13.271597 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:59:13.271433    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124-console-config\") pod \"44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124\" (UID: \"44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124\") "
Sep 18 16:59:13.271597 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:59:13.271564    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124-console-serving-cert\") pod \"44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124\" (UID: \"44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124\") "
Sep 18 16:59:13.271700 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:59:13.271636    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-87lpz\" (UniqueName: \"kubernetes.io/projected/44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124-kube-api-access-87lpz\") pod \"44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124\" (UID: \"44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124\") "
Sep 18 16:59:13.271700 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:59:13.271668    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124-service-ca\") pod \"44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124\" (UID: \"44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124\") "
Sep 18 16:59:13.271810 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:59:13.271720    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124-console-oauth-config\") pod \"44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124\" (UID: \"44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124\") "
Sep 18 16:59:13.271883 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:59:13.271861    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124-console-config" (OuterVolumeSpecName: "console-config") pod "44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124" (UID: "44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:59:13.271942 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:59:13.271899    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124" (UID: "44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:59:13.271942 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:59:13.271888    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124" (UID: "44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:59:13.272122 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:59:13.272095    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124-service-ca" (OuterVolumeSpecName: "service-ca") pod "44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124" (UID: "44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:59:13.272250 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:59:13.272207    2881 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124-oauth-serving-cert\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 16:59:13.272321 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:59:13.272253    2881 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124-console-config\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 16:59:13.272321 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:59:13.272268    2881 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124-service-ca\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 16:59:13.272321 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:59:13.272280    2881 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124-trusted-ca-bundle\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 16:59:13.274155 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:59:13.274131    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124" (UID: "44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:59:13.274155 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:59:13.274139    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124-kube-api-access-87lpz" (OuterVolumeSpecName: "kube-api-access-87lpz") pod "44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124" (UID: "44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124"). InnerVolumeSpecName "kube-api-access-87lpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:59:13.274329 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:59:13.274216    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124" (UID: "44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:59:13.372944 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:59:13.372910    2881 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124-console-serving-cert\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 16:59:13.372944 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:59:13.372941    2881 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87lpz\" (UniqueName: \"kubernetes.io/projected/44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124-kube-api-access-87lpz\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 16:59:13.373156 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:59:13.372956    2881 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124-console-oauth-config\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 16:59:13.874839 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:59:13.874804    2881 generic.go:356] "Generic (PLEG): container finished" podID="44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124" containerID="e2b8212407ef3b1521331a8fa22162d9bc0f042f936ac1a98b25ff7c3172910c" exitCode=0
Sep 18 16:59:13.875308 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:59:13.874857    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bf8875fb5-z4njj"
Sep 18 16:59:13.875308 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:59:13.874888    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bf8875fb5-z4njj" event={"ID":"44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124","Type":"ContainerDied","Data":"e2b8212407ef3b1521331a8fa22162d9bc0f042f936ac1a98b25ff7c3172910c"}
Sep 18 16:59:13.875308 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:59:13.874969    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bf8875fb5-z4njj" event={"ID":"44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124","Type":"ContainerDied","Data":"14c52aaeae102c3cd976efe66fdda8aab7fdf9f5d437c94c66490b3c73fc0850"}
Sep 18 16:59:13.875308 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:59:13.874990    2881 scope.go:117] "RemoveContainer" containerID="e2b8212407ef3b1521331a8fa22162d9bc0f042f936ac1a98b25ff7c3172910c"
Sep 18 16:59:13.884203 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:59:13.884186    2881 scope.go:117] "RemoveContainer" containerID="e2b8212407ef3b1521331a8fa22162d9bc0f042f936ac1a98b25ff7c3172910c"
Sep 18 16:59:13.884577 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:59:13.884556    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2b8212407ef3b1521331a8fa22162d9bc0f042f936ac1a98b25ff7c3172910c\": container with ID starting with e2b8212407ef3b1521331a8fa22162d9bc0f042f936ac1a98b25ff7c3172910c not found: ID does not exist" containerID="e2b8212407ef3b1521331a8fa22162d9bc0f042f936ac1a98b25ff7c3172910c"
Sep 18 16:59:13.884637 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:59:13.884588    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2b8212407ef3b1521331a8fa22162d9bc0f042f936ac1a98b25ff7c3172910c"} err="failed to get container status \"e2b8212407ef3b1521331a8fa22162d9bc0f042f936ac1a98b25ff7c3172910c\": rpc error: code = NotFound desc = could not find container \"e2b8212407ef3b1521331a8fa22162d9bc0f042f936ac1a98b25ff7c3172910c\": container with ID starting with e2b8212407ef3b1521331a8fa22162d9bc0f042f936ac1a98b25ff7c3172910c not found: ID does not exist"
Sep 18 16:59:13.898998 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:59:13.898965    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5bf8875fb5-z4njj"]
Sep 18 16:59:13.904713 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:59:13.904685    2881 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5bf8875fb5-z4njj"]
Sep 18 16:59:15.528272 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:59:15.528217    2881 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-67b6f56d48-lvpbv"
Sep 18 16:59:15.532050 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:59:15.532026    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-67b6f56d48-lvpbv"
Sep 18 16:59:15.893994 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:59:15.893910    2881 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124" path="/var/lib/kubelet/pods/44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124/volumes"
Sep 18 16:59:18.888116 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:59:18.888076    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" podUID="a3127caef50db0be093e96af172c433b"
Sep 18 16:59:31.888496 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:59:31.888453    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" podUID="a3127caef50db0be093e96af172c433b"
Sep 18 16:59:37.472058 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:59:37.472008    2881 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:59:37.492138 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:59:37.492105    2881 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:59:37.986325 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:59:37.986280    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:59:45.888823 ip-10-0-143-5 kubenswrapper[2881]: E0918 16:59:45.888773    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator(a3127caef50db0be093e96af172c433b)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" podUID="a3127caef50db0be093e96af172c433b"
Sep 18 16:59:58.977248 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:59:58.977199    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-64458494f-tqdzx"]
Sep 18 16:59:58.977772 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:59:58.977749    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124" containerName="console"
Sep 18 16:59:58.977862 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:59:58.977776    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124" containerName="console"
Sep 18 16:59:58.977959 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:59:58.977931    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="44e9b3b1-f2cd-47e8-beb6-3b6a2c3e3124" containerName="console"
Sep 18 16:59:58.980913 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:59:58.980889    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64458494f-tqdzx"
Sep 18 16:59:58.992808 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:59:58.992778    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64458494f-tqdzx"]
Sep 18 16:59:59.050365 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:59:59.050322    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a6dab97-7fcb-4809-b660-dc69f00a0292-trusted-ca-bundle\") pod \"console-64458494f-tqdzx\" (UID: \"8a6dab97-7fcb-4809-b660-dc69f00a0292\") " pod="openshift-console/console-64458494f-tqdzx"
Sep 18 16:59:59.050569 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:59:59.050374    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgdth\" (UniqueName: \"kubernetes.io/projected/8a6dab97-7fcb-4809-b660-dc69f00a0292-kube-api-access-vgdth\") pod \"console-64458494f-tqdzx\" (UID: \"8a6dab97-7fcb-4809-b660-dc69f00a0292\") " pod="openshift-console/console-64458494f-tqdzx"
Sep 18 16:59:59.050569 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:59:59.050424    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8a6dab97-7fcb-4809-b660-dc69f00a0292-console-oauth-config\") pod \"console-64458494f-tqdzx\" (UID: \"8a6dab97-7fcb-4809-b660-dc69f00a0292\") " pod="openshift-console/console-64458494f-tqdzx"
Sep 18 16:59:59.050569 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:59:59.050448    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8a6dab97-7fcb-4809-b660-dc69f00a0292-console-config\") pod \"console-64458494f-tqdzx\" (UID: \"8a6dab97-7fcb-4809-b660-dc69f00a0292\") " pod="openshift-console/console-64458494f-tqdzx"
Sep 18 16:59:59.050569 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:59:59.050529    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8a6dab97-7fcb-4809-b660-dc69f00a0292-console-serving-cert\") pod \"console-64458494f-tqdzx\" (UID: \"8a6dab97-7fcb-4809-b660-dc69f00a0292\") " pod="openshift-console/console-64458494f-tqdzx"
Sep 18 16:59:59.050777 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:59:59.050592    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8a6dab97-7fcb-4809-b660-dc69f00a0292-service-ca\") pod \"console-64458494f-tqdzx\" (UID: \"8a6dab97-7fcb-4809-b660-dc69f00a0292\") " pod="openshift-console/console-64458494f-tqdzx"
Sep 18 16:59:59.050777 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:59:59.050700    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8a6dab97-7fcb-4809-b660-dc69f00a0292-oauth-serving-cert\") pod \"console-64458494f-tqdzx\" (UID: \"8a6dab97-7fcb-4809-b660-dc69f00a0292\") " pod="openshift-console/console-64458494f-tqdzx"
Sep 18 16:59:59.151799 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:59:59.151751    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8a6dab97-7fcb-4809-b660-dc69f00a0292-oauth-serving-cert\") pod \"console-64458494f-tqdzx\" (UID: \"8a6dab97-7fcb-4809-b660-dc69f00a0292\") " pod="openshift-console/console-64458494f-tqdzx"
Sep 18 16:59:59.151984 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:59:59.151815    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a6dab97-7fcb-4809-b660-dc69f00a0292-trusted-ca-bundle\") pod \"console-64458494f-tqdzx\" (UID: \"8a6dab97-7fcb-4809-b660-dc69f00a0292\") " pod="openshift-console/console-64458494f-tqdzx"
Sep 18 16:59:59.151984 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:59:59.151837    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgdth\" (UniqueName: \"kubernetes.io/projected/8a6dab97-7fcb-4809-b660-dc69f00a0292-kube-api-access-vgdth\") pod \"console-64458494f-tqdzx\" (UID: \"8a6dab97-7fcb-4809-b660-dc69f00a0292\") " pod="openshift-console/console-64458494f-tqdzx"
Sep 18 16:59:59.151984 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:59:59.151859    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8a6dab97-7fcb-4809-b660-dc69f00a0292-console-oauth-config\") pod \"console-64458494f-tqdzx\" (UID: \"8a6dab97-7fcb-4809-b660-dc69f00a0292\") " pod="openshift-console/console-64458494f-tqdzx"
Sep 18 16:59:59.151984 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:59:59.151883    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8a6dab97-7fcb-4809-b660-dc69f00a0292-console-config\") pod \"console-64458494f-tqdzx\" (UID: \"8a6dab97-7fcb-4809-b660-dc69f00a0292\") " pod="openshift-console/console-64458494f-tqdzx"
Sep 18 16:59:59.151984 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:59:59.151915    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8a6dab97-7fcb-4809-b660-dc69f00a0292-console-serving-cert\") pod \"console-64458494f-tqdzx\" (UID: \"8a6dab97-7fcb-4809-b660-dc69f00a0292\") " pod="openshift-console/console-64458494f-tqdzx"
Sep 18 16:59:59.151984 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:59:59.151965    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8a6dab97-7fcb-4809-b660-dc69f00a0292-service-ca\") pod \"console-64458494f-tqdzx\" (UID: \"8a6dab97-7fcb-4809-b660-dc69f00a0292\") " pod="openshift-console/console-64458494f-tqdzx"
Sep 18 16:59:59.152595 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:59:59.152565    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8a6dab97-7fcb-4809-b660-dc69f00a0292-oauth-serving-cert\") pod \"console-64458494f-tqdzx\" (UID: \"8a6dab97-7fcb-4809-b660-dc69f00a0292\") " pod="openshift-console/console-64458494f-tqdzx"
Sep 18 16:59:59.152718 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:59:59.152565    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8a6dab97-7fcb-4809-b660-dc69f00a0292-console-config\") pod \"console-64458494f-tqdzx\" (UID: \"8a6dab97-7fcb-4809-b660-dc69f00a0292\") " pod="openshift-console/console-64458494f-tqdzx"
Sep 18 16:59:59.152788 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:59:59.152767    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8a6dab97-7fcb-4809-b660-dc69f00a0292-service-ca\") pod \"console-64458494f-tqdzx\" (UID: \"8a6dab97-7fcb-4809-b660-dc69f00a0292\") " pod="openshift-console/console-64458494f-tqdzx"
Sep 18 16:59:59.152833 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:59:59.152823    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a6dab97-7fcb-4809-b660-dc69f00a0292-trusted-ca-bundle\") pod \"console-64458494f-tqdzx\" (UID: \"8a6dab97-7fcb-4809-b660-dc69f00a0292\") " pod="openshift-console/console-64458494f-tqdzx"
Sep 18 16:59:59.154537 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:59:59.154514    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8a6dab97-7fcb-4809-b660-dc69f00a0292-console-oauth-config\") pod \"console-64458494f-tqdzx\" (UID: \"8a6dab97-7fcb-4809-b660-dc69f00a0292\") " pod="openshift-console/console-64458494f-tqdzx"
Sep 18 16:59:59.154649 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:59:59.154609    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8a6dab97-7fcb-4809-b660-dc69f00a0292-console-serving-cert\") pod \"console-64458494f-tqdzx\" (UID: \"8a6dab97-7fcb-4809-b660-dc69f00a0292\") " pod="openshift-console/console-64458494f-tqdzx"
Sep 18 16:59:59.161669 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:59:59.161636    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgdth\" (UniqueName: \"kubernetes.io/projected/8a6dab97-7fcb-4809-b660-dc69f00a0292-kube-api-access-vgdth\") pod \"console-64458494f-tqdzx\" (UID: \"8a6dab97-7fcb-4809-b660-dc69f00a0292\") " pod="openshift-console/console-64458494f-tqdzx"
Sep 18 16:59:59.290868 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:59:59.290824    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64458494f-tqdzx"
Sep 18 16:59:59.570826 ip-10-0-143-5 kubenswrapper[2881]: I0918 16:59:59.570787    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64458494f-tqdzx"]
Sep 18 16:59:59.574563 ip-10-0-143-5 kubenswrapper[2881]: W0918 16:59:59.574528    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a6dab97_7fcb_4809_b660_dc69f00a0292.slice/crio-4a9c3b0156c252820496094064c8d695e2b41371f6040a312d2f45ddfe2a2330 WatchSource:0}: Error finding container 4a9c3b0156c252820496094064c8d695e2b41371f6040a312d2f45ddfe2a2330: Status 404 returned error can't find the container with id 4a9c3b0156c252820496094064c8d695e2b41371f6040a312d2f45ddfe2a2330
Sep 18 17:00:00.061173 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:00.061140    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_a3127caef50db0be093e96af172c433b/setup/10.log"
Sep 18 17:00:00.061706 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:00.061187    2881 generic.go:356] "Generic (PLEG): container finished" podID="a3127caef50db0be093e96af172c433b" containerID="e93dadec60a3c8cc3745a8295a9e654edf86f642d9b66661fca575ddade4b994" exitCode=0
Sep 18 17:00:00.061706 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:00.061269    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" event={"ID":"a3127caef50db0be093e96af172c433b","Type":"ContainerDied","Data":"e93dadec60a3c8cc3745a8295a9e654edf86f642d9b66661fca575ddade4b994"}
Sep 18 17:00:00.061706 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:00.061334    2881 scope.go:117] "RemoveContainer" containerID="5a1a260f7fcf9e054488a21120f0bc2e2d4a1984e7dfdda5af9badd275584b95"
Sep 18 17:00:00.061706 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:00.061681    2881 scope.go:117] "RemoveContainer" containerID="5a1a260f7fcf9e054488a21120f0bc2e2d4a1984e7dfdda5af9badd275584b95"
Sep 18 17:00:00.063187 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:00.063164    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64458494f-tqdzx" event={"ID":"8a6dab97-7fcb-4809-b660-dc69f00a0292","Type":"ContainerStarted","Data":"4354c9d10653f84ce8245ce69e1b5c5a1d6eb2107d6cbdca0c6cf4f6fbbf2178"}
Sep 18 17:00:00.063305 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:00.063233    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64458494f-tqdzx" event={"ID":"8a6dab97-7fcb-4809-b660-dc69f00a0292","Type":"ContainerStarted","Data":"4a9c3b0156c252820496094064c8d695e2b41371f6040a312d2f45ddfe2a2330"}
Sep 18 17:00:00.072383 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:00:00.072347    2881 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator_a3127caef50db0be093e96af172c433b_10 in pod sandbox cf961e79c5b92a2e43a70c6a29822bf8a5a2069147989bc7b9bb0b53e33609ad from index: no such id: '5a1a260f7fcf9e054488a21120f0bc2e2d4a1984e7dfdda5af9badd275584b95'" containerID="5a1a260f7fcf9e054488a21120f0bc2e2d4a1984e7dfdda5af9badd275584b95"
Sep 18 17:00:00.072517 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:00.072392    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a1a260f7fcf9e054488a21120f0bc2e2d4a1984e7dfdda5af9badd275584b95"} err="rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal_openshift-machine-config-operator_a3127caef50db0be093e96af172c433b_10 in pod sandbox cf961e79c5b92a2e43a70c6a29822bf8a5a2069147989bc7b9bb0b53e33609ad from index: no such id: '5a1a260f7fcf9e054488a21120f0bc2e2d4a1984e7dfdda5af9badd275584b95'"
Sep 18 17:00:00.099839 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:00.099773    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-64458494f-tqdzx" podStartSLOduration=2.09974053 podStartE2EDuration="2.09974053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:59:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:00:00.098502875 +0000 UTC m=+390.298358162" watchObservedRunningTime="2026-09-18 17:00:00.09974053 +0000 UTC m=+390.299595817"
Sep 18 17:00:01.070165 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:01.070121    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" event={"ID":"a3127caef50db0be093e96af172c433b","Type":"ContainerStarted","Data":"26c3a3b3ed829fe87b99ad22f6dcc2c20342abb9527b7e228a1c62777139eb68"}
Sep 18 17:00:01.087860 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:01.087802    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-5.ec2.internal" podStartSLOduration=157.087784253 podStartE2EDuration="2m37.087784253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:57:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:00:01.087190854 +0000 UTC m=+391.287046140" watchObservedRunningTime="2026-09-18 17:00:01.087784253 +0000 UTC m=+391.287639535"
Sep 18 17:00:07.821962 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:07.821923    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64458494f-tqdzx"]
Sep 18 17:00:09.291444 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:09.291399    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-64458494f-tqdzx"
Sep 18 17:00:18.328675 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:18.328638    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-789fdf96f9-9vphn"]
Sep 18 17:00:32.841184 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:32.841113    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-64458494f-tqdzx" podUID="8a6dab97-7fcb-4809-b660-dc69f00a0292" containerName="console" containerID="cri-o://4354c9d10653f84ce8245ce69e1b5c5a1d6eb2107d6cbdca0c6cf4f6fbbf2178" gracePeriod=15
Sep 18 17:00:33.083624 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:33.083593    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64458494f-tqdzx"
Sep 18 17:00:33.165025 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:33.164928    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vgdth\" (UniqueName: \"kubernetes.io/projected/8a6dab97-7fcb-4809-b660-dc69f00a0292-kube-api-access-vgdth\") pod \"8a6dab97-7fcb-4809-b660-dc69f00a0292\" (UID: \"8a6dab97-7fcb-4809-b660-dc69f00a0292\") "
Sep 18 17:00:33.165025 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:33.165013    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8a6dab97-7fcb-4809-b660-dc69f00a0292-console-config\") pod \"8a6dab97-7fcb-4809-b660-dc69f00a0292\" (UID: \"8a6dab97-7fcb-4809-b660-dc69f00a0292\") "
Sep 18 17:00:33.165311 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:33.165034    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8a6dab97-7fcb-4809-b660-dc69f00a0292-console-serving-cert\") pod \"8a6dab97-7fcb-4809-b660-dc69f00a0292\" (UID: \"8a6dab97-7fcb-4809-b660-dc69f00a0292\") "
Sep 18 17:00:33.165311 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:33.165060    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8a6dab97-7fcb-4809-b660-dc69f00a0292-oauth-serving-cert\") pod \"8a6dab97-7fcb-4809-b660-dc69f00a0292\" (UID: \"8a6dab97-7fcb-4809-b660-dc69f00a0292\") "
Sep 18 17:00:33.165311 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:33.165102    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a6dab97-7fcb-4809-b660-dc69f00a0292-trusted-ca-bundle\") pod \"8a6dab97-7fcb-4809-b660-dc69f00a0292\" (UID: \"8a6dab97-7fcb-4809-b660-dc69f00a0292\") "
Sep 18 17:00:33.165311 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:33.165144    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8a6dab97-7fcb-4809-b660-dc69f00a0292-console-oauth-config\") pod \"8a6dab97-7fcb-4809-b660-dc69f00a0292\" (UID: \"8a6dab97-7fcb-4809-b660-dc69f00a0292\") "
Sep 18 17:00:33.165311 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:33.165171    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8a6dab97-7fcb-4809-b660-dc69f00a0292-service-ca\") pod \"8a6dab97-7fcb-4809-b660-dc69f00a0292\" (UID: \"8a6dab97-7fcb-4809-b660-dc69f00a0292\") "
Sep 18 17:00:33.165596 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:33.165492    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a6dab97-7fcb-4809-b660-dc69f00a0292-console-config" (OuterVolumeSpecName: "console-config") pod "8a6dab97-7fcb-4809-b660-dc69f00a0292" (UID: "8a6dab97-7fcb-4809-b660-dc69f00a0292"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:00:33.165596 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:33.165514    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a6dab97-7fcb-4809-b660-dc69f00a0292-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8a6dab97-7fcb-4809-b660-dc69f00a0292" (UID: "8a6dab97-7fcb-4809-b660-dc69f00a0292"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:00:33.165596 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:33.165557    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a6dab97-7fcb-4809-b660-dc69f00a0292-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "8a6dab97-7fcb-4809-b660-dc69f00a0292" (UID: "8a6dab97-7fcb-4809-b660-dc69f00a0292"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:00:33.165866 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:33.165835    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a6dab97-7fcb-4809-b660-dc69f00a0292-service-ca" (OuterVolumeSpecName: "service-ca") pod "8a6dab97-7fcb-4809-b660-dc69f00a0292" (UID: "8a6dab97-7fcb-4809-b660-dc69f00a0292"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:00:33.167238 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:33.167201    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a6dab97-7fcb-4809-b660-dc69f00a0292-kube-api-access-vgdth" (OuterVolumeSpecName: "kube-api-access-vgdth") pod "8a6dab97-7fcb-4809-b660-dc69f00a0292" (UID: "8a6dab97-7fcb-4809-b660-dc69f00a0292"). InnerVolumeSpecName "kube-api-access-vgdth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:00:33.167647 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:33.167628    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a6dab97-7fcb-4809-b660-dc69f00a0292-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8a6dab97-7fcb-4809-b660-dc69f00a0292" (UID: "8a6dab97-7fcb-4809-b660-dc69f00a0292"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:00:33.167710 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:33.167648    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a6dab97-7fcb-4809-b660-dc69f00a0292-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8a6dab97-7fcb-4809-b660-dc69f00a0292" (UID: "8a6dab97-7fcb-4809-b660-dc69f00a0292"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:00:33.206434 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:33.206400    2881 generic.go:356] "Generic (PLEG): container finished" podID="8a6dab97-7fcb-4809-b660-dc69f00a0292" containerID="4354c9d10653f84ce8245ce69e1b5c5a1d6eb2107d6cbdca0c6cf4f6fbbf2178" exitCode=0
Sep 18 17:00:33.206645 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:33.206443    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64458494f-tqdzx"
Sep 18 17:00:33.206645 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:33.206495    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64458494f-tqdzx" event={"ID":"8a6dab97-7fcb-4809-b660-dc69f00a0292","Type":"ContainerDied","Data":"4354c9d10653f84ce8245ce69e1b5c5a1d6eb2107d6cbdca0c6cf4f6fbbf2178"}
Sep 18 17:00:33.206645 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:33.206551    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64458494f-tqdzx" event={"ID":"8a6dab97-7fcb-4809-b660-dc69f00a0292","Type":"ContainerDied","Data":"4a9c3b0156c252820496094064c8d695e2b41371f6040a312d2f45ddfe2a2330"}
Sep 18 17:00:33.206645 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:33.206577    2881 scope.go:117] "RemoveContainer" containerID="4354c9d10653f84ce8245ce69e1b5c5a1d6eb2107d6cbdca0c6cf4f6fbbf2178"
Sep 18 17:00:33.216186 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:33.216172    2881 scope.go:117] "RemoveContainer" containerID="4354c9d10653f84ce8245ce69e1b5c5a1d6eb2107d6cbdca0c6cf4f6fbbf2178"
Sep 18 17:00:33.216531 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:00:33.216514    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4354c9d10653f84ce8245ce69e1b5c5a1d6eb2107d6cbdca0c6cf4f6fbbf2178\": container with ID starting with 4354c9d10653f84ce8245ce69e1b5c5a1d6eb2107d6cbdca0c6cf4f6fbbf2178 not found: ID does not exist" containerID="4354c9d10653f84ce8245ce69e1b5c5a1d6eb2107d6cbdca0c6cf4f6fbbf2178"
Sep 18 17:00:33.216593 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:33.216540    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4354c9d10653f84ce8245ce69e1b5c5a1d6eb2107d6cbdca0c6cf4f6fbbf2178"} err="failed to get container status \"4354c9d10653f84ce8245ce69e1b5c5a1d6eb2107d6cbdca0c6cf4f6fbbf2178\": rpc error: code = NotFound desc = could not find container \"4354c9d10653f84ce8245ce69e1b5c5a1d6eb2107d6cbdca0c6cf4f6fbbf2178\": container with ID starting with 4354c9d10653f84ce8245ce69e1b5c5a1d6eb2107d6cbdca0c6cf4f6fbbf2178 not found: ID does not exist"
Sep 18 17:00:33.231169 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:33.231141    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64458494f-tqdzx"]
Sep 18 17:00:33.237073 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:33.237050    2881 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-64458494f-tqdzx"]
Sep 18 17:00:33.266031 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:33.266000    2881 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8a6dab97-7fcb-4809-b660-dc69f00a0292-console-oauth-config\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:00:33.266031 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:33.266028    2881 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8a6dab97-7fcb-4809-b660-dc69f00a0292-service-ca\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:00:33.266168 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:33.266041    2881 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgdth\" (UniqueName: \"kubernetes.io/projected/8a6dab97-7fcb-4809-b660-dc69f00a0292-kube-api-access-vgdth\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:00:33.266168 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:33.266049    2881 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8a6dab97-7fcb-4809-b660-dc69f00a0292-console-config\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:00:33.266168 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:33.266057    2881 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8a6dab97-7fcb-4809-b660-dc69f00a0292-console-serving-cert\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:00:33.266168 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:33.266065    2881 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8a6dab97-7fcb-4809-b660-dc69f00a0292-oauth-serving-cert\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:00:33.266168 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:33.266073    2881 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a6dab97-7fcb-4809-b660-dc69f00a0292-trusted-ca-bundle\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:00:33.892538 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:33.892504    2881 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a6dab97-7fcb-4809-b660-dc69f00a0292" path="/var/lib/kubelet/pods/8a6dab97-7fcb-4809-b660-dc69f00a0292/volumes"
Sep 18 17:00:43.348364 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:43.348295    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-789fdf96f9-9vphn" podUID="ade8157b-6ddf-4e19-9129-11e6177db425" containerName="console" containerID="cri-o://c8bdfe8065a5b19df10ed0c12a6bf45d267b00d3ff6a6b8526b82f4e343dcee0" gracePeriod=15
Sep 18 17:00:43.596996 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:43.596970    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-789fdf96f9-9vphn"
Sep 18 17:00:43.667173 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:43.667074    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ade8157b-6ddf-4e19-9129-11e6177db425-trusted-ca-bundle\") pod \"ade8157b-6ddf-4e19-9129-11e6177db425\" (UID: \"ade8157b-6ddf-4e19-9129-11e6177db425\") "
Sep 18 17:00:43.667173 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:43.667156    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ade8157b-6ddf-4e19-9129-11e6177db425-oauth-serving-cert\") pod \"ade8157b-6ddf-4e19-9129-11e6177db425\" (UID: \"ade8157b-6ddf-4e19-9129-11e6177db425\") "
Sep 18 17:00:43.667478 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:43.667181    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ade8157b-6ddf-4e19-9129-11e6177db425-console-oauth-config\") pod \"ade8157b-6ddf-4e19-9129-11e6177db425\" (UID: \"ade8157b-6ddf-4e19-9129-11e6177db425\") "
Sep 18 17:00:43.667478 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:43.667287    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-45wr9\" (UniqueName: \"kubernetes.io/projected/ade8157b-6ddf-4e19-9129-11e6177db425-kube-api-access-45wr9\") pod \"ade8157b-6ddf-4e19-9129-11e6177db425\" (UID: \"ade8157b-6ddf-4e19-9129-11e6177db425\") "
Sep 18 17:00:43.667478 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:43.667342    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ade8157b-6ddf-4e19-9129-11e6177db425-service-ca\") pod \"ade8157b-6ddf-4e19-9129-11e6177db425\" (UID: \"ade8157b-6ddf-4e19-9129-11e6177db425\") "
Sep 18 17:00:43.667478 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:43.667378    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ade8157b-6ddf-4e19-9129-11e6177db425-console-serving-cert\") pod \"ade8157b-6ddf-4e19-9129-11e6177db425\" (UID: \"ade8157b-6ddf-4e19-9129-11e6177db425\") "
Sep 18 17:00:43.667478 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:43.667432    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ade8157b-6ddf-4e19-9129-11e6177db425-console-config\") pod \"ade8157b-6ddf-4e19-9129-11e6177db425\" (UID: \"ade8157b-6ddf-4e19-9129-11e6177db425\") "
Sep 18 17:00:43.667752 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:43.667641    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ade8157b-6ddf-4e19-9129-11e6177db425-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ade8157b-6ddf-4e19-9129-11e6177db425" (UID: "ade8157b-6ddf-4e19-9129-11e6177db425"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:00:43.667752 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:43.667673    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ade8157b-6ddf-4e19-9129-11e6177db425-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ade8157b-6ddf-4e19-9129-11e6177db425" (UID: "ade8157b-6ddf-4e19-9129-11e6177db425"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:00:43.667854 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:43.667792    2881 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ade8157b-6ddf-4e19-9129-11e6177db425-oauth-serving-cert\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:00:43.667854 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:43.667812    2881 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ade8157b-6ddf-4e19-9129-11e6177db425-trusted-ca-bundle\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:00:43.667950 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:43.667921    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ade8157b-6ddf-4e19-9129-11e6177db425-service-ca" (OuterVolumeSpecName: "service-ca") pod "ade8157b-6ddf-4e19-9129-11e6177db425" (UID: "ade8157b-6ddf-4e19-9129-11e6177db425"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:00:43.668000 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:43.667982    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ade8157b-6ddf-4e19-9129-11e6177db425-console-config" (OuterVolumeSpecName: "console-config") pod "ade8157b-6ddf-4e19-9129-11e6177db425" (UID: "ade8157b-6ddf-4e19-9129-11e6177db425"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:00:43.669751 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:43.669724    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ade8157b-6ddf-4e19-9129-11e6177db425-kube-api-access-45wr9" (OuterVolumeSpecName: "kube-api-access-45wr9") pod "ade8157b-6ddf-4e19-9129-11e6177db425" (UID: "ade8157b-6ddf-4e19-9129-11e6177db425"). InnerVolumeSpecName "kube-api-access-45wr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:00:43.670140 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:43.670120    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ade8157b-6ddf-4e19-9129-11e6177db425-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ade8157b-6ddf-4e19-9129-11e6177db425" (UID: "ade8157b-6ddf-4e19-9129-11e6177db425"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:00:43.670204 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:43.670143    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ade8157b-6ddf-4e19-9129-11e6177db425-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ade8157b-6ddf-4e19-9129-11e6177db425" (UID: "ade8157b-6ddf-4e19-9129-11e6177db425"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:00:43.768188 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:43.768154    2881 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ade8157b-6ddf-4e19-9129-11e6177db425-console-oauth-config\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:00:43.768188 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:43.768183    2881 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45wr9\" (UniqueName: \"kubernetes.io/projected/ade8157b-6ddf-4e19-9129-11e6177db425-kube-api-access-45wr9\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:00:43.768188 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:43.768198    2881 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ade8157b-6ddf-4e19-9129-11e6177db425-service-ca\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:00:43.768476 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:43.768211    2881 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ade8157b-6ddf-4e19-9129-11e6177db425-console-serving-cert\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:00:43.768476 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:43.768276    2881 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ade8157b-6ddf-4e19-9129-11e6177db425-console-config\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:00:44.254560 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:44.254522    2881 generic.go:356] "Generic (PLEG): container finished" podID="ade8157b-6ddf-4e19-9129-11e6177db425" containerID="c8bdfe8065a5b19df10ed0c12a6bf45d267b00d3ff6a6b8526b82f4e343dcee0" exitCode=0
Sep 18 17:00:44.254560 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:44.254565    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-789fdf96f9-9vphn"
Sep 18 17:00:44.254821 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:44.254609    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-789fdf96f9-9vphn" event={"ID":"ade8157b-6ddf-4e19-9129-11e6177db425","Type":"ContainerDied","Data":"c8bdfe8065a5b19df10ed0c12a6bf45d267b00d3ff6a6b8526b82f4e343dcee0"}
Sep 18 17:00:44.254821 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:44.254661    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-789fdf96f9-9vphn" event={"ID":"ade8157b-6ddf-4e19-9129-11e6177db425","Type":"ContainerDied","Data":"e8b7054e9db9a68636f76e8dcabd2ab420c46424bb01c8c0ae9a081bdd4e41e4"}
Sep 18 17:00:44.254821 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:44.254682    2881 scope.go:117] "RemoveContainer" containerID="c8bdfe8065a5b19df10ed0c12a6bf45d267b00d3ff6a6b8526b82f4e343dcee0"
Sep 18 17:00:44.262945 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:44.262928    2881 scope.go:117] "RemoveContainer" containerID="c8bdfe8065a5b19df10ed0c12a6bf45d267b00d3ff6a6b8526b82f4e343dcee0"
Sep 18 17:00:44.263239 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:00:44.263207    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8bdfe8065a5b19df10ed0c12a6bf45d267b00d3ff6a6b8526b82f4e343dcee0\": container with ID starting with c8bdfe8065a5b19df10ed0c12a6bf45d267b00d3ff6a6b8526b82f4e343dcee0 not found: ID does not exist" containerID="c8bdfe8065a5b19df10ed0c12a6bf45d267b00d3ff6a6b8526b82f4e343dcee0"
Sep 18 17:00:44.263287 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:44.263246    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8bdfe8065a5b19df10ed0c12a6bf45d267b00d3ff6a6b8526b82f4e343dcee0"} err="failed to get container status \"c8bdfe8065a5b19df10ed0c12a6bf45d267b00d3ff6a6b8526b82f4e343dcee0\": rpc error: code = NotFound desc = could not find container \"c8bdfe8065a5b19df10ed0c12a6bf45d267b00d3ff6a6b8526b82f4e343dcee0\": container with ID starting with c8bdfe8065a5b19df10ed0c12a6bf45d267b00d3ff6a6b8526b82f4e343dcee0 not found: ID does not exist"
Sep 18 17:00:44.275126 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:44.275102    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-789fdf96f9-9vphn"]
Sep 18 17:00:44.281180 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:44.281157    2881 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-789fdf96f9-9vphn"]
Sep 18 17:00:45.893535 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:00:45.893495    2881 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ade8157b-6ddf-4e19-9129-11e6177db425" path="/var/lib/kubelet/pods/ade8157b-6ddf-4e19-9129-11e6177db425/volumes"
Sep 18 17:02:04.057312 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:02:04.057275    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-vqnhc"]
Sep 18 17:02:04.057810 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:02:04.057605    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ade8157b-6ddf-4e19-9129-11e6177db425" containerName="console"
Sep 18 17:02:04.057810 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:02:04.057618    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="ade8157b-6ddf-4e19-9129-11e6177db425" containerName="console"
Sep 18 17:02:04.057810 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:02:04.057641    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a6dab97-7fcb-4809-b660-dc69f00a0292" containerName="console"
Sep 18 17:02:04.057810 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:02:04.057646    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a6dab97-7fcb-4809-b660-dc69f00a0292" containerName="console"
Sep 18 17:02:04.057810 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:02:04.057702    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="ade8157b-6ddf-4e19-9129-11e6177db425" containerName="console"
Sep 18 17:02:04.057810 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:02:04.057712    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a6dab97-7fcb-4809-b660-dc69f00a0292" containerName="console"
Sep 18 17:02:04.059778 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:02:04.059762    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-vqnhc"
Sep 18 17:02:04.064185 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:02:04.064133    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 18 17:02:04.065242 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:02:04.065209    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-9bd46\""
Sep 18 17:02:04.065392 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:02:04.065209    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 18 17:02:04.065599 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:02:04.065581    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 18 17:02:04.065679 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:02:04.065616    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 18 17:02:04.072167 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:02:04.072142    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-vqnhc"]
Sep 18 17:02:04.179539 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:02:04.179499    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmxvs\" (UniqueName: \"kubernetes.io/projected/1437f020-a1a6-4647-a1ca-1a3e0aec5364-kube-api-access-bmxvs\") pod \"keda-admission-6c4d864b86-vqnhc\" (UID: \"1437f020-a1a6-4647-a1ca-1a3e0aec5364\") " pod="openshift-keda/keda-admission-6c4d864b86-vqnhc"
Sep 18 17:02:04.179746 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:02:04.179556    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1437f020-a1a6-4647-a1ca-1a3e0aec5364-certificates\") pod \"keda-admission-6c4d864b86-vqnhc\" (UID: \"1437f020-a1a6-4647-a1ca-1a3e0aec5364\") " pod="openshift-keda/keda-admission-6c4d864b86-vqnhc"
Sep 18 17:02:04.280511 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:02:04.280472    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmxvs\" (UniqueName: \"kubernetes.io/projected/1437f020-a1a6-4647-a1ca-1a3e0aec5364-kube-api-access-bmxvs\") pod \"keda-admission-6c4d864b86-vqnhc\" (UID: \"1437f020-a1a6-4647-a1ca-1a3e0aec5364\") " pod="openshift-keda/keda-admission-6c4d864b86-vqnhc"
Sep 18 17:02:04.280716 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:02:04.280547    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1437f020-a1a6-4647-a1ca-1a3e0aec5364-certificates\") pod \"keda-admission-6c4d864b86-vqnhc\" (UID: \"1437f020-a1a6-4647-a1ca-1a3e0aec5364\") " pod="openshift-keda/keda-admission-6c4d864b86-vqnhc"
Sep 18 17:02:04.283428 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:02:04.283398    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1437f020-a1a6-4647-a1ca-1a3e0aec5364-certificates\") pod \"keda-admission-6c4d864b86-vqnhc\" (UID: \"1437f020-a1a6-4647-a1ca-1a3e0aec5364\") " pod="openshift-keda/keda-admission-6c4d864b86-vqnhc"
Sep 18 17:02:04.291732 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:02:04.291699    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmxvs\" (UniqueName: \"kubernetes.io/projected/1437f020-a1a6-4647-a1ca-1a3e0aec5364-kube-api-access-bmxvs\") pod \"keda-admission-6c4d864b86-vqnhc\" (UID: \"1437f020-a1a6-4647-a1ca-1a3e0aec5364\") " pod="openshift-keda/keda-admission-6c4d864b86-vqnhc"
Sep 18 17:02:04.370735 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:02:04.370621    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-vqnhc"
Sep 18 17:02:04.678091 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:02:04.677793    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-vqnhc"]
Sep 18 17:02:04.679993 ip-10-0-143-5 kubenswrapper[2881]: W0918 17:02:04.679965    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1437f020_a1a6_4647_a1ca_1a3e0aec5364.slice/crio-2a3381b7df0ec0ceb82b9ea97a0e801896b81238b7362e95eb37d9b16c724d98 WatchSource:0}: Error finding container 2a3381b7df0ec0ceb82b9ea97a0e801896b81238b7362e95eb37d9b16c724d98: Status 404 returned error can't find the container with id 2a3381b7df0ec0ceb82b9ea97a0e801896b81238b7362e95eb37d9b16c724d98
Sep 18 17:02:05.594167 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:02:05.594127    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-vqnhc" event={"ID":"1437f020-a1a6-4647-a1ca-1a3e0aec5364","Type":"ContainerStarted","Data":"2a3381b7df0ec0ceb82b9ea97a0e801896b81238b7362e95eb37d9b16c724d98"}
Sep 18 17:02:07.605073 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:02:07.605029    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-vqnhc" event={"ID":"1437f020-a1a6-4647-a1ca-1a3e0aec5364","Type":"ContainerStarted","Data":"b65a77b4519ccbed0c87c25ad5a0f6b42ee392b287a528e63ed1b942888e4b69"}
Sep 18 17:02:07.605492 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:02:07.605119    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-vqnhc"
Sep 18 17:02:07.628820 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:02:07.628763    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-vqnhc" podStartSLOduration=0.909290991 podStartE2EDuration="3.628746309s" totalImagesPullingTime="2.719455318s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:02:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:02:07.626844014 +0000 UTC m=+517.826699300" watchObservedRunningTime="2026-09-18 17:02:07.628746309 +0000 UTC m=+517.828601845"
Sep 18 17:02:28.612390 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:02:28.612357    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-vqnhc"
Sep 18 17:03:17.232164 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:03:17.232076    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7dd6748f4f-46p69"]
Sep 18 17:03:17.235777 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:03:17.235748    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dd6748f4f-46p69"
Sep 18 17:03:17.238452 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:03:17.238427    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 18 17:03:17.239611 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:03:17.239586    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 18 17:03:17.239889 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:03:17.239678    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 18 17:03:17.239991 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:03:17.239693    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-9s7t7\""
Sep 18 17:03:17.239991 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:03:17.239719    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 18 17:03:17.240104 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:03:17.239723    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 18 17:03:17.240104 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:03:17.239741    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 18 17:03:17.240888 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:03:17.240871    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 18 17:03:17.244319 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:03:17.244301    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 18 17:03:17.255132 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:03:17.255107    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7dd6748f4f-46p69"]
Sep 18 17:03:17.434734 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:03:17.434693    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/87e97870-514e-482f-b4bc-c62cfe89dc39-console-oauth-config\") pod \"console-7dd6748f4f-46p69\" (UID: \"87e97870-514e-482f-b4bc-c62cfe89dc39\") " pod="openshift-console/console-7dd6748f4f-46p69"
Sep 18 17:03:17.434734 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:03:17.434743    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5v5w\" (UniqueName: \"kubernetes.io/projected/87e97870-514e-482f-b4bc-c62cfe89dc39-kube-api-access-c5v5w\") pod \"console-7dd6748f4f-46p69\" (UID: \"87e97870-514e-482f-b4bc-c62cfe89dc39\") " pod="openshift-console/console-7dd6748f4f-46p69"
Sep 18 17:03:17.434981 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:03:17.434816    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87e97870-514e-482f-b4bc-c62cfe89dc39-trusted-ca-bundle\") pod \"console-7dd6748f4f-46p69\" (UID: \"87e97870-514e-482f-b4bc-c62cfe89dc39\") " pod="openshift-console/console-7dd6748f4f-46p69"
Sep 18 17:03:17.434981 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:03:17.434846    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/87e97870-514e-482f-b4bc-c62cfe89dc39-service-ca\") pod \"console-7dd6748f4f-46p69\" (UID: \"87e97870-514e-482f-b4bc-c62cfe89dc39\") " pod="openshift-console/console-7dd6748f4f-46p69"
Sep 18 17:03:17.434981 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:03:17.434871    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/87e97870-514e-482f-b4bc-c62cfe89dc39-console-config\") pod \"console-7dd6748f4f-46p69\" (UID: \"87e97870-514e-482f-b4bc-c62cfe89dc39\") " pod="openshift-console/console-7dd6748f4f-46p69"
Sep 18 17:03:17.435084 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:03:17.434975    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/87e97870-514e-482f-b4bc-c62cfe89dc39-console-serving-cert\") pod \"console-7dd6748f4f-46p69\" (UID: \"87e97870-514e-482f-b4bc-c62cfe89dc39\") " pod="openshift-console/console-7dd6748f4f-46p69"
Sep 18 17:03:17.435084 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:03:17.435018    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/87e97870-514e-482f-b4bc-c62cfe89dc39-oauth-serving-cert\") pod \"console-7dd6748f4f-46p69\" (UID: \"87e97870-514e-482f-b4bc-c62cfe89dc39\") " pod="openshift-console/console-7dd6748f4f-46p69"
Sep 18 17:03:17.535917 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:03:17.535872    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/87e97870-514e-482f-b4bc-c62cfe89dc39-console-oauth-config\") pod \"console-7dd6748f4f-46p69\" (UID: \"87e97870-514e-482f-b4bc-c62cfe89dc39\") " pod="openshift-console/console-7dd6748f4f-46p69"
Sep 18 17:03:17.536097 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:03:17.535927    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5v5w\" (UniqueName: \"kubernetes.io/projected/87e97870-514e-482f-b4bc-c62cfe89dc39-kube-api-access-c5v5w\") pod \"console-7dd6748f4f-46p69\" (UID: \"87e97870-514e-482f-b4bc-c62cfe89dc39\") " pod="openshift-console/console-7dd6748f4f-46p69"
Sep 18 17:03:17.536097 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:03:17.535958    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87e97870-514e-482f-b4bc-c62cfe89dc39-trusted-ca-bundle\") pod \"console-7dd6748f4f-46p69\" (UID: \"87e97870-514e-482f-b4bc-c62cfe89dc39\") " pod="openshift-console/console-7dd6748f4f-46p69"
Sep 18 17:03:17.536097 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:03:17.535978    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/87e97870-514e-482f-b4bc-c62cfe89dc39-service-ca\") pod \"console-7dd6748f4f-46p69\" (UID: \"87e97870-514e-482f-b4bc-c62cfe89dc39\") " pod="openshift-console/console-7dd6748f4f-46p69"
Sep 18 17:03:17.536097 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:03:17.536002    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/87e97870-514e-482f-b4bc-c62cfe89dc39-console-config\") pod \"console-7dd6748f4f-46p69\" (UID: \"87e97870-514e-482f-b4bc-c62cfe89dc39\") " pod="openshift-console/console-7dd6748f4f-46p69"
Sep 18 17:03:17.536097 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:03:17.536042    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/87e97870-514e-482f-b4bc-c62cfe89dc39-console-serving-cert\") pod \"console-7dd6748f4f-46p69\" (UID: \"87e97870-514e-482f-b4bc-c62cfe89dc39\") " pod="openshift-console/console-7dd6748f4f-46p69"
Sep 18 17:03:17.536097 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:03:17.536075    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/87e97870-514e-482f-b4bc-c62cfe89dc39-oauth-serving-cert\") pod \"console-7dd6748f4f-46p69\" (UID: \"87e97870-514e-482f-b4bc-c62cfe89dc39\") " pod="openshift-console/console-7dd6748f4f-46p69"
Sep 18 17:03:17.536832 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:03:17.536805    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/87e97870-514e-482f-b4bc-c62cfe89dc39-oauth-serving-cert\") pod \"console-7dd6748f4f-46p69\" (UID: \"87e97870-514e-482f-b4bc-c62cfe89dc39\") " pod="openshift-console/console-7dd6748f4f-46p69"
Sep 18 17:03:17.536961 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:03:17.536895    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/87e97870-514e-482f-b4bc-c62cfe89dc39-service-ca\") pod \"console-7dd6748f4f-46p69\" (UID: \"87e97870-514e-482f-b4bc-c62cfe89dc39\") " pod="openshift-console/console-7dd6748f4f-46p69"
Sep 18 17:03:17.536961 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:03:17.536903    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/87e97870-514e-482f-b4bc-c62cfe89dc39-console-config\") pod \"console-7dd6748f4f-46p69\" (UID: \"87e97870-514e-482f-b4bc-c62cfe89dc39\") " pod="openshift-console/console-7dd6748f4f-46p69"
Sep 18 17:03:17.537049 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:03:17.536965    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87e97870-514e-482f-b4bc-c62cfe89dc39-trusted-ca-bundle\") pod \"console-7dd6748f4f-46p69\" (UID: \"87e97870-514e-482f-b4bc-c62cfe89dc39\") " pod="openshift-console/console-7dd6748f4f-46p69"
Sep 18 17:03:17.539005 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:03:17.538979    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/87e97870-514e-482f-b4bc-c62cfe89dc39-console-serving-cert\") pod \"console-7dd6748f4f-46p69\" (UID: \"87e97870-514e-482f-b4bc-c62cfe89dc39\") " pod="openshift-console/console-7dd6748f4f-46p69"
Sep 18 17:03:17.539104 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:03:17.539084    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/87e97870-514e-482f-b4bc-c62cfe89dc39-console-oauth-config\") pod \"console-7dd6748f4f-46p69\" (UID: \"87e97870-514e-482f-b4bc-c62cfe89dc39\") " pod="openshift-console/console-7dd6748f4f-46p69"
Sep 18 17:03:17.546590 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:03:17.546566    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5v5w\" (UniqueName: \"kubernetes.io/projected/87e97870-514e-482f-b4bc-c62cfe89dc39-kube-api-access-c5v5w\") pod \"console-7dd6748f4f-46p69\" (UID: \"87e97870-514e-482f-b4bc-c62cfe89dc39\") " pod="openshift-console/console-7dd6748f4f-46p69"
Sep 18 17:03:17.565785 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:03:17.565745    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dd6748f4f-46p69"
Sep 18 17:03:17.843849 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:03:17.843823    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7dd6748f4f-46p69"]
Sep 18 17:03:17.846395 ip-10-0-143-5 kubenswrapper[2881]: W0918 17:03:17.846366    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87e97870_514e_482f_b4bc_c62cfe89dc39.slice/crio-6ad80f21e0ad9233034dd34e895cf4163df5c2f0547edd36cb42b09811b31966 WatchSource:0}: Error finding container 6ad80f21e0ad9233034dd34e895cf4163df5c2f0547edd36cb42b09811b31966: Status 404 returned error can't find the container with id 6ad80f21e0ad9233034dd34e895cf4163df5c2f0547edd36cb42b09811b31966
Sep 18 17:03:17.898974 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:03:17.898940    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dd6748f4f-46p69" event={"ID":"87e97870-514e-482f-b4bc-c62cfe89dc39","Type":"ContainerStarted","Data":"6ad80f21e0ad9233034dd34e895cf4163df5c2f0547edd36cb42b09811b31966"}
Sep 18 17:03:18.904879 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:03:18.904825    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dd6748f4f-46p69" event={"ID":"87e97870-514e-482f-b4bc-c62cfe89dc39","Type":"ContainerStarted","Data":"83ed1606586ad0eca600864883773f9525daf76f6aa13b49edec3018cbdc47e5"}
Sep 18 17:03:18.927462 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:03:18.927407    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7dd6748f4f-46p69" podStartSLOduration=1.927388541 podStartE2EDuration="1.927388541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:03:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:03:18.925272734 +0000 UTC m=+589.125128018" watchObservedRunningTime="2026-09-18 17:03:18.927388541 +0000 UTC m=+589.127243826"
Sep 18 17:03:27.566923 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:03:27.566874    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7dd6748f4f-46p69"
Sep 18 17:03:27.567480 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:03:27.566966    2881 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7dd6748f4f-46p69"
Sep 18 17:03:27.571691 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:03:27.571665    2881 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7dd6748f4f-46p69"
Sep 18 17:03:27.946647 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:03:27.946559    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7dd6748f4f-46p69"
Sep 18 17:05:00.987413 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:05:00.987372    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-x2g9p"]
Sep 18 17:05:00.990839 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:05:00.990821    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-x2g9p"
Sep 18 17:05:00.993710 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:05:00.993671    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 18 17:05:00.993710 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:05:00.993686    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-q8nb7\""
Sep 18 17:05:00.993710 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:05:00.993690    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 18 17:05:00.993956 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:05:00.993721    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 18 17:05:00.993956 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:05:00.993739    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 18 17:05:00.999205 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:05:00.999175    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-x2g9p"]
Sep 18 17:05:01.053760 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:05:01.053723    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21ce4cae-ef77-4050-8d30-e0522e6c7116-model-cache\") pod \"s3-init-x2g9p\" (UID: \"21ce4cae-ef77-4050-8d30-e0522e6c7116\") " pod="kserve/s3-init-x2g9p"
Sep 18 17:05:01.053954 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:05:01.053853    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-426dx\" (UniqueName: \"kubernetes.io/projected/21ce4cae-ef77-4050-8d30-e0522e6c7116-kube-api-access-426dx\") pod \"s3-init-x2g9p\" (UID: \"21ce4cae-ef77-4050-8d30-e0522e6c7116\") " pod="kserve/s3-init-x2g9p"
Sep 18 17:05:01.154959 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:05:01.154914    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-426dx\" (UniqueName: \"kubernetes.io/projected/21ce4cae-ef77-4050-8d30-e0522e6c7116-kube-api-access-426dx\") pod \"s3-init-x2g9p\" (UID: \"21ce4cae-ef77-4050-8d30-e0522e6c7116\") " pod="kserve/s3-init-x2g9p"
Sep 18 17:05:01.155150 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:05:01.154992    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21ce4cae-ef77-4050-8d30-e0522e6c7116-model-cache\") pod \"s3-init-x2g9p\" (UID: \"21ce4cae-ef77-4050-8d30-e0522e6c7116\") " pod="kserve/s3-init-x2g9p"
Sep 18 17:05:01.155387 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:05:01.155371    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21ce4cae-ef77-4050-8d30-e0522e6c7116-model-cache\") pod \"s3-init-x2g9p\" (UID: \"21ce4cae-ef77-4050-8d30-e0522e6c7116\") " pod="kserve/s3-init-x2g9p"
Sep 18 17:05:01.164715 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:05:01.164691    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-426dx\" (UniqueName: \"kubernetes.io/projected/21ce4cae-ef77-4050-8d30-e0522e6c7116-kube-api-access-426dx\") pod \"s3-init-x2g9p\" (UID: \"21ce4cae-ef77-4050-8d30-e0522e6c7116\") " pod="kserve/s3-init-x2g9p"
Sep 18 17:05:01.312292 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:05:01.312257    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-x2g9p"
Sep 18 17:05:01.589323 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:05:01.589278    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-x2g9p"]
Sep 18 17:05:01.591665 ip-10-0-143-5 kubenswrapper[2881]: W0918 17:05:01.591634    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod21ce4cae_ef77_4050_8d30_e0522e6c7116.slice/crio-6438d535bd0610d9d802e39ea6abbecf9a84aeed77049cc4e506a1f108f08181 WatchSource:0}: Error finding container 6438d535bd0610d9d802e39ea6abbecf9a84aeed77049cc4e506a1f108f08181: Status 404 returned error can't find the container with id 6438d535bd0610d9d802e39ea6abbecf9a84aeed77049cc4e506a1f108f08181
Sep 18 17:05:01.593584 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:05:01.593566    2881 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 17:05:02.332839 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:05:02.332801    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-x2g9p" event={"ID":"21ce4cae-ef77-4050-8d30-e0522e6c7116","Type":"ContainerStarted","Data":"6438d535bd0610d9d802e39ea6abbecf9a84aeed77049cc4e506a1f108f08181"}
Sep 18 17:05:06.354554 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:05:06.354507    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-x2g9p" event={"ID":"21ce4cae-ef77-4050-8d30-e0522e6c7116","Type":"ContainerStarted","Data":"7f0c0ef6d8a9e00d1166f261c707e88944a1b73b36ffbab41c83c93eb166cd98"}
Sep 18 17:05:08.365990 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:05:08.365953    2881 generic.go:356] "Generic (PLEG): container finished" podID="21ce4cae-ef77-4050-8d30-e0522e6c7116" containerID="7f0c0ef6d8a9e00d1166f261c707e88944a1b73b36ffbab41c83c93eb166cd98" exitCode=0
Sep 18 17:05:08.366531 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:05:08.366017    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-x2g9p" event={"ID":"21ce4cae-ef77-4050-8d30-e0522e6c7116","Type":"ContainerDied","Data":"7f0c0ef6d8a9e00d1166f261c707e88944a1b73b36ffbab41c83c93eb166cd98"}
Sep 18 17:05:13.395464 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:05:13.395428    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-x2g9p" event={"ID":"21ce4cae-ef77-4050-8d30-e0522e6c7116","Type":"ContainerStarted","Data":"352352eb130d42f31da0f1a787e5f3a537d2cb2e13a7780875397336b16b1107"}
Sep 18 17:05:13.427392 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:05:13.427335    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-x2g9p" podStartSLOduration=3.267999148 podStartE2EDuration="13.427317704s" totalImagesPullingTime="8.405949111s" totalInitContainerRuntime="1.753369445s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:05:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:05:13.426135561 +0000 UTC m=+703.625990846" watchObservedRunningTime="2026-09-18 17:05:13.427317704 +0000 UTC m=+703.627172990"
Sep 18 17:06:18.681341 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:06:18.681301    2881 generic.go:356] "Generic (PLEG): container finished" podID="21ce4cae-ef77-4050-8d30-e0522e6c7116" containerID="352352eb130d42f31da0f1a787e5f3a537d2cb2e13a7780875397336b16b1107" exitCode=0
Sep 18 17:06:18.681816 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:06:18.681370    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-x2g9p" event={"ID":"21ce4cae-ef77-4050-8d30-e0522e6c7116","Type":"ContainerDied","Data":"352352eb130d42f31da0f1a787e5f3a537d2cb2e13a7780875397336b16b1107"}
Sep 18 17:06:19.811687 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:06:19.811659    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-x2g9p"
Sep 18 17:06:19.882346 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:06:19.882306    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21ce4cae-ef77-4050-8d30-e0522e6c7116-model-cache\") pod \"21ce4cae-ef77-4050-8d30-e0522e6c7116\" (UID: \"21ce4cae-ef77-4050-8d30-e0522e6c7116\") "
Sep 18 17:06:19.903513 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:06:19.903472    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21ce4cae-ef77-4050-8d30-e0522e6c7116-model-cache" (OuterVolumeSpecName: "model-cache") pod "21ce4cae-ef77-4050-8d30-e0522e6c7116" (UID: "21ce4cae-ef77-4050-8d30-e0522e6c7116"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:06:19.983560 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:06:19.983477    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-426dx\" (UniqueName: \"kubernetes.io/projected/21ce4cae-ef77-4050-8d30-e0522e6c7116-kube-api-access-426dx\") pod \"21ce4cae-ef77-4050-8d30-e0522e6c7116\" (UID: \"21ce4cae-ef77-4050-8d30-e0522e6c7116\") "
Sep 18 17:06:19.983705 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:06:19.983646    2881 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21ce4cae-ef77-4050-8d30-e0522e6c7116-model-cache\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:06:19.985797 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:06:19.985753    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21ce4cae-ef77-4050-8d30-e0522e6c7116-kube-api-access-426dx" (OuterVolumeSpecName: "kube-api-access-426dx") pod "21ce4cae-ef77-4050-8d30-e0522e6c7116" (UID: "21ce4cae-ef77-4050-8d30-e0522e6c7116"). InnerVolumeSpecName "kube-api-access-426dx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:06:20.084682 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:06:20.084649    2881 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-426dx\" (UniqueName: \"kubernetes.io/projected/21ce4cae-ef77-4050-8d30-e0522e6c7116-kube-api-access-426dx\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:06:20.691629 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:06:20.691582    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-x2g9p" event={"ID":"21ce4cae-ef77-4050-8d30-e0522e6c7116","Type":"ContainerDied","Data":"6438d535bd0610d9d802e39ea6abbecf9a84aeed77049cc4e506a1f108f08181"}
Sep 18 17:06:20.691798 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:06:20.691639    2881 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6438d535bd0610d9d802e39ea6abbecf9a84aeed77049cc4e506a1f108f08181"
Sep 18 17:06:20.691798 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:06:20.691602    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-x2g9p"
Sep 18 17:06:56.970877 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:06:56.970839    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-6zcfm"]
Sep 18 17:06:56.971409 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:06:56.971176    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21ce4cae-ef77-4050-8d30-e0522e6c7116" containerName="s3-init"
Sep 18 17:06:56.971409 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:06:56.971186    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="21ce4cae-ef77-4050-8d30-e0522e6c7116" containerName="s3-init"
Sep 18 17:06:56.971409 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:06:56.971195    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21ce4cae-ef77-4050-8d30-e0522e6c7116" containerName="download-hf-model"
Sep 18 17:06:56.971409 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:06:56.971201    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="21ce4cae-ef77-4050-8d30-e0522e6c7116" containerName="download-hf-model"
Sep 18 17:06:56.971409 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:06:56.971275    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="21ce4cae-ef77-4050-8d30-e0522e6c7116" containerName="s3-init"
Sep 18 17:06:56.974408 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:06:56.974392    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-6zcfm"
Sep 18 17:06:56.977165 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:06:56.977136    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Sep 18 17:06:56.977328 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:06:56.977172    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 18 17:06:56.978190 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:06:56.978176    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-q8nb7\""
Sep 18 17:06:56.978264 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:06:56.978181    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 18 17:06:56.978264 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:06:56.978204    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 18 17:06:56.984052 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:06:56.984023    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-6zcfm"]
Sep 18 17:06:57.118441 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:06:57.118380    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18be7a4a-ca58-4ce2-b81e-e07710a62746-model-cache\") pod \"s3-tls-init-custom-6zcfm\" (UID: \"18be7a4a-ca58-4ce2-b81e-e07710a62746\") " pod="kserve/s3-tls-init-custom-6zcfm"
Sep 18 17:06:57.118649 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:06:57.118515    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzbw2\" (UniqueName: \"kubernetes.io/projected/18be7a4a-ca58-4ce2-b81e-e07710a62746-kube-api-access-vzbw2\") pod \"s3-tls-init-custom-6zcfm\" (UID: \"18be7a4a-ca58-4ce2-b81e-e07710a62746\") " pod="kserve/s3-tls-init-custom-6zcfm"
Sep 18 17:06:57.219694 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:06:57.219637    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18be7a4a-ca58-4ce2-b81e-e07710a62746-model-cache\") pod \"s3-tls-init-custom-6zcfm\" (UID: \"18be7a4a-ca58-4ce2-b81e-e07710a62746\") " pod="kserve/s3-tls-init-custom-6zcfm"
Sep 18 17:06:57.219890 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:06:57.219748    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzbw2\" (UniqueName: \"kubernetes.io/projected/18be7a4a-ca58-4ce2-b81e-e07710a62746-kube-api-access-vzbw2\") pod \"s3-tls-init-custom-6zcfm\" (UID: \"18be7a4a-ca58-4ce2-b81e-e07710a62746\") " pod="kserve/s3-tls-init-custom-6zcfm"
Sep 18 17:06:57.220057 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:06:57.220037    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18be7a4a-ca58-4ce2-b81e-e07710a62746-model-cache\") pod \"s3-tls-init-custom-6zcfm\" (UID: \"18be7a4a-ca58-4ce2-b81e-e07710a62746\") " pod="kserve/s3-tls-init-custom-6zcfm"
Sep 18 17:06:57.230575 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:06:57.230520    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzbw2\" (UniqueName: \"kubernetes.io/projected/18be7a4a-ca58-4ce2-b81e-e07710a62746-kube-api-access-vzbw2\") pod \"s3-tls-init-custom-6zcfm\" (UID: \"18be7a4a-ca58-4ce2-b81e-e07710a62746\") " pod="kserve/s3-tls-init-custom-6zcfm"
Sep 18 17:06:57.295307 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:06:57.295261    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-6zcfm"
Sep 18 17:06:57.572040 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:06:57.572006    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-6zcfm"]
Sep 18 17:06:57.575208 ip-10-0-143-5 kubenswrapper[2881]: W0918 17:06:57.575182    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod18be7a4a_ca58_4ce2_b81e_e07710a62746.slice/crio-e58b7c9018a5e53e3dfb54612edaaf0706520490e6565ce5f13af56f0ed4fe24 WatchSource:0}: Error finding container e58b7c9018a5e53e3dfb54612edaaf0706520490e6565ce5f13af56f0ed4fe24: Status 404 returned error can't find the container with id e58b7c9018a5e53e3dfb54612edaaf0706520490e6565ce5f13af56f0ed4fe24
Sep 18 17:06:57.850457 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:06:57.850353    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-6zcfm" event={"ID":"18be7a4a-ca58-4ce2-b81e-e07710a62746","Type":"ContainerStarted","Data":"1a940c1cae13c38a18a75628c492c23fcdabd29d2e791dbfc1d50e03cecd0f4b"}
Sep 18 17:06:57.850457 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:06:57.850406    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-6zcfm" event={"ID":"18be7a4a-ca58-4ce2-b81e-e07710a62746","Type":"ContainerStarted","Data":"e58b7c9018a5e53e3dfb54612edaaf0706520490e6565ce5f13af56f0ed4fe24"}
Sep 18 17:06:59.861098 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:06:59.861063    2881 generic.go:356] "Generic (PLEG): container finished" podID="18be7a4a-ca58-4ce2-b81e-e07710a62746" containerID="1a940c1cae13c38a18a75628c492c23fcdabd29d2e791dbfc1d50e03cecd0f4b" exitCode=0
Sep 18 17:06:59.861544 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:06:59.861137    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-6zcfm" event={"ID":"18be7a4a-ca58-4ce2-b81e-e07710a62746","Type":"ContainerDied","Data":"1a940c1cae13c38a18a75628c492c23fcdabd29d2e791dbfc1d50e03cecd0f4b"}
Sep 18 17:07:00.867898 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:00.867859    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-6zcfm" event={"ID":"18be7a4a-ca58-4ce2-b81e-e07710a62746","Type":"ContainerStarted","Data":"63c876a4f28efe4457ff102feb4264efd6f41f9f4a8dfdc21b80204a77904e77"}
Sep 18 17:07:00.889158 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:00.889099    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-custom-6zcfm" podStartSLOduration=3.174909687 podStartE2EDuration="4.889082114s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.714172427s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:06:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:07:00.88730253 +0000 UTC m=+811.087157816" watchObservedRunningTime="2026-09-18 17:07:00.889082114 +0000 UTC m=+811.088937399"
Sep 18 17:07:07.901830 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:07.901795    2881 generic.go:356] "Generic (PLEG): container finished" podID="18be7a4a-ca58-4ce2-b81e-e07710a62746" containerID="63c876a4f28efe4457ff102feb4264efd6f41f9f4a8dfdc21b80204a77904e77" exitCode=0
Sep 18 17:07:07.902216 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:07.901871    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-6zcfm" event={"ID":"18be7a4a-ca58-4ce2-b81e-e07710a62746","Type":"ContainerDied","Data":"63c876a4f28efe4457ff102feb4264efd6f41f9f4a8dfdc21b80204a77904e77"}
Sep 18 17:07:09.046567 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:09.046537    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-6zcfm"
Sep 18 17:07:09.120811 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:09.120746    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18be7a4a-ca58-4ce2-b81e-e07710a62746-model-cache\") pod \"18be7a4a-ca58-4ce2-b81e-e07710a62746\" (UID: \"18be7a4a-ca58-4ce2-b81e-e07710a62746\") "
Sep 18 17:07:09.120811 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:09.120806    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vzbw2\" (UniqueName: \"kubernetes.io/projected/18be7a4a-ca58-4ce2-b81e-e07710a62746-kube-api-access-vzbw2\") pod \"18be7a4a-ca58-4ce2-b81e-e07710a62746\" (UID: \"18be7a4a-ca58-4ce2-b81e-e07710a62746\") "
Sep 18 17:07:09.123826 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:09.123780    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18be7a4a-ca58-4ce2-b81e-e07710a62746-kube-api-access-vzbw2" (OuterVolumeSpecName: "kube-api-access-vzbw2") pod "18be7a4a-ca58-4ce2-b81e-e07710a62746" (UID: "18be7a4a-ca58-4ce2-b81e-e07710a62746"). InnerVolumeSpecName "kube-api-access-vzbw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:07:09.162292 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:09.162178    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18be7a4a-ca58-4ce2-b81e-e07710a62746-model-cache" (OuterVolumeSpecName: "model-cache") pod "18be7a4a-ca58-4ce2-b81e-e07710a62746" (UID: "18be7a4a-ca58-4ce2-b81e-e07710a62746"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:07:09.221990 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:09.221959    2881 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18be7a4a-ca58-4ce2-b81e-e07710a62746-model-cache\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:07:09.221990 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:09.221986    2881 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzbw2\" (UniqueName: \"kubernetes.io/projected/18be7a4a-ca58-4ce2-b81e-e07710a62746-kube-api-access-vzbw2\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:07:09.914012 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:09.913982    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-6zcfm"
Sep 18 17:07:09.914189 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:09.914023    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-6zcfm" event={"ID":"18be7a4a-ca58-4ce2-b81e-e07710a62746","Type":"ContainerDied","Data":"e58b7c9018a5e53e3dfb54612edaaf0706520490e6565ce5f13af56f0ed4fe24"}
Sep 18 17:07:09.914189 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:09.914061    2881 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e58b7c9018a5e53e3dfb54612edaaf0706520490e6565ce5f13af56f0ed4fe24"
Sep 18 17:07:13.272279 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:13.272236    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-m5bq2"]
Sep 18 17:07:13.272816 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:13.272586    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18be7a4a-ca58-4ce2-b81e-e07710a62746" containerName="download-hf-model"
Sep 18 17:07:13.272816 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:13.272598    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="18be7a4a-ca58-4ce2-b81e-e07710a62746" containerName="download-hf-model"
Sep 18 17:07:13.272816 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:13.272610    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18be7a4a-ca58-4ce2-b81e-e07710a62746" containerName="s3-tls-init-custom"
Sep 18 17:07:13.272816 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:13.272616    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="18be7a4a-ca58-4ce2-b81e-e07710a62746" containerName="s3-tls-init-custom"
Sep 18 17:07:13.272816 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:13.272674    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="18be7a4a-ca58-4ce2-b81e-e07710a62746" containerName="s3-tls-init-custom"
Sep 18 17:07:13.276163 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:13.276143    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-m5bq2"
Sep 18 17:07:13.278709 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:13.278681    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 18 17:07:13.278882 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:13.278866    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 18 17:07:13.279846 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:13.279819    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 18 17:07:13.279846 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:13.279837    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Sep 18 17:07:13.280030 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:13.279862    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-q8nb7\""
Sep 18 17:07:13.284862 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:13.284837    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-m5bq2"]
Sep 18 17:07:13.354495 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:13.354453    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9037cfd0-1f18-4bc6-b35a-f920c082823e-model-cache\") pod \"s3-tls-init-serving-m5bq2\" (UID: \"9037cfd0-1f18-4bc6-b35a-f920c082823e\") " pod="kserve/s3-tls-init-serving-m5bq2"
Sep 18 17:07:13.354495 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:13.354496    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znvf5\" (UniqueName: \"kubernetes.io/projected/9037cfd0-1f18-4bc6-b35a-f920c082823e-kube-api-access-znvf5\") pod \"s3-tls-init-serving-m5bq2\" (UID: \"9037cfd0-1f18-4bc6-b35a-f920c082823e\") " pod="kserve/s3-tls-init-serving-m5bq2"
Sep 18 17:07:13.455859 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:13.455803    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9037cfd0-1f18-4bc6-b35a-f920c082823e-model-cache\") pod \"s3-tls-init-serving-m5bq2\" (UID: \"9037cfd0-1f18-4bc6-b35a-f920c082823e\") " pod="kserve/s3-tls-init-serving-m5bq2"
Sep 18 17:07:13.455859 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:13.455861    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znvf5\" (UniqueName: \"kubernetes.io/projected/9037cfd0-1f18-4bc6-b35a-f920c082823e-kube-api-access-znvf5\") pod \"s3-tls-init-serving-m5bq2\" (UID: \"9037cfd0-1f18-4bc6-b35a-f920c082823e\") " pod="kserve/s3-tls-init-serving-m5bq2"
Sep 18 17:07:13.456250 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:13.456203    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9037cfd0-1f18-4bc6-b35a-f920c082823e-model-cache\") pod \"s3-tls-init-serving-m5bq2\" (UID: \"9037cfd0-1f18-4bc6-b35a-f920c082823e\") " pod="kserve/s3-tls-init-serving-m5bq2"
Sep 18 17:07:13.466139 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:13.466117    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znvf5\" (UniqueName: \"kubernetes.io/projected/9037cfd0-1f18-4bc6-b35a-f920c082823e-kube-api-access-znvf5\") pod \"s3-tls-init-serving-m5bq2\" (UID: \"9037cfd0-1f18-4bc6-b35a-f920c082823e\") " pod="kserve/s3-tls-init-serving-m5bq2"
Sep 18 17:07:13.603010 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:13.602914    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-m5bq2"
Sep 18 17:07:13.876194 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:13.876166    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-m5bq2"]
Sep 18 17:07:13.878082 ip-10-0-143-5 kubenswrapper[2881]: W0918 17:07:13.878054    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9037cfd0_1f18_4bc6_b35a_f920c082823e.slice/crio-c14ec9c046a3e0444da5bf5d79cacc825e33083e55fda6f83096315cf6274fae WatchSource:0}: Error finding container c14ec9c046a3e0444da5bf5d79cacc825e33083e55fda6f83096315cf6274fae: Status 404 returned error can't find the container with id c14ec9c046a3e0444da5bf5d79cacc825e33083e55fda6f83096315cf6274fae
Sep 18 17:07:13.933271 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:13.933219    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-m5bq2" event={"ID":"9037cfd0-1f18-4bc6-b35a-f920c082823e","Type":"ContainerStarted","Data":"c14ec9c046a3e0444da5bf5d79cacc825e33083e55fda6f83096315cf6274fae"}
Sep 18 17:07:14.939940 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:14.939892    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-m5bq2" event={"ID":"9037cfd0-1f18-4bc6-b35a-f920c082823e","Type":"ContainerStarted","Data":"af6d6ea38e982a89670bcd83b90e9fd609d393ae4db1b2d744832ef9b9bce5de"}
Sep 18 17:07:16.950734 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:16.950703    2881 generic.go:356] "Generic (PLEG): container finished" podID="9037cfd0-1f18-4bc6-b35a-f920c082823e" containerID="af6d6ea38e982a89670bcd83b90e9fd609d393ae4db1b2d744832ef9b9bce5de" exitCode=0
Sep 18 17:07:16.951120 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:16.950782    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-m5bq2" event={"ID":"9037cfd0-1f18-4bc6-b35a-f920c082823e","Type":"ContainerDied","Data":"af6d6ea38e982a89670bcd83b90e9fd609d393ae4db1b2d744832ef9b9bce5de"}
Sep 18 17:07:17.957671 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:17.957638    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-m5bq2" event={"ID":"9037cfd0-1f18-4bc6-b35a-f920c082823e","Type":"ContainerStarted","Data":"7b027070cca348ae345fe9639146f43ac1eff0c25439fbc9cda1f5301f2fc4b5"}
Sep 18 17:07:17.978609 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:17.978541    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-m5bq2" podStartSLOduration=2.905733927 podStartE2EDuration="4.978521378s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.072787451s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:07:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:07:17.976465662 +0000 UTC m=+828.176320941" watchObservedRunningTime="2026-09-18 17:07:17.978521378 +0000 UTC m=+828.178376665"
Sep 18 17:07:24.994896 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:24.994861    2881 generic.go:356] "Generic (PLEG): container finished" podID="9037cfd0-1f18-4bc6-b35a-f920c082823e" containerID="7b027070cca348ae345fe9639146f43ac1eff0c25439fbc9cda1f5301f2fc4b5" exitCode=0
Sep 18 17:07:24.995341 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:24.994919    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-m5bq2" event={"ID":"9037cfd0-1f18-4bc6-b35a-f920c082823e","Type":"ContainerDied","Data":"7b027070cca348ae345fe9639146f43ac1eff0c25439fbc9cda1f5301f2fc4b5"}
Sep 18 17:07:26.136149 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:26.136117    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-m5bq2"
Sep 18 17:07:26.282449 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:26.282412    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9037cfd0-1f18-4bc6-b35a-f920c082823e-model-cache\") pod \"9037cfd0-1f18-4bc6-b35a-f920c082823e\" (UID: \"9037cfd0-1f18-4bc6-b35a-f920c082823e\") "
Sep 18 17:07:26.282623 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:26.282497    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-znvf5\" (UniqueName: \"kubernetes.io/projected/9037cfd0-1f18-4bc6-b35a-f920c082823e-kube-api-access-znvf5\") pod \"9037cfd0-1f18-4bc6-b35a-f920c082823e\" (UID: \"9037cfd0-1f18-4bc6-b35a-f920c082823e\") "
Sep 18 17:07:26.284727 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:26.284685    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9037cfd0-1f18-4bc6-b35a-f920c082823e-kube-api-access-znvf5" (OuterVolumeSpecName: "kube-api-access-znvf5") pod "9037cfd0-1f18-4bc6-b35a-f920c082823e" (UID: "9037cfd0-1f18-4bc6-b35a-f920c082823e"). InnerVolumeSpecName "kube-api-access-znvf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:07:26.303008 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:26.302971    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9037cfd0-1f18-4bc6-b35a-f920c082823e-model-cache" (OuterVolumeSpecName: "model-cache") pod "9037cfd0-1f18-4bc6-b35a-f920c082823e" (UID: "9037cfd0-1f18-4bc6-b35a-f920c082823e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:07:26.383829 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:26.383793    2881 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9037cfd0-1f18-4bc6-b35a-f920c082823e-model-cache\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:07:26.383829 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:26.383824    2881 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znvf5\" (UniqueName: \"kubernetes.io/projected/9037cfd0-1f18-4bc6-b35a-f920c082823e-kube-api-access-znvf5\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:07:27.005168 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:27.005137    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-m5bq2"
Sep 18 17:07:27.005427 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:27.005171    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-m5bq2" event={"ID":"9037cfd0-1f18-4bc6-b35a-f920c082823e","Type":"ContainerDied","Data":"c14ec9c046a3e0444da5bf5d79cacc825e33083e55fda6f83096315cf6274fae"}
Sep 18 17:07:27.005427 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:27.005207    2881 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c14ec9c046a3e0444da5bf5d79cacc825e33083e55fda6f83096315cf6274fae"
Sep 18 17:07:39.972208 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:39.972165    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d"]
Sep 18 17:07:39.972766 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:39.972744    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9037cfd0-1f18-4bc6-b35a-f920c082823e" containerName="s3-tls-init-serving"
Sep 18 17:07:39.972848 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:39.972769    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="9037cfd0-1f18-4bc6-b35a-f920c082823e" containerName="s3-tls-init-serving"
Sep 18 17:07:39.972848 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:39.972785    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9037cfd0-1f18-4bc6-b35a-f920c082823e" containerName="download-hf-model"
Sep 18 17:07:39.972848 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:39.972793    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="9037cfd0-1f18-4bc6-b35a-f920c082823e" containerName="download-hf-model"
Sep 18 17:07:39.973006 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:39.972878    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="9037cfd0-1f18-4bc6-b35a-f920c082823e" containerName="s3-tls-init-serving"
Sep 18 17:07:39.975915 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:39.975893    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d"
Sep 18 17:07:39.978747 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:39.978720    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Sep 18 17:07:39.978921 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:39.978895    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 17:07:39.978921 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:39.978911    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 17:07:39.979104 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:39.978722    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 17:07:39.979104 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:39.978720    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Sep 18 17:07:39.979104 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:39.978722    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 17:07:39.979104 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:39.978788    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5hv9v\""
Sep 18 17:07:39.987010 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:39.986984    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d"]
Sep 18 17:07:40.083398 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:40.083371    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-64d445-predictor-586f9c5f7f-zmvhm"]
Sep 18 17:07:40.087021 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:40.086998    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-64d445-predictor-586f9c5f7f-zmvhm"
Sep 18 17:07:40.089717 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:40.089694    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-64d445\""
Sep 18 17:07:40.089717 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:40.089711    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-64d445-dockercfg-bxz7b\""
Sep 18 17:07:40.089917 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:40.089700    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-64d445-kube-rbac-proxy-sar-config\""
Sep 18 17:07:40.089917 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:40.089701    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-64d445-predictor-serving-cert\""
Sep 18 17:07:40.098717 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:40.098692    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-64d445-predictor-586f9c5f7f-zmvhm"]
Sep 18 17:07:40.109154 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:40.109125    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bc48814e-b773-45f3-be4f-e13aa2f69dab-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d\" (UID: \"bc48814e-b773-45f3-be4f-e13aa2f69dab\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d"
Sep 18 17:07:40.109154 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:40.109161    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45j5q\" (UniqueName: \"kubernetes.io/projected/bc48814e-b773-45f3-be4f-e13aa2f69dab-kube-api-access-45j5q\") pod \"isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d\" (UID: \"bc48814e-b773-45f3-be4f-e13aa2f69dab\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d"
Sep 18 17:07:40.109422 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:40.109191    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc48814e-b773-45f3-be4f-e13aa2f69dab-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d\" (UID: \"bc48814e-b773-45f3-be4f-e13aa2f69dab\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d"
Sep 18 17:07:40.109422 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:40.109282    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bc48814e-b773-45f3-be4f-e13aa2f69dab-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d\" (UID: \"bc48814e-b773-45f3-be4f-e13aa2f69dab\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d"
Sep 18 17:07:40.109422 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:40.109309    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc48814e-b773-45f3-be4f-e13aa2f69dab-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d\" (UID: \"bc48814e-b773-45f3-be4f-e13aa2f69dab\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d"
Sep 18 17:07:40.210303 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:40.210260    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bc48814e-b773-45f3-be4f-e13aa2f69dab-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d\" (UID: \"bc48814e-b773-45f3-be4f-e13aa2f69dab\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d"
Sep 18 17:07:40.210303 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:40.210306    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc48814e-b773-45f3-be4f-e13aa2f69dab-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d\" (UID: \"bc48814e-b773-45f3-be4f-e13aa2f69dab\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d"
Sep 18 17:07:40.210529 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:40.210326    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1-cabundle-cert\") pod \"isvc-init-fail-64d445-predictor-586f9c5f7f-zmvhm\" (UID: \"ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1\") " pod="kserve-ci-e2e-test/isvc-init-fail-64d445-predictor-586f9c5f7f-zmvhm"
Sep 18 17:07:40.210529 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:40.210360    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-64d445-predictor-586f9c5f7f-zmvhm\" (UID: \"ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1\") " pod="kserve-ci-e2e-test/isvc-init-fail-64d445-predictor-586f9c5f7f-zmvhm"
Sep 18 17:07:40.210529 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:40.210381    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bc48814e-b773-45f3-be4f-e13aa2f69dab-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d\" (UID: \"bc48814e-b773-45f3-be4f-e13aa2f69dab\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d"
Sep 18 17:07:40.210529 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:40.210408    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45j5q\" (UniqueName: \"kubernetes.io/projected/bc48814e-b773-45f3-be4f-e13aa2f69dab-kube-api-access-45j5q\") pod \"isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d\" (UID: \"bc48814e-b773-45f3-be4f-e13aa2f69dab\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d"
Sep 18 17:07:40.210529 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:40.210489    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc48814e-b773-45f3-be4f-e13aa2f69dab-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d\" (UID: \"bc48814e-b773-45f3-be4f-e13aa2f69dab\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d"
Sep 18 17:07:40.210831 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:40.210535    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1-proxy-tls\") pod \"isvc-init-fail-64d445-predictor-586f9c5f7f-zmvhm\" (UID: \"ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1\") " pod="kserve-ci-e2e-test/isvc-init-fail-64d445-predictor-586f9c5f7f-zmvhm"
Sep 18 17:07:40.210831 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:40.210592    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1-kserve-provision-location\") pod \"isvc-init-fail-64d445-predictor-586f9c5f7f-zmvhm\" (UID: \"ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1\") " pod="kserve-ci-e2e-test/isvc-init-fail-64d445-predictor-586f9c5f7f-zmvhm"
Sep 18 17:07:40.210831 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:40.210680    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9jb9\" (UniqueName: \"kubernetes.io/projected/ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1-kube-api-access-r9jb9\") pod \"isvc-init-fail-64d445-predictor-586f9c5f7f-zmvhm\" (UID: \"ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1\") " pod="kserve-ci-e2e-test/isvc-init-fail-64d445-predictor-586f9c5f7f-zmvhm"
Sep 18 17:07:40.210831 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:40.210744    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bc48814e-b773-45f3-be4f-e13aa2f69dab-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d\" (UID: \"bc48814e-b773-45f3-be4f-e13aa2f69dab\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d"
Sep 18 17:07:40.211047 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:40.211033    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc48814e-b773-45f3-be4f-e13aa2f69dab-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d\" (UID: \"bc48814e-b773-45f3-be4f-e13aa2f69dab\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d"
Sep 18 17:07:40.211187 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:40.211167    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bc48814e-b773-45f3-be4f-e13aa2f69dab-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d\" (UID: \"bc48814e-b773-45f3-be4f-e13aa2f69dab\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d"
Sep 18 17:07:40.213056 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:40.213038    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc48814e-b773-45f3-be4f-e13aa2f69dab-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d\" (UID: \"bc48814e-b773-45f3-be4f-e13aa2f69dab\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d"
Sep 18 17:07:40.219154 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:40.219132    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45j5q\" (UniqueName: \"kubernetes.io/projected/bc48814e-b773-45f3-be4f-e13aa2f69dab-kube-api-access-45j5q\") pod \"isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d\" (UID: \"bc48814e-b773-45f3-be4f-e13aa2f69dab\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d"
Sep 18 17:07:40.287063 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:40.287029    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d"
Sep 18 17:07:40.312024 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:40.311987    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1-cabundle-cert\") pod \"isvc-init-fail-64d445-predictor-586f9c5f7f-zmvhm\" (UID: \"ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1\") " pod="kserve-ci-e2e-test/isvc-init-fail-64d445-predictor-586f9c5f7f-zmvhm"
Sep 18 17:07:40.312200 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:40.312057    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-64d445-predictor-586f9c5f7f-zmvhm\" (UID: \"ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1\") " pod="kserve-ci-e2e-test/isvc-init-fail-64d445-predictor-586f9c5f7f-zmvhm"
Sep 18 17:07:40.312200 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:40.312125    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1-proxy-tls\") pod \"isvc-init-fail-64d445-predictor-586f9c5f7f-zmvhm\" (UID: \"ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1\") " pod="kserve-ci-e2e-test/isvc-init-fail-64d445-predictor-586f9c5f7f-zmvhm"
Sep 18 17:07:40.312200 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:40.312166    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1-kserve-provision-location\") pod \"isvc-init-fail-64d445-predictor-586f9c5f7f-zmvhm\" (UID: \"ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1\") " pod="kserve-ci-e2e-test/isvc-init-fail-64d445-predictor-586f9c5f7f-zmvhm"
Sep 18 17:07:40.312411 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:40.312249    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9jb9\" (UniqueName: \"kubernetes.io/projected/ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1-kube-api-access-r9jb9\") pod \"isvc-init-fail-64d445-predictor-586f9c5f7f-zmvhm\" (UID: \"ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1\") " pod="kserve-ci-e2e-test/isvc-init-fail-64d445-predictor-586f9c5f7f-zmvhm"
Sep 18 17:07:40.312588 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:40.312563    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1-kserve-provision-location\") pod \"isvc-init-fail-64d445-predictor-586f9c5f7f-zmvhm\" (UID: \"ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1\") " pod="kserve-ci-e2e-test/isvc-init-fail-64d445-predictor-586f9c5f7f-zmvhm"
Sep 18 17:07:40.312689 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:40.312672    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1-cabundle-cert\") pod \"isvc-init-fail-64d445-predictor-586f9c5f7f-zmvhm\" (UID: \"ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1\") " pod="kserve-ci-e2e-test/isvc-init-fail-64d445-predictor-586f9c5f7f-zmvhm"
Sep 18 17:07:40.312821 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:40.312805    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-64d445-predictor-586f9c5f7f-zmvhm\" (UID: \"ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1\") " pod="kserve-ci-e2e-test/isvc-init-fail-64d445-predictor-586f9c5f7f-zmvhm"
Sep 18 17:07:40.314880 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:40.314860    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1-proxy-tls\") pod \"isvc-init-fail-64d445-predictor-586f9c5f7f-zmvhm\" (UID: \"ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1\") " pod="kserve-ci-e2e-test/isvc-init-fail-64d445-predictor-586f9c5f7f-zmvhm"
Sep 18 17:07:40.322622 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:40.322597    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9jb9\" (UniqueName: \"kubernetes.io/projected/ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1-kube-api-access-r9jb9\") pod \"isvc-init-fail-64d445-predictor-586f9c5f7f-zmvhm\" (UID: \"ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1\") " pod="kserve-ci-e2e-test/isvc-init-fail-64d445-predictor-586f9c5f7f-zmvhm"
Sep 18 17:07:40.399847 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:40.399801    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-64d445-predictor-586f9c5f7f-zmvhm"
Sep 18 17:07:40.578192 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:40.578166    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d"]
Sep 18 17:07:40.580737 ip-10-0-143-5 kubenswrapper[2881]: W0918 17:07:40.580706    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc48814e_b773_45f3_be4f_e13aa2f69dab.slice/crio-a3d78882fce450338fa7a2fe3b3a6e22e8e16256b2870fa298e29def568d64bd WatchSource:0}: Error finding container a3d78882fce450338fa7a2fe3b3a6e22e8e16256b2870fa298e29def568d64bd: Status 404 returned error can't find the container with id a3d78882fce450338fa7a2fe3b3a6e22e8e16256b2870fa298e29def568d64bd
Sep 18 17:07:40.682329 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:40.682272    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-64d445-predictor-586f9c5f7f-zmvhm"]
Sep 18 17:07:40.684727 ip-10-0-143-5 kubenswrapper[2881]: W0918 17:07:40.684684    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab7e7366_faa7_4fa3_8483_f17dfb4f2dd1.slice/crio-6fe59b6f536661b3ea5b407ee740faa61a13662308a69a6ffc0734ac23689da0 WatchSource:0}: Error finding container 6fe59b6f536661b3ea5b407ee740faa61a13662308a69a6ffc0734ac23689da0: Status 404 returned error can't find the container with id 6fe59b6f536661b3ea5b407ee740faa61a13662308a69a6ffc0734ac23689da0
Sep 18 17:07:41.068353 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:41.068300    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-64d445-predictor-586f9c5f7f-zmvhm" event={"ID":"ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1","Type":"ContainerStarted","Data":"45cabdb9aeae5ecc9a33e88c4ce466595f5cf42214fae9338f478c81538712f5"}
Sep 18 17:07:41.068833 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:41.068389    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-64d445-predictor-586f9c5f7f-zmvhm" event={"ID":"ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1","Type":"ContainerStarted","Data":"6fe59b6f536661b3ea5b407ee740faa61a13662308a69a6ffc0734ac23689da0"}
Sep 18 17:07:41.070239 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:41.070188    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d" event={"ID":"bc48814e-b773-45f3-be4f-e13aa2f69dab","Type":"ContainerStarted","Data":"e5242ae64fc92e532256b64dda35efcdf0d44d0ad931a9f2e69a6468d0bb4cf7"}
Sep 18 17:07:41.070377 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:41.070297    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d" event={"ID":"bc48814e-b773-45f3-be4f-e13aa2f69dab","Type":"ContainerStarted","Data":"a3d78882fce450338fa7a2fe3b3a6e22e8e16256b2870fa298e29def568d64bd"}
Sep 18 17:07:45.089538 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:45.089501    2881 generic.go:356] "Generic (PLEG): container finished" podID="bc48814e-b773-45f3-be4f-e13aa2f69dab" containerID="e5242ae64fc92e532256b64dda35efcdf0d44d0ad931a9f2e69a6468d0bb4cf7" exitCode=0
Sep 18 17:07:45.090039 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:45.089572    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d" event={"ID":"bc48814e-b773-45f3-be4f-e13aa2f69dab","Type":"ContainerDied","Data":"e5242ae64fc92e532256b64dda35efcdf0d44d0ad931a9f2e69a6468d0bb4cf7"}
Sep 18 17:07:47.116249 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:47.114922    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-64d445-predictor-586f9c5f7f-zmvhm_ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1/storage-initializer/0.log"
Sep 18 17:07:47.116249 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:47.114971    2881 generic.go:356] "Generic (PLEG): container finished" podID="ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1" containerID="45cabdb9aeae5ecc9a33e88c4ce466595f5cf42214fae9338f478c81538712f5" exitCode=1
Sep 18 17:07:47.116249 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:47.115100    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-64d445-predictor-586f9c5f7f-zmvhm" event={"ID":"ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1","Type":"ContainerDied","Data":"45cabdb9aeae5ecc9a33e88c4ce466595f5cf42214fae9338f478c81538712f5"}
Sep 18 17:07:48.127934 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:48.126893    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-64d445-predictor-586f9c5f7f-zmvhm_ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1/storage-initializer/0.log"
Sep 18 17:07:48.127934 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:48.127036    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-64d445-predictor-586f9c5f7f-zmvhm" event={"ID":"ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1","Type":"ContainerStarted","Data":"ac2df9ebc123ac3e83b1a81e681983fe816e86cd2aecbf659eec49d4449ba593"}
Sep 18 17:07:49.793747 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:49.792840    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-64d445-predictor-586f9c5f7f-zmvhm"]
Sep 18 17:07:49.793747 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:49.793126    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-64d445-predictor-586f9c5f7f-zmvhm" podUID="ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1" containerName="storage-initializer" containerID="cri-o://ac2df9ebc123ac3e83b1a81e681983fe816e86cd2aecbf659eec49d4449ba593" gracePeriod=30
Sep 18 17:07:51.146759 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:51.146730    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-64d445-predictor-586f9c5f7f-zmvhm_ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1/storage-initializer/1.log"
Sep 18 17:07:51.147239 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:51.147203    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-64d445-predictor-586f9c5f7f-zmvhm_ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1/storage-initializer/0.log"
Sep 18 17:07:51.147325 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:51.147269    2881 generic.go:356] "Generic (PLEG): container finished" podID="ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1" containerID="ac2df9ebc123ac3e83b1a81e681983fe816e86cd2aecbf659eec49d4449ba593" exitCode=1
Sep 18 17:07:51.147377 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:51.147353    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-64d445-predictor-586f9c5f7f-zmvhm" event={"ID":"ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1","Type":"ContainerDied","Data":"ac2df9ebc123ac3e83b1a81e681983fe816e86cd2aecbf659eec49d4449ba593"}
Sep 18 17:07:51.147461 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:51.147444    2881 scope.go:117] "RemoveContainer" containerID="45cabdb9aeae5ecc9a33e88c4ce466595f5cf42214fae9338f478c81538712f5"
Sep 18 17:07:55.062574 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:55.062514    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t5n2z"]
Sep 18 17:07:55.089088 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:55.089054    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t5n2z"]
Sep 18 17:07:55.089301 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:55.089206    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t5n2z"
Sep 18 17:07:55.091944 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:55.091919    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-predictor-serving-cert\""
Sep 18 17:07:55.092109 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:55.091943    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\""
Sep 18 17:07:55.161495 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:55.161444    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp6tx\" (UniqueName: \"kubernetes.io/projected/f9e8938f-6241-4ade-8b71-4bd14398fc4c-kube-api-access-kp6tx\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-t5n2z\" (UID: \"f9e8938f-6241-4ade-8b71-4bd14398fc4c\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t5n2z"
Sep 18 17:07:55.161714 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:55.161583    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9e8938f-6241-4ade-8b71-4bd14398fc4c-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-t5n2z\" (UID: \"f9e8938f-6241-4ade-8b71-4bd14398fc4c\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t5n2z"
Sep 18 17:07:55.161714 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:55.161624    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9e8938f-6241-4ade-8b71-4bd14398fc4c-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-t5n2z\" (UID: \"f9e8938f-6241-4ade-8b71-4bd14398fc4c\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t5n2z"
Sep 18 17:07:55.161714 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:55.161652    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9e8938f-6241-4ade-8b71-4bd14398fc4c-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-t5n2z\" (UID: \"f9e8938f-6241-4ade-8b71-4bd14398fc4c\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t5n2z"
Sep 18 17:07:55.161714 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:55.161688    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9e8938f-6241-4ade-8b71-4bd14398fc4c-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-t5n2z\" (UID: \"f9e8938f-6241-4ade-8b71-4bd14398fc4c\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t5n2z"
Sep 18 17:07:55.262586 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:55.262557    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp6tx\" (UniqueName: \"kubernetes.io/projected/f9e8938f-6241-4ade-8b71-4bd14398fc4c-kube-api-access-kp6tx\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-t5n2z\" (UID: \"f9e8938f-6241-4ade-8b71-4bd14398fc4c\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t5n2z"
Sep 18 17:07:55.262761 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:55.262657    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9e8938f-6241-4ade-8b71-4bd14398fc4c-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-t5n2z\" (UID: \"f9e8938f-6241-4ade-8b71-4bd14398fc4c\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t5n2z"
Sep 18 17:07:55.262761 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:55.262684    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9e8938f-6241-4ade-8b71-4bd14398fc4c-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-t5n2z\" (UID: \"f9e8938f-6241-4ade-8b71-4bd14398fc4c\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t5n2z"
Sep 18 17:07:55.262761 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:55.262710    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9e8938f-6241-4ade-8b71-4bd14398fc4c-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-t5n2z\" (UID: \"f9e8938f-6241-4ade-8b71-4bd14398fc4c\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t5n2z"
Sep 18 17:07:55.262761 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:55.262742    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9e8938f-6241-4ade-8b71-4bd14398fc4c-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-t5n2z\" (UID: \"f9e8938f-6241-4ade-8b71-4bd14398fc4c\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t5n2z"
Sep 18 17:07:55.263272 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:55.263241    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9e8938f-6241-4ade-8b71-4bd14398fc4c-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-t5n2z\" (UID: \"f9e8938f-6241-4ade-8b71-4bd14398fc4c\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t5n2z"
Sep 18 17:07:55.263504 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:55.263482    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9e8938f-6241-4ade-8b71-4bd14398fc4c-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-t5n2z\" (UID: \"f9e8938f-6241-4ade-8b71-4bd14398fc4c\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t5n2z"
Sep 18 17:07:55.263597 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:55.263506    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9e8938f-6241-4ade-8b71-4bd14398fc4c-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-t5n2z\" (UID: \"f9e8938f-6241-4ade-8b71-4bd14398fc4c\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t5n2z"
Sep 18 17:07:55.265367 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:55.265343    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9e8938f-6241-4ade-8b71-4bd14398fc4c-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-t5n2z\" (UID: \"f9e8938f-6241-4ade-8b71-4bd14398fc4c\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t5n2z"
Sep 18 17:07:55.272304 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:55.272276    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp6tx\" (UniqueName: \"kubernetes.io/projected/f9e8938f-6241-4ade-8b71-4bd14398fc4c-kube-api-access-kp6tx\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-t5n2z\" (UID: \"f9e8938f-6241-4ade-8b71-4bd14398fc4c\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t5n2z"
Sep 18 17:07:55.290474 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:55.290448    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-64d445-predictor-586f9c5f7f-zmvhm_ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1/storage-initializer/1.log"
Sep 18 17:07:55.294161 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:55.294130    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-64d445-predictor-586f9c5f7f-zmvhm"
Sep 18 17:07:55.363948 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:55.363863    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1-kserve-provision-location\") pod \"ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1\" (UID: \"ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1\") "
Sep 18 17:07:55.363948 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:55.363927    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1-proxy-tls\") pod \"ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1\" (UID: \"ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1\") "
Sep 18 17:07:55.364205 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:55.363963    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r9jb9\" (UniqueName: \"kubernetes.io/projected/ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1-kube-api-access-r9jb9\") pod \"ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1\" (UID: \"ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1\") "
Sep 18 17:07:55.364205 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:55.364042    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1-cabundle-cert\") pod \"ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1\" (UID: \"ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1\") "
Sep 18 17:07:55.364205 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:55.364103    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1-kube-rbac-proxy-sar-config\") pod \"ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1\" (UID: \"ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1\") "
Sep 18 17:07:55.364205 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:55.364186    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1" (UID: "ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:07:55.364442 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:55.364430    2881 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1-kserve-provision-location\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:07:55.364563 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:55.364527    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1" (UID: "ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:07:55.364626 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:55.364557    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1" (UID: "ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:07:55.367416 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:55.367349    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1-kube-api-access-r9jb9" (OuterVolumeSpecName: "kube-api-access-r9jb9") pod "ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1" (UID: "ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1"). InnerVolumeSpecName "kube-api-access-r9jb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:07:55.367416 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:55.367369    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1" (UID: "ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:07:55.402348 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:55.402310    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t5n2z"
Sep 18 17:07:55.465362 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:55.465324    2881 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1-proxy-tls\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:07:55.465362 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:55.465358    2881 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9jb9\" (UniqueName: \"kubernetes.io/projected/ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1-kube-api-access-r9jb9\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:07:55.465362 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:55.465371    2881 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1-cabundle-cert\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:07:55.465681 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:55.465380    2881 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:07:56.173786 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:56.173701    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-64d445-predictor-586f9c5f7f-zmvhm_ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1/storage-initializer/1.log"
Sep 18 17:07:56.174319 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:56.174239    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-64d445-predictor-586f9c5f7f-zmvhm" event={"ID":"ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1","Type":"ContainerDied","Data":"6fe59b6f536661b3ea5b407ee740faa61a13662308a69a6ffc0734ac23689da0"}
Sep 18 17:07:56.174319 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:56.174290    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-64d445-predictor-586f9c5f7f-zmvhm"
Sep 18 17:07:56.214724 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:56.214680    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-64d445-predictor-586f9c5f7f-zmvhm"]
Sep 18 17:07:56.219198 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:56.219163    2881 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-64d445-predictor-586f9c5f7f-zmvhm"]
Sep 18 17:07:57.894217 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:57.894183    2881 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1" path="/var/lib/kubelet/pods/ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1/volumes"
Sep 18 17:07:58.476279 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:58.476086    2881 scope.go:117] "RemoveContainer" containerID="ac2df9ebc123ac3e83b1a81e681983fe816e86cd2aecbf659eec49d4449ba593"
Sep 18 17:07:58.527111 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:58.526956    2881 scope.go:117] "RemoveContainer" containerID="45cabdb9aeae5ecc9a33e88c4ce466595f5cf42214fae9338f478c81538712f5"
Sep 18 17:07:58.527618 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:07:58.527563    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45cabdb9aeae5ecc9a33e88c4ce466595f5cf42214fae9338f478c81538712f5\": container with ID starting with 45cabdb9aeae5ecc9a33e88c4ce466595f5cf42214fae9338f478c81538712f5 not found: ID does not exist" containerID="45cabdb9aeae5ecc9a33e88c4ce466595f5cf42214fae9338f478c81538712f5"
Sep 18 17:07:58.527946 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:58.527615    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45cabdb9aeae5ecc9a33e88c4ce466595f5cf42214fae9338f478c81538712f5"} err="failed to get container status \"45cabdb9aeae5ecc9a33e88c4ce466595f5cf42214fae9338f478c81538712f5\": rpc error: code = NotFound desc = could not find container \"45cabdb9aeae5ecc9a33e88c4ce466595f5cf42214fae9338f478c81538712f5\": container with ID starting with 45cabdb9aeae5ecc9a33e88c4ce466595f5cf42214fae9338f478c81538712f5 not found: ID does not exist"
Sep 18 17:07:58.752170 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:58.752142    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t5n2z"]
Sep 18 17:07:58.754768 ip-10-0-143-5 kubenswrapper[2881]: W0918 17:07:58.754740    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9e8938f_6241_4ade_8b71_4bd14398fc4c.slice/crio-d1941789539df4b1bd195363fc0b99493fb7c2eded35f3c2305099ed7db549b5 WatchSource:0}: Error finding container d1941789539df4b1bd195363fc0b99493fb7c2eded35f3c2305099ed7db549b5: Status 404 returned error can't find the container with id d1941789539df4b1bd195363fc0b99493fb7c2eded35f3c2305099ed7db549b5
Sep 18 17:07:59.193243 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:59.193184    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d" event={"ID":"bc48814e-b773-45f3-be4f-e13aa2f69dab","Type":"ContainerStarted","Data":"8663b43327ed0b2630ada698429fdb817d468e66456c47edb7ee233d0944b484"}
Sep 18 17:07:59.196042 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:59.196009    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t5n2z" event={"ID":"f9e8938f-6241-4ade-8b71-4bd14398fc4c","Type":"ContainerStarted","Data":"0e60dd3e7159ad3d721a139802c1220307008df1ddce2cdca13f5abedf57b512"}
Sep 18 17:07:59.196185 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:07:59.196080    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t5n2z" event={"ID":"f9e8938f-6241-4ade-8b71-4bd14398fc4c","Type":"ContainerStarted","Data":"d1941789539df4b1bd195363fc0b99493fb7c2eded35f3c2305099ed7db549b5"}
Sep 18 17:08:02.215900 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:08:02.215862    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d" event={"ID":"bc48814e-b773-45f3-be4f-e13aa2f69dab","Type":"ContainerStarted","Data":"8e6a873386f6d8763767378e27954a2796d755a98f42296db16190cf2f2f8956"}
Sep 18 17:08:03.223034 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:08:03.222995    2881 generic.go:356] "Generic (PLEG): container finished" podID="f9e8938f-6241-4ade-8b71-4bd14398fc4c" containerID="0e60dd3e7159ad3d721a139802c1220307008df1ddce2cdca13f5abedf57b512" exitCode=0
Sep 18 17:08:03.223536 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:08:03.223042    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t5n2z" event={"ID":"f9e8938f-6241-4ade-8b71-4bd14398fc4c","Type":"ContainerDied","Data":"0e60dd3e7159ad3d721a139802c1220307008df1ddce2cdca13f5abedf57b512"}
Sep 18 17:08:05.240794 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:08:05.240647    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d" event={"ID":"bc48814e-b773-45f3-be4f-e13aa2f69dab","Type":"ContainerStarted","Data":"52fdd6c16bba1635e1138741889f6b0b9b9366753c490df6eda8c7655d019458"}
Sep 18 17:08:05.241682 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:08:05.241653    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d"
Sep 18 17:08:05.241808 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:08:05.241738    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d"
Sep 18 17:08:05.241808 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:08:05.241777    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d"
Sep 18 17:08:05.243689 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:08:05.243624    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d" podUID="bc48814e-b773-45f3-be4f-e13aa2f69dab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 18 17:08:05.244677 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:08:05.244640    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d" podUID="bc48814e-b773-45f3-be4f-e13aa2f69dab" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 17:08:05.271400 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:08:05.271305    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d" podStartSLOduration=3.367833839 podStartE2EDuration="26.271281821s" totalImagesPullingTime="19.152648582s" totalInitContainerRuntime="3.7507994s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:07:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:08:05.268255075 +0000 UTC m=+875.468110366" watchObservedRunningTime="2026-09-18 17:08:05.271281821 +0000 UTC m=+875.471137112"
Sep 18 17:08:06.248502 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:08:06.248445    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d" podUID="bc48814e-b773-45f3-be4f-e13aa2f69dab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 18 17:08:06.249451 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:08:06.249367    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d" podUID="bc48814e-b773-45f3-be4f-e13aa2f69dab" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 17:08:06.254146 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:08:06.254117    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d"
Sep 18 17:08:07.254889 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:08:07.254833    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d" podUID="bc48814e-b773-45f3-be4f-e13aa2f69dab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 18 17:08:07.255423 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:08:07.255397    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d" podUID="bc48814e-b773-45f3-be4f-e13aa2f69dab" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 17:08:17.254853 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:08:17.254798    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d" podUID="bc48814e-b773-45f3-be4f-e13aa2f69dab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 18 17:08:17.255519 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:08:17.255334    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d" podUID="bc48814e-b773-45f3-be4f-e13aa2f69dab" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 17:08:27.255379 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:08:27.255329    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d" podUID="bc48814e-b773-45f3-be4f-e13aa2f69dab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 18 17:08:27.255920 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:08:27.255744    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d" podUID="bc48814e-b773-45f3-be4f-e13aa2f69dab" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 17:08:28.388019 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:08:28.387983    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t5n2z" event={"ID":"f9e8938f-6241-4ade-8b71-4bd14398fc4c","Type":"ContainerStarted","Data":"6adc13e7e8b4d155d01258372796e13abd6f2d10ea26084a5fcfc7fa6d0378fa"}
Sep 18 17:08:28.388470 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:08:28.388048    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t5n2z" event={"ID":"f9e8938f-6241-4ade-8b71-4bd14398fc4c","Type":"ContainerStarted","Data":"b7bcfcdeb0d6090c024a1b055d6b2ae01ce777571afd1009362e7dafec344df5"}
Sep 18 17:08:28.388470 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:08:28.388263    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t5n2z"
Sep 18 17:08:28.436172 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:08:28.436103    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t5n2z" podStartSLOduration=4.97959545 podStartE2EDuration="33.436078406s" totalImagesPullingTime="24.200735192s" totalInitContainerRuntime="4.255747764s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:07:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:08:28.43578854 +0000 UTC m=+898.635643828" watchObservedRunningTime="2026-09-18 17:08:28.436078406 +0000 UTC m=+898.635933694"
Sep 18 17:08:29.393300 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:08:29.393259    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t5n2z"
Sep 18 17:08:29.394567 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:08:29.394540    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t5n2z" podUID="f9e8938f-6241-4ade-8b71-4bd14398fc4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 17:08:30.397355 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:08:30.397318    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t5n2z" podUID="f9e8938f-6241-4ade-8b71-4bd14398fc4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 17:08:35.402723 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:08:35.402689    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t5n2z"
Sep 18 17:08:35.403448 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:08:35.403417    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t5n2z" podUID="f9e8938f-6241-4ade-8b71-4bd14398fc4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 17:08:37.255753 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:08:37.255708    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d" podUID="bc48814e-b773-45f3-be4f-e13aa2f69dab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 18 17:08:37.256255 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:08:37.256210    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d" podUID="bc48814e-b773-45f3-be4f-e13aa2f69dab" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 17:08:45.403152 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:08:45.403108    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t5n2z" podUID="f9e8938f-6241-4ade-8b71-4bd14398fc4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 17:08:47.255565 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:08:47.255510    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d" podUID="bc48814e-b773-45f3-be4f-e13aa2f69dab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 18 17:08:47.256000 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:08:47.255919    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d" podUID="bc48814e-b773-45f3-be4f-e13aa2f69dab" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 17:08:55.403547 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:08:55.403501    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t5n2z" podUID="f9e8938f-6241-4ade-8b71-4bd14398fc4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 17:08:57.255552 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:08:57.255507    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d" podUID="bc48814e-b773-45f3-be4f-e13aa2f69dab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 18 17:08:57.256473 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:08:57.256447    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d" podUID="bc48814e-b773-45f3-be4f-e13aa2f69dab" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 17:09:05.403504 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:05.403410    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t5n2z" podUID="f9e8938f-6241-4ade-8b71-4bd14398fc4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 17:09:07.255579 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:07.255521    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d" podUID="bc48814e-b773-45f3-be4f-e13aa2f69dab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 18 17:09:07.255999 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:07.255958    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d" podUID="bc48814e-b773-45f3-be4f-e13aa2f69dab" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 17:09:15.403487 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:15.403436    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t5n2z" podUID="f9e8938f-6241-4ade-8b71-4bd14398fc4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 17:09:17.255287 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:17.255212    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d" podUID="bc48814e-b773-45f3-be4f-e13aa2f69dab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 18 17:09:17.255778 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:17.255725    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d" podUID="bc48814e-b773-45f3-be4f-e13aa2f69dab" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 17:09:25.403517 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:25.403465    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t5n2z" podUID="f9e8938f-6241-4ade-8b71-4bd14398fc4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 17:09:27.256295 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:27.256237    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d"
Sep 18 17:09:27.256811 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:27.256404    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d"
Sep 18 17:09:35.403145 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:35.403102    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t5n2z" podUID="f9e8938f-6241-4ade-8b71-4bd14398fc4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 17:09:35.484486 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:35.484448    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d"]
Sep 18 17:09:35.484770 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:35.484733    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d" podUID="bc48814e-b773-45f3-be4f-e13aa2f69dab" containerName="kserve-container" containerID="cri-o://8663b43327ed0b2630ada698429fdb817d468e66456c47edb7ee233d0944b484" gracePeriod=30
Sep 18 17:09:35.484914 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:35.484765    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d" podUID="bc48814e-b773-45f3-be4f-e13aa2f69dab" containerName="agent" containerID="cri-o://52fdd6c16bba1635e1138741889f6b0b9b9366753c490df6eda8c7655d019458" gracePeriod=30
Sep 18 17:09:35.484914 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:35.484777    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d" podUID="bc48814e-b773-45f3-be4f-e13aa2f69dab" containerName="kube-rbac-proxy" containerID="cri-o://8e6a873386f6d8763767378e27954a2796d755a98f42296db16190cf2f2f8956" gracePeriod=30
Sep 18 17:09:35.713800 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:35.713703    2881 generic.go:356] "Generic (PLEG): container finished" podID="bc48814e-b773-45f3-be4f-e13aa2f69dab" containerID="8e6a873386f6d8763767378e27954a2796d755a98f42296db16190cf2f2f8956" exitCode=2
Sep 18 17:09:35.713800 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:35.713776    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d" event={"ID":"bc48814e-b773-45f3-be4f-e13aa2f69dab","Type":"ContainerDied","Data":"8e6a873386f6d8763767378e27954a2796d755a98f42296db16190cf2f2f8956"}
Sep 18 17:09:35.733011 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:35.732966    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9"]
Sep 18 17:09:35.733535 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:35.733507    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1" containerName="storage-initializer"
Sep 18 17:09:35.733535 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:35.733531    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1" containerName="storage-initializer"
Sep 18 17:09:35.733724 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:35.733546    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1" containerName="storage-initializer"
Sep 18 17:09:35.733724 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:35.733554    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1" containerName="storage-initializer"
Sep 18 17:09:35.733724 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:35.733614    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1" containerName="storage-initializer"
Sep 18 17:09:35.733724 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:35.733624    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab7e7366-faa7-4fa3-8483-f17dfb4f2dd1" containerName="storage-initializer"
Sep 18 17:09:35.736293 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:35.736268    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9"
Sep 18 17:09:35.740097 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:35.740066    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Sep 18 17:09:35.740288 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:35.740116    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Sep 18 17:09:35.757454 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:35.757423    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9"]
Sep 18 17:09:35.824375 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:35.824330    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d00ab19-e542-4ced-ae6f-b26074cb25e4-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9\" (UID: \"5d00ab19-e542-4ced-ae6f-b26074cb25e4\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9"
Sep 18 17:09:35.824375 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:35.824376    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d00ab19-e542-4ced-ae6f-b26074cb25e4-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9\" (UID: \"5d00ab19-e542-4ced-ae6f-b26074cb25e4\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9"
Sep 18 17:09:35.824621 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:35.824413    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d00ab19-e542-4ced-ae6f-b26074cb25e4-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9\" (UID: \"5d00ab19-e542-4ced-ae6f-b26074cb25e4\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9"
Sep 18 17:09:35.824621 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:35.824431    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5d00ab19-e542-4ced-ae6f-b26074cb25e4-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9\" (UID: \"5d00ab19-e542-4ced-ae6f-b26074cb25e4\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9"
Sep 18 17:09:35.824621 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:35.824459    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnlrq\" (UniqueName: \"kubernetes.io/projected/5d00ab19-e542-4ced-ae6f-b26074cb25e4-kube-api-access-gnlrq\") pod \"isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9\" (UID: \"5d00ab19-e542-4ced-ae6f-b26074cb25e4\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9"
Sep 18 17:09:35.925155 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:35.925118    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d00ab19-e542-4ced-ae6f-b26074cb25e4-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9\" (UID: \"5d00ab19-e542-4ced-ae6f-b26074cb25e4\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9"
Sep 18 17:09:35.925155 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:35.925169    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d00ab19-e542-4ced-ae6f-b26074cb25e4-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9\" (UID: \"5d00ab19-e542-4ced-ae6f-b26074cb25e4\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9"
Sep 18 17:09:35.925476 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:35.925215    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d00ab19-e542-4ced-ae6f-b26074cb25e4-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9\" (UID: \"5d00ab19-e542-4ced-ae6f-b26074cb25e4\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9"
Sep 18 17:09:35.925476 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:35.925270    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5d00ab19-e542-4ced-ae6f-b26074cb25e4-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9\" (UID: \"5d00ab19-e542-4ced-ae6f-b26074cb25e4\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9"
Sep 18 17:09:35.925476 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:35.925316    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnlrq\" (UniqueName: \"kubernetes.io/projected/5d00ab19-e542-4ced-ae6f-b26074cb25e4-kube-api-access-gnlrq\") pod \"isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9\" (UID: \"5d00ab19-e542-4ced-ae6f-b26074cb25e4\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9"
Sep 18 17:09:35.925476 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:09:35.925368    2881 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-serving-cert: secret "isvc-sklearn-batcher-custom-predictor-serving-cert" not found
Sep 18 17:09:35.925476 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:09:35.925437    2881 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d00ab19-e542-4ced-ae6f-b26074cb25e4-proxy-tls podName:5d00ab19-e542-4ced-ae6f-b26074cb25e4 nodeName:}" failed. No retries permitted until 2026-09-18 17:09:36.425417751 +0000 UTC m=+966.625273014 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5d00ab19-e542-4ced-ae6f-b26074cb25e4-proxy-tls") pod "isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9" (UID: "5d00ab19-e542-4ced-ae6f-b26074cb25e4") : secret "isvc-sklearn-batcher-custom-predictor-serving-cert" not found
Sep 18 17:09:35.925701 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:35.925666    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d00ab19-e542-4ced-ae6f-b26074cb25e4-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9\" (UID: \"5d00ab19-e542-4ced-ae6f-b26074cb25e4\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9"
Sep 18 17:09:35.926027 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:35.926004    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5d00ab19-e542-4ced-ae6f-b26074cb25e4-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9\" (UID: \"5d00ab19-e542-4ced-ae6f-b26074cb25e4\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9"
Sep 18 17:09:35.926070 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:35.926004    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d00ab19-e542-4ced-ae6f-b26074cb25e4-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9\" (UID: \"5d00ab19-e542-4ced-ae6f-b26074cb25e4\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9"
Sep 18 17:09:35.937576 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:35.937539    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnlrq\" (UniqueName: \"kubernetes.io/projected/5d00ab19-e542-4ced-ae6f-b26074cb25e4-kube-api-access-gnlrq\") pod \"isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9\" (UID: \"5d00ab19-e542-4ced-ae6f-b26074cb25e4\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9"
Sep 18 17:09:36.249356 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:36.249311    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d" podUID="bc48814e-b773-45f3-be4f-e13aa2f69dab" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Sep 18 17:09:36.431068 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:36.431021    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d00ab19-e542-4ced-ae6f-b26074cb25e4-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9\" (UID: \"5d00ab19-e542-4ced-ae6f-b26074cb25e4\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9"
Sep 18 17:09:36.433928 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:36.433883    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d00ab19-e542-4ced-ae6f-b26074cb25e4-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9\" (UID: \"5d00ab19-e542-4ced-ae6f-b26074cb25e4\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9"
Sep 18 17:09:36.647957 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:36.647904    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9"
Sep 18 17:09:36.927482 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:36.927452    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9"]
Sep 18 17:09:36.930236 ip-10-0-143-5 kubenswrapper[2881]: W0918 17:09:36.930190    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d00ab19_e542_4ced_ae6f_b26074cb25e4.slice/crio-7f2d41ef8e265c2a5221e685a87974a9677ca670e38e0ca6c1ae62b8077f9419 WatchSource:0}: Error finding container 7f2d41ef8e265c2a5221e685a87974a9677ca670e38e0ca6c1ae62b8077f9419: Status 404 returned error can't find the container with id 7f2d41ef8e265c2a5221e685a87974a9677ca670e38e0ca6c1ae62b8077f9419
Sep 18 17:09:37.255062 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:37.254955    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d" podUID="bc48814e-b773-45f3-be4f-e13aa2f69dab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 18 17:09:37.255284 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:37.255258    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d" podUID="bc48814e-b773-45f3-be4f-e13aa2f69dab" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 17:09:37.726386 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:37.726347    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9" event={"ID":"5d00ab19-e542-4ced-ae6f-b26074cb25e4","Type":"ContainerStarted","Data":"3b8267f66a6de7ca3e4ef9d17ee059d474dc65677fc2c14925a6f9c8dcb81e9c"}
Sep 18 17:09:37.726901 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:37.726416    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9" event={"ID":"5d00ab19-e542-4ced-ae6f-b26074cb25e4","Type":"ContainerStarted","Data":"7f2d41ef8e265c2a5221e685a87974a9677ca670e38e0ca6c1ae62b8077f9419"}
Sep 18 17:09:41.249212 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:41.249102    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d" podUID="bc48814e-b773-45f3-be4f-e13aa2f69dab" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Sep 18 17:09:41.749084 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:41.749049    2881 generic.go:356] "Generic (PLEG): container finished" podID="5d00ab19-e542-4ced-ae6f-b26074cb25e4" containerID="3b8267f66a6de7ca3e4ef9d17ee059d474dc65677fc2c14925a6f9c8dcb81e9c" exitCode=0
Sep 18 17:09:41.749354 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:41.749119    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9" event={"ID":"5d00ab19-e542-4ced-ae6f-b26074cb25e4","Type":"ContainerDied","Data":"3b8267f66a6de7ca3e4ef9d17ee059d474dc65677fc2c14925a6f9c8dcb81e9c"}
Sep 18 17:09:41.752085 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:41.752063    2881 generic.go:356] "Generic (PLEG): container finished" podID="bc48814e-b773-45f3-be4f-e13aa2f69dab" containerID="8663b43327ed0b2630ada698429fdb817d468e66456c47edb7ee233d0944b484" exitCode=0
Sep 18 17:09:41.752192 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:41.752117    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d" event={"ID":"bc48814e-b773-45f3-be4f-e13aa2f69dab","Type":"ContainerDied","Data":"8663b43327ed0b2630ada698429fdb817d468e66456c47edb7ee233d0944b484"}
Sep 18 17:09:42.760580 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:42.760545    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9" event={"ID":"5d00ab19-e542-4ced-ae6f-b26074cb25e4","Type":"ContainerStarted","Data":"00baad68da4a8bb0e7930d95abf32289d682f05c53f28d5ad67a4827b2cdfb08"}
Sep 18 17:09:42.761082 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:42.760632    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9" event={"ID":"5d00ab19-e542-4ced-ae6f-b26074cb25e4","Type":"ContainerStarted","Data":"052766904b7b8d3d4e4b59e3621f63ff54a65f4cd42098443c7ea6e7d8f93bd3"}
Sep 18 17:09:42.761082 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:42.760677    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9" event={"ID":"5d00ab19-e542-4ced-ae6f-b26074cb25e4","Type":"ContainerStarted","Data":"edb44ae0f5fc4a2f7d8856a9c070982a27771e394fa2c3d151168b3d0be46f15"}
Sep 18 17:09:42.761082 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:42.760937    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9"
Sep 18 17:09:42.802813 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:42.802759    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9" podStartSLOduration=4.044241508 podStartE2EDuration="7.802740438s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.75849893s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:09:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:09:42.801473103 +0000 UTC m=+973.001328391" watchObservedRunningTime="2026-09-18 17:09:42.802740438 +0000 UTC m=+973.002595724"
Sep 18 17:09:43.765761 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:43.765725    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9"
Sep 18 17:09:43.766164 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:43.765796    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9"
Sep 18 17:09:43.767233 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:43.767178    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9" podUID="5d00ab19-e542-4ced-ae6f-b26074cb25e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:5000: connect: connection refused"
Sep 18 17:09:43.767862 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:43.767829    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9" podUID="5d00ab19-e542-4ced-ae6f-b26074cb25e4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 17:09:44.770758 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:44.770705    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9" podUID="5d00ab19-e542-4ced-ae6f-b26074cb25e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:5000: connect: connection refused"
Sep 18 17:09:44.771201 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:44.771177    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9" podUID="5d00ab19-e542-4ced-ae6f-b26074cb25e4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 17:09:45.403502 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:45.403449    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t5n2z" podUID="f9e8938f-6241-4ade-8b71-4bd14398fc4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 17:09:46.249141 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:46.249094    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d" podUID="bc48814e-b773-45f3-be4f-e13aa2f69dab" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Sep 18 17:09:46.249610 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:46.249249    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d"
Sep 18 17:09:47.255197 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:47.255141    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d" podUID="bc48814e-b773-45f3-be4f-e13aa2f69dab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 18 17:09:47.255701 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:47.255467    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d" podUID="bc48814e-b773-45f3-be4f-e13aa2f69dab" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 17:09:49.775419 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:49.775387    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9"
Sep 18 17:09:49.776102 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:49.776073    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9" podUID="5d00ab19-e542-4ced-ae6f-b26074cb25e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:5000: connect: connection refused"
Sep 18 17:09:49.776454 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:49.776424    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9" podUID="5d00ab19-e542-4ced-ae6f-b26074cb25e4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 17:09:51.249213 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:51.249172    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d" podUID="bc48814e-b773-45f3-be4f-e13aa2f69dab" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Sep 18 17:09:55.404200 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:55.404169    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t5n2z"
Sep 18 17:09:56.249026 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:56.248981    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d" podUID="bc48814e-b773-45f3-be4f-e13aa2f69dab" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Sep 18 17:09:57.255499 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:57.255449    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d" podUID="bc48814e-b773-45f3-be4f-e13aa2f69dab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 18 17:09:57.255959 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:57.255601    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d"
Sep 18 17:09:57.255959 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:57.255691    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d" podUID="bc48814e-b773-45f3-be4f-e13aa2f69dab" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 17:09:57.255959 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:57.255803    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d"
Sep 18 17:09:59.776786 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:59.776735    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9" podUID="5d00ab19-e542-4ced-ae6f-b26074cb25e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:5000: connect: connection refused"
Sep 18 17:09:59.777334 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:09:59.777210    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9" podUID="5d00ab19-e542-4ced-ae6f-b26074cb25e4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 17:10:01.249135 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:01.249088    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d" podUID="bc48814e-b773-45f3-be4f-e13aa2f69dab" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Sep 18 17:10:05.588550 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:05.588505    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t5n2z"]
Sep 18 17:10:05.589013 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:05.588875    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t5n2z" podUID="f9e8938f-6241-4ade-8b71-4bd14398fc4c" containerName="kserve-container" containerID="cri-o://b7bcfcdeb0d6090c024a1b055d6b2ae01ce777571afd1009362e7dafec344df5" gracePeriod=30
Sep 18 17:10:05.589392 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:05.589169    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t5n2z" podUID="f9e8938f-6241-4ade-8b71-4bd14398fc4c" containerName="kube-rbac-proxy" containerID="cri-o://6adc13e7e8b4d155d01258372796e13abd6f2d10ea26084a5fcfc7fa6d0378fa" gracePeriod=30
Sep 18 17:10:05.716985 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:05.716956    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d"
Sep 18 17:10:05.814830 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:05.814795    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc48814e-b773-45f3-be4f-e13aa2f69dab-kube-rbac-proxy-sar-config\") pod \"bc48814e-b773-45f3-be4f-e13aa2f69dab\" (UID: \"bc48814e-b773-45f3-be4f-e13aa2f69dab\") "
Sep 18 17:10:05.815020 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:05.814894    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc48814e-b773-45f3-be4f-e13aa2f69dab-proxy-tls\") pod \"bc48814e-b773-45f3-be4f-e13aa2f69dab\" (UID: \"bc48814e-b773-45f3-be4f-e13aa2f69dab\") "
Sep 18 17:10:05.815020 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:05.814929    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bc48814e-b773-45f3-be4f-e13aa2f69dab-cabundle-cert\") pod \"bc48814e-b773-45f3-be4f-e13aa2f69dab\" (UID: \"bc48814e-b773-45f3-be4f-e13aa2f69dab\") "
Sep 18 17:10:05.815020 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:05.814948    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bc48814e-b773-45f3-be4f-e13aa2f69dab-kserve-provision-location\") pod \"bc48814e-b773-45f3-be4f-e13aa2f69dab\" (UID: \"bc48814e-b773-45f3-be4f-e13aa2f69dab\") "
Sep 18 17:10:05.815130 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:05.815019    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-45j5q\" (UniqueName: \"kubernetes.io/projected/bc48814e-b773-45f3-be4f-e13aa2f69dab-kube-api-access-45j5q\") pod \"bc48814e-b773-45f3-be4f-e13aa2f69dab\" (UID: \"bc48814e-b773-45f3-be4f-e13aa2f69dab\") "
Sep 18 17:10:05.815316 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:05.815286    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc48814e-b773-45f3-be4f-e13aa2f69dab-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bc48814e-b773-45f3-be4f-e13aa2f69dab" (UID: "bc48814e-b773-45f3-be4f-e13aa2f69dab"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:10:05.815381 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:05.815333    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc48814e-b773-45f3-be4f-e13aa2f69dab-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bc48814e-b773-45f3-be4f-e13aa2f69dab" (UID: "bc48814e-b773-45f3-be4f-e13aa2f69dab"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:10:05.815381 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:05.815335    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc48814e-b773-45f3-be4f-e13aa2f69dab-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bc48814e-b773-45f3-be4f-e13aa2f69dab" (UID: "bc48814e-b773-45f3-be4f-e13aa2f69dab"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:10:05.817138 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:05.817113    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc48814e-b773-45f3-be4f-e13aa2f69dab-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bc48814e-b773-45f3-be4f-e13aa2f69dab" (UID: "bc48814e-b773-45f3-be4f-e13aa2f69dab"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:10:05.817208 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:05.817151    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc48814e-b773-45f3-be4f-e13aa2f69dab-kube-api-access-45j5q" (OuterVolumeSpecName: "kube-api-access-45j5q") pod "bc48814e-b773-45f3-be4f-e13aa2f69dab" (UID: "bc48814e-b773-45f3-be4f-e13aa2f69dab"). InnerVolumeSpecName "kube-api-access-45j5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:10:05.881496 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:05.881460    2881 generic.go:356] "Generic (PLEG): container finished" podID="f9e8938f-6241-4ade-8b71-4bd14398fc4c" containerID="6adc13e7e8b4d155d01258372796e13abd6f2d10ea26084a5fcfc7fa6d0378fa" exitCode=2
Sep 18 17:10:05.881688 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:05.881534    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t5n2z" event={"ID":"f9e8938f-6241-4ade-8b71-4bd14398fc4c","Type":"ContainerDied","Data":"6adc13e7e8b4d155d01258372796e13abd6f2d10ea26084a5fcfc7fa6d0378fa"}
Sep 18 17:10:05.883811 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:05.883787    2881 generic.go:356] "Generic (PLEG): container finished" podID="bc48814e-b773-45f3-be4f-e13aa2f69dab" containerID="52fdd6c16bba1635e1138741889f6b0b9b9366753c490df6eda8c7655d019458" exitCode=0
Sep 18 17:10:05.883889 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:05.883840    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d" event={"ID":"bc48814e-b773-45f3-be4f-e13aa2f69dab","Type":"ContainerDied","Data":"52fdd6c16bba1635e1138741889f6b0b9b9366753c490df6eda8c7655d019458"}
Sep 18 17:10:05.883889 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:05.883863    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d"
Sep 18 17:10:05.883976 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:05.883908    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d" event={"ID":"bc48814e-b773-45f3-be4f-e13aa2f69dab","Type":"ContainerDied","Data":"a3d78882fce450338fa7a2fe3b3a6e22e8e16256b2870fa298e29def568d64bd"}
Sep 18 17:10:05.883976 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:05.883952    2881 scope.go:117] "RemoveContainer" containerID="52fdd6c16bba1635e1138741889f6b0b9b9366753c490df6eda8c7655d019458"
Sep 18 17:10:05.893718 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:05.893689    2881 scope.go:117] "RemoveContainer" containerID="8e6a873386f6d8763767378e27954a2796d755a98f42296db16190cf2f2f8956"
Sep 18 17:10:05.902795 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:05.902776    2881 scope.go:117] "RemoveContainer" containerID="8663b43327ed0b2630ada698429fdb817d468e66456c47edb7ee233d0944b484"
Sep 18 17:10:05.911464 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:05.911442    2881 scope.go:117] "RemoveContainer" containerID="e5242ae64fc92e532256b64dda35efcdf0d44d0ad931a9f2e69a6468d0bb4cf7"
Sep 18 17:10:05.912058 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:05.912030    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d"]
Sep 18 17:10:05.916124 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:05.916096    2881 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc48814e-b773-45f3-be4f-e13aa2f69dab-proxy-tls\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:10:05.916336 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:05.916129    2881 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bc48814e-b773-45f3-be4f-e13aa2f69dab-cabundle-cert\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:10:05.916336 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:05.916145    2881 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bc48814e-b773-45f3-be4f-e13aa2f69dab-kserve-provision-location\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:10:05.916336 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:05.916161    2881 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45j5q\" (UniqueName: \"kubernetes.io/projected/bc48814e-b773-45f3-be4f-e13aa2f69dab-kube-api-access-45j5q\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:10:05.916336 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:05.916175    2881 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc48814e-b773-45f3-be4f-e13aa2f69dab-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:10:05.920218 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:05.920186    2881 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5f9b9c4f65-ffx4d"]
Sep 18 17:10:05.921010 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:05.920992    2881 scope.go:117] "RemoveContainer" containerID="52fdd6c16bba1635e1138741889f6b0b9b9366753c490df6eda8c7655d019458"
Sep 18 17:10:05.921397 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:10:05.921376    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52fdd6c16bba1635e1138741889f6b0b9b9366753c490df6eda8c7655d019458\": container with ID starting with 52fdd6c16bba1635e1138741889f6b0b9b9366753c490df6eda8c7655d019458 not found: ID does not exist" containerID="52fdd6c16bba1635e1138741889f6b0b9b9366753c490df6eda8c7655d019458"
Sep 18 17:10:05.921463 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:05.921407    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52fdd6c16bba1635e1138741889f6b0b9b9366753c490df6eda8c7655d019458"} err="failed to get container status \"52fdd6c16bba1635e1138741889f6b0b9b9366753c490df6eda8c7655d019458\": rpc error: code = NotFound desc = could not find container \"52fdd6c16bba1635e1138741889f6b0b9b9366753c490df6eda8c7655d019458\": container with ID starting with 52fdd6c16bba1635e1138741889f6b0b9b9366753c490df6eda8c7655d019458 not found: ID does not exist"
Sep 18 17:10:05.921463 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:05.921426    2881 scope.go:117] "RemoveContainer" containerID="8e6a873386f6d8763767378e27954a2796d755a98f42296db16190cf2f2f8956"
Sep 18 17:10:05.921679 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:10:05.921659    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e6a873386f6d8763767378e27954a2796d755a98f42296db16190cf2f2f8956\": container with ID starting with 8e6a873386f6d8763767378e27954a2796d755a98f42296db16190cf2f2f8956 not found: ID does not exist" containerID="8e6a873386f6d8763767378e27954a2796d755a98f42296db16190cf2f2f8956"
Sep 18 17:10:05.921719 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:05.921686    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e6a873386f6d8763767378e27954a2796d755a98f42296db16190cf2f2f8956"} err="failed to get container status \"8e6a873386f6d8763767378e27954a2796d755a98f42296db16190cf2f2f8956\": rpc error: code = NotFound desc = could not find container \"8e6a873386f6d8763767378e27954a2796d755a98f42296db16190cf2f2f8956\": container with ID starting with 8e6a873386f6d8763767378e27954a2796d755a98f42296db16190cf2f2f8956 not found: ID does not exist"
Sep 18 17:10:05.921719 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:05.921704    2881 scope.go:117] "RemoveContainer" containerID="8663b43327ed0b2630ada698429fdb817d468e66456c47edb7ee233d0944b484"
Sep 18 17:10:05.921943 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:10:05.921927    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8663b43327ed0b2630ada698429fdb817d468e66456c47edb7ee233d0944b484\": container with ID starting with 8663b43327ed0b2630ada698429fdb817d468e66456c47edb7ee233d0944b484 not found: ID does not exist" containerID="8663b43327ed0b2630ada698429fdb817d468e66456c47edb7ee233d0944b484"
Sep 18 17:10:05.921985 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:05.921947    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8663b43327ed0b2630ada698429fdb817d468e66456c47edb7ee233d0944b484"} err="failed to get container status \"8663b43327ed0b2630ada698429fdb817d468e66456c47edb7ee233d0944b484\": rpc error: code = NotFound desc = could not find container \"8663b43327ed0b2630ada698429fdb817d468e66456c47edb7ee233d0944b484\": container with ID starting with 8663b43327ed0b2630ada698429fdb817d468e66456c47edb7ee233d0944b484 not found: ID does not exist"
Sep 18 17:10:05.921985 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:05.921960    2881 scope.go:117] "RemoveContainer" containerID="e5242ae64fc92e532256b64dda35efcdf0d44d0ad931a9f2e69a6468d0bb4cf7"
Sep 18 17:10:05.922173 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:10:05.922157    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5242ae64fc92e532256b64dda35efcdf0d44d0ad931a9f2e69a6468d0bb4cf7\": container with ID starting with e5242ae64fc92e532256b64dda35efcdf0d44d0ad931a9f2e69a6468d0bb4cf7 not found: ID does not exist" containerID="e5242ae64fc92e532256b64dda35efcdf0d44d0ad931a9f2e69a6468d0bb4cf7"
Sep 18 17:10:05.922213 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:05.922176    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5242ae64fc92e532256b64dda35efcdf0d44d0ad931a9f2e69a6468d0bb4cf7"} err="failed to get container status \"e5242ae64fc92e532256b64dda35efcdf0d44d0ad931a9f2e69a6468d0bb4cf7\": rpc error: code = NotFound desc = could not find container \"e5242ae64fc92e532256b64dda35efcdf0d44d0ad931a9f2e69a6468d0bb4cf7\": container with ID starting with e5242ae64fc92e532256b64dda35efcdf0d44d0ad931a9f2e69a6468d0bb4cf7 not found: ID does not exist"
Sep 18 17:10:07.893943 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:07.893910    2881 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc48814e-b773-45f3-be4f-e13aa2f69dab" path="/var/lib/kubelet/pods/bc48814e-b773-45f3-be4f-e13aa2f69dab/volumes"
Sep 18 17:10:09.776120 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:09.776077    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9" podUID="5d00ab19-e542-4ced-ae6f-b26074cb25e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:5000: connect: connection refused"
Sep 18 17:10:09.776644 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:09.776619    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9" podUID="5d00ab19-e542-4ced-ae6f-b26074cb25e4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 17:10:10.397741 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:10.397691    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t5n2z" podUID="f9e8938f-6241-4ade-8b71-4bd14398fc4c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Sep 18 17:10:10.809772 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:10.809734    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z8fcb"]
Sep 18 17:10:10.810189 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:10.810127    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc48814e-b773-45f3-be4f-e13aa2f69dab" containerName="storage-initializer"
Sep 18 17:10:10.810189 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:10.810139    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc48814e-b773-45f3-be4f-e13aa2f69dab" containerName="storage-initializer"
Sep 18 17:10:10.810189 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:10.810147    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc48814e-b773-45f3-be4f-e13aa2f69dab" containerName="agent"
Sep 18 17:10:10.810189 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:10.810153    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc48814e-b773-45f3-be4f-e13aa2f69dab" containerName="agent"
Sep 18 17:10:10.810189 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:10.810164    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc48814e-b773-45f3-be4f-e13aa2f69dab" containerName="kube-rbac-proxy"
Sep 18 17:10:10.810189 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:10.810170    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc48814e-b773-45f3-be4f-e13aa2f69dab" containerName="kube-rbac-proxy"
Sep 18 17:10:10.810189 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:10.810189    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc48814e-b773-45f3-be4f-e13aa2f69dab" containerName="kserve-container"
Sep 18 17:10:10.810189 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:10.810195    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc48814e-b773-45f3-be4f-e13aa2f69dab" containerName="kserve-container"
Sep 18 17:10:10.810504 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:10.810292    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc48814e-b773-45f3-be4f-e13aa2f69dab" containerName="kube-rbac-proxy"
Sep 18 17:10:10.810504 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:10.810299    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc48814e-b773-45f3-be4f-e13aa2f69dab" containerName="agent"
Sep 18 17:10:10.810504 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:10.810306    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc48814e-b773-45f3-be4f-e13aa2f69dab" containerName="kserve-container"
Sep 18 17:10:10.812754 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:10.812736    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z8fcb"
Sep 18 17:10:10.815394 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:10.815370    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-predictor-serving-cert\""
Sep 18 17:10:10.815516 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:10.815417    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\""
Sep 18 17:10:10.829119 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:10.829088    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z8fcb"]
Sep 18 17:10:10.962406 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:10.962361    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b9b0bfc6-1476-4dd1-a232-5281f1052623-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-z8fcb\" (UID: \"b9b0bfc6-1476-4dd1-a232-5281f1052623\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z8fcb"
Sep 18 17:10:10.962611 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:10.962474    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b9b0bfc6-1476-4dd1-a232-5281f1052623-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-z8fcb\" (UID: \"b9b0bfc6-1476-4dd1-a232-5281f1052623\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z8fcb"
Sep 18 17:10:10.962611 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:10.962537    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9b0bfc6-1476-4dd1-a232-5281f1052623-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-z8fcb\" (UID: \"b9b0bfc6-1476-4dd1-a232-5281f1052623\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z8fcb"
Sep 18 17:10:10.962611 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:10.962575    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km2zd\" (UniqueName: \"kubernetes.io/projected/b9b0bfc6-1476-4dd1-a232-5281f1052623-kube-api-access-km2zd\") pod \"isvc-predictive-xgboost-predictor-695f978978-z8fcb\" (UID: \"b9b0bfc6-1476-4dd1-a232-5281f1052623\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z8fcb"
Sep 18 17:10:10.962768 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:10.962639    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9b0bfc6-1476-4dd1-a232-5281f1052623-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-z8fcb\" (UID: \"b9b0bfc6-1476-4dd1-a232-5281f1052623\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z8fcb"
Sep 18 17:10:11.063935 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:11.063833    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9b0bfc6-1476-4dd1-a232-5281f1052623-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-z8fcb\" (UID: \"b9b0bfc6-1476-4dd1-a232-5281f1052623\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z8fcb"
Sep 18 17:10:11.063935 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:11.063892    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km2zd\" (UniqueName: \"kubernetes.io/projected/b9b0bfc6-1476-4dd1-a232-5281f1052623-kube-api-access-km2zd\") pod \"isvc-predictive-xgboost-predictor-695f978978-z8fcb\" (UID: \"b9b0bfc6-1476-4dd1-a232-5281f1052623\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z8fcb"
Sep 18 17:10:11.063935 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:11.063930    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9b0bfc6-1476-4dd1-a232-5281f1052623-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-z8fcb\" (UID: \"b9b0bfc6-1476-4dd1-a232-5281f1052623\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z8fcb"
Sep 18 17:10:11.064278 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:11.063985    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b9b0bfc6-1476-4dd1-a232-5281f1052623-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-z8fcb\" (UID: \"b9b0bfc6-1476-4dd1-a232-5281f1052623\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z8fcb"
Sep 18 17:10:11.064278 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:11.064044    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b9b0bfc6-1476-4dd1-a232-5281f1052623-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-z8fcb\" (UID: \"b9b0bfc6-1476-4dd1-a232-5281f1052623\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z8fcb"
Sep 18 17:10:11.064528 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:11.064504    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b9b0bfc6-1476-4dd1-a232-5281f1052623-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-z8fcb\" (UID: \"b9b0bfc6-1476-4dd1-a232-5281f1052623\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z8fcb"
Sep 18 17:10:11.064790 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:11.064767    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b9b0bfc6-1476-4dd1-a232-5281f1052623-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-z8fcb\" (UID: \"b9b0bfc6-1476-4dd1-a232-5281f1052623\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z8fcb"
Sep 18 17:10:11.064848 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:11.064768    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9b0bfc6-1476-4dd1-a232-5281f1052623-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-z8fcb\" (UID: \"b9b0bfc6-1476-4dd1-a232-5281f1052623\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z8fcb"
Sep 18 17:10:11.066584 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:11.066565    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9b0bfc6-1476-4dd1-a232-5281f1052623-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-z8fcb\" (UID: \"b9b0bfc6-1476-4dd1-a232-5281f1052623\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z8fcb"
Sep 18 17:10:11.074500 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:11.074470    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-km2zd\" (UniqueName: \"kubernetes.io/projected/b9b0bfc6-1476-4dd1-a232-5281f1052623-kube-api-access-km2zd\") pod \"isvc-predictive-xgboost-predictor-695f978978-z8fcb\" (UID: \"b9b0bfc6-1476-4dd1-a232-5281f1052623\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z8fcb"
Sep 18 17:10:11.123975 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:11.123934    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z8fcb"
Sep 18 17:10:11.416673 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:11.416496    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z8fcb"]
Sep 18 17:10:11.419485 ip-10-0-143-5 kubenswrapper[2881]: W0918 17:10:11.419456    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9b0bfc6_1476_4dd1_a232_5281f1052623.slice/crio-aa537ec76b74f91e198debf153f68e20e4949c28c2cd14e14644ca3701e94d84 WatchSource:0}: Error finding container aa537ec76b74f91e198debf153f68e20e4949c28c2cd14e14644ca3701e94d84: Status 404 returned error can't find the container with id aa537ec76b74f91e198debf153f68e20e4949c28c2cd14e14644ca3701e94d84
Sep 18 17:10:11.729331 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:11.729305    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t5n2z"
Sep 18 17:10:11.871882 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:11.871842    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9e8938f-6241-4ade-8b71-4bd14398fc4c-proxy-tls\") pod \"f9e8938f-6241-4ade-8b71-4bd14398fc4c\" (UID: \"f9e8938f-6241-4ade-8b71-4bd14398fc4c\") "
Sep 18 17:10:11.871882 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:11.871888    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9e8938f-6241-4ade-8b71-4bd14398fc4c-cabundle-cert\") pod \"f9e8938f-6241-4ade-8b71-4bd14398fc4c\" (UID: \"f9e8938f-6241-4ade-8b71-4bd14398fc4c\") "
Sep 18 17:10:11.872451 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:11.871944    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9e8938f-6241-4ade-8b71-4bd14398fc4c-kserve-provision-location\") pod \"f9e8938f-6241-4ade-8b71-4bd14398fc4c\" (UID: \"f9e8938f-6241-4ade-8b71-4bd14398fc4c\") "
Sep 18 17:10:11.872451 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:11.871963    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kp6tx\" (UniqueName: \"kubernetes.io/projected/f9e8938f-6241-4ade-8b71-4bd14398fc4c-kube-api-access-kp6tx\") pod \"f9e8938f-6241-4ade-8b71-4bd14398fc4c\" (UID: \"f9e8938f-6241-4ade-8b71-4bd14398fc4c\") "
Sep 18 17:10:11.872451 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:11.872012    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9e8938f-6241-4ade-8b71-4bd14398fc4c-kube-rbac-proxy-sar-config\") pod \"f9e8938f-6241-4ade-8b71-4bd14398fc4c\" (UID: \"f9e8938f-6241-4ade-8b71-4bd14398fc4c\") "
Sep 18 17:10:11.872451 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:11.872346    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9e8938f-6241-4ade-8b71-4bd14398fc4c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f9e8938f-6241-4ade-8b71-4bd14398fc4c" (UID: "f9e8938f-6241-4ade-8b71-4bd14398fc4c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:10:11.872451 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:11.872349    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9e8938f-6241-4ade-8b71-4bd14398fc4c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f9e8938f-6241-4ade-8b71-4bd14398fc4c" (UID: "f9e8938f-6241-4ade-8b71-4bd14398fc4c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:10:11.872652 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:11.872483    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9e8938f-6241-4ade-8b71-4bd14398fc4c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f9e8938f-6241-4ade-8b71-4bd14398fc4c" (UID: "f9e8938f-6241-4ade-8b71-4bd14398fc4c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:10:11.874686 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:11.874665    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9e8938f-6241-4ade-8b71-4bd14398fc4c-kube-api-access-kp6tx" (OuterVolumeSpecName: "kube-api-access-kp6tx") pod "f9e8938f-6241-4ade-8b71-4bd14398fc4c" (UID: "f9e8938f-6241-4ade-8b71-4bd14398fc4c"). InnerVolumeSpecName "kube-api-access-kp6tx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:10:11.874782 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:11.874681    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9e8938f-6241-4ade-8b71-4bd14398fc4c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f9e8938f-6241-4ade-8b71-4bd14398fc4c" (UID: "f9e8938f-6241-4ade-8b71-4bd14398fc4c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:10:11.917240 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:11.917197    2881 generic.go:356] "Generic (PLEG): container finished" podID="f9e8938f-6241-4ade-8b71-4bd14398fc4c" containerID="b7bcfcdeb0d6090c024a1b055d6b2ae01ce777571afd1009362e7dafec344df5" exitCode=0
Sep 18 17:10:11.917466 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:11.917271    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t5n2z" event={"ID":"f9e8938f-6241-4ade-8b71-4bd14398fc4c","Type":"ContainerDied","Data":"b7bcfcdeb0d6090c024a1b055d6b2ae01ce777571afd1009362e7dafec344df5"}
Sep 18 17:10:11.917466 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:11.917298    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t5n2z"
Sep 18 17:10:11.917466 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:11.917354    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t5n2z" event={"ID":"f9e8938f-6241-4ade-8b71-4bd14398fc4c","Type":"ContainerDied","Data":"d1941789539df4b1bd195363fc0b99493fb7c2eded35f3c2305099ed7db549b5"}
Sep 18 17:10:11.917466 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:11.917394    2881 scope.go:117] "RemoveContainer" containerID="6adc13e7e8b4d155d01258372796e13abd6f2d10ea26084a5fcfc7fa6d0378fa"
Sep 18 17:10:11.919703 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:11.919660    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z8fcb" event={"ID":"b9b0bfc6-1476-4dd1-a232-5281f1052623","Type":"ContainerStarted","Data":"5c4bda3f1791f7a8fdb3a3fcff5f2e27e09c84aea2f1fdcbfd03fa407188b0ef"}
Sep 18 17:10:11.919848 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:11.919744    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z8fcb" event={"ID":"b9b0bfc6-1476-4dd1-a232-5281f1052623","Type":"ContainerStarted","Data":"aa537ec76b74f91e198debf153f68e20e4949c28c2cd14e14644ca3701e94d84"}
Sep 18 17:10:11.927314 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:11.927294    2881 scope.go:117] "RemoveContainer" containerID="b7bcfcdeb0d6090c024a1b055d6b2ae01ce777571afd1009362e7dafec344df5"
Sep 18 17:10:11.937710 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:11.937687    2881 scope.go:117] "RemoveContainer" containerID="0e60dd3e7159ad3d721a139802c1220307008df1ddce2cdca13f5abedf57b512"
Sep 18 17:10:11.947588 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:11.947566    2881 scope.go:117] "RemoveContainer" containerID="6adc13e7e8b4d155d01258372796e13abd6f2d10ea26084a5fcfc7fa6d0378fa"
Sep 18 17:10:11.947979 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:10:11.947961    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6adc13e7e8b4d155d01258372796e13abd6f2d10ea26084a5fcfc7fa6d0378fa\": container with ID starting with 6adc13e7e8b4d155d01258372796e13abd6f2d10ea26084a5fcfc7fa6d0378fa not found: ID does not exist" containerID="6adc13e7e8b4d155d01258372796e13abd6f2d10ea26084a5fcfc7fa6d0378fa"
Sep 18 17:10:11.948032 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:11.947988    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6adc13e7e8b4d155d01258372796e13abd6f2d10ea26084a5fcfc7fa6d0378fa"} err="failed to get container status \"6adc13e7e8b4d155d01258372796e13abd6f2d10ea26084a5fcfc7fa6d0378fa\": rpc error: code = NotFound desc = could not find container \"6adc13e7e8b4d155d01258372796e13abd6f2d10ea26084a5fcfc7fa6d0378fa\": container with ID starting with 6adc13e7e8b4d155d01258372796e13abd6f2d10ea26084a5fcfc7fa6d0378fa not found: ID does not exist"
Sep 18 17:10:11.948032 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:11.948007    2881 scope.go:117] "RemoveContainer" containerID="b7bcfcdeb0d6090c024a1b055d6b2ae01ce777571afd1009362e7dafec344df5"
Sep 18 17:10:11.948318 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:10:11.948293    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7bcfcdeb0d6090c024a1b055d6b2ae01ce777571afd1009362e7dafec344df5\": container with ID starting with b7bcfcdeb0d6090c024a1b055d6b2ae01ce777571afd1009362e7dafec344df5 not found: ID does not exist" containerID="b7bcfcdeb0d6090c024a1b055d6b2ae01ce777571afd1009362e7dafec344df5"
Sep 18 17:10:11.948391 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:11.948327    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7bcfcdeb0d6090c024a1b055d6b2ae01ce777571afd1009362e7dafec344df5"} err="failed to get container status \"b7bcfcdeb0d6090c024a1b055d6b2ae01ce777571afd1009362e7dafec344df5\": rpc error: code = NotFound desc = could not find container \"b7bcfcdeb0d6090c024a1b055d6b2ae01ce777571afd1009362e7dafec344df5\": container with ID starting with b7bcfcdeb0d6090c024a1b055d6b2ae01ce777571afd1009362e7dafec344df5 not found: ID does not exist"
Sep 18 17:10:11.948391 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:11.948349    2881 scope.go:117] "RemoveContainer" containerID="0e60dd3e7159ad3d721a139802c1220307008df1ddce2cdca13f5abedf57b512"
Sep 18 17:10:11.948619 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:10:11.948596    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e60dd3e7159ad3d721a139802c1220307008df1ddce2cdca13f5abedf57b512\": container with ID starting with 0e60dd3e7159ad3d721a139802c1220307008df1ddce2cdca13f5abedf57b512 not found: ID does not exist" containerID="0e60dd3e7159ad3d721a139802c1220307008df1ddce2cdca13f5abedf57b512"
Sep 18 17:10:11.948662 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:11.948623    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e60dd3e7159ad3d721a139802c1220307008df1ddce2cdca13f5abedf57b512"} err="failed to get container status \"0e60dd3e7159ad3d721a139802c1220307008df1ddce2cdca13f5abedf57b512\": rpc error: code = NotFound desc = could not find container \"0e60dd3e7159ad3d721a139802c1220307008df1ddce2cdca13f5abedf57b512\": container with ID starting with 0e60dd3e7159ad3d721a139802c1220307008df1ddce2cdca13f5abedf57b512 not found: ID does not exist"
Sep 18 17:10:11.953850 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:11.953824    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t5n2z"]
Sep 18 17:10:11.966556 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:11.966527    2881 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-t5n2z"]
Sep 18 17:10:11.972918 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:11.972880    2881 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9e8938f-6241-4ade-8b71-4bd14398fc4c-proxy-tls\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:10:11.972918 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:11.972918    2881 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9e8938f-6241-4ade-8b71-4bd14398fc4c-cabundle-cert\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:10:11.973093 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:11.972932    2881 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9e8938f-6241-4ade-8b71-4bd14398fc4c-kserve-provision-location\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:10:11.973093 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:11.972946    2881 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kp6tx\" (UniqueName: \"kubernetes.io/projected/f9e8938f-6241-4ade-8b71-4bd14398fc4c-kube-api-access-kp6tx\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:10:11.973093 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:11.972963    2881 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9e8938f-6241-4ade-8b71-4bd14398fc4c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:10:13.893059 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:13.893016    2881 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9e8938f-6241-4ade-8b71-4bd14398fc4c" path="/var/lib/kubelet/pods/f9e8938f-6241-4ade-8b71-4bd14398fc4c/volumes"
Sep 18 17:10:15.943557 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:15.943525    2881 generic.go:356] "Generic (PLEG): container finished" podID="b9b0bfc6-1476-4dd1-a232-5281f1052623" containerID="5c4bda3f1791f7a8fdb3a3fcff5f2e27e09c84aea2f1fdcbfd03fa407188b0ef" exitCode=0
Sep 18 17:10:15.944020 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:15.943603    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z8fcb" event={"ID":"b9b0bfc6-1476-4dd1-a232-5281f1052623","Type":"ContainerDied","Data":"5c4bda3f1791f7a8fdb3a3fcff5f2e27e09c84aea2f1fdcbfd03fa407188b0ef"}
Sep 18 17:10:16.950674 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:16.950636    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z8fcb" event={"ID":"b9b0bfc6-1476-4dd1-a232-5281f1052623","Type":"ContainerStarted","Data":"8887877e817b63e88b3457d1a2447057a62cf3dda2a87ea6bc96fc872a80913d"}
Sep 18 17:10:16.951094 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:16.950728    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z8fcb" event={"ID":"b9b0bfc6-1476-4dd1-a232-5281f1052623","Type":"ContainerStarted","Data":"6fa18d248a27e1f817e0181f008d59ea60ea74b42197797cfb1ad3109b3ad1a4"}
Sep 18 17:10:16.951094 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:16.951002    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z8fcb"
Sep 18 17:10:16.951173 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:16.951142    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z8fcb"
Sep 18 17:10:16.952574 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:16.952534    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z8fcb" podUID="b9b0bfc6-1476-4dd1-a232-5281f1052623" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 18 17:10:16.978308 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:16.978245    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z8fcb" podStartSLOduration=2.965798442 podStartE2EDuration="6.978210527s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.012412085s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:10:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:10:16.976960792 +0000 UTC m=+1007.176816077" watchObservedRunningTime="2026-09-18 17:10:16.978210527 +0000 UTC m=+1007.178065812"
Sep 18 17:10:17.956682 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:17.956637    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z8fcb" podUID="b9b0bfc6-1476-4dd1-a232-5281f1052623" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 18 17:10:19.776081 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:19.776034    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9" podUID="5d00ab19-e542-4ced-ae6f-b26074cb25e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:5000: connect: connection refused"
Sep 18 17:10:19.776581 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:19.776557    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9" podUID="5d00ab19-e542-4ced-ae6f-b26074cb25e4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 17:10:22.962760 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:22.962727    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z8fcb"
Sep 18 17:10:22.963448 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:22.963415    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z8fcb" podUID="b9b0bfc6-1476-4dd1-a232-5281f1052623" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 18 17:10:29.776394 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:29.776354    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9" podUID="5d00ab19-e542-4ced-ae6f-b26074cb25e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:5000: connect: connection refused"
Sep 18 17:10:29.776839 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:29.776813    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9" podUID="5d00ab19-e542-4ced-ae6f-b26074cb25e4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 17:10:32.964383 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:32.964335    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z8fcb" podUID="b9b0bfc6-1476-4dd1-a232-5281f1052623" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 18 17:10:39.776096 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:39.776045    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9" podUID="5d00ab19-e542-4ced-ae6f-b26074cb25e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:5000: connect: connection refused"
Sep 18 17:10:39.776631 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:39.776607    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9" podUID="5d00ab19-e542-4ced-ae6f-b26074cb25e4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 17:10:42.963439 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:42.963398    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z8fcb" podUID="b9b0bfc6-1476-4dd1-a232-5281f1052623" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 18 17:10:49.776909 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:49.776861    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9" podUID="5d00ab19-e542-4ced-ae6f-b26074cb25e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:5000: connect: connection refused"
Sep 18 17:10:49.777443 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:49.777247    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9" podUID="5d00ab19-e542-4ced-ae6f-b26074cb25e4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 17:10:52.964197 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:52.964151    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z8fcb" podUID="b9b0bfc6-1476-4dd1-a232-5281f1052623" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 18 17:10:53.888513 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:53.888467    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9" podUID="5d00ab19-e542-4ced-ae6f-b26074cb25e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:5000: connect: connection refused"
Sep 18 17:10:53.888804 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:10:53.888776    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9" podUID="5d00ab19-e542-4ced-ae6f-b26074cb25e4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 17:11:02.963873 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:02.963828    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z8fcb" podUID="b9b0bfc6-1476-4dd1-a232-5281f1052623" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 18 17:11:03.889364 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:03.889306    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9" podUID="5d00ab19-e542-4ced-ae6f-b26074cb25e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:5000: connect: connection refused"
Sep 18 17:11:03.889779 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:03.889749    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9" podUID="5d00ab19-e542-4ced-ae6f-b26074cb25e4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 17:11:12.963805 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:12.963754    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z8fcb" podUID="b9b0bfc6-1476-4dd1-a232-5281f1052623" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 18 17:11:13.892830 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:13.892798    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9"
Sep 18 17:11:13.893021 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:13.892894    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9"
Sep 18 17:11:21.059582 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:21.059533    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9"]
Sep 18 17:11:21.060144 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:21.059888    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9" podUID="5d00ab19-e542-4ced-ae6f-b26074cb25e4" containerName="kserve-container" containerID="cri-o://edb44ae0f5fc4a2f7d8856a9c070982a27771e394fa2c3d151168b3d0be46f15" gracePeriod=30
Sep 18 17:11:21.060144 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:21.059930    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9" podUID="5d00ab19-e542-4ced-ae6f-b26074cb25e4" containerName="kube-rbac-proxy" containerID="cri-o://052766904b7b8d3d4e4b59e3621f63ff54a65f4cd42098443c7ea6e7d8f93bd3" gracePeriod=30
Sep 18 17:11:21.060144 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:21.059894    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9" podUID="5d00ab19-e542-4ced-ae6f-b26074cb25e4" containerName="agent" containerID="cri-o://00baad68da4a8bb0e7930d95abf32289d682f05c53f28d5ad67a4827b2cdfb08" gracePeriod=30
Sep 18 17:11:21.267856 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:21.267824    2881 generic.go:356] "Generic (PLEG): container finished" podID="5d00ab19-e542-4ced-ae6f-b26074cb25e4" containerID="052766904b7b8d3d4e4b59e3621f63ff54a65f4cd42098443c7ea6e7d8f93bd3" exitCode=2
Sep 18 17:11:21.268047 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:21.267905    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9" event={"ID":"5d00ab19-e542-4ced-ae6f-b26074cb25e4","Type":"ContainerDied","Data":"052766904b7b8d3d4e4b59e3621f63ff54a65f4cd42098443c7ea6e7d8f93bd3"}
Sep 18 17:11:22.964283 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:22.964246    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z8fcb" podUID="b9b0bfc6-1476-4dd1-a232-5281f1052623" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 18 17:11:23.888433 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:23.888378    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9" podUID="5d00ab19-e542-4ced-ae6f-b26074cb25e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:5000: connect: connection refused"
Sep 18 17:11:23.888687 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:23.888661    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9" podUID="5d00ab19-e542-4ced-ae6f-b26074cb25e4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 17:11:24.771937 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:24.771888    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9" podUID="5d00ab19-e542-4ced-ae6f-b26074cb25e4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Sep 18 17:11:25.888021 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:25.887982    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z8fcb" podUID="b9b0bfc6-1476-4dd1-a232-5281f1052623" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 18 17:11:27.297372 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:27.297334    2881 generic.go:356] "Generic (PLEG): container finished" podID="5d00ab19-e542-4ced-ae6f-b26074cb25e4" containerID="edb44ae0f5fc4a2f7d8856a9c070982a27771e394fa2c3d151168b3d0be46f15" exitCode=0
Sep 18 17:11:27.297771 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:27.297405    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9" event={"ID":"5d00ab19-e542-4ced-ae6f-b26074cb25e4","Type":"ContainerDied","Data":"edb44ae0f5fc4a2f7d8856a9c070982a27771e394fa2c3d151168b3d0be46f15"}
Sep 18 17:11:29.771865 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:29.771819    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9" podUID="5d00ab19-e542-4ced-ae6f-b26074cb25e4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Sep 18 17:11:33.888914 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:33.888849    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9" podUID="5d00ab19-e542-4ced-ae6f-b26074cb25e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:5000: connect: connection refused"
Sep 18 17:11:33.889427 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:33.889212    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9" podUID="5d00ab19-e542-4ced-ae6f-b26074cb25e4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 17:11:34.771630 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:34.771585    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9" podUID="5d00ab19-e542-4ced-ae6f-b26074cb25e4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Sep 18 17:11:34.771833 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:34.771737    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9"
Sep 18 17:11:35.887992 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:35.887940    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z8fcb" podUID="b9b0bfc6-1476-4dd1-a232-5281f1052623" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 18 17:11:39.771880 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:39.771828    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9" podUID="5d00ab19-e542-4ced-ae6f-b26074cb25e4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Sep 18 17:11:41.257691 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:41.257653    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr"]
Sep 18 17:11:41.258159 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:41.258044    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9e8938f-6241-4ade-8b71-4bd14398fc4c" containerName="kserve-container"
Sep 18 17:11:41.258159 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:41.258056    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9e8938f-6241-4ade-8b71-4bd14398fc4c" containerName="kserve-container"
Sep 18 17:11:41.258159 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:41.258066    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9e8938f-6241-4ade-8b71-4bd14398fc4c" containerName="kube-rbac-proxy"
Sep 18 17:11:41.258159 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:41.258071    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9e8938f-6241-4ade-8b71-4bd14398fc4c" containerName="kube-rbac-proxy"
Sep 18 17:11:41.258159 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:41.258083    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9e8938f-6241-4ade-8b71-4bd14398fc4c" containerName="storage-initializer"
Sep 18 17:11:41.258159 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:41.258091    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9e8938f-6241-4ade-8b71-4bd14398fc4c" containerName="storage-initializer"
Sep 18 17:11:41.258159 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:41.258151    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9e8938f-6241-4ade-8b71-4bd14398fc4c" containerName="kube-rbac-proxy"
Sep 18 17:11:41.258159 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:41.258162    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9e8938f-6241-4ade-8b71-4bd14398fc4c" containerName="kserve-container"
Sep 18 17:11:41.261502 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:41.261485    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr"
Sep 18 17:11:41.263944 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:41.263918    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Sep 18 17:11:41.264096 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:41.263922    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Sep 18 17:11:41.273724 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:41.273696    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr"]
Sep 18 17:11:41.410303 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:41.410261    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/82fa6790-f089-4a1b-b09b-043c637124da-cabundle-cert\") pod \"isvc-logger-predictor-65985d8494-dcxqr\" (UID: \"82fa6790-f089-4a1b-b09b-043c637124da\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr"
Sep 18 17:11:41.410508 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:41.410411    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82fa6790-f089-4a1b-b09b-043c637124da-proxy-tls\") pod \"isvc-logger-predictor-65985d8494-dcxqr\" (UID: \"82fa6790-f089-4a1b-b09b-043c637124da\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr"
Sep 18 17:11:41.410508 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:41.410460    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj9vw\" (UniqueName: \"kubernetes.io/projected/82fa6790-f089-4a1b-b09b-043c637124da-kube-api-access-fj9vw\") pod \"isvc-logger-predictor-65985d8494-dcxqr\" (UID: \"82fa6790-f089-4a1b-b09b-043c637124da\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr"
Sep 18 17:11:41.410508 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:41.410488    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/82fa6790-f089-4a1b-b09b-043c637124da-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-65985d8494-dcxqr\" (UID: \"82fa6790-f089-4a1b-b09b-043c637124da\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr"
Sep 18 17:11:41.410611 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:41.410518    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/82fa6790-f089-4a1b-b09b-043c637124da-kserve-provision-location\") pod \"isvc-logger-predictor-65985d8494-dcxqr\" (UID: \"82fa6790-f089-4a1b-b09b-043c637124da\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr"
Sep 18 17:11:41.511132 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:41.511091    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj9vw\" (UniqueName: \"kubernetes.io/projected/82fa6790-f089-4a1b-b09b-043c637124da-kube-api-access-fj9vw\") pod \"isvc-logger-predictor-65985d8494-dcxqr\" (UID: \"82fa6790-f089-4a1b-b09b-043c637124da\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr"
Sep 18 17:11:41.511376 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:41.511145    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/82fa6790-f089-4a1b-b09b-043c637124da-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-65985d8494-dcxqr\" (UID: \"82fa6790-f089-4a1b-b09b-043c637124da\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr"
Sep 18 17:11:41.511376 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:41.511186    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/82fa6790-f089-4a1b-b09b-043c637124da-kserve-provision-location\") pod \"isvc-logger-predictor-65985d8494-dcxqr\" (UID: \"82fa6790-f089-4a1b-b09b-043c637124da\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr"
Sep 18 17:11:41.511376 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:41.511215    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/82fa6790-f089-4a1b-b09b-043c637124da-cabundle-cert\") pod \"isvc-logger-predictor-65985d8494-dcxqr\" (UID: \"82fa6790-f089-4a1b-b09b-043c637124da\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr"
Sep 18 17:11:41.511376 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:41.511332    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82fa6790-f089-4a1b-b09b-043c637124da-proxy-tls\") pod \"isvc-logger-predictor-65985d8494-dcxqr\" (UID: \"82fa6790-f089-4a1b-b09b-043c637124da\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr"
Sep 18 17:11:41.511604 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:11:41.511479    2881 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-predictor-serving-cert: secret "isvc-logger-predictor-serving-cert" not found
Sep 18 17:11:41.511604 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:11:41.511554    2881 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82fa6790-f089-4a1b-b09b-043c637124da-proxy-tls podName:82fa6790-f089-4a1b-b09b-043c637124da nodeName:}" failed. No retries permitted until 2026-09-18 17:11:42.011530384 +0000 UTC m=+1092.211385651 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/82fa6790-f089-4a1b-b09b-043c637124da-proxy-tls") pod "isvc-logger-predictor-65985d8494-dcxqr" (UID: "82fa6790-f089-4a1b-b09b-043c637124da") : secret "isvc-logger-predictor-serving-cert" not found
Sep 18 17:11:41.511717 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:41.511605    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/82fa6790-f089-4a1b-b09b-043c637124da-kserve-provision-location\") pod \"isvc-logger-predictor-65985d8494-dcxqr\" (UID: \"82fa6790-f089-4a1b-b09b-043c637124da\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr"
Sep 18 17:11:41.511933 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:41.511915    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/82fa6790-f089-4a1b-b09b-043c637124da-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-65985d8494-dcxqr\" (UID: \"82fa6790-f089-4a1b-b09b-043c637124da\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr"
Sep 18 17:11:41.511981 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:41.511960    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/82fa6790-f089-4a1b-b09b-043c637124da-cabundle-cert\") pod \"isvc-logger-predictor-65985d8494-dcxqr\" (UID: \"82fa6790-f089-4a1b-b09b-043c637124da\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr"
Sep 18 17:11:41.520670 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:41.520644    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj9vw\" (UniqueName: \"kubernetes.io/projected/82fa6790-f089-4a1b-b09b-043c637124da-kube-api-access-fj9vw\") pod \"isvc-logger-predictor-65985d8494-dcxqr\" (UID: \"82fa6790-f089-4a1b-b09b-043c637124da\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr"
Sep 18 17:11:42.017271 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:42.017205    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82fa6790-f089-4a1b-b09b-043c637124da-proxy-tls\") pod \"isvc-logger-predictor-65985d8494-dcxqr\" (UID: \"82fa6790-f089-4a1b-b09b-043c637124da\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr"
Sep 18 17:11:42.019910 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:42.019884    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82fa6790-f089-4a1b-b09b-043c637124da-proxy-tls\") pod \"isvc-logger-predictor-65985d8494-dcxqr\" (UID: \"82fa6790-f089-4a1b-b09b-043c637124da\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr"
Sep 18 17:11:42.172726 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:42.172683    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr"
Sep 18 17:11:42.451870 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:42.451829    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr"]
Sep 18 17:11:42.454493 ip-10-0-143-5 kubenswrapper[2881]: W0918 17:11:42.454457    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82fa6790_f089_4a1b_b09b_043c637124da.slice/crio-d1cf38c963f8e0566beab2a1cda49d27070563fa69c4866033204614b0b2f97b WatchSource:0}: Error finding container d1cf38c963f8e0566beab2a1cda49d27070563fa69c4866033204614b0b2f97b: Status 404 returned error can't find the container with id d1cf38c963f8e0566beab2a1cda49d27070563fa69c4866033204614b0b2f97b
Sep 18 17:11:43.376590 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:43.376552    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr" event={"ID":"82fa6790-f089-4a1b-b09b-043c637124da","Type":"ContainerStarted","Data":"efef13fa82ada7080335d119215ef425aef24aadb4265622e1935763bcd9f45b"}
Sep 18 17:11:43.376827 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:43.376620    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr" event={"ID":"82fa6790-f089-4a1b-b09b-043c637124da","Type":"ContainerStarted","Data":"d1cf38c963f8e0566beab2a1cda49d27070563fa69c4866033204614b0b2f97b"}
Sep 18 17:11:43.892596 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:43.892530    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9" podUID="5d00ab19-e542-4ced-ae6f-b26074cb25e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:5000: connect: connection refused"
Sep 18 17:11:43.893873 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:43.893836    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9" podUID="5d00ab19-e542-4ced-ae6f-b26074cb25e4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 17:11:43.895964 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:43.895936    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9"
Sep 18 17:11:43.896114 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:43.896051    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9"
Sep 18 17:11:44.771739 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:44.771695    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9" podUID="5d00ab19-e542-4ced-ae6f-b26074cb25e4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Sep 18 17:11:45.893169 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:45.893129    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z8fcb"
Sep 18 17:11:46.392572 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:46.392538    2881 generic.go:356] "Generic (PLEG): container finished" podID="82fa6790-f089-4a1b-b09b-043c637124da" containerID="efef13fa82ada7080335d119215ef425aef24aadb4265622e1935763bcd9f45b" exitCode=0
Sep 18 17:11:46.392738 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:46.392609    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr" event={"ID":"82fa6790-f089-4a1b-b09b-043c637124da","Type":"ContainerDied","Data":"efef13fa82ada7080335d119215ef425aef24aadb4265622e1935763bcd9f45b"}
Sep 18 17:11:47.400522 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:47.400482    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr" event={"ID":"82fa6790-f089-4a1b-b09b-043c637124da","Type":"ContainerStarted","Data":"bb9b9571d657eb8d07009a896b1f24416352d0924a5be5b460f4c3d999d76fa8"}
Sep 18 17:11:47.400925 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:47.400575    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr" event={"ID":"82fa6790-f089-4a1b-b09b-043c637124da","Type":"ContainerStarted","Data":"410af00669365ab171129443b4f1b03e067f880bb43390b000ecd7b1d7ce2b87"}
Sep 18 17:11:47.400925 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:47.400612    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr" event={"ID":"82fa6790-f089-4a1b-b09b-043c637124da","Type":"ContainerStarted","Data":"bcfe7e8533845d5e015719dc6b20d9f54e834072463a33279b2dbe16ee59cc7a"}
Sep 18 17:11:47.400925 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:47.400876    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr"
Sep 18 17:11:48.406160 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:48.406131    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr"
Sep 18 17:11:48.406594 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:48.406191    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr"
Sep 18 17:11:48.407668 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:48.407634    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr" podUID="82fa6790-f089-4a1b-b09b-043c637124da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 18 17:11:48.408381 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:48.408352    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr" podUID="82fa6790-f089-4a1b-b09b-043c637124da" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 17:11:49.411661 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:49.411616    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr" podUID="82fa6790-f089-4a1b-b09b-043c637124da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 18 17:11:49.412103 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:49.412055    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr" podUID="82fa6790-f089-4a1b-b09b-043c637124da" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 17:11:49.771964 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:49.771925    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9" podUID="5d00ab19-e542-4ced-ae6f-b26074cb25e4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Sep 18 17:11:51.208747 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:51.208719    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9"
Sep 18 17:11:51.234483 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:51.234411    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr" podStartSLOduration=6.376570699 podStartE2EDuration="10.234393826s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.857823127s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:11:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:11:47.434904783 +0000 UTC m=+1097.634760082" watchObservedRunningTime="2026-09-18 17:11:51.234393826 +0000 UTC m=+1101.434249129"
Sep 18 17:11:51.303682 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:51.303644    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d00ab19-e542-4ced-ae6f-b26074cb25e4-kserve-provision-location\") pod \"5d00ab19-e542-4ced-ae6f-b26074cb25e4\" (UID: \"5d00ab19-e542-4ced-ae6f-b26074cb25e4\") "
Sep 18 17:11:51.303682 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:51.303687    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d00ab19-e542-4ced-ae6f-b26074cb25e4-proxy-tls\") pod \"5d00ab19-e542-4ced-ae6f-b26074cb25e4\" (UID: \"5d00ab19-e542-4ced-ae6f-b26074cb25e4\") "
Sep 18 17:11:51.303907 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:51.303724    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5d00ab19-e542-4ced-ae6f-b26074cb25e4-kube-rbac-proxy-sar-config\") pod \"5d00ab19-e542-4ced-ae6f-b26074cb25e4\" (UID: \"5d00ab19-e542-4ced-ae6f-b26074cb25e4\") "
Sep 18 17:11:51.303907 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:51.303749    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d00ab19-e542-4ced-ae6f-b26074cb25e4-cabundle-cert\") pod \"5d00ab19-e542-4ced-ae6f-b26074cb25e4\" (UID: \"5d00ab19-e542-4ced-ae6f-b26074cb25e4\") "
Sep 18 17:11:51.303907 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:51.303785    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gnlrq\" (UniqueName: \"kubernetes.io/projected/5d00ab19-e542-4ced-ae6f-b26074cb25e4-kube-api-access-gnlrq\") pod \"5d00ab19-e542-4ced-ae6f-b26074cb25e4\" (UID: \"5d00ab19-e542-4ced-ae6f-b26074cb25e4\") "
Sep 18 17:11:51.304127 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:51.304100    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d00ab19-e542-4ced-ae6f-b26074cb25e4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5d00ab19-e542-4ced-ae6f-b26074cb25e4" (UID: "5d00ab19-e542-4ced-ae6f-b26074cb25e4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:11:51.304192 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:51.304104    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d00ab19-e542-4ced-ae6f-b26074cb25e4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5d00ab19-e542-4ced-ae6f-b26074cb25e4" (UID: "5d00ab19-e542-4ced-ae6f-b26074cb25e4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:11:51.304192 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:51.304169    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d00ab19-e542-4ced-ae6f-b26074cb25e4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5d00ab19-e542-4ced-ae6f-b26074cb25e4" (UID: "5d00ab19-e542-4ced-ae6f-b26074cb25e4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:11:51.306088 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:51.306050    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d00ab19-e542-4ced-ae6f-b26074cb25e4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5d00ab19-e542-4ced-ae6f-b26074cb25e4" (UID: "5d00ab19-e542-4ced-ae6f-b26074cb25e4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:11:51.306088 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:51.306059    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d00ab19-e542-4ced-ae6f-b26074cb25e4-kube-api-access-gnlrq" (OuterVolumeSpecName: "kube-api-access-gnlrq") pod "5d00ab19-e542-4ced-ae6f-b26074cb25e4" (UID: "5d00ab19-e542-4ced-ae6f-b26074cb25e4"). InnerVolumeSpecName "kube-api-access-gnlrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:11:51.354914 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:51.354814    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z8fcb"]
Sep 18 17:11:51.355140 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:51.355098    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z8fcb" podUID="b9b0bfc6-1476-4dd1-a232-5281f1052623" containerName="kserve-container" containerID="cri-o://6fa18d248a27e1f817e0181f008d59ea60ea74b42197797cfb1ad3109b3ad1a4" gracePeriod=30
Sep 18 17:11:51.355244 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:51.355156    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z8fcb" podUID="b9b0bfc6-1476-4dd1-a232-5281f1052623" containerName="kube-rbac-proxy" containerID="cri-o://8887877e817b63e88b3457d1a2447057a62cf3dda2a87ea6bc96fc872a80913d" gracePeriod=30
Sep 18 17:11:51.405546 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:51.405502    2881 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d00ab19-e542-4ced-ae6f-b26074cb25e4-kserve-provision-location\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:11:51.405546 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:51.405536    2881 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d00ab19-e542-4ced-ae6f-b26074cb25e4-proxy-tls\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:11:51.405546 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:51.405546    2881 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5d00ab19-e542-4ced-ae6f-b26074cb25e4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:11:51.405546 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:51.405555    2881 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d00ab19-e542-4ced-ae6f-b26074cb25e4-cabundle-cert\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:11:51.405840 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:51.405564    2881 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnlrq\" (UniqueName: \"kubernetes.io/projected/5d00ab19-e542-4ced-ae6f-b26074cb25e4-kube-api-access-gnlrq\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:11:51.425586 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:51.425550    2881 generic.go:356] "Generic (PLEG): container finished" podID="5d00ab19-e542-4ced-ae6f-b26074cb25e4" containerID="00baad68da4a8bb0e7930d95abf32289d682f05c53f28d5ad67a4827b2cdfb08" exitCode=0
Sep 18 17:11:51.425789 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:51.425636    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9"
Sep 18 17:11:51.425789 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:51.425639    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9" event={"ID":"5d00ab19-e542-4ced-ae6f-b26074cb25e4","Type":"ContainerDied","Data":"00baad68da4a8bb0e7930d95abf32289d682f05c53f28d5ad67a4827b2cdfb08"}
Sep 18 17:11:51.425789 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:51.425710    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9" event={"ID":"5d00ab19-e542-4ced-ae6f-b26074cb25e4","Type":"ContainerDied","Data":"7f2d41ef8e265c2a5221e685a87974a9677ca670e38e0ca6c1ae62b8077f9419"}
Sep 18 17:11:51.425789 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:51.425743    2881 scope.go:117] "RemoveContainer" containerID="00baad68da4a8bb0e7930d95abf32289d682f05c53f28d5ad67a4827b2cdfb08"
Sep 18 17:11:51.449738 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:51.449701    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9"]
Sep 18 17:11:51.454292 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:51.454262    2881 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9cb98cbf9-ssxs9"]
Sep 18 17:11:51.466984 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:51.466937    2881 scope.go:117] "RemoveContainer" containerID="052766904b7b8d3d4e4b59e3621f63ff54a65f4cd42098443c7ea6e7d8f93bd3"
Sep 18 17:11:51.475166 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:51.475145    2881 scope.go:117] "RemoveContainer" containerID="edb44ae0f5fc4a2f7d8856a9c070982a27771e394fa2c3d151168b3d0be46f15"
Sep 18 17:11:51.483707 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:51.483682    2881 scope.go:117] "RemoveContainer" containerID="3b8267f66a6de7ca3e4ef9d17ee059d474dc65677fc2c14925a6f9c8dcb81e9c"
Sep 18 17:11:51.492455 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:51.492426    2881 scope.go:117] "RemoveContainer" containerID="00baad68da4a8bb0e7930d95abf32289d682f05c53f28d5ad67a4827b2cdfb08"
Sep 18 17:11:51.492777 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:11:51.492758    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00baad68da4a8bb0e7930d95abf32289d682f05c53f28d5ad67a4827b2cdfb08\": container with ID starting with 00baad68da4a8bb0e7930d95abf32289d682f05c53f28d5ad67a4827b2cdfb08 not found: ID does not exist" containerID="00baad68da4a8bb0e7930d95abf32289d682f05c53f28d5ad67a4827b2cdfb08"
Sep 18 17:11:51.492842 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:51.492786    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00baad68da4a8bb0e7930d95abf32289d682f05c53f28d5ad67a4827b2cdfb08"} err="failed to get container status \"00baad68da4a8bb0e7930d95abf32289d682f05c53f28d5ad67a4827b2cdfb08\": rpc error: code = NotFound desc = could not find container \"00baad68da4a8bb0e7930d95abf32289d682f05c53f28d5ad67a4827b2cdfb08\": container with ID starting with 00baad68da4a8bb0e7930d95abf32289d682f05c53f28d5ad67a4827b2cdfb08 not found: ID does not exist"
Sep 18 17:11:51.492842 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:51.492804    2881 scope.go:117] "RemoveContainer" containerID="052766904b7b8d3d4e4b59e3621f63ff54a65f4cd42098443c7ea6e7d8f93bd3"
Sep 18 17:11:51.493135 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:11:51.493112    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"052766904b7b8d3d4e4b59e3621f63ff54a65f4cd42098443c7ea6e7d8f93bd3\": container with ID starting with 052766904b7b8d3d4e4b59e3621f63ff54a65f4cd42098443c7ea6e7d8f93bd3 not found: ID does not exist" containerID="052766904b7b8d3d4e4b59e3621f63ff54a65f4cd42098443c7ea6e7d8f93bd3"
Sep 18 17:11:51.493186 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:51.493146    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"052766904b7b8d3d4e4b59e3621f63ff54a65f4cd42098443c7ea6e7d8f93bd3"} err="failed to get container status \"052766904b7b8d3d4e4b59e3621f63ff54a65f4cd42098443c7ea6e7d8f93bd3\": rpc error: code = NotFound desc = could not find container \"052766904b7b8d3d4e4b59e3621f63ff54a65f4cd42098443c7ea6e7d8f93bd3\": container with ID starting with 052766904b7b8d3d4e4b59e3621f63ff54a65f4cd42098443c7ea6e7d8f93bd3 not found: ID does not exist"
Sep 18 17:11:51.493186 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:51.493165    2881 scope.go:117] "RemoveContainer" containerID="edb44ae0f5fc4a2f7d8856a9c070982a27771e394fa2c3d151168b3d0be46f15"
Sep 18 17:11:51.493443 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:11:51.493427    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edb44ae0f5fc4a2f7d8856a9c070982a27771e394fa2c3d151168b3d0be46f15\": container with ID starting with edb44ae0f5fc4a2f7d8856a9c070982a27771e394fa2c3d151168b3d0be46f15 not found: ID does not exist" containerID="edb44ae0f5fc4a2f7d8856a9c070982a27771e394fa2c3d151168b3d0be46f15"
Sep 18 17:11:51.493501 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:51.493445    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edb44ae0f5fc4a2f7d8856a9c070982a27771e394fa2c3d151168b3d0be46f15"} err="failed to get container status \"edb44ae0f5fc4a2f7d8856a9c070982a27771e394fa2c3d151168b3d0be46f15\": rpc error: code = NotFound desc = could not find container \"edb44ae0f5fc4a2f7d8856a9c070982a27771e394fa2c3d151168b3d0be46f15\": container with ID starting with edb44ae0f5fc4a2f7d8856a9c070982a27771e394fa2c3d151168b3d0be46f15 not found: ID does not exist"
Sep 18 17:11:51.493501 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:51.493457    2881 scope.go:117] "RemoveContainer" containerID="3b8267f66a6de7ca3e4ef9d17ee059d474dc65677fc2c14925a6f9c8dcb81e9c"
Sep 18 17:11:51.493722 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:11:51.493684    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b8267f66a6de7ca3e4ef9d17ee059d474dc65677fc2c14925a6f9c8dcb81e9c\": container with ID starting with 3b8267f66a6de7ca3e4ef9d17ee059d474dc65677fc2c14925a6f9c8dcb81e9c not found: ID does not exist" containerID="3b8267f66a6de7ca3e4ef9d17ee059d474dc65677fc2c14925a6f9c8dcb81e9c"
Sep 18 17:11:51.493759 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:51.493730    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b8267f66a6de7ca3e4ef9d17ee059d474dc65677fc2c14925a6f9c8dcb81e9c"} err="failed to get container status \"3b8267f66a6de7ca3e4ef9d17ee059d474dc65677fc2c14925a6f9c8dcb81e9c\": rpc error: code = NotFound desc = could not find container \"3b8267f66a6de7ca3e4ef9d17ee059d474dc65677fc2c14925a6f9c8dcb81e9c\": container with ID starting with 3b8267f66a6de7ca3e4ef9d17ee059d474dc65677fc2c14925a6f9c8dcb81e9c not found: ID does not exist"
Sep 18 17:11:51.894146 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:51.894110    2881 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d00ab19-e542-4ced-ae6f-b26074cb25e4" path="/var/lib/kubelet/pods/5d00ab19-e542-4ced-ae6f-b26074cb25e4/volumes"
Sep 18 17:11:52.433863 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:52.433830    2881 generic.go:356] "Generic (PLEG): container finished" podID="b9b0bfc6-1476-4dd1-a232-5281f1052623" containerID="8887877e817b63e88b3457d1a2447057a62cf3dda2a87ea6bc96fc872a80913d" exitCode=2
Sep 18 17:11:52.434380 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:52.433917    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z8fcb" event={"ID":"b9b0bfc6-1476-4dd1-a232-5281f1052623","Type":"ContainerDied","Data":"8887877e817b63e88b3457d1a2447057a62cf3dda2a87ea6bc96fc872a80913d"}
Sep 18 17:11:52.957537 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:52.957486    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z8fcb" podUID="b9b0bfc6-1476-4dd1-a232-5281f1052623" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Sep 18 17:11:54.415776 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:54.415738    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr"
Sep 18 17:11:54.416326 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:54.416296    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr" podUID="82fa6790-f089-4a1b-b09b-043c637124da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 18 17:11:54.416618 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:54.416596    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr" podUID="82fa6790-f089-4a1b-b09b-043c637124da" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 17:11:55.887990 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:55.887939    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z8fcb" podUID="b9b0bfc6-1476-4dd1-a232-5281f1052623" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 18 17:11:56.622364 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:56.622325    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-gd55l"]
Sep 18 17:11:56.622768 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:56.622755    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d00ab19-e542-4ced-ae6f-b26074cb25e4" containerName="agent"
Sep 18 17:11:56.622828 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:56.622769    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d00ab19-e542-4ced-ae6f-b26074cb25e4" containerName="agent"
Sep 18 17:11:56.622828 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:56.622782    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d00ab19-e542-4ced-ae6f-b26074cb25e4" containerName="kserve-container"
Sep 18 17:11:56.622828 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:56.622788    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d00ab19-e542-4ced-ae6f-b26074cb25e4" containerName="kserve-container"
Sep 18 17:11:56.622828 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:56.622806    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d00ab19-e542-4ced-ae6f-b26074cb25e4" containerName="kube-rbac-proxy"
Sep 18 17:11:56.622828 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:56.622811    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d00ab19-e542-4ced-ae6f-b26074cb25e4" containerName="kube-rbac-proxy"
Sep 18 17:11:56.622828 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:56.622830    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d00ab19-e542-4ced-ae6f-b26074cb25e4" containerName="storage-initializer"
Sep 18 17:11:56.623018 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:56.622836    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d00ab19-e542-4ced-ae6f-b26074cb25e4" containerName="storage-initializer"
Sep 18 17:11:56.623018 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:56.622887    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d00ab19-e542-4ced-ae6f-b26074cb25e4" containerName="kserve-container"
Sep 18 17:11:56.623018 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:56.622897    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d00ab19-e542-4ced-ae6f-b26074cb25e4" containerName="agent"
Sep 18 17:11:56.623018 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:56.622904    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d00ab19-e542-4ced-ae6f-b26074cb25e4" containerName="kube-rbac-proxy"
Sep 18 17:11:56.627967 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:56.627943    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-gd55l"
Sep 18 17:11:56.643299 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:56.643273    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-predictor-serving-cert\""
Sep 18 17:11:56.643457 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:56.643411    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\""
Sep 18 17:11:56.651622 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:56.651593    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-gd55l"]
Sep 18 17:11:56.653133 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:56.653104    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4cb5e70-7d73-42be-a217-a32ab964bde0-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-gd55l\" (UID: \"e4cb5e70-7d73-42be-a217-a32ab964bde0\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-gd55l"
Sep 18 17:11:56.653324 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:56.653140    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4cb5e70-7d73-42be-a217-a32ab964bde0-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-gd55l\" (UID: \"e4cb5e70-7d73-42be-a217-a32ab964bde0\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-gd55l"
Sep 18 17:11:56.653324 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:56.653188    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4cb5e70-7d73-42be-a217-a32ab964bde0-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-gd55l\" (UID: \"e4cb5e70-7d73-42be-a217-a32ab964bde0\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-gd55l"
Sep 18 17:11:56.653324 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:56.653302    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr8hc\" (UniqueName: \"kubernetes.io/projected/e4cb5e70-7d73-42be-a217-a32ab964bde0-kube-api-access-kr8hc\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-gd55l\" (UID: \"e4cb5e70-7d73-42be-a217-a32ab964bde0\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-gd55l"
Sep 18 17:11:56.653446 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:56.653355    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4cb5e70-7d73-42be-a217-a32ab964bde0-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-gd55l\" (UID: \"e4cb5e70-7d73-42be-a217-a32ab964bde0\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-gd55l"
Sep 18 17:11:56.754544 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:56.754497    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4cb5e70-7d73-42be-a217-a32ab964bde0-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-gd55l\" (UID: \"e4cb5e70-7d73-42be-a217-a32ab964bde0\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-gd55l"
Sep 18 17:11:56.754544 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:56.754547    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr8hc\" (UniqueName: \"kubernetes.io/projected/e4cb5e70-7d73-42be-a217-a32ab964bde0-kube-api-access-kr8hc\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-gd55l\" (UID: \"e4cb5e70-7d73-42be-a217-a32ab964bde0\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-gd55l"
Sep 18 17:11:56.754793 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:56.754581    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4cb5e70-7d73-42be-a217-a32ab964bde0-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-gd55l\" (UID: \"e4cb5e70-7d73-42be-a217-a32ab964bde0\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-gd55l"
Sep 18 17:11:56.754793 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:56.754628    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4cb5e70-7d73-42be-a217-a32ab964bde0-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-gd55l\" (UID: \"e4cb5e70-7d73-42be-a217-a32ab964bde0\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-gd55l"
Sep 18 17:11:56.754793 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:56.754651    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4cb5e70-7d73-42be-a217-a32ab964bde0-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-gd55l\" (UID: \"e4cb5e70-7d73-42be-a217-a32ab964bde0\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-gd55l"
Sep 18 17:11:56.755128 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:56.755105    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4cb5e70-7d73-42be-a217-a32ab964bde0-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-gd55l\" (UID: \"e4cb5e70-7d73-42be-a217-a32ab964bde0\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-gd55l"
Sep 18 17:11:56.755578 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:56.755551    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4cb5e70-7d73-42be-a217-a32ab964bde0-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-gd55l\" (UID: \"e4cb5e70-7d73-42be-a217-a32ab964bde0\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-gd55l"
Sep 18 17:11:56.755692 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:56.755551    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4cb5e70-7d73-42be-a217-a32ab964bde0-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-gd55l\" (UID: \"e4cb5e70-7d73-42be-a217-a32ab964bde0\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-gd55l"
Sep 18 17:11:56.757182 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:56.757158    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4cb5e70-7d73-42be-a217-a32ab964bde0-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-gd55l\" (UID: \"e4cb5e70-7d73-42be-a217-a32ab964bde0\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-gd55l"
Sep 18 17:11:56.770270 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:56.770242    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr8hc\" (UniqueName: \"kubernetes.io/projected/e4cb5e70-7d73-42be-a217-a32ab964bde0-kube-api-access-kr8hc\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-gd55l\" (UID: \"e4cb5e70-7d73-42be-a217-a32ab964bde0\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-gd55l"
Sep 18 17:11:56.941645 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:56.941522    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-gd55l"
Sep 18 17:11:57.224389 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:57.224365    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-gd55l"]
Sep 18 17:11:57.226208 ip-10-0-143-5 kubenswrapper[2881]: W0918 17:11:57.226179    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4cb5e70_7d73_42be_a217_a32ab964bde0.slice/crio-cc07e13a3c8888dc952a034594115508fb86c77152f489ee991aceebb6739183 WatchSource:0}: Error finding container cc07e13a3c8888dc952a034594115508fb86c77152f489ee991aceebb6739183: Status 404 returned error can't find the container with id cc07e13a3c8888dc952a034594115508fb86c77152f489ee991aceebb6739183
Sep 18 17:11:57.401796 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:57.401766    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z8fcb"
Sep 18 17:11:57.460488 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:57.460457    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b9b0bfc6-1476-4dd1-a232-5281f1052623-cabundle-cert\") pod \"b9b0bfc6-1476-4dd1-a232-5281f1052623\" (UID: \"b9b0bfc6-1476-4dd1-a232-5281f1052623\") "
Sep 18 17:11:57.460488 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:57.460504    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-km2zd\" (UniqueName: \"kubernetes.io/projected/b9b0bfc6-1476-4dd1-a232-5281f1052623-kube-api-access-km2zd\") pod \"b9b0bfc6-1476-4dd1-a232-5281f1052623\" (UID: \"b9b0bfc6-1476-4dd1-a232-5281f1052623\") "
Sep 18 17:11:57.460783 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:57.460576    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9b0bfc6-1476-4dd1-a232-5281f1052623-proxy-tls\") pod \"b9b0bfc6-1476-4dd1-a232-5281f1052623\" (UID: \"b9b0bfc6-1476-4dd1-a232-5281f1052623\") "
Sep 18 17:11:57.460783 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:57.460620    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b9b0bfc6-1476-4dd1-a232-5281f1052623-kserve-provision-location\") pod \"b9b0bfc6-1476-4dd1-a232-5281f1052623\" (UID: \"b9b0bfc6-1476-4dd1-a232-5281f1052623\") "
Sep 18 17:11:57.460783 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:57.460661    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9b0bfc6-1476-4dd1-a232-5281f1052623-kube-rbac-proxy-sar-config\") pod \"b9b0bfc6-1476-4dd1-a232-5281f1052623\" (UID: \"b9b0bfc6-1476-4dd1-a232-5281f1052623\") "
Sep 18 17:11:57.460959 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:57.460896    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9b0bfc6-1476-4dd1-a232-5281f1052623-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b9b0bfc6-1476-4dd1-a232-5281f1052623" (UID: "b9b0bfc6-1476-4dd1-a232-5281f1052623"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:11:57.461276 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:57.461235    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9b0bfc6-1476-4dd1-a232-5281f1052623-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b9b0bfc6-1476-4dd1-a232-5281f1052623" (UID: "b9b0bfc6-1476-4dd1-a232-5281f1052623"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:11:57.461431 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:57.461331    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9b0bfc6-1476-4dd1-a232-5281f1052623-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b9b0bfc6-1476-4dd1-a232-5281f1052623" (UID: "b9b0bfc6-1476-4dd1-a232-5281f1052623"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:11:57.461431 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:57.461351    2881 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b9b0bfc6-1476-4dd1-a232-5281f1052623-kserve-provision-location\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:11:57.461431 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:57.461370    2881 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b9b0bfc6-1476-4dd1-a232-5281f1052623-cabundle-cert\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:11:57.463069 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:57.463042    2881 generic.go:356] "Generic (PLEG): container finished" podID="b9b0bfc6-1476-4dd1-a232-5281f1052623" containerID="6fa18d248a27e1f817e0181f008d59ea60ea74b42197797cfb1ad3109b3ad1a4" exitCode=0
Sep 18 17:11:57.463200 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:57.463132    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z8fcb"
Sep 18 17:11:57.463301 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:57.463123    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z8fcb" event={"ID":"b9b0bfc6-1476-4dd1-a232-5281f1052623","Type":"ContainerDied","Data":"6fa18d248a27e1f817e0181f008d59ea60ea74b42197797cfb1ad3109b3ad1a4"}
Sep 18 17:11:57.463364 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:57.463317    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z8fcb" event={"ID":"b9b0bfc6-1476-4dd1-a232-5281f1052623","Type":"ContainerDied","Data":"aa537ec76b74f91e198debf153f68e20e4949c28c2cd14e14644ca3701e94d84"}
Sep 18 17:11:57.463415 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:57.463369    2881 scope.go:117] "RemoveContainer" containerID="8887877e817b63e88b3457d1a2447057a62cf3dda2a87ea6bc96fc872a80913d"
Sep 18 17:11:57.463415 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:57.463370    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9b0bfc6-1476-4dd1-a232-5281f1052623-kube-api-access-km2zd" (OuterVolumeSpecName: "kube-api-access-km2zd") pod "b9b0bfc6-1476-4dd1-a232-5281f1052623" (UID: "b9b0bfc6-1476-4dd1-a232-5281f1052623"). InnerVolumeSpecName "kube-api-access-km2zd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:11:57.463415 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:57.463379    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9b0bfc6-1476-4dd1-a232-5281f1052623-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b9b0bfc6-1476-4dd1-a232-5281f1052623" (UID: "b9b0bfc6-1476-4dd1-a232-5281f1052623"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:11:57.465075 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:57.465048    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-gd55l" event={"ID":"e4cb5e70-7d73-42be-a217-a32ab964bde0","Type":"ContainerStarted","Data":"0fb32346aa79712865e959ae5310f7f7160e15c5de0b519e7050535234ac93c3"}
Sep 18 17:11:57.465181 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:57.465126    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-gd55l" event={"ID":"e4cb5e70-7d73-42be-a217-a32ab964bde0","Type":"ContainerStarted","Data":"cc07e13a3c8888dc952a034594115508fb86c77152f489ee991aceebb6739183"}
Sep 18 17:11:57.474088 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:57.474068    2881 scope.go:117] "RemoveContainer" containerID="6fa18d248a27e1f817e0181f008d59ea60ea74b42197797cfb1ad3109b3ad1a4"
Sep 18 17:11:57.482602 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:57.482578    2881 scope.go:117] "RemoveContainer" containerID="5c4bda3f1791f7a8fdb3a3fcff5f2e27e09c84aea2f1fdcbfd03fa407188b0ef"
Sep 18 17:11:57.493122 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:57.493102    2881 scope.go:117] "RemoveContainer" containerID="8887877e817b63e88b3457d1a2447057a62cf3dda2a87ea6bc96fc872a80913d"
Sep 18 17:11:57.493555 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:11:57.493529    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8887877e817b63e88b3457d1a2447057a62cf3dda2a87ea6bc96fc872a80913d\": container with ID starting with 8887877e817b63e88b3457d1a2447057a62cf3dda2a87ea6bc96fc872a80913d not found: ID does not exist" containerID="8887877e817b63e88b3457d1a2447057a62cf3dda2a87ea6bc96fc872a80913d"
Sep 18 17:11:57.493652 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:57.493568    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8887877e817b63e88b3457d1a2447057a62cf3dda2a87ea6bc96fc872a80913d"} err="failed to get container status \"8887877e817b63e88b3457d1a2447057a62cf3dda2a87ea6bc96fc872a80913d\": rpc error: code = NotFound desc = could not find container \"8887877e817b63e88b3457d1a2447057a62cf3dda2a87ea6bc96fc872a80913d\": container with ID starting with 8887877e817b63e88b3457d1a2447057a62cf3dda2a87ea6bc96fc872a80913d not found: ID does not exist"
Sep 18 17:11:57.493652 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:57.493590    2881 scope.go:117] "RemoveContainer" containerID="6fa18d248a27e1f817e0181f008d59ea60ea74b42197797cfb1ad3109b3ad1a4"
Sep 18 17:11:57.493907 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:11:57.493889    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fa18d248a27e1f817e0181f008d59ea60ea74b42197797cfb1ad3109b3ad1a4\": container with ID starting with 6fa18d248a27e1f817e0181f008d59ea60ea74b42197797cfb1ad3109b3ad1a4 not found: ID does not exist" containerID="6fa18d248a27e1f817e0181f008d59ea60ea74b42197797cfb1ad3109b3ad1a4"
Sep 18 17:11:57.493975 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:57.493917    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fa18d248a27e1f817e0181f008d59ea60ea74b42197797cfb1ad3109b3ad1a4"} err="failed to get container status \"6fa18d248a27e1f817e0181f008d59ea60ea74b42197797cfb1ad3109b3ad1a4\": rpc error: code = NotFound desc = could not find container \"6fa18d248a27e1f817e0181f008d59ea60ea74b42197797cfb1ad3109b3ad1a4\": container with ID starting with 6fa18d248a27e1f817e0181f008d59ea60ea74b42197797cfb1ad3109b3ad1a4 not found: ID does not exist"
Sep 18 17:11:57.493975 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:57.493943    2881 scope.go:117] "RemoveContainer" containerID="5c4bda3f1791f7a8fdb3a3fcff5f2e27e09c84aea2f1fdcbfd03fa407188b0ef"
Sep 18 17:11:57.494209 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:11:57.494193    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c4bda3f1791f7a8fdb3a3fcff5f2e27e09c84aea2f1fdcbfd03fa407188b0ef\": container with ID starting with 5c4bda3f1791f7a8fdb3a3fcff5f2e27e09c84aea2f1fdcbfd03fa407188b0ef not found: ID does not exist" containerID="5c4bda3f1791f7a8fdb3a3fcff5f2e27e09c84aea2f1fdcbfd03fa407188b0ef"
Sep 18 17:11:57.494270 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:57.494216    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c4bda3f1791f7a8fdb3a3fcff5f2e27e09c84aea2f1fdcbfd03fa407188b0ef"} err="failed to get container status \"5c4bda3f1791f7a8fdb3a3fcff5f2e27e09c84aea2f1fdcbfd03fa407188b0ef\": rpc error: code = NotFound desc = could not find container \"5c4bda3f1791f7a8fdb3a3fcff5f2e27e09c84aea2f1fdcbfd03fa407188b0ef\": container with ID starting with 5c4bda3f1791f7a8fdb3a3fcff5f2e27e09c84aea2f1fdcbfd03fa407188b0ef not found: ID does not exist"
Sep 18 17:11:57.562824 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:57.562776    2881 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-km2zd\" (UniqueName: \"kubernetes.io/projected/b9b0bfc6-1476-4dd1-a232-5281f1052623-kube-api-access-km2zd\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:11:57.563034 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:57.562938    2881 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9b0bfc6-1476-4dd1-a232-5281f1052623-proxy-tls\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:11:57.563140 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:57.563114    2881 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9b0bfc6-1476-4dd1-a232-5281f1052623-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:11:57.792431 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:57.792389    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z8fcb"]
Sep 18 17:11:57.798569 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:57.798537    2881 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z8fcb"]
Sep 18 17:11:57.893938 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:11:57.893908    2881 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9b0bfc6-1476-4dd1-a232-5281f1052623" path="/var/lib/kubelet/pods/b9b0bfc6-1476-4dd1-a232-5281f1052623/volumes"
Sep 18 17:12:01.496129 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:12:01.496034    2881 generic.go:356] "Generic (PLEG): container finished" podID="e4cb5e70-7d73-42be-a217-a32ab964bde0" containerID="0fb32346aa79712865e959ae5310f7f7160e15c5de0b519e7050535234ac93c3" exitCode=0
Sep 18 17:12:01.496129 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:12:01.496113    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-gd55l" event={"ID":"e4cb5e70-7d73-42be-a217-a32ab964bde0","Type":"ContainerDied","Data":"0fb32346aa79712865e959ae5310f7f7160e15c5de0b519e7050535234ac93c3"}
Sep 18 17:12:02.503607 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:12:02.503571    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-gd55l" event={"ID":"e4cb5e70-7d73-42be-a217-a32ab964bde0","Type":"ContainerStarted","Data":"6d5d9ae9d33f42df90cfff8982a5bade542027ef3e4ed4da489cffbcb6494493"}
Sep 18 17:12:02.504063 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:12:02.503641    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-gd55l" event={"ID":"e4cb5e70-7d73-42be-a217-a32ab964bde0","Type":"ContainerStarted","Data":"3c742744b9eab72943c8750ed65e8b26510d15e1a6dba5d51c5dfc86020cc8fd"}
Sep 18 17:12:02.504063 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:12:02.503871    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-gd55l"
Sep 18 17:12:02.528673 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:12:02.528619    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-gd55l" podStartSLOduration=2.6296826429999998 podStartE2EDuration="6.528599815s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.898917172s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:11:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:12:02.527174081 +0000 UTC m=+1112.727029381" watchObservedRunningTime="2026-09-18 17:12:02.528599815 +0000 UTC m=+1112.728455100"
Sep 18 17:12:03.508502 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:12:03.508466    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-gd55l"
Sep 18 17:12:03.509809 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:12:03.509779    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-gd55l" podUID="e4cb5e70-7d73-42be-a217-a32ab964bde0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 18 17:12:04.416301 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:12:04.416257    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr" podUID="82fa6790-f089-4a1b-b09b-043c637124da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 18 17:12:04.416824 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:12:04.416803    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr" podUID="82fa6790-f089-4a1b-b09b-043c637124da" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 17:12:04.513329 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:12:04.513289    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-gd55l" podUID="e4cb5e70-7d73-42be-a217-a32ab964bde0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 18 17:12:09.518028 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:12:09.517996    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-gd55l"
Sep 18 17:12:09.518598 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:12:09.518563    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-gd55l" podUID="e4cb5e70-7d73-42be-a217-a32ab964bde0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 18 17:12:14.416661 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:12:14.416618    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr" podUID="82fa6790-f089-4a1b-b09b-043c637124da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 18 17:12:14.417138 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:12:14.417103    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr" podUID="82fa6790-f089-4a1b-b09b-043c637124da" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 17:12:19.519239 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:12:19.519192    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-gd55l" podUID="e4cb5e70-7d73-42be-a217-a32ab964bde0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 18 17:12:24.416390 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:12:24.416340    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr" podUID="82fa6790-f089-4a1b-b09b-043c637124da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 18 17:12:24.416874 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:12:24.416851    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr" podUID="82fa6790-f089-4a1b-b09b-043c637124da" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 17:12:29.518610 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:12:29.518566    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-gd55l" podUID="e4cb5e70-7d73-42be-a217-a32ab964bde0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 18 17:12:34.416316 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:12:34.416272    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr" podUID="82fa6790-f089-4a1b-b09b-043c637124da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 18 17:12:34.416916 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:12:34.416893    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr" podUID="82fa6790-f089-4a1b-b09b-043c637124da" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 17:12:39.519349 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:12:39.519302    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-gd55l" podUID="e4cb5e70-7d73-42be-a217-a32ab964bde0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 18 17:12:44.416475 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:12:44.416421    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr" podUID="82fa6790-f089-4a1b-b09b-043c637124da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 18 17:12:44.416993 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:12:44.416901    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr" podUID="82fa6790-f089-4a1b-b09b-043c637124da" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 17:12:49.519473 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:12:49.519429    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-gd55l" podUID="e4cb5e70-7d73-42be-a217-a32ab964bde0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 18 17:12:54.416218 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:12:54.416166    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr" podUID="82fa6790-f089-4a1b-b09b-043c637124da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 18 17:12:54.416808 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:12:54.416610    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr" podUID="82fa6790-f089-4a1b-b09b-043c637124da" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 17:12:59.519063 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:12:59.519021    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-gd55l" podUID="e4cb5e70-7d73-42be-a217-a32ab964bde0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 18 17:13:04.416566 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:04.416499    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr" podUID="82fa6790-f089-4a1b-b09b-043c637124da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 18 17:13:04.417082 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:04.416992    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr" podUID="82fa6790-f089-4a1b-b09b-043c637124da" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 17:13:09.518941 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:09.518900    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-gd55l" podUID="e4cb5e70-7d73-42be-a217-a32ab964bde0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 18 17:13:14.417464 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:14.417429    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr"
Sep 18 17:13:14.417979 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:14.417920    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr"
Sep 18 17:13:19.519493 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:19.519448    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-gd55l" podUID="e4cb5e70-7d73-42be-a217-a32ab964bde0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 18 17:13:21.888955 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:21.888910    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-gd55l" podUID="e4cb5e70-7d73-42be-a217-a32ab964bde0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 18 17:13:27.046100 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:27.046062    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr"]
Sep 18 17:13:27.046598 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:27.046432    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr" podUID="82fa6790-f089-4a1b-b09b-043c637124da" containerName="kserve-container" containerID="cri-o://bcfe7e8533845d5e015719dc6b20d9f54e834072463a33279b2dbe16ee59cc7a" gracePeriod=30
Sep 18 17:13:27.046598 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:27.046535    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr" podUID="82fa6790-f089-4a1b-b09b-043c637124da" containerName="kube-rbac-proxy" containerID="cri-o://410af00669365ab171129443b4f1b03e067f880bb43390b000ecd7b1d7ce2b87" gracePeriod=30
Sep 18 17:13:27.046717 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:27.046535    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr" podUID="82fa6790-f089-4a1b-b09b-043c637124da" containerName="agent" containerID="cri-o://bb9b9571d657eb8d07009a896b1f24416352d0924a5be5b460f4c3d999d76fa8" gracePeriod=30
Sep 18 17:13:27.270730 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:27.270684    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgcgh"]
Sep 18 17:13:27.271087 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:27.271074    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9b0bfc6-1476-4dd1-a232-5281f1052623" containerName="kserve-container"
Sep 18 17:13:27.271140 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:27.271088    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9b0bfc6-1476-4dd1-a232-5281f1052623" containerName="kserve-container"
Sep 18 17:13:27.271140 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:27.271098    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9b0bfc6-1476-4dd1-a232-5281f1052623" containerName="storage-initializer"
Sep 18 17:13:27.271140 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:27.271105    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9b0bfc6-1476-4dd1-a232-5281f1052623" containerName="storage-initializer"
Sep 18 17:13:27.271140 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:27.271134    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9b0bfc6-1476-4dd1-a232-5281f1052623" containerName="kube-rbac-proxy"
Sep 18 17:13:27.271140 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:27.271141    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9b0bfc6-1476-4dd1-a232-5281f1052623" containerName="kube-rbac-proxy"
Sep 18 17:13:27.271317 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:27.271210    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9b0bfc6-1476-4dd1-a232-5281f1052623" containerName="kube-rbac-proxy"
Sep 18 17:13:27.271317 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:27.271232    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9b0bfc6-1476-4dd1-a232-5281f1052623" containerName="kserve-container"
Sep 18 17:13:27.274618 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:27.274597    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgcgh"
Sep 18 17:13:27.277508 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:27.277479    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Sep 18 17:13:27.277631 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:27.277479    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Sep 18 17:13:27.288133 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:27.288105    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgcgh"]
Sep 18 17:13:27.392408 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:27.392307    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1d0374f-a6e1-46fa-9344-afbddffbaaef-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-cgcgh\" (UID: \"c1d0374f-a6e1-46fa-9344-afbddffbaaef\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgcgh"
Sep 18 17:13:27.392408 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:27.392356    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1d0374f-a6e1-46fa-9344-afbddffbaaef-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-cgcgh\" (UID: \"c1d0374f-a6e1-46fa-9344-afbddffbaaef\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgcgh"
Sep 18 17:13:27.392621 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:27.392445    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c1d0374f-a6e1-46fa-9344-afbddffbaaef-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-cgcgh\" (UID: \"c1d0374f-a6e1-46fa-9344-afbddffbaaef\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgcgh"
Sep 18 17:13:27.392621 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:27.392489    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnx2b\" (UniqueName: \"kubernetes.io/projected/c1d0374f-a6e1-46fa-9344-afbddffbaaef-kube-api-access-rnx2b\") pod \"isvc-autogluon-v1-predictor-574b5b7495-cgcgh\" (UID: \"c1d0374f-a6e1-46fa-9344-afbddffbaaef\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgcgh"
Sep 18 17:13:27.392621 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:27.392516    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1d0374f-a6e1-46fa-9344-afbddffbaaef-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-cgcgh\" (UID: \"c1d0374f-a6e1-46fa-9344-afbddffbaaef\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgcgh"
Sep 18 17:13:27.493623 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:27.493583    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1d0374f-a6e1-46fa-9344-afbddffbaaef-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-cgcgh\" (UID: \"c1d0374f-a6e1-46fa-9344-afbddffbaaef\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgcgh"
Sep 18 17:13:27.493852 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:27.493642    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c1d0374f-a6e1-46fa-9344-afbddffbaaef-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-cgcgh\" (UID: \"c1d0374f-a6e1-46fa-9344-afbddffbaaef\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgcgh"
Sep 18 17:13:27.493852 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:27.493667    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnx2b\" (UniqueName: \"kubernetes.io/projected/c1d0374f-a6e1-46fa-9344-afbddffbaaef-kube-api-access-rnx2b\") pod \"isvc-autogluon-v1-predictor-574b5b7495-cgcgh\" (UID: \"c1d0374f-a6e1-46fa-9344-afbddffbaaef\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgcgh"
Sep 18 17:13:27.493852 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:27.493683    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1d0374f-a6e1-46fa-9344-afbddffbaaef-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-cgcgh\" (UID: \"c1d0374f-a6e1-46fa-9344-afbddffbaaef\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgcgh"
Sep 18 17:13:27.493852 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:27.493739    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1d0374f-a6e1-46fa-9344-afbddffbaaef-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-cgcgh\" (UID: \"c1d0374f-a6e1-46fa-9344-afbddffbaaef\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgcgh"
Sep 18 17:13:27.494196 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:27.494168    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1d0374f-a6e1-46fa-9344-afbddffbaaef-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-cgcgh\" (UID: \"c1d0374f-a6e1-46fa-9344-afbddffbaaef\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgcgh"
Sep 18 17:13:27.494436 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:27.494415    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c1d0374f-a6e1-46fa-9344-afbddffbaaef-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-cgcgh\" (UID: \"c1d0374f-a6e1-46fa-9344-afbddffbaaef\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgcgh"
Sep 18 17:13:27.494502 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:27.494415    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1d0374f-a6e1-46fa-9344-afbddffbaaef-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-cgcgh\" (UID: \"c1d0374f-a6e1-46fa-9344-afbddffbaaef\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgcgh"
Sep 18 17:13:27.496376 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:27.496353    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1d0374f-a6e1-46fa-9344-afbddffbaaef-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-cgcgh\" (UID: \"c1d0374f-a6e1-46fa-9344-afbddffbaaef\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgcgh"
Sep 18 17:13:27.504694 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:27.504666    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnx2b\" (UniqueName: \"kubernetes.io/projected/c1d0374f-a6e1-46fa-9344-afbddffbaaef-kube-api-access-rnx2b\") pod \"isvc-autogluon-v1-predictor-574b5b7495-cgcgh\" (UID: \"c1d0374f-a6e1-46fa-9344-afbddffbaaef\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgcgh"
Sep 18 17:13:27.586214 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:27.586168    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgcgh"
Sep 18 17:13:27.863114 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:27.863085    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgcgh"]
Sep 18 17:13:27.865308 ip-10-0-143-5 kubenswrapper[2881]: W0918 17:13:27.865281    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1d0374f_a6e1_46fa_9344_afbddffbaaef.slice/crio-769ca9fc333a62d020053b7bab676a6fd4d8e79e6af6dcc22926fc035d9fad54 WatchSource:0}: Error finding container 769ca9fc333a62d020053b7bab676a6fd4d8e79e6af6dcc22926fc035d9fad54: Status 404 returned error can't find the container with id 769ca9fc333a62d020053b7bab676a6fd4d8e79e6af6dcc22926fc035d9fad54
Sep 18 17:13:27.911588 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:27.911553    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgcgh" event={"ID":"c1d0374f-a6e1-46fa-9344-afbddffbaaef","Type":"ContainerStarted","Data":"769ca9fc333a62d020053b7bab676a6fd4d8e79e6af6dcc22926fc035d9fad54"}
Sep 18 17:13:27.913973 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:27.913944    2881 generic.go:356] "Generic (PLEG): container finished" podID="82fa6790-f089-4a1b-b09b-043c637124da" containerID="410af00669365ab171129443b4f1b03e067f880bb43390b000ecd7b1d7ce2b87" exitCode=2
Sep 18 17:13:27.914116 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:27.913980    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr" event={"ID":"82fa6790-f089-4a1b-b09b-043c637124da","Type":"ContainerDied","Data":"410af00669365ab171129443b4f1b03e067f880bb43390b000ecd7b1d7ce2b87"}
Sep 18 17:13:28.920619 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:28.920578    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgcgh" event={"ID":"c1d0374f-a6e1-46fa-9344-afbddffbaaef","Type":"ContainerStarted","Data":"f5f27aaecd4ef31bec2d01b53e184ee1e16b934034ad6d2dbf8668ca1f80c9fc"}
Sep 18 17:13:29.412529 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:29.412488    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr" podUID="82fa6790-f089-4a1b-b09b-043c637124da" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Sep 18 17:13:31.893673 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:31.893589    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-gd55l"
Sep 18 17:13:32.944612 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:32.944581    2881 generic.go:356] "Generic (PLEG): container finished" podID="82fa6790-f089-4a1b-b09b-043c637124da" containerID="bcfe7e8533845d5e015719dc6b20d9f54e834072463a33279b2dbe16ee59cc7a" exitCode=0
Sep 18 17:13:32.945036 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:32.944661    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr" event={"ID":"82fa6790-f089-4a1b-b09b-043c637124da","Type":"ContainerDied","Data":"bcfe7e8533845d5e015719dc6b20d9f54e834072463a33279b2dbe16ee59cc7a"}
Sep 18 17:13:34.412644 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:34.412588    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr" podUID="82fa6790-f089-4a1b-b09b-043c637124da" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Sep 18 17:13:34.416983 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:34.416957    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr" podUID="82fa6790-f089-4a1b-b09b-043c637124da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 18 17:13:34.418609 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:34.418582    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr" podUID="82fa6790-f089-4a1b-b09b-043c637124da" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 17:13:37.180716 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:37.180675    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-gd55l"]
Sep 18 17:13:37.181139 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:37.180942    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-gd55l" podUID="e4cb5e70-7d73-42be-a217-a32ab964bde0" containerName="kserve-container" containerID="cri-o://3c742744b9eab72943c8750ed65e8b26510d15e1a6dba5d51c5dfc86020cc8fd" gracePeriod=30
Sep 18 17:13:37.181139 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:37.181024    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-gd55l" podUID="e4cb5e70-7d73-42be-a217-a32ab964bde0" containerName="kube-rbac-proxy" containerID="cri-o://6d5d9ae9d33f42df90cfff8982a5bade542027ef3e4ed4da489cffbcb6494493" gracePeriod=30
Sep 18 17:13:37.971190 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:37.971157    2881 generic.go:356] "Generic (PLEG): container finished" podID="e4cb5e70-7d73-42be-a217-a32ab964bde0" containerID="6d5d9ae9d33f42df90cfff8982a5bade542027ef3e4ed4da489cffbcb6494493" exitCode=2
Sep 18 17:13:37.971405 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:37.971244    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-gd55l" event={"ID":"e4cb5e70-7d73-42be-a217-a32ab964bde0","Type":"ContainerDied","Data":"6d5d9ae9d33f42df90cfff8982a5bade542027ef3e4ed4da489cffbcb6494493"}
Sep 18 17:13:39.412635 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:39.412585    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr" podUID="82fa6790-f089-4a1b-b09b-043c637124da" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Sep 18 17:13:39.413071 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:39.412728    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr"
Sep 18 17:13:39.514185 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:39.514140    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-gd55l" podUID="e4cb5e70-7d73-42be-a217-a32ab964bde0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Sep 18 17:13:41.889047 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:41.889000    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-gd55l" podUID="e4cb5e70-7d73-42be-a217-a32ab964bde0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 18 17:13:43.224902 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:43.224873    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-gd55l"
Sep 18 17:13:43.341909 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:43.341797    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4cb5e70-7d73-42be-a217-a32ab964bde0-kube-rbac-proxy-sar-config\") pod \"e4cb5e70-7d73-42be-a217-a32ab964bde0\" (UID: \"e4cb5e70-7d73-42be-a217-a32ab964bde0\") "
Sep 18 17:13:43.341909 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:43.341856    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4cb5e70-7d73-42be-a217-a32ab964bde0-proxy-tls\") pod \"e4cb5e70-7d73-42be-a217-a32ab964bde0\" (UID: \"e4cb5e70-7d73-42be-a217-a32ab964bde0\") "
Sep 18 17:13:43.341909 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:43.341897    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4cb5e70-7d73-42be-a217-a32ab964bde0-kserve-provision-location\") pod \"e4cb5e70-7d73-42be-a217-a32ab964bde0\" (UID: \"e4cb5e70-7d73-42be-a217-a32ab964bde0\") "
Sep 18 17:13:43.342161 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:43.341955    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kr8hc\" (UniqueName: \"kubernetes.io/projected/e4cb5e70-7d73-42be-a217-a32ab964bde0-kube-api-access-kr8hc\") pod \"e4cb5e70-7d73-42be-a217-a32ab964bde0\" (UID: \"e4cb5e70-7d73-42be-a217-a32ab964bde0\") "
Sep 18 17:13:43.342161 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:43.342022    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4cb5e70-7d73-42be-a217-a32ab964bde0-cabundle-cert\") pod \"e4cb5e70-7d73-42be-a217-a32ab964bde0\" (UID: \"e4cb5e70-7d73-42be-a217-a32ab964bde0\") "
Sep 18 17:13:43.342289 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:43.342267    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4cb5e70-7d73-42be-a217-a32ab964bde0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e4cb5e70-7d73-42be-a217-a32ab964bde0" (UID: "e4cb5e70-7d73-42be-a217-a32ab964bde0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:13:43.342343 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:43.342317    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4cb5e70-7d73-42be-a217-a32ab964bde0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e4cb5e70-7d73-42be-a217-a32ab964bde0" (UID: "e4cb5e70-7d73-42be-a217-a32ab964bde0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:13:43.342471 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:43.342447    2881 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4cb5e70-7d73-42be-a217-a32ab964bde0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:13:43.342601 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:43.342479    2881 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4cb5e70-7d73-42be-a217-a32ab964bde0-kserve-provision-location\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:13:43.342601 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:43.342445    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4cb5e70-7d73-42be-a217-a32ab964bde0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e4cb5e70-7d73-42be-a217-a32ab964bde0" (UID: "e4cb5e70-7d73-42be-a217-a32ab964bde0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:13:43.344198 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:43.344171    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4cb5e70-7d73-42be-a217-a32ab964bde0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e4cb5e70-7d73-42be-a217-a32ab964bde0" (UID: "e4cb5e70-7d73-42be-a217-a32ab964bde0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:13:43.344360 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:43.344267    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4cb5e70-7d73-42be-a217-a32ab964bde0-kube-api-access-kr8hc" (OuterVolumeSpecName: "kube-api-access-kr8hc") pod "e4cb5e70-7d73-42be-a217-a32ab964bde0" (UID: "e4cb5e70-7d73-42be-a217-a32ab964bde0"). InnerVolumeSpecName "kube-api-access-kr8hc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:13:43.443216 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:43.443179    2881 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr8hc\" (UniqueName: \"kubernetes.io/projected/e4cb5e70-7d73-42be-a217-a32ab964bde0-kube-api-access-kr8hc\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:13:43.443216 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:43.443209    2881 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4cb5e70-7d73-42be-a217-a32ab964bde0-cabundle-cert\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:13:43.443216 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:43.443245    2881 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4cb5e70-7d73-42be-a217-a32ab964bde0-proxy-tls\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:13:44.000768 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:44.000734    2881 generic.go:356] "Generic (PLEG): container finished" podID="e4cb5e70-7d73-42be-a217-a32ab964bde0" containerID="3c742744b9eab72943c8750ed65e8b26510d15e1a6dba5d51c5dfc86020cc8fd" exitCode=0
Sep 18 17:13:44.000947 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:44.000796    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-gd55l" event={"ID":"e4cb5e70-7d73-42be-a217-a32ab964bde0","Type":"ContainerDied","Data":"3c742744b9eab72943c8750ed65e8b26510d15e1a6dba5d51c5dfc86020cc8fd"}
Sep 18 17:13:44.000947 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:44.000865    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-gd55l" event={"ID":"e4cb5e70-7d73-42be-a217-a32ab964bde0","Type":"ContainerDied","Data":"cc07e13a3c8888dc952a034594115508fb86c77152f489ee991aceebb6739183"}
Sep 18 17:13:44.000947 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:44.000802    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-gd55l"
Sep 18 17:13:44.000947 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:44.000894    2881 scope.go:117] "RemoveContainer" containerID="6d5d9ae9d33f42df90cfff8982a5bade542027ef3e4ed4da489cffbcb6494493"
Sep 18 17:13:44.009527 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:44.009504    2881 scope.go:117] "RemoveContainer" containerID="3c742744b9eab72943c8750ed65e8b26510d15e1a6dba5d51c5dfc86020cc8fd"
Sep 18 17:13:44.017794 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:44.017774    2881 scope.go:117] "RemoveContainer" containerID="0fb32346aa79712865e959ae5310f7f7160e15c5de0b519e7050535234ac93c3"
Sep 18 17:13:44.021186 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:44.021159    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-gd55l"]
Sep 18 17:13:44.027316 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:44.027274    2881 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-gd55l"]
Sep 18 17:13:44.028566 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:44.028547    2881 scope.go:117] "RemoveContainer" containerID="6d5d9ae9d33f42df90cfff8982a5bade542027ef3e4ed4da489cffbcb6494493"
Sep 18 17:13:44.028956 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:13:44.028934    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d5d9ae9d33f42df90cfff8982a5bade542027ef3e4ed4da489cffbcb6494493\": container with ID starting with 6d5d9ae9d33f42df90cfff8982a5bade542027ef3e4ed4da489cffbcb6494493 not found: ID does not exist" containerID="6d5d9ae9d33f42df90cfff8982a5bade542027ef3e4ed4da489cffbcb6494493"
Sep 18 17:13:44.029041 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:44.028969    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d5d9ae9d33f42df90cfff8982a5bade542027ef3e4ed4da489cffbcb6494493"} err="failed to get container status \"6d5d9ae9d33f42df90cfff8982a5bade542027ef3e4ed4da489cffbcb6494493\": rpc error: code = NotFound desc = could not find container \"6d5d9ae9d33f42df90cfff8982a5bade542027ef3e4ed4da489cffbcb6494493\": container with ID starting with 6d5d9ae9d33f42df90cfff8982a5bade542027ef3e4ed4da489cffbcb6494493 not found: ID does not exist"
Sep 18 17:13:44.029041 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:44.028998    2881 scope.go:117] "RemoveContainer" containerID="3c742744b9eab72943c8750ed65e8b26510d15e1a6dba5d51c5dfc86020cc8fd"
Sep 18 17:13:44.029388 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:13:44.029362    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c742744b9eab72943c8750ed65e8b26510d15e1a6dba5d51c5dfc86020cc8fd\": container with ID starting with 3c742744b9eab72943c8750ed65e8b26510d15e1a6dba5d51c5dfc86020cc8fd not found: ID does not exist" containerID="3c742744b9eab72943c8750ed65e8b26510d15e1a6dba5d51c5dfc86020cc8fd"
Sep 18 17:13:44.029444 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:44.029395    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c742744b9eab72943c8750ed65e8b26510d15e1a6dba5d51c5dfc86020cc8fd"} err="failed to get container status \"3c742744b9eab72943c8750ed65e8b26510d15e1a6dba5d51c5dfc86020cc8fd\": rpc error: code = NotFound desc = could not find container \"3c742744b9eab72943c8750ed65e8b26510d15e1a6dba5d51c5dfc86020cc8fd\": container with ID starting with 3c742744b9eab72943c8750ed65e8b26510d15e1a6dba5d51c5dfc86020cc8fd not found: ID does not exist"
Sep 18 17:13:44.029444 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:44.029425    2881 scope.go:117] "RemoveContainer" containerID="0fb32346aa79712865e959ae5310f7f7160e15c5de0b519e7050535234ac93c3"
Sep 18 17:13:44.029714 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:13:44.029694    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fb32346aa79712865e959ae5310f7f7160e15c5de0b519e7050535234ac93c3\": container with ID starting with 0fb32346aa79712865e959ae5310f7f7160e15c5de0b519e7050535234ac93c3 not found: ID does not exist" containerID="0fb32346aa79712865e959ae5310f7f7160e15c5de0b519e7050535234ac93c3"
Sep 18 17:13:44.029758 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:44.029723    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fb32346aa79712865e959ae5310f7f7160e15c5de0b519e7050535234ac93c3"} err="failed to get container status \"0fb32346aa79712865e959ae5310f7f7160e15c5de0b519e7050535234ac93c3\": rpc error: code = NotFound desc = could not find container \"0fb32346aa79712865e959ae5310f7f7160e15c5de0b519e7050535234ac93c3\": container with ID starting with 0fb32346aa79712865e959ae5310f7f7160e15c5de0b519e7050535234ac93c3 not found: ID does not exist"
Sep 18 17:13:44.412415 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:44.412358    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr" podUID="82fa6790-f089-4a1b-b09b-043c637124da" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Sep 18 17:13:44.416724 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:44.416690    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr" podUID="82fa6790-f089-4a1b-b09b-043c637124da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 18 17:13:44.419474 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:44.419439    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr" podUID="82fa6790-f089-4a1b-b09b-043c637124da" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 17:13:45.893433 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:45.893392    2881 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4cb5e70-7d73-42be-a217-a32ab964bde0" path="/var/lib/kubelet/pods/e4cb5e70-7d73-42be-a217-a32ab964bde0/volumes"
Sep 18 17:13:49.026304 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:49.026257    2881 generic.go:356] "Generic (PLEG): container finished" podID="c1d0374f-a6e1-46fa-9344-afbddffbaaef" containerID="f5f27aaecd4ef31bec2d01b53e184ee1e16b934034ad6d2dbf8668ca1f80c9fc" exitCode=0
Sep 18 17:13:49.026684 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:49.026325    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgcgh" event={"ID":"c1d0374f-a6e1-46fa-9344-afbddffbaaef","Type":"ContainerDied","Data":"f5f27aaecd4ef31bec2d01b53e184ee1e16b934034ad6d2dbf8668ca1f80c9fc"}
Sep 18 17:13:49.027707 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:49.027690    2881 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 17:13:49.412166 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:49.412064    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr" podUID="82fa6790-f089-4a1b-b09b-043c637124da" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Sep 18 17:13:54.412573 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:54.412517    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr" podUID="82fa6790-f089-4a1b-b09b-043c637124da" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Sep 18 17:13:54.417027 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:54.416986    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr" podUID="82fa6790-f089-4a1b-b09b-043c637124da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 18 17:13:54.417166 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:54.417140    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr"
Sep 18 17:13:54.418987 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:54.418950    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr" podUID="82fa6790-f089-4a1b-b09b-043c637124da" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 17:13:54.419138 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:54.419099    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr"
Sep 18 17:13:57.079936 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:57.079898    2881 generic.go:356] "Generic (PLEG): container finished" podID="82fa6790-f089-4a1b-b09b-043c637124da" containerID="bb9b9571d657eb8d07009a896b1f24416352d0924a5be5b460f4c3d999d76fa8" exitCode=0
Sep 18 17:13:57.080464 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:57.079952    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr" event={"ID":"82fa6790-f089-4a1b-b09b-043c637124da","Type":"ContainerDied","Data":"bb9b9571d657eb8d07009a896b1f24416352d0924a5be5b460f4c3d999d76fa8"}
Sep 18 17:13:57.223075 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:57.223036    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr"
Sep 18 17:13:57.273303 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:57.273268    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82fa6790-f089-4a1b-b09b-043c637124da-proxy-tls\") pod \"82fa6790-f089-4a1b-b09b-043c637124da\" (UID: \"82fa6790-f089-4a1b-b09b-043c637124da\") "
Sep 18 17:13:57.273577 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:57.273329    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/82fa6790-f089-4a1b-b09b-043c637124da-cabundle-cert\") pod \"82fa6790-f089-4a1b-b09b-043c637124da\" (UID: \"82fa6790-f089-4a1b-b09b-043c637124da\") "
Sep 18 17:13:57.273577 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:57.273395    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/82fa6790-f089-4a1b-b09b-043c637124da-kserve-provision-location\") pod \"82fa6790-f089-4a1b-b09b-043c637124da\" (UID: \"82fa6790-f089-4a1b-b09b-043c637124da\") "
Sep 18 17:13:57.273577 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:57.273440    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fj9vw\" (UniqueName: \"kubernetes.io/projected/82fa6790-f089-4a1b-b09b-043c637124da-kube-api-access-fj9vw\") pod \"82fa6790-f089-4a1b-b09b-043c637124da\" (UID: \"82fa6790-f089-4a1b-b09b-043c637124da\") "
Sep 18 17:13:57.273577 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:57.273477    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/82fa6790-f089-4a1b-b09b-043c637124da-kube-rbac-proxy-sar-config\") pod \"82fa6790-f089-4a1b-b09b-043c637124da\" (UID: \"82fa6790-f089-4a1b-b09b-043c637124da\") "
Sep 18 17:13:57.273858 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:57.273778    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82fa6790-f089-4a1b-b09b-043c637124da-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "82fa6790-f089-4a1b-b09b-043c637124da" (UID: "82fa6790-f089-4a1b-b09b-043c637124da"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:13:57.273858 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:57.273835    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82fa6790-f089-4a1b-b09b-043c637124da-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "82fa6790-f089-4a1b-b09b-043c637124da" (UID: "82fa6790-f089-4a1b-b09b-043c637124da"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:13:57.274055 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:57.274023    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82fa6790-f089-4a1b-b09b-043c637124da-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "82fa6790-f089-4a1b-b09b-043c637124da" (UID: "82fa6790-f089-4a1b-b09b-043c637124da"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:13:57.276114 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:57.276083    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82fa6790-f089-4a1b-b09b-043c637124da-kube-api-access-fj9vw" (OuterVolumeSpecName: "kube-api-access-fj9vw") pod "82fa6790-f089-4a1b-b09b-043c637124da" (UID: "82fa6790-f089-4a1b-b09b-043c637124da"). InnerVolumeSpecName "kube-api-access-fj9vw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:13:57.276637 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:57.276599    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82fa6790-f089-4a1b-b09b-043c637124da-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "82fa6790-f089-4a1b-b09b-043c637124da" (UID: "82fa6790-f089-4a1b-b09b-043c637124da"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:13:57.374385 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:57.374267    2881 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/82fa6790-f089-4a1b-b09b-043c637124da-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:13:57.374385 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:57.374314    2881 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82fa6790-f089-4a1b-b09b-043c637124da-proxy-tls\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:13:57.374385 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:57.374325    2881 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/82fa6790-f089-4a1b-b09b-043c637124da-cabundle-cert\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:13:57.374385 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:57.374336    2881 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/82fa6790-f089-4a1b-b09b-043c637124da-kserve-provision-location\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:13:57.374385 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:57.374345    2881 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fj9vw\" (UniqueName: \"kubernetes.io/projected/82fa6790-f089-4a1b-b09b-043c637124da-kube-api-access-fj9vw\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:13:58.087261 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:58.087203    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr" event={"ID":"82fa6790-f089-4a1b-b09b-043c637124da","Type":"ContainerDied","Data":"d1cf38c963f8e0566beab2a1cda49d27070563fa69c4866033204614b0b2f97b"}
Sep 18 17:13:58.087261 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:58.087265    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr"
Sep 18 17:13:58.087766 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:58.087316    2881 scope.go:117] "RemoveContainer" containerID="bb9b9571d657eb8d07009a896b1f24416352d0924a5be5b460f4c3d999d76fa8"
Sep 18 17:13:58.095824 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:58.095805    2881 scope.go:117] "RemoveContainer" containerID="410af00669365ab171129443b4f1b03e067f880bb43390b000ecd7b1d7ce2b87"
Sep 18 17:13:58.104030 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:58.104009    2881 scope.go:117] "RemoveContainer" containerID="bcfe7e8533845d5e015719dc6b20d9f54e834072463a33279b2dbe16ee59cc7a"
Sep 18 17:13:58.112029 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:58.112010    2881 scope.go:117] "RemoveContainer" containerID="efef13fa82ada7080335d119215ef425aef24aadb4265622e1935763bcd9f45b"
Sep 18 17:13:58.120204 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:58.120172    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr"]
Sep 18 17:13:58.121935 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:58.121913    2881 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-65985d8494-dcxqr"]
Sep 18 17:13:59.894279 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:13:59.894244    2881 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82fa6790-f089-4a1b-b09b-043c637124da" path="/var/lib/kubelet/pods/82fa6790-f089-4a1b-b09b-043c637124da/volumes"
Sep 18 17:15:53.693516 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:15:53.693409    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgcgh" event={"ID":"c1d0374f-a6e1-46fa-9344-afbddffbaaef","Type":"ContainerStarted","Data":"60533e218d61307627b23207bdd35c3fd806c95147519f01bb9abe9e0816d426"}
Sep 18 17:15:53.693516 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:15:53.693478    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgcgh" event={"ID":"c1d0374f-a6e1-46fa-9344-afbddffbaaef","Type":"ContainerStarted","Data":"2d6f0f1402ab32582831c7b4fc9e50d1f0349cebcf72a8c9390568848f5c271b"}
Sep 18 17:15:53.694111 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:15:53.693552    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgcgh"
Sep 18 17:15:53.717034 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:15:53.716974    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgcgh" podStartSLOduration=2.457600189 podStartE2EDuration="2m26.716951899s" totalImagesPullingTime="2m3.562211132s" totalInitContainerRuntime="20.697140578s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:13:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:15:53.714120089 +0000 UTC m=+1343.913975403" watchObservedRunningTime="2026-09-18 17:15:53.716951899 +0000 UTC m=+1343.916807185"
Sep 18 17:15:54.697588 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:15:54.697555    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgcgh"
Sep 18 17:15:54.698919 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:15:54.698885    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgcgh" podUID="c1d0374f-a6e1-46fa-9344-afbddffbaaef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 18 17:15:55.702795 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:15:55.702755    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgcgh" podUID="c1d0374f-a6e1-46fa-9344-afbddffbaaef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 18 17:16:00.707827 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:16:00.707794    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgcgh"
Sep 18 17:16:00.708393 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:16:00.708363    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgcgh" podUID="c1d0374f-a6e1-46fa-9344-afbddffbaaef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 18 17:16:10.709156 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:16:10.709126    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgcgh"
Sep 18 17:16:23.893398 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:16:23.893365    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgcgh"]
Sep 18 17:16:23.893886 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:16:23.893624    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgcgh" podUID="c1d0374f-a6e1-46fa-9344-afbddffbaaef" containerName="kserve-container" containerID="cri-o://2d6f0f1402ab32582831c7b4fc9e50d1f0349cebcf72a8c9390568848f5c271b" gracePeriod=30
Sep 18 17:16:23.893886 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:16:23.893673    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgcgh" podUID="c1d0374f-a6e1-46fa-9344-afbddffbaaef" containerName="kube-rbac-proxy" containerID="cri-o://60533e218d61307627b23207bdd35c3fd806c95147519f01bb9abe9e0816d426" gracePeriod=30
Sep 18 17:16:24.839192 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:16:24.839155    2881 generic.go:356] "Generic (PLEG): container finished" podID="c1d0374f-a6e1-46fa-9344-afbddffbaaef" containerID="60533e218d61307627b23207bdd35c3fd806c95147519f01bb9abe9e0816d426" exitCode=2
Sep 18 17:16:24.839409 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:16:24.839238    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgcgh" event={"ID":"c1d0374f-a6e1-46fa-9344-afbddffbaaef","Type":"ContainerDied","Data":"60533e218d61307627b23207bdd35c3fd806c95147519f01bb9abe9e0816d426"}
Sep 18 17:16:25.684074 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:16:25.684036    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgcgh"
Sep 18 17:16:25.812900 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:16:25.812856    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1d0374f-a6e1-46fa-9344-afbddffbaaef-kube-rbac-proxy-sar-config\") pod \"c1d0374f-a6e1-46fa-9344-afbddffbaaef\" (UID: \"c1d0374f-a6e1-46fa-9344-afbddffbaaef\") "
Sep 18 17:16:25.813122 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:16:25.812915    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1d0374f-a6e1-46fa-9344-afbddffbaaef-kserve-provision-location\") pod \"c1d0374f-a6e1-46fa-9344-afbddffbaaef\" (UID: \"c1d0374f-a6e1-46fa-9344-afbddffbaaef\") "
Sep 18 17:16:25.813122 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:16:25.812960    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c1d0374f-a6e1-46fa-9344-afbddffbaaef-cabundle-cert\") pod \"c1d0374f-a6e1-46fa-9344-afbddffbaaef\" (UID: \"c1d0374f-a6e1-46fa-9344-afbddffbaaef\") "
Sep 18 17:16:25.813122 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:16:25.813001    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rnx2b\" (UniqueName: \"kubernetes.io/projected/c1d0374f-a6e1-46fa-9344-afbddffbaaef-kube-api-access-rnx2b\") pod \"c1d0374f-a6e1-46fa-9344-afbddffbaaef\" (UID: \"c1d0374f-a6e1-46fa-9344-afbddffbaaef\") "
Sep 18 17:16:25.813122 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:16:25.813061    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1d0374f-a6e1-46fa-9344-afbddffbaaef-proxy-tls\") pod \"c1d0374f-a6e1-46fa-9344-afbddffbaaef\" (UID: \"c1d0374f-a6e1-46fa-9344-afbddffbaaef\") "
Sep 18 17:16:25.813460 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:16:25.813438    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1d0374f-a6e1-46fa-9344-afbddffbaaef-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c1d0374f-a6e1-46fa-9344-afbddffbaaef" (UID: "c1d0374f-a6e1-46fa-9344-afbddffbaaef"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:16:25.813460 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:16:25.813443    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1d0374f-a6e1-46fa-9344-afbddffbaaef-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c1d0374f-a6e1-46fa-9344-afbddffbaaef" (UID: "c1d0374f-a6e1-46fa-9344-afbddffbaaef"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:16:25.815306 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:16:25.815275    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1d0374f-a6e1-46fa-9344-afbddffbaaef-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c1d0374f-a6e1-46fa-9344-afbddffbaaef" (UID: "c1d0374f-a6e1-46fa-9344-afbddffbaaef"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:16:25.815438 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:16:25.815365    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1d0374f-a6e1-46fa-9344-afbddffbaaef-kube-api-access-rnx2b" (OuterVolumeSpecName: "kube-api-access-rnx2b") pod "c1d0374f-a6e1-46fa-9344-afbddffbaaef" (UID: "c1d0374f-a6e1-46fa-9344-afbddffbaaef"). InnerVolumeSpecName "kube-api-access-rnx2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:16:25.818742 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:16:25.818712    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1d0374f-a6e1-46fa-9344-afbddffbaaef-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c1d0374f-a6e1-46fa-9344-afbddffbaaef" (UID: "c1d0374f-a6e1-46fa-9344-afbddffbaaef"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:16:25.845519 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:16:25.845476    2881 generic.go:356] "Generic (PLEG): container finished" podID="c1d0374f-a6e1-46fa-9344-afbddffbaaef" containerID="2d6f0f1402ab32582831c7b4fc9e50d1f0349cebcf72a8c9390568848f5c271b" exitCode=0
Sep 18 17:16:25.845695 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:16:25.845554    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgcgh"
Sep 18 17:16:25.845695 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:16:25.845552    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgcgh" event={"ID":"c1d0374f-a6e1-46fa-9344-afbddffbaaef","Type":"ContainerDied","Data":"2d6f0f1402ab32582831c7b4fc9e50d1f0349cebcf72a8c9390568848f5c271b"}
Sep 18 17:16:25.845695 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:16:25.845678    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgcgh" event={"ID":"c1d0374f-a6e1-46fa-9344-afbddffbaaef","Type":"ContainerDied","Data":"769ca9fc333a62d020053b7bab676a6fd4d8e79e6af6dcc22926fc035d9fad54"}
Sep 18 17:16:25.845802 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:16:25.845705    2881 scope.go:117] "RemoveContainer" containerID="60533e218d61307627b23207bdd35c3fd806c95147519f01bb9abe9e0816d426"
Sep 18 17:16:25.855308 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:16:25.855290    2881 scope.go:117] "RemoveContainer" containerID="2d6f0f1402ab32582831c7b4fc9e50d1f0349cebcf72a8c9390568848f5c271b"
Sep 18 17:16:25.864053 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:16:25.864030    2881 scope.go:117] "RemoveContainer" containerID="f5f27aaecd4ef31bec2d01b53e184ee1e16b934034ad6d2dbf8668ca1f80c9fc"
Sep 18 17:16:25.870275 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:16:25.870248    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgcgh"]
Sep 18 17:16:25.873827 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:16:25.873785    2881 scope.go:117] "RemoveContainer" containerID="60533e218d61307627b23207bdd35c3fd806c95147519f01bb9abe9e0816d426"
Sep 18 17:16:25.874218 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:16:25.874197    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60533e218d61307627b23207bdd35c3fd806c95147519f01bb9abe9e0816d426\": container with ID starting with 60533e218d61307627b23207bdd35c3fd806c95147519f01bb9abe9e0816d426 not found: ID does not exist" containerID="60533e218d61307627b23207bdd35c3fd806c95147519f01bb9abe9e0816d426"
Sep 18 17:16:25.874218 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:16:25.874255    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60533e218d61307627b23207bdd35c3fd806c95147519f01bb9abe9e0816d426"} err="failed to get container status \"60533e218d61307627b23207bdd35c3fd806c95147519f01bb9abe9e0816d426\": rpc error: code = NotFound desc = could not find container \"60533e218d61307627b23207bdd35c3fd806c95147519f01bb9abe9e0816d426\": container with ID starting with 60533e218d61307627b23207bdd35c3fd806c95147519f01bb9abe9e0816d426 not found: ID does not exist"
Sep 18 17:16:25.874218 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:16:25.874279    2881 scope.go:117] "RemoveContainer" containerID="2d6f0f1402ab32582831c7b4fc9e50d1f0349cebcf72a8c9390568848f5c271b"
Sep 18 17:16:25.874633 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:16:25.874612    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d6f0f1402ab32582831c7b4fc9e50d1f0349cebcf72a8c9390568848f5c271b\": container with ID starting with 2d6f0f1402ab32582831c7b4fc9e50d1f0349cebcf72a8c9390568848f5c271b not found: ID does not exist" containerID="2d6f0f1402ab32582831c7b4fc9e50d1f0349cebcf72a8c9390568848f5c271b"
Sep 18 17:16:25.874691 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:16:25.874637    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d6f0f1402ab32582831c7b4fc9e50d1f0349cebcf72a8c9390568848f5c271b"} err="failed to get container status \"2d6f0f1402ab32582831c7b4fc9e50d1f0349cebcf72a8c9390568848f5c271b\": rpc error: code = NotFound desc = could not find container \"2d6f0f1402ab32582831c7b4fc9e50d1f0349cebcf72a8c9390568848f5c271b\": container with ID starting with 2d6f0f1402ab32582831c7b4fc9e50d1f0349cebcf72a8c9390568848f5c271b not found: ID does not exist"
Sep 18 17:16:25.874691 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:16:25.874651    2881 scope.go:117] "RemoveContainer" containerID="f5f27aaecd4ef31bec2d01b53e184ee1e16b934034ad6d2dbf8668ca1f80c9fc"
Sep 18 17:16:25.874960 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:16:25.874941    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5f27aaecd4ef31bec2d01b53e184ee1e16b934034ad6d2dbf8668ca1f80c9fc\": container with ID starting with f5f27aaecd4ef31bec2d01b53e184ee1e16b934034ad6d2dbf8668ca1f80c9fc not found: ID does not exist" containerID="f5f27aaecd4ef31bec2d01b53e184ee1e16b934034ad6d2dbf8668ca1f80c9fc"
Sep 18 17:16:25.875014 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:16:25.874964    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5f27aaecd4ef31bec2d01b53e184ee1e16b934034ad6d2dbf8668ca1f80c9fc"} err="failed to get container status \"f5f27aaecd4ef31bec2d01b53e184ee1e16b934034ad6d2dbf8668ca1f80c9fc\": rpc error: code = NotFound desc = could not find container \"f5f27aaecd4ef31bec2d01b53e184ee1e16b934034ad6d2dbf8668ca1f80c9fc\": container with ID starting with f5f27aaecd4ef31bec2d01b53e184ee1e16b934034ad6d2dbf8668ca1f80c9fc not found: ID does not exist"
Sep 18 17:16:25.875492 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:16:25.875474    2881 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgcgh"]
Sep 18 17:16:25.894054 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:16:25.894026    2881 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1d0374f-a6e1-46fa-9344-afbddffbaaef" path="/var/lib/kubelet/pods/c1d0374f-a6e1-46fa-9344-afbddffbaaef/volumes"
Sep 18 17:16:25.913728 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:16:25.913696    2881 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1d0374f-a6e1-46fa-9344-afbddffbaaef-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:16:25.913728 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:16:25.913726    2881 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1d0374f-a6e1-46fa-9344-afbddffbaaef-kserve-provision-location\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:16:25.913862 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:16:25.913740    2881 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c1d0374f-a6e1-46fa-9344-afbddffbaaef-cabundle-cert\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:16:25.913862 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:16:25.913749    2881 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnx2b\" (UniqueName: \"kubernetes.io/projected/c1d0374f-a6e1-46fa-9344-afbddffbaaef-kube-api-access-rnx2b\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:16:25.913862 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:16:25.913757    2881 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1d0374f-a6e1-46fa-9344-afbddffbaaef-proxy-tls\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:17:42.969379 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:17:42.969343    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dn9ft"]
Sep 18 17:17:42.969899 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:17:42.969692    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4cb5e70-7d73-42be-a217-a32ab964bde0" containerName="kube-rbac-proxy"
Sep 18 17:17:42.969899 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:17:42.969703    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4cb5e70-7d73-42be-a217-a32ab964bde0" containerName="kube-rbac-proxy"
Sep 18 17:17:42.969899 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:17:42.969715    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82fa6790-f089-4a1b-b09b-043c637124da" containerName="kserve-container"
Sep 18 17:17:42.969899 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:17:42.969722    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="82fa6790-f089-4a1b-b09b-043c637124da" containerName="kserve-container"
Sep 18 17:17:42.969899 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:17:42.969735    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82fa6790-f089-4a1b-b09b-043c637124da" containerName="agent"
Sep 18 17:17:42.969899 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:17:42.969741    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="82fa6790-f089-4a1b-b09b-043c637124da" containerName="agent"
Sep 18 17:17:42.969899 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:17:42.969747    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82fa6790-f089-4a1b-b09b-043c637124da" containerName="kube-rbac-proxy"
Sep 18 17:17:42.969899 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:17:42.969753    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="82fa6790-f089-4a1b-b09b-043c637124da" containerName="kube-rbac-proxy"
Sep 18 17:17:42.969899 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:17:42.969763    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82fa6790-f089-4a1b-b09b-043c637124da" containerName="storage-initializer"
Sep 18 17:17:42.969899 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:17:42.969769    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="82fa6790-f089-4a1b-b09b-043c637124da" containerName="storage-initializer"
Sep 18 17:17:42.969899 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:17:42.969776    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4cb5e70-7d73-42be-a217-a32ab964bde0" containerName="kserve-container"
Sep 18 17:17:42.969899 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:17:42.969781    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4cb5e70-7d73-42be-a217-a32ab964bde0" containerName="kserve-container"
Sep 18 17:17:42.969899 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:17:42.969789    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4cb5e70-7d73-42be-a217-a32ab964bde0" containerName="storage-initializer"
Sep 18 17:17:42.969899 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:17:42.969794    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4cb5e70-7d73-42be-a217-a32ab964bde0" containerName="storage-initializer"
Sep 18 17:17:42.969899 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:17:42.969801    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1d0374f-a6e1-46fa-9344-afbddffbaaef" containerName="storage-initializer"
Sep 18 17:17:42.969899 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:17:42.969807    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1d0374f-a6e1-46fa-9344-afbddffbaaef" containerName="storage-initializer"
Sep 18 17:17:42.969899 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:17:42.969818    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1d0374f-a6e1-46fa-9344-afbddffbaaef" containerName="kube-rbac-proxy"
Sep 18 17:17:42.969899 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:17:42.969823    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1d0374f-a6e1-46fa-9344-afbddffbaaef" containerName="kube-rbac-proxy"
Sep 18 17:17:42.969899 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:17:42.969829    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1d0374f-a6e1-46fa-9344-afbddffbaaef" containerName="kserve-container"
Sep 18 17:17:42.969899 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:17:42.969834    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1d0374f-a6e1-46fa-9344-afbddffbaaef" containerName="kserve-container"
Sep 18 17:17:42.969899 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:17:42.969884    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4cb5e70-7d73-42be-a217-a32ab964bde0" containerName="kserve-container"
Sep 18 17:17:42.969899 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:17:42.969894    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="82fa6790-f089-4a1b-b09b-043c637124da" containerName="kserve-container"
Sep 18 17:17:42.969899 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:17:42.969900    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1d0374f-a6e1-46fa-9344-afbddffbaaef" containerName="kube-rbac-proxy"
Sep 18 17:17:42.969899 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:17:42.969909    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="82fa6790-f089-4a1b-b09b-043c637124da" containerName="agent"
Sep 18 17:17:42.969899 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:17:42.969915    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1d0374f-a6e1-46fa-9344-afbddffbaaef" containerName="kserve-container"
Sep 18 17:17:42.970796 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:17:42.969921    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4cb5e70-7d73-42be-a217-a32ab964bde0" containerName="kube-rbac-proxy"
Sep 18 17:17:42.970796 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:17:42.969926    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="82fa6790-f089-4a1b-b09b-043c637124da" containerName="kube-rbac-proxy"
Sep 18 17:17:42.972306 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:17:42.972290    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dn9ft"
Sep 18 17:17:42.976385 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:17:42.976353    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 17:17:42.976551 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:17:42.976510    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 17:17:42.976727 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:17:42.976709    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5hv9v\""
Sep 18 17:17:42.976863 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:17:42.976845    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 17:17:42.977274 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:17:42.977256    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 17:17:42.977375 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:17:42.977277    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\""
Sep 18 17:17:42.979285 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:17:42.979262    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-predictor-serving-cert\""
Sep 18 17:17:42.987172 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:17:42.987142    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dn9ft"]
Sep 18 17:17:43.105075 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:17:43.105036    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e10b86b-aec5-4133-8950-858d0af61b95-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-dn9ft\" (UID: \"7e10b86b-aec5-4133-8950-858d0af61b95\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dn9ft"
Sep 18 17:17:43.105307 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:17:43.105094    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e10b86b-aec5-4133-8950-858d0af61b95-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-dn9ft\" (UID: \"7e10b86b-aec5-4133-8950-858d0af61b95\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dn9ft"
Sep 18 17:17:43.105307 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:17:43.105114    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x6s4\" (UniqueName: \"kubernetes.io/projected/7e10b86b-aec5-4133-8950-858d0af61b95-kube-api-access-4x6s4\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-dn9ft\" (UID: \"7e10b86b-aec5-4133-8950-858d0af61b95\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dn9ft"
Sep 18 17:17:43.105307 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:17:43.105136    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e10b86b-aec5-4133-8950-858d0af61b95-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-dn9ft\" (UID: \"7e10b86b-aec5-4133-8950-858d0af61b95\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dn9ft"
Sep 18 17:17:43.105307 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:17:43.105152    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e10b86b-aec5-4133-8950-858d0af61b95-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-dn9ft\" (UID: \"7e10b86b-aec5-4133-8950-858d0af61b95\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dn9ft"
Sep 18 17:17:43.206521 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:17:43.206483    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e10b86b-aec5-4133-8950-858d0af61b95-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-dn9ft\" (UID: \"7e10b86b-aec5-4133-8950-858d0af61b95\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dn9ft"
Sep 18 17:17:43.206722 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:17:43.206528    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e10b86b-aec5-4133-8950-858d0af61b95-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-dn9ft\" (UID: \"7e10b86b-aec5-4133-8950-858d0af61b95\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dn9ft"
Sep 18 17:17:43.206722 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:17:43.206662    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x6s4\" (UniqueName: \"kubernetes.io/projected/7e10b86b-aec5-4133-8950-858d0af61b95-kube-api-access-4x6s4\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-dn9ft\" (UID: \"7e10b86b-aec5-4133-8950-858d0af61b95\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dn9ft"
Sep 18 17:17:43.206722 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:17:43.206708    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e10b86b-aec5-4133-8950-858d0af61b95-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-dn9ft\" (UID: \"7e10b86b-aec5-4133-8950-858d0af61b95\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dn9ft"
Sep 18 17:17:43.206889 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:17:43.206734    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e10b86b-aec5-4133-8950-858d0af61b95-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-dn9ft\" (UID: \"7e10b86b-aec5-4133-8950-858d0af61b95\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dn9ft"
Sep 18 17:17:43.206949 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:17:43.206929    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e10b86b-aec5-4133-8950-858d0af61b95-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-dn9ft\" (UID: \"7e10b86b-aec5-4133-8950-858d0af61b95\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dn9ft"
Sep 18 17:17:43.207259 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:17:43.207235    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e10b86b-aec5-4133-8950-858d0af61b95-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-dn9ft\" (UID: \"7e10b86b-aec5-4133-8950-858d0af61b95\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dn9ft"
Sep 18 17:17:43.207340 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:17:43.207320    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e10b86b-aec5-4133-8950-858d0af61b95-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-dn9ft\" (UID: \"7e10b86b-aec5-4133-8950-858d0af61b95\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dn9ft"
Sep 18 17:17:43.209218 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:17:43.209201    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e10b86b-aec5-4133-8950-858d0af61b95-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-dn9ft\" (UID: \"7e10b86b-aec5-4133-8950-858d0af61b95\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dn9ft"
Sep 18 17:17:43.217939 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:17:43.217917    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x6s4\" (UniqueName: \"kubernetes.io/projected/7e10b86b-aec5-4133-8950-858d0af61b95-kube-api-access-4x6s4\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-dn9ft\" (UID: \"7e10b86b-aec5-4133-8950-858d0af61b95\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dn9ft"
Sep 18 17:17:43.283317 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:17:43.283283    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dn9ft"
Sep 18 17:17:43.565292 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:17:43.565266    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dn9ft"]
Sep 18 17:17:43.567726 ip-10-0-143-5 kubenswrapper[2881]: W0918 17:17:43.567690    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e10b86b_aec5_4133_8950_858d0af61b95.slice/crio-035f07c0a2ce79d09935a02dedc2798f04a38fc7397df73e802bb3dfecb6ee8c WatchSource:0}: Error finding container 035f07c0a2ce79d09935a02dedc2798f04a38fc7397df73e802bb3dfecb6ee8c: Status 404 returned error can't find the container with id 035f07c0a2ce79d09935a02dedc2798f04a38fc7397df73e802bb3dfecb6ee8c
Sep 18 17:17:44.202755 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:17:44.202716    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dn9ft" event={"ID":"7e10b86b-aec5-4133-8950-858d0af61b95","Type":"ContainerStarted","Data":"69f094e54946364fdbe0b074f94f6d6b9b947d44d393f04c37d1730bfe4582a0"}
Sep 18 17:17:44.203188 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:17:44.202780    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dn9ft" event={"ID":"7e10b86b-aec5-4133-8950-858d0af61b95","Type":"ContainerStarted","Data":"035f07c0a2ce79d09935a02dedc2798f04a38fc7397df73e802bb3dfecb6ee8c"}
Sep 18 17:17:48.220302 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:17:48.220264    2881 generic.go:356] "Generic (PLEG): container finished" podID="7e10b86b-aec5-4133-8950-858d0af61b95" containerID="69f094e54946364fdbe0b074f94f6d6b9b947d44d393f04c37d1730bfe4582a0" exitCode=0
Sep 18 17:17:48.220729 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:17:48.220342    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dn9ft" event={"ID":"7e10b86b-aec5-4133-8950-858d0af61b95","Type":"ContainerDied","Data":"69f094e54946364fdbe0b074f94f6d6b9b947d44d393f04c37d1730bfe4582a0"}
Sep 18 17:17:49.226992 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:17:49.226960    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dn9ft" event={"ID":"7e10b86b-aec5-4133-8950-858d0af61b95","Type":"ContainerStarted","Data":"15a77b9d06e1b687b3b47f6ed0ceb3ae96bc3dca597957f644879914b0ec7a45"}
Sep 18 17:17:49.227414 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:17:49.227022    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dn9ft" event={"ID":"7e10b86b-aec5-4133-8950-858d0af61b95","Type":"ContainerStarted","Data":"c5ee2dbfdf54ef7bbf2e6508f1ea430015d061f80a49a5132af9be798ed7dbe2"}
Sep 18 17:17:49.227414 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:17:49.227260    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dn9ft"
Sep 18 17:17:49.260150 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:17:49.260096    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dn9ft" podStartSLOduration=3.186663514 podStartE2EDuration="7.260077938s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.073414424s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:17:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:17:49.258369272 +0000 UTC m=+1459.458224558" watchObservedRunningTime="2026-09-18 17:17:49.260077938 +0000 UTC m=+1459.459933224"
Sep 18 17:17:50.231298 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:17:50.231266    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dn9ft"
Sep 18 17:17:56.242207 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:17:56.242173    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dn9ft"
Sep 18 17:18:26.243516 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:18:26.243473    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dn9ft" podUID="7e10b86b-aec5-4133-8950-858d0af61b95" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.33:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 18 17:18:36.243035 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:18:36.242992    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dn9ft" podUID="7e10b86b-aec5-4133-8950-858d0af61b95" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.33:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 18 17:18:46.242714 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:18:46.242673    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dn9ft" podUID="7e10b86b-aec5-4133-8950-858d0af61b95" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.33:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 18 17:18:56.243477 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:18:56.243433    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dn9ft" podUID="7e10b86b-aec5-4133-8950-858d0af61b95" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.33:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 18 17:19:05.888441 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:05.888391    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dn9ft" podUID="7e10b86b-aec5-4133-8950-858d0af61b95" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.33:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 18 17:19:16.437691 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:16.437653    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dn9ft"
Sep 18 17:19:23.100210 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:23.100173    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dn9ft"]
Sep 18 17:19:23.100674 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:23.100454    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dn9ft" podUID="7e10b86b-aec5-4133-8950-858d0af61b95" containerName="kserve-container" containerID="cri-o://c5ee2dbfdf54ef7bbf2e6508f1ea430015d061f80a49a5132af9be798ed7dbe2" gracePeriod=30
Sep 18 17:19:23.100674 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:23.100501    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dn9ft" podUID="7e10b86b-aec5-4133-8950-858d0af61b95" containerName="kube-rbac-proxy" containerID="cri-o://15a77b9d06e1b687b3b47f6ed0ceb3ae96bc3dca597957f644879914b0ec7a45" gracePeriod=30
Sep 18 17:19:23.658459 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:23.658420    2881 generic.go:356] "Generic (PLEG): container finished" podID="7e10b86b-aec5-4133-8950-858d0af61b95" containerID="15a77b9d06e1b687b3b47f6ed0ceb3ae96bc3dca597957f644879914b0ec7a45" exitCode=2
Sep 18 17:19:23.658660 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:23.658488    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dn9ft" event={"ID":"7e10b86b-aec5-4133-8950-858d0af61b95","Type":"ContainerDied","Data":"15a77b9d06e1b687b3b47f6ed0ceb3ae96bc3dca597957f644879914b0ec7a45"}
Sep 18 17:19:25.888552 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:25.888505    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dn9ft" podUID="7e10b86b-aec5-4133-8950-858d0af61b95" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.33:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 18 17:19:26.237860 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:26.237754    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dn9ft" podUID="7e10b86b-aec5-4133-8950-858d0af61b95" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Sep 18 17:19:29.451408 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:29.451376    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dn9ft"
Sep 18 17:19:29.484603 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:29.484562    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e10b86b-aec5-4133-8950-858d0af61b95-proxy-tls\") pod \"7e10b86b-aec5-4133-8950-858d0af61b95\" (UID: \"7e10b86b-aec5-4133-8950-858d0af61b95\") "
Sep 18 17:19:29.484807 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:29.484634    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e10b86b-aec5-4133-8950-858d0af61b95-kserve-provision-location\") pod \"7e10b86b-aec5-4133-8950-858d0af61b95\" (UID: \"7e10b86b-aec5-4133-8950-858d0af61b95\") "
Sep 18 17:19:29.484807 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:29.484672    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e10b86b-aec5-4133-8950-858d0af61b95-kube-rbac-proxy-sar-config\") pod \"7e10b86b-aec5-4133-8950-858d0af61b95\" (UID: \"7e10b86b-aec5-4133-8950-858d0af61b95\") "
Sep 18 17:19:29.484807 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:29.484714    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e10b86b-aec5-4133-8950-858d0af61b95-cabundle-cert\") pod \"7e10b86b-aec5-4133-8950-858d0af61b95\" (UID: \"7e10b86b-aec5-4133-8950-858d0af61b95\") "
Sep 18 17:19:29.484807 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:29.484774    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4x6s4\" (UniqueName: \"kubernetes.io/projected/7e10b86b-aec5-4133-8950-858d0af61b95-kube-api-access-4x6s4\") pod \"7e10b86b-aec5-4133-8950-858d0af61b95\" (UID: \"7e10b86b-aec5-4133-8950-858d0af61b95\") "
Sep 18 17:19:29.485061 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:29.485030    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e10b86b-aec5-4133-8950-858d0af61b95-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7e10b86b-aec5-4133-8950-858d0af61b95" (UID: "7e10b86b-aec5-4133-8950-858d0af61b95"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:19:29.485131 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:29.485097    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e10b86b-aec5-4133-8950-858d0af61b95-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7e10b86b-aec5-4133-8950-858d0af61b95" (UID: "7e10b86b-aec5-4133-8950-858d0af61b95"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:19:29.485131 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:29.485095    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e10b86b-aec5-4133-8950-858d0af61b95-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7e10b86b-aec5-4133-8950-858d0af61b95" (UID: "7e10b86b-aec5-4133-8950-858d0af61b95"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:19:29.486904 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:29.486868    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e10b86b-aec5-4133-8950-858d0af61b95-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7e10b86b-aec5-4133-8950-858d0af61b95" (UID: "7e10b86b-aec5-4133-8950-858d0af61b95"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:19:29.487064 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:29.486958    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e10b86b-aec5-4133-8950-858d0af61b95-kube-api-access-4x6s4" (OuterVolumeSpecName: "kube-api-access-4x6s4") pod "7e10b86b-aec5-4133-8950-858d0af61b95" (UID: "7e10b86b-aec5-4133-8950-858d0af61b95"). InnerVolumeSpecName "kube-api-access-4x6s4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:19:29.586541 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:29.586419    2881 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e10b86b-aec5-4133-8950-858d0af61b95-proxy-tls\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:19:29.586541 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:29.586470    2881 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e10b86b-aec5-4133-8950-858d0af61b95-kserve-provision-location\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:19:29.586541 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:29.586487    2881 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e10b86b-aec5-4133-8950-858d0af61b95-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:19:29.586541 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:29.586497    2881 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e10b86b-aec5-4133-8950-858d0af61b95-cabundle-cert\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:19:29.586541 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:29.586507    2881 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4x6s4\" (UniqueName: \"kubernetes.io/projected/7e10b86b-aec5-4133-8950-858d0af61b95-kube-api-access-4x6s4\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:19:29.688365 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:29.688318    2881 generic.go:356] "Generic (PLEG): container finished" podID="7e10b86b-aec5-4133-8950-858d0af61b95" containerID="c5ee2dbfdf54ef7bbf2e6508f1ea430015d061f80a49a5132af9be798ed7dbe2" exitCode=0
Sep 18 17:19:29.688554 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:29.688390    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dn9ft"
Sep 18 17:19:29.688554 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:29.688389    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dn9ft" event={"ID":"7e10b86b-aec5-4133-8950-858d0af61b95","Type":"ContainerDied","Data":"c5ee2dbfdf54ef7bbf2e6508f1ea430015d061f80a49a5132af9be798ed7dbe2"}
Sep 18 17:19:29.688554 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:29.688454    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dn9ft" event={"ID":"7e10b86b-aec5-4133-8950-858d0af61b95","Type":"ContainerDied","Data":"035f07c0a2ce79d09935a02dedc2798f04a38fc7397df73e802bb3dfecb6ee8c"}
Sep 18 17:19:29.688554 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:29.688483    2881 scope.go:117] "RemoveContainer" containerID="15a77b9d06e1b687b3b47f6ed0ceb3ae96bc3dca597957f644879914b0ec7a45"
Sep 18 17:19:29.697831 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:29.697810    2881 scope.go:117] "RemoveContainer" containerID="c5ee2dbfdf54ef7bbf2e6508f1ea430015d061f80a49a5132af9be798ed7dbe2"
Sep 18 17:19:29.706343 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:29.706322    2881 scope.go:117] "RemoveContainer" containerID="69f094e54946364fdbe0b074f94f6d6b9b947d44d393f04c37d1730bfe4582a0"
Sep 18 17:19:29.714886 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:29.714865    2881 scope.go:117] "RemoveContainer" containerID="15a77b9d06e1b687b3b47f6ed0ceb3ae96bc3dca597957f644879914b0ec7a45"
Sep 18 17:19:29.715278 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:19:29.715249    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15a77b9d06e1b687b3b47f6ed0ceb3ae96bc3dca597957f644879914b0ec7a45\": container with ID starting with 15a77b9d06e1b687b3b47f6ed0ceb3ae96bc3dca597957f644879914b0ec7a45 not found: ID does not exist" containerID="15a77b9d06e1b687b3b47f6ed0ceb3ae96bc3dca597957f644879914b0ec7a45"
Sep 18 17:19:29.715372 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:29.715292    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15a77b9d06e1b687b3b47f6ed0ceb3ae96bc3dca597957f644879914b0ec7a45"} err="failed to get container status \"15a77b9d06e1b687b3b47f6ed0ceb3ae96bc3dca597957f644879914b0ec7a45\": rpc error: code = NotFound desc = could not find container \"15a77b9d06e1b687b3b47f6ed0ceb3ae96bc3dca597957f644879914b0ec7a45\": container with ID starting with 15a77b9d06e1b687b3b47f6ed0ceb3ae96bc3dca597957f644879914b0ec7a45 not found: ID does not exist"
Sep 18 17:19:29.715372 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:29.715319    2881 scope.go:117] "RemoveContainer" containerID="c5ee2dbfdf54ef7bbf2e6508f1ea430015d061f80a49a5132af9be798ed7dbe2"
Sep 18 17:19:29.715629 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:19:29.715608    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5ee2dbfdf54ef7bbf2e6508f1ea430015d061f80a49a5132af9be798ed7dbe2\": container with ID starting with c5ee2dbfdf54ef7bbf2e6508f1ea430015d061f80a49a5132af9be798ed7dbe2 not found: ID does not exist" containerID="c5ee2dbfdf54ef7bbf2e6508f1ea430015d061f80a49a5132af9be798ed7dbe2"
Sep 18 17:19:29.715693 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:29.715636    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5ee2dbfdf54ef7bbf2e6508f1ea430015d061f80a49a5132af9be798ed7dbe2"} err="failed to get container status \"c5ee2dbfdf54ef7bbf2e6508f1ea430015d061f80a49a5132af9be798ed7dbe2\": rpc error: code = NotFound desc = could not find container \"c5ee2dbfdf54ef7bbf2e6508f1ea430015d061f80a49a5132af9be798ed7dbe2\": container with ID starting with c5ee2dbfdf54ef7bbf2e6508f1ea430015d061f80a49a5132af9be798ed7dbe2 not found: ID does not exist"
Sep 18 17:19:29.715693 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:29.715652    2881 scope.go:117] "RemoveContainer" containerID="69f094e54946364fdbe0b074f94f6d6b9b947d44d393f04c37d1730bfe4582a0"
Sep 18 17:19:29.715903 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:19:29.715882    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69f094e54946364fdbe0b074f94f6d6b9b947d44d393f04c37d1730bfe4582a0\": container with ID starting with 69f094e54946364fdbe0b074f94f6d6b9b947d44d393f04c37d1730bfe4582a0 not found: ID does not exist" containerID="69f094e54946364fdbe0b074f94f6d6b9b947d44d393f04c37d1730bfe4582a0"
Sep 18 17:19:29.715944 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:29.715909    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69f094e54946364fdbe0b074f94f6d6b9b947d44d393f04c37d1730bfe4582a0"} err="failed to get container status \"69f094e54946364fdbe0b074f94f6d6b9b947d44d393f04c37d1730bfe4582a0\": rpc error: code = NotFound desc = could not find container \"69f094e54946364fdbe0b074f94f6d6b9b947d44d393f04c37d1730bfe4582a0\": container with ID starting with 69f094e54946364fdbe0b074f94f6d6b9b947d44d393f04c37d1730bfe4582a0 not found: ID does not exist"
Sep 18 17:19:29.720938 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:29.720907    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dn9ft"]
Sep 18 17:19:29.726243 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:29.726204    2881 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dn9ft"]
Sep 18 17:19:29.894023 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:29.893941    2881 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e10b86b-aec5-4133-8950-858d0af61b95" path="/var/lib/kubelet/pods/7e10b86b-aec5-4133-8950-858d0af61b95/volumes"
Sep 18 17:19:45.752946 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:45.752900    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/curl-metrics-202c4c4c"]
Sep 18 17:19:45.753511 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:45.753488    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e10b86b-aec5-4133-8950-858d0af61b95" containerName="kserve-container"
Sep 18 17:19:45.753586 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:45.753515    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e10b86b-aec5-4133-8950-858d0af61b95" containerName="kserve-container"
Sep 18 17:19:45.753586 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:45.753530    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e10b86b-aec5-4133-8950-858d0af61b95" containerName="storage-initializer"
Sep 18 17:19:45.753586 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:45.753539    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e10b86b-aec5-4133-8950-858d0af61b95" containerName="storage-initializer"
Sep 18 17:19:45.753586 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:45.753553    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e10b86b-aec5-4133-8950-858d0af61b95" containerName="kube-rbac-proxy"
Sep 18 17:19:45.753586 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:45.753562    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e10b86b-aec5-4133-8950-858d0af61b95" containerName="kube-rbac-proxy"
Sep 18 17:19:45.753834 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:45.753641    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e10b86b-aec5-4133-8950-858d0af61b95" containerName="kserve-container"
Sep 18 17:19:45.753834 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:45.753658    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e10b86b-aec5-4133-8950-858d0af61b95" containerName="kube-rbac-proxy"
Sep 18 17:19:45.755644 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:45.755622    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-202c4c4c"
Sep 18 17:19:45.762017 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:45.761989    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 18 17:19:45.762017 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:45.761995    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 18 17:19:45.763208 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:45.762945    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-q8nb7\""
Sep 18 17:19:45.767849 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:45.767821    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-202c4c4c"]
Sep 18 17:19:45.835163 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:45.835113    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vgmq\" (UniqueName: \"kubernetes.io/projected/50045576-8443-4523-8c04-bb26dfd5f17b-kube-api-access-7vgmq\") pod \"curl-metrics-202c4c4c\" (UID: \"50045576-8443-4523-8c04-bb26dfd5f17b\") " pod="kserve/curl-metrics-202c4c4c"
Sep 18 17:19:45.936647 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:45.936597    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vgmq\" (UniqueName: \"kubernetes.io/projected/50045576-8443-4523-8c04-bb26dfd5f17b-kube-api-access-7vgmq\") pod \"curl-metrics-202c4c4c\" (UID: \"50045576-8443-4523-8c04-bb26dfd5f17b\") " pod="kserve/curl-metrics-202c4c4c"
Sep 18 17:19:45.946733 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:45.946701    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vgmq\" (UniqueName: \"kubernetes.io/projected/50045576-8443-4523-8c04-bb26dfd5f17b-kube-api-access-7vgmq\") pod \"curl-metrics-202c4c4c\" (UID: \"50045576-8443-4523-8c04-bb26dfd5f17b\") " pod="kserve/curl-metrics-202c4c4c"
Sep 18 17:19:46.077533 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:46.077492    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-202c4c4c"
Sep 18 17:19:46.353632 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:46.353603    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-202c4c4c"]
Sep 18 17:19:46.356494 ip-10-0-143-5 kubenswrapper[2881]: W0918 17:19:46.356463    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod50045576_8443_4523_8c04_bb26dfd5f17b.slice/crio-6300cfd1340bd1cfa08cc837b1610da8c24ce41b48ec90dfbda6f2cea76d523e WatchSource:0}: Error finding container 6300cfd1340bd1cfa08cc837b1610da8c24ce41b48ec90dfbda6f2cea76d523e: Status 404 returned error can't find the container with id 6300cfd1340bd1cfa08cc837b1610da8c24ce41b48ec90dfbda6f2cea76d523e
Sep 18 17:19:46.358561 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:46.358538    2881 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 17:19:46.771603 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:46.771557    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-202c4c4c" event={"ID":"50045576-8443-4523-8c04-bb26dfd5f17b","Type":"ContainerStarted","Data":"6300cfd1340bd1cfa08cc837b1610da8c24ce41b48ec90dfbda6f2cea76d523e"}
Sep 18 17:19:47.778149 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:47.778116    2881 generic.go:356] "Generic (PLEG): container finished" podID="50045576-8443-4523-8c04-bb26dfd5f17b" containerID="151a2c0a0495bd3347e28ec7e86179f5c9b8ab85b22be81aaa94f3daedf3c01e" exitCode=0
Sep 18 17:19:47.778594 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:47.778202    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-202c4c4c" event={"ID":"50045576-8443-4523-8c04-bb26dfd5f17b","Type":"ContainerDied","Data":"151a2c0a0495bd3347e28ec7e86179f5c9b8ab85b22be81aaa94f3daedf3c01e"}
Sep 18 17:19:48.917837 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:48.917808    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-202c4c4c"
Sep 18 17:19:48.971006 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:48.970975    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_curl-metrics-202c4c4c_50045576-8443-4523-8c04-bb26dfd5f17b/curl-metrics-202c4c4c/0.log"
Sep 18 17:19:49.000280 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:49.000214    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/curl-metrics-202c4c4c"]
Sep 18 17:19:49.007696 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:49.007659    2881 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/curl-metrics-202c4c4c"]
Sep 18 17:19:49.066276 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:49.066163    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7vgmq\" (UniqueName: \"kubernetes.io/projected/50045576-8443-4523-8c04-bb26dfd5f17b-kube-api-access-7vgmq\") pod \"50045576-8443-4523-8c04-bb26dfd5f17b\" (UID: \"50045576-8443-4523-8c04-bb26dfd5f17b\") "
Sep 18 17:19:49.068391 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:49.068357    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50045576-8443-4523-8c04-bb26dfd5f17b-kube-api-access-7vgmq" (OuterVolumeSpecName: "kube-api-access-7vgmq") pod "50045576-8443-4523-8c04-bb26dfd5f17b" (UID: "50045576-8443-4523-8c04-bb26dfd5f17b"). InnerVolumeSpecName "kube-api-access-7vgmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:19:49.167566 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:49.167525    2881 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vgmq\" (UniqueName: \"kubernetes.io/projected/50045576-8443-4523-8c04-bb26dfd5f17b-kube-api-access-7vgmq\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:19:49.789078 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:49.789042    2881 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6300cfd1340bd1cfa08cc837b1610da8c24ce41b48ec90dfbda6f2cea76d523e"
Sep 18 17:19:49.789078 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:49.789057    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-202c4c4c"
Sep 18 17:19:49.799342 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:19:49.799308    2881 status_manager.go:1018] "Failed to get status for pod" err="pods \"curl-metrics-202c4c4c\" is forbidden: User \"system:node:ip-10-0-143-5.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kserve\": no relationship found between node 'ip-10-0-143-5.ec2.internal' and this object" podUID="50045576-8443-4523-8c04-bb26dfd5f17b" pod="kserve/curl-metrics-202c4c4c"
Sep 18 17:19:49.892445 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:19:49.892401    2881 status_manager.go:1018] "Failed to get status for pod" err="pods \"curl-metrics-202c4c4c\" is forbidden: User \"system:node:ip-10-0-143-5.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kserve\": no relationship found between node 'ip-10-0-143-5.ec2.internal' and this object" podUID="50045576-8443-4523-8c04-bb26dfd5f17b" pod="kserve/curl-metrics-202c4c4c"
Sep 18 17:19:49.893704 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:49.893681    2881 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50045576-8443-4523-8c04-bb26dfd5f17b" path="/var/lib/kubelet/pods/50045576-8443-4523-8c04-bb26dfd5f17b/volumes"
Sep 18 17:19:54.404990 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:54.404946    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/curl-metrics-49cb11ae"]
Sep 18 17:19:54.405469 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:54.405320    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50045576-8443-4523-8c04-bb26dfd5f17b" containerName="curl-metrics-202c4c4c"
Sep 18 17:19:54.405469 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:54.405335    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="50045576-8443-4523-8c04-bb26dfd5f17b" containerName="curl-metrics-202c4c4c"
Sep 18 17:19:54.405469 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:54.405401    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="50045576-8443-4523-8c04-bb26dfd5f17b" containerName="curl-metrics-202c4c4c"
Sep 18 17:19:54.407448 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:54.407432    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-49cb11ae"
Sep 18 17:19:54.410963 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:54.410929    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 18 17:19:54.411160 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:54.411115    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-q8nb7\""
Sep 18 17:19:54.411992 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:54.411974    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 18 17:19:54.425065 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:54.425032    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-49cb11ae"]
Sep 18 17:19:54.514408 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:54.514364    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rg2nz\" (UniqueName: \"kubernetes.io/projected/110c3823-5a7b-4b8a-889b-59dd88ad5ba9-kube-api-access-rg2nz\") pod \"curl-metrics-49cb11ae\" (UID: \"110c3823-5a7b-4b8a-889b-59dd88ad5ba9\") " pod="kserve/curl-metrics-49cb11ae"
Sep 18 17:19:54.615279 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:54.615199    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg2nz\" (UniqueName: \"kubernetes.io/projected/110c3823-5a7b-4b8a-889b-59dd88ad5ba9-kube-api-access-rg2nz\") pod \"curl-metrics-49cb11ae\" (UID: \"110c3823-5a7b-4b8a-889b-59dd88ad5ba9\") " pod="kserve/curl-metrics-49cb11ae"
Sep 18 17:19:54.625094 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:54.625057    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg2nz\" (UniqueName: \"kubernetes.io/projected/110c3823-5a7b-4b8a-889b-59dd88ad5ba9-kube-api-access-rg2nz\") pod \"curl-metrics-49cb11ae\" (UID: \"110c3823-5a7b-4b8a-889b-59dd88ad5ba9\") " pod="kserve/curl-metrics-49cb11ae"
Sep 18 17:19:54.727430 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:54.727311    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-49cb11ae"
Sep 18 17:19:54.998718 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:54.998689    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-49cb11ae"]
Sep 18 17:19:55.001412 ip-10-0-143-5 kubenswrapper[2881]: W0918 17:19:55.001382    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod110c3823_5a7b_4b8a_889b_59dd88ad5ba9.slice/crio-9c56e151e04a8349a7a352e34710f53cda00020ced7f8ea0b14c1ed329b9c6ba WatchSource:0}: Error finding container 9c56e151e04a8349a7a352e34710f53cda00020ced7f8ea0b14c1ed329b9c6ba: Status 404 returned error can't find the container with id 9c56e151e04a8349a7a352e34710f53cda00020ced7f8ea0b14c1ed329b9c6ba
Sep 18 17:19:55.815950 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:55.815911    2881 generic.go:356] "Generic (PLEG): container finished" podID="110c3823-5a7b-4b8a-889b-59dd88ad5ba9" containerID="e29141587efcfec8c1863b7167c5f4b5a6181de71b8c85e59797ed5ee2098476" exitCode=0
Sep 18 17:19:55.816371 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:55.815996    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-49cb11ae" event={"ID":"110c3823-5a7b-4b8a-889b-59dd88ad5ba9","Type":"ContainerDied","Data":"e29141587efcfec8c1863b7167c5f4b5a6181de71b8c85e59797ed5ee2098476"}
Sep 18 17:19:55.816371 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:55.816040    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-49cb11ae" event={"ID":"110c3823-5a7b-4b8a-889b-59dd88ad5ba9","Type":"ContainerStarted","Data":"9c56e151e04a8349a7a352e34710f53cda00020ced7f8ea0b14c1ed329b9c6ba"}
Sep 18 17:19:56.951134 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:56.951107    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-49cb11ae"
Sep 18 17:19:56.980515 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:56.980485    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_curl-metrics-49cb11ae_110c3823-5a7b-4b8a-889b-59dd88ad5ba9/curl-metrics-49cb11ae/0.log"
Sep 18 17:19:57.003427 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:57.003386    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/curl-metrics-49cb11ae"]
Sep 18 17:19:57.007773 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:57.007736    2881 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/curl-metrics-49cb11ae"]
Sep 18 17:19:57.036396 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:57.036360    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rg2nz\" (UniqueName: \"kubernetes.io/projected/110c3823-5a7b-4b8a-889b-59dd88ad5ba9-kube-api-access-rg2nz\") pod \"110c3823-5a7b-4b8a-889b-59dd88ad5ba9\" (UID: \"110c3823-5a7b-4b8a-889b-59dd88ad5ba9\") "
Sep 18 17:19:57.038626 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:57.038582    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/110c3823-5a7b-4b8a-889b-59dd88ad5ba9-kube-api-access-rg2nz" (OuterVolumeSpecName: "kube-api-access-rg2nz") pod "110c3823-5a7b-4b8a-889b-59dd88ad5ba9" (UID: "110c3823-5a7b-4b8a-889b-59dd88ad5ba9"). InnerVolumeSpecName "kube-api-access-rg2nz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:19:57.137327 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:57.137207    2881 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rg2nz\" (UniqueName: \"kubernetes.io/projected/110c3823-5a7b-4b8a-889b-59dd88ad5ba9-kube-api-access-rg2nz\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:19:57.826150 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:57.826116    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-49cb11ae"
Sep 18 17:19:57.826354 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:57.826158    2881 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c56e151e04a8349a7a352e34710f53cda00020ced7f8ea0b14c1ed329b9c6ba"
Sep 18 17:19:57.836616 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:19:57.836582    2881 status_manager.go:1018] "Failed to get status for pod" err="pods \"curl-metrics-49cb11ae\" is forbidden: User \"system:node:ip-10-0-143-5.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kserve\": no relationship found between node 'ip-10-0-143-5.ec2.internal' and this object" podUID="110c3823-5a7b-4b8a-889b-59dd88ad5ba9" pod="kserve/curl-metrics-49cb11ae"
Sep 18 17:19:57.893321 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:19:57.893285    2881 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="110c3823-5a7b-4b8a-889b-59dd88ad5ba9" path="/var/lib/kubelet/pods/110c3823-5a7b-4b8a-889b-59dd88ad5ba9/volumes"
Sep 18 17:20:02.374350 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:20:02.374310    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-d448d4989-knt95"]
Sep 18 17:20:02.374849 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:20:02.374688    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="110c3823-5a7b-4b8a-889b-59dd88ad5ba9" containerName="curl-metrics-49cb11ae"
Sep 18 17:20:02.374849 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:20:02.374702    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="110c3823-5a7b-4b8a-889b-59dd88ad5ba9" containerName="curl-metrics-49cb11ae"
Sep 18 17:20:02.374849 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:20:02.374772    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="110c3823-5a7b-4b8a-889b-59dd88ad5ba9" containerName="curl-metrics-49cb11ae"
Sep 18 17:20:02.378211 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:20:02.378188    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d448d4989-knt95"
Sep 18 17:20:02.381103 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:20:02.381067    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-predictor-serving-cert\""
Sep 18 17:20:02.381277 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:20:02.381152    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5hv9v\""
Sep 18 17:20:02.381277 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:20:02.381163    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-kube-rbac-proxy-sar-config\""
Sep 18 17:20:02.381277 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:20:02.381169    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 17:20:02.381277 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:20:02.381201    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 17:20:02.381277 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:20:02.381155    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 17:20:02.381541 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:20:02.381090    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 17:20:02.391218 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:20:02.391189    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-d448d4989-knt95"]
Sep 18 17:20:02.482440 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:20:02.482401    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srzmb\" (UniqueName: \"kubernetes.io/projected/4375b2a5-4110-4096-ac03-759325e95a54-kube-api-access-srzmb\") pod \"isvc-sklearn-predictor-d448d4989-knt95\" (UID: \"4375b2a5-4110-4096-ac03-759325e95a54\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d448d4989-knt95"
Sep 18 17:20:02.482655 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:20:02.482465    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4375b2a5-4110-4096-ac03-759325e95a54-cabundle-cert\") pod \"isvc-sklearn-predictor-d448d4989-knt95\" (UID: \"4375b2a5-4110-4096-ac03-759325e95a54\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d448d4989-knt95"
Sep 18 17:20:02.482655 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:20:02.482581    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4375b2a5-4110-4096-ac03-759325e95a54-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-d448d4989-knt95\" (UID: \"4375b2a5-4110-4096-ac03-759325e95a54\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d448d4989-knt95"
Sep 18 17:20:02.482655 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:20:02.482618    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4375b2a5-4110-4096-ac03-759325e95a54-kserve-provision-location\") pod \"isvc-sklearn-predictor-d448d4989-knt95\" (UID: \"4375b2a5-4110-4096-ac03-759325e95a54\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d448d4989-knt95"
Sep 18 17:20:02.482829 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:20:02.482682    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4375b2a5-4110-4096-ac03-759325e95a54-proxy-tls\") pod \"isvc-sklearn-predictor-d448d4989-knt95\" (UID: \"4375b2a5-4110-4096-ac03-759325e95a54\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d448d4989-knt95"
Sep 18 17:20:02.583417 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:20:02.583375    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4375b2a5-4110-4096-ac03-759325e95a54-cabundle-cert\") pod \"isvc-sklearn-predictor-d448d4989-knt95\" (UID: \"4375b2a5-4110-4096-ac03-759325e95a54\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d448d4989-knt95"
Sep 18 17:20:02.583636 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:20:02.583444    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4375b2a5-4110-4096-ac03-759325e95a54-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-d448d4989-knt95\" (UID: \"4375b2a5-4110-4096-ac03-759325e95a54\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d448d4989-knt95"
Sep 18 17:20:02.583636 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:20:02.583464    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4375b2a5-4110-4096-ac03-759325e95a54-kserve-provision-location\") pod \"isvc-sklearn-predictor-d448d4989-knt95\" (UID: \"4375b2a5-4110-4096-ac03-759325e95a54\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d448d4989-knt95"
Sep 18 17:20:02.583636 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:20:02.583501    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4375b2a5-4110-4096-ac03-759325e95a54-proxy-tls\") pod \"isvc-sklearn-predictor-d448d4989-knt95\" (UID: \"4375b2a5-4110-4096-ac03-759325e95a54\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d448d4989-knt95"
Sep 18 17:20:02.583636 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:20:02.583526    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srzmb\" (UniqueName: \"kubernetes.io/projected/4375b2a5-4110-4096-ac03-759325e95a54-kube-api-access-srzmb\") pod \"isvc-sklearn-predictor-d448d4989-knt95\" (UID: \"4375b2a5-4110-4096-ac03-759325e95a54\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d448d4989-knt95"
Sep 18 17:20:02.583983 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:20:02.583947    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4375b2a5-4110-4096-ac03-759325e95a54-kserve-provision-location\") pod \"isvc-sklearn-predictor-d448d4989-knt95\" (UID: \"4375b2a5-4110-4096-ac03-759325e95a54\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d448d4989-knt95"
Sep 18 17:20:02.584194 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:20:02.584173    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4375b2a5-4110-4096-ac03-759325e95a54-cabundle-cert\") pod \"isvc-sklearn-predictor-d448d4989-knt95\" (UID: \"4375b2a5-4110-4096-ac03-759325e95a54\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d448d4989-knt95"
Sep 18 17:20:02.584333 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:20:02.584310    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4375b2a5-4110-4096-ac03-759325e95a54-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-d448d4989-knt95\" (UID: \"4375b2a5-4110-4096-ac03-759325e95a54\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d448d4989-knt95"
Sep 18 17:20:02.586408 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:20:02.586380    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4375b2a5-4110-4096-ac03-759325e95a54-proxy-tls\") pod \"isvc-sklearn-predictor-d448d4989-knt95\" (UID: \"4375b2a5-4110-4096-ac03-759325e95a54\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d448d4989-knt95"
Sep 18 17:20:02.594641 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:20:02.594605    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srzmb\" (UniqueName: \"kubernetes.io/projected/4375b2a5-4110-4096-ac03-759325e95a54-kube-api-access-srzmb\") pod \"isvc-sklearn-predictor-d448d4989-knt95\" (UID: \"4375b2a5-4110-4096-ac03-759325e95a54\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d448d4989-knt95"
Sep 18 17:20:02.689820 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:20:02.689712    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d448d4989-knt95"
Sep 18 17:20:02.969999 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:20:02.969971    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-d448d4989-knt95"]
Sep 18 17:20:02.972095 ip-10-0-143-5 kubenswrapper[2881]: W0918 17:20:02.972063    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4375b2a5_4110_4096_ac03_759325e95a54.slice/crio-1a198117e515222ee5214ae3351bbe9fe602df8bacda86456c5c22ac47251954 WatchSource:0}: Error finding container 1a198117e515222ee5214ae3351bbe9fe602df8bacda86456c5c22ac47251954: Status 404 returned error can't find the container with id 1a198117e515222ee5214ae3351bbe9fe602df8bacda86456c5c22ac47251954
Sep 18 17:20:03.854042 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:20:03.853998    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d448d4989-knt95" event={"ID":"4375b2a5-4110-4096-ac03-759325e95a54","Type":"ContainerStarted","Data":"a276abc6bf0837299eecc62a0d97d4b594952e52cea7b242216122fa7e5d685a"}
Sep 18 17:20:03.854524 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:20:03.854091    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d448d4989-knt95" event={"ID":"4375b2a5-4110-4096-ac03-759325e95a54","Type":"ContainerStarted","Data":"1a198117e515222ee5214ae3351bbe9fe602df8bacda86456c5c22ac47251954"}
Sep 18 17:20:07.873736 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:20:07.873700    2881 generic.go:356] "Generic (PLEG): container finished" podID="4375b2a5-4110-4096-ac03-759325e95a54" containerID="a276abc6bf0837299eecc62a0d97d4b594952e52cea7b242216122fa7e5d685a" exitCode=0
Sep 18 17:20:07.874134 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:20:07.873775    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d448d4989-knt95" event={"ID":"4375b2a5-4110-4096-ac03-759325e95a54","Type":"ContainerDied","Data":"a276abc6bf0837299eecc62a0d97d4b594952e52cea7b242216122fa7e5d685a"}
Sep 18 17:20:08.880991 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:20:08.880954    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d448d4989-knt95" event={"ID":"4375b2a5-4110-4096-ac03-759325e95a54","Type":"ContainerStarted","Data":"848585143b032b747bf91ae324687abd2f2a6c1938bd4fefc10294d4c215d15e"}
Sep 18 17:20:08.881425 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:20:08.881020    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d448d4989-knt95" event={"ID":"4375b2a5-4110-4096-ac03-759325e95a54","Type":"ContainerStarted","Data":"141293d8c25660f8a5fd9dc6d80e0bb7ea423966cb904ac5d7010178d2809b98"}
Sep 18 17:20:08.881425 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:20:08.881236    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d448d4989-knt95"
Sep 18 17:20:08.918903 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:20:08.918843    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d448d4989-knt95" podStartSLOduration=3.027396714 podStartE2EDuration="6.918824642s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.891427928s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:20:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:20:08.918089805 +0000 UTC m=+1599.117945091" watchObservedRunningTime="2026-09-18 17:20:08.918824642 +0000 UTC m=+1599.118679988"
Sep 18 17:20:09.885774 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:20:09.885738    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d448d4989-knt95"
Sep 18 17:20:09.887341 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:20:09.887312    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d448d4989-knt95" podUID="4375b2a5-4110-4096-ac03-759325e95a54" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 17:20:10.890534 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:20:10.890491    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d448d4989-knt95" podUID="4375b2a5-4110-4096-ac03-759325e95a54" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 17:20:15.895211 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:20:15.895172    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d448d4989-knt95"
Sep 18 17:20:15.895921 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:20:15.895887    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d448d4989-knt95" podUID="4375b2a5-4110-4096-ac03-759325e95a54" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 17:20:25.896858 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:20:25.896812    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d448d4989-knt95" podUID="4375b2a5-4110-4096-ac03-759325e95a54" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 17:20:35.896252 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:20:35.896181    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d448d4989-knt95" podUID="4375b2a5-4110-4096-ac03-759325e95a54" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 17:20:45.895835 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:20:45.895787    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d448d4989-knt95" podUID="4375b2a5-4110-4096-ac03-759325e95a54" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 17:20:55.895812 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:20:55.895764    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d448d4989-knt95" podUID="4375b2a5-4110-4096-ac03-759325e95a54" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 17:21:05.896240 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:21:05.896130    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d448d4989-knt95" podUID="4375b2a5-4110-4096-ac03-759325e95a54" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 17:21:15.896761 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:21:15.896712    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d448d4989-knt95" podUID="4375b2a5-4110-4096-ac03-759325e95a54" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 17:21:20.888182 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:21:20.888137    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d448d4989-knt95" podUID="4375b2a5-4110-4096-ac03-759325e95a54" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 17:21:30.889437 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:21:30.889396    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d448d4989-knt95"
Sep 18 17:21:32.899523 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:21:32.899487    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-d448d4989-knt95_4375b2a5-4110-4096-ac03-759325e95a54/kserve-container/0.log"
Sep 18 17:21:34.910337 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:21:34.910302    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-d448d4989-knt95_4375b2a5-4110-4096-ac03-759325e95a54/kserve-container/0.log"
Sep 18 17:21:36.922797 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:21:36.922768    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-d448d4989-knt95_4375b2a5-4110-4096-ac03-759325e95a54/kserve-container/0.log"
Sep 18 17:21:38.934880 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:21:38.934845    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-d448d4989-knt95_4375b2a5-4110-4096-ac03-759325e95a54/kserve-container/0.log"
Sep 18 17:21:40.950018 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:21:40.949986    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-d448d4989-knt95_4375b2a5-4110-4096-ac03-759325e95a54/kserve-container/0.log"
Sep 18 17:21:42.966035 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:21:42.966006    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-d448d4989-knt95_4375b2a5-4110-4096-ac03-759325e95a54/kserve-container/0.log"
Sep 18 17:21:44.977829 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:21:44.977800    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-d448d4989-knt95_4375b2a5-4110-4096-ac03-759325e95a54/kserve-container/0.log"
Sep 18 17:21:46.993209 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:21:46.993180    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-d448d4989-knt95_4375b2a5-4110-4096-ac03-759325e95a54/kserve-container/0.log"
Sep 18 17:21:49.004837 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:21:49.004810    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-d448d4989-knt95_4375b2a5-4110-4096-ac03-759325e95a54/kserve-container/0.log"
Sep 18 17:21:51.023342 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:21:51.023313    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-d448d4989-knt95_4375b2a5-4110-4096-ac03-759325e95a54/kserve-container/0.log"
Sep 18 17:21:53.038555 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:21:53.038521    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-d448d4989-knt95_4375b2a5-4110-4096-ac03-759325e95a54/kserve-container/0.log"
Sep 18 17:21:55.050480 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:21:55.050445    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-d448d4989-knt95_4375b2a5-4110-4096-ac03-759325e95a54/kserve-container/0.log"
Sep 18 17:21:57.062262 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:21:57.062211    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-d448d4989-knt95_4375b2a5-4110-4096-ac03-759325e95a54/kserve-container/0.log"
Sep 18 17:21:59.075452 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:21:59.075403    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-d448d4989-knt95_4375b2a5-4110-4096-ac03-759325e95a54/kserve-container/0.log"
Sep 18 17:22:01.088335 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:22:01.088301    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-d448d4989-knt95_4375b2a5-4110-4096-ac03-759325e95a54/kserve-container/0.log"
Sep 18 17:22:03.101375 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:22:03.101334    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-d448d4989-knt95_4375b2a5-4110-4096-ac03-759325e95a54/kserve-container/0.log"
Sep 18 17:22:03.323128 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:22:03.323096    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-d448d4989-knt95"]
Sep 18 17:22:03.323395 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:22:03.323368    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d448d4989-knt95" podUID="4375b2a5-4110-4096-ac03-759325e95a54" containerName="kserve-container" containerID="cri-o://141293d8c25660f8a5fd9dc6d80e0bb7ea423966cb904ac5d7010178d2809b98" gracePeriod=30
Sep 18 17:22:03.323470 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:22:03.323402    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d448d4989-knt95" podUID="4375b2a5-4110-4096-ac03-759325e95a54" containerName="kube-rbac-proxy" containerID="cri-o://848585143b032b747bf91ae324687abd2f2a6c1938bd4fefc10294d4c215d15e" gracePeriod=30
Sep 18 17:22:04.394687 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:22:04.394655    2881 generic.go:356] "Generic (PLEG): container finished" podID="4375b2a5-4110-4096-ac03-759325e95a54" containerID="848585143b032b747bf91ae324687abd2f2a6c1938bd4fefc10294d4c215d15e" exitCode=2
Sep 18 17:22:04.395157 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:22:04.394726    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d448d4989-knt95" event={"ID":"4375b2a5-4110-4096-ac03-759325e95a54","Type":"ContainerDied","Data":"848585143b032b747bf91ae324687abd2f2a6c1938bd4fefc10294d4c215d15e"}
Sep 18 17:22:05.890983 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:22:05.890930    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d448d4989-knt95" podUID="4375b2a5-4110-4096-ac03-759325e95a54" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Sep 18 17:22:08.526443 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:22:08.526401    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qwr9x"]
Sep 18 17:22:08.530075 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:22:08.530051    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qwr9x"
Sep 18 17:22:08.533290 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:22:08.533270    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-predictor-serving-cert\""
Sep 18 17:22:08.533989 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:22:08.533974    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 18 17:22:08.550199 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:22:08.550165    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qwr9x"]
Sep 18 17:22:08.626315 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:22:08.626269    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9mk8\" (UniqueName: \"kubernetes.io/projected/ab69c5ba-e722-4484-b8d4-4bd1f26d86f1-kube-api-access-l9mk8\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-qwr9x\" (UID: \"ab69c5ba-e722-4484-b8d4-4bd1f26d86f1\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qwr9x"
Sep 18 17:22:08.626542 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:22:08.626324    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab69c5ba-e722-4484-b8d4-4bd1f26d86f1-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-qwr9x\" (UID: \"ab69c5ba-e722-4484-b8d4-4bd1f26d86f1\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qwr9x"
Sep 18 17:22:08.626542 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:22:08.626421    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab69c5ba-e722-4484-b8d4-4bd1f26d86f1-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-qwr9x\" (UID: \"ab69c5ba-e722-4484-b8d4-4bd1f26d86f1\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qwr9x"
Sep 18 17:22:08.626542 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:22:08.626447    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab69c5ba-e722-4484-b8d4-4bd1f26d86f1-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-qwr9x\" (UID: \"ab69c5ba-e722-4484-b8d4-4bd1f26d86f1\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qwr9x"
Sep 18 17:22:08.626542 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:22:08.626493    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab69c5ba-e722-4484-b8d4-4bd1f26d86f1-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-qwr9x\" (UID: \"ab69c5ba-e722-4484-b8d4-4bd1f26d86f1\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qwr9x"
Sep 18 17:22:08.727355 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:22:08.727308    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab69c5ba-e722-4484-b8d4-4bd1f26d86f1-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-qwr9x\" (UID: \"ab69c5ba-e722-4484-b8d4-4bd1f26d86f1\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qwr9x"
Sep 18 17:22:08.727539 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:22:08.727371    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9mk8\" (UniqueName: \"kubernetes.io/projected/ab69c5ba-e722-4484-b8d4-4bd1f26d86f1-kube-api-access-l9mk8\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-qwr9x\" (UID: \"ab69c5ba-e722-4484-b8d4-4bd1f26d86f1\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qwr9x"
Sep 18 17:22:08.727539 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:22:08.727400    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab69c5ba-e722-4484-b8d4-4bd1f26d86f1-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-qwr9x\" (UID: \"ab69c5ba-e722-4484-b8d4-4bd1f26d86f1\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qwr9x"
Sep 18 17:22:08.727539 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:22:08.727455    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab69c5ba-e722-4484-b8d4-4bd1f26d86f1-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-qwr9x\" (UID: \"ab69c5ba-e722-4484-b8d4-4bd1f26d86f1\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qwr9x"
Sep 18 17:22:08.727689 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:22:08.727586    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab69c5ba-e722-4484-b8d4-4bd1f26d86f1-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-qwr9x\" (UID: \"ab69c5ba-e722-4484-b8d4-4bd1f26d86f1\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qwr9x"
Sep 18 17:22:08.727889 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:22:08.727869    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab69c5ba-e722-4484-b8d4-4bd1f26d86f1-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-qwr9x\" (UID: \"ab69c5ba-e722-4484-b8d4-4bd1f26d86f1\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qwr9x"
Sep 18 17:22:08.728357 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:22:08.728334    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab69c5ba-e722-4484-b8d4-4bd1f26d86f1-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-qwr9x\" (UID: \"ab69c5ba-e722-4484-b8d4-4bd1f26d86f1\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qwr9x"
Sep 18 17:22:08.728357 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:22:08.728345    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab69c5ba-e722-4484-b8d4-4bd1f26d86f1-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-qwr9x\" (UID: \"ab69c5ba-e722-4484-b8d4-4bd1f26d86f1\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qwr9x"
Sep 18 17:22:08.730032 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:22:08.730012    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab69c5ba-e722-4484-b8d4-4bd1f26d86f1-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-qwr9x\" (UID: \"ab69c5ba-e722-4484-b8d4-4bd1f26d86f1\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qwr9x"
Sep 18 17:22:08.737936 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:22:08.737908    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9mk8\" (UniqueName: \"kubernetes.io/projected/ab69c5ba-e722-4484-b8d4-4bd1f26d86f1-kube-api-access-l9mk8\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-qwr9x\" (UID: \"ab69c5ba-e722-4484-b8d4-4bd1f26d86f1\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qwr9x"
Sep 18 17:22:08.843042 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:22:08.842930    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qwr9x"
Sep 18 17:22:09.126625 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:22:09.126531    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qwr9x"]
Sep 18 17:22:09.130634 ip-10-0-143-5 kubenswrapper[2881]: W0918 17:22:09.130602    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab69c5ba_e722_4484_b8d4_4bd1f26d86f1.slice/crio-54fff305cf373b94dc33b2db2ee541832429879278508dbdd8426252af7d333d WatchSource:0}: Error finding container 54fff305cf373b94dc33b2db2ee541832429879278508dbdd8426252af7d333d: Status 404 returned error can't find the container with id 54fff305cf373b94dc33b2db2ee541832429879278508dbdd8426252af7d333d
Sep 18 17:22:09.173917 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:22:09.173884    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d448d4989-knt95"
Sep 18 17:22:09.333788 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:22:09.333751    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4375b2a5-4110-4096-ac03-759325e95a54-kube-rbac-proxy-sar-config\") pod \"4375b2a5-4110-4096-ac03-759325e95a54\" (UID: \"4375b2a5-4110-4096-ac03-759325e95a54\") "
Sep 18 17:22:09.333788 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:22:09.333796    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4375b2a5-4110-4096-ac03-759325e95a54-kserve-provision-location\") pod \"4375b2a5-4110-4096-ac03-759325e95a54\" (UID: \"4375b2a5-4110-4096-ac03-759325e95a54\") "
Sep 18 17:22:09.334067 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:22:09.333820    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4375b2a5-4110-4096-ac03-759325e95a54-proxy-tls\") pod \"4375b2a5-4110-4096-ac03-759325e95a54\" (UID: \"4375b2a5-4110-4096-ac03-759325e95a54\") "
Sep 18 17:22:09.334067 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:22:09.333868    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-srzmb\" (UniqueName: \"kubernetes.io/projected/4375b2a5-4110-4096-ac03-759325e95a54-kube-api-access-srzmb\") pod \"4375b2a5-4110-4096-ac03-759325e95a54\" (UID: \"4375b2a5-4110-4096-ac03-759325e95a54\") "
Sep 18 17:22:09.334067 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:22:09.333904    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4375b2a5-4110-4096-ac03-759325e95a54-cabundle-cert\") pod \"4375b2a5-4110-4096-ac03-759325e95a54\" (UID: \"4375b2a5-4110-4096-ac03-759325e95a54\") "
Sep 18 17:22:09.334262 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:22:09.334165    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4375b2a5-4110-4096-ac03-759325e95a54-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4375b2a5-4110-4096-ac03-759325e95a54" (UID: "4375b2a5-4110-4096-ac03-759325e95a54"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:22:09.334356 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:22:09.334330    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4375b2a5-4110-4096-ac03-759325e95a54-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4375b2a5-4110-4096-ac03-759325e95a54" (UID: "4375b2a5-4110-4096-ac03-759325e95a54"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:22:09.334421 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:22:09.334393    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4375b2a5-4110-4096-ac03-759325e95a54-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4375b2a5-4110-4096-ac03-759325e95a54" (UID: "4375b2a5-4110-4096-ac03-759325e95a54"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:22:09.336111 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:22:09.336078    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4375b2a5-4110-4096-ac03-759325e95a54-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4375b2a5-4110-4096-ac03-759325e95a54" (UID: "4375b2a5-4110-4096-ac03-759325e95a54"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:22:09.336111 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:22:09.336086    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4375b2a5-4110-4096-ac03-759325e95a54-kube-api-access-srzmb" (OuterVolumeSpecName: "kube-api-access-srzmb") pod "4375b2a5-4110-4096-ac03-759325e95a54" (UID: "4375b2a5-4110-4096-ac03-759325e95a54"). InnerVolumeSpecName "kube-api-access-srzmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:22:09.419618 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:22:09.419533    2881 generic.go:356] "Generic (PLEG): container finished" podID="4375b2a5-4110-4096-ac03-759325e95a54" containerID="141293d8c25660f8a5fd9dc6d80e0bb7ea423966cb904ac5d7010178d2809b98" exitCode=0
Sep 18 17:22:09.419618 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:22:09.419597    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d448d4989-knt95"
Sep 18 17:22:09.419822 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:22:09.419618    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d448d4989-knt95" event={"ID":"4375b2a5-4110-4096-ac03-759325e95a54","Type":"ContainerDied","Data":"141293d8c25660f8a5fd9dc6d80e0bb7ea423966cb904ac5d7010178d2809b98"}
Sep 18 17:22:09.419822 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:22:09.419689    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d448d4989-knt95" event={"ID":"4375b2a5-4110-4096-ac03-759325e95a54","Type":"ContainerDied","Data":"1a198117e515222ee5214ae3351bbe9fe602df8bacda86456c5c22ac47251954"}
Sep 18 17:22:09.419822 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:22:09.419730    2881 scope.go:117] "RemoveContainer" containerID="848585143b032b747bf91ae324687abd2f2a6c1938bd4fefc10294d4c215d15e"
Sep 18 17:22:09.421702 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:22:09.421674    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qwr9x" event={"ID":"ab69c5ba-e722-4484-b8d4-4bd1f26d86f1","Type":"ContainerStarted","Data":"1e300c3cb0318f61668d46777a33a5655c449e6eeb2d9afe2815bf65ca7df001"}
Sep 18 17:22:09.421828 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:22:09.421741    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qwr9x" event={"ID":"ab69c5ba-e722-4484-b8d4-4bd1f26d86f1","Type":"ContainerStarted","Data":"54fff305cf373b94dc33b2db2ee541832429879278508dbdd8426252af7d333d"}
Sep 18 17:22:09.430912 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:22:09.430875    2881 scope.go:117] "RemoveContainer" containerID="141293d8c25660f8a5fd9dc6d80e0bb7ea423966cb904ac5d7010178d2809b98"
Sep 18 17:22:09.435568 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:22:09.435544    2881 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4375b2a5-4110-4096-ac03-759325e95a54-proxy-tls\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:22:09.435664 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:22:09.435573    2881 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srzmb\" (UniqueName: \"kubernetes.io/projected/4375b2a5-4110-4096-ac03-759325e95a54-kube-api-access-srzmb\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:22:09.435664 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:22:09.435588    2881 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4375b2a5-4110-4096-ac03-759325e95a54-cabundle-cert\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:22:09.435664 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:22:09.435602    2881 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4375b2a5-4110-4096-ac03-759325e95a54-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:22:09.435664 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:22:09.435614    2881 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4375b2a5-4110-4096-ac03-759325e95a54-kserve-provision-location\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:22:09.440999 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:22:09.440977    2881 scope.go:117] "RemoveContainer" containerID="a276abc6bf0837299eecc62a0d97d4b594952e52cea7b242216122fa7e5d685a"
Sep 18 17:22:09.450558 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:22:09.450533    2881 scope.go:117] "RemoveContainer" containerID="848585143b032b747bf91ae324687abd2f2a6c1938bd4fefc10294d4c215d15e"
Sep 18 17:22:09.451015 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:22:09.450989    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"848585143b032b747bf91ae324687abd2f2a6c1938bd4fefc10294d4c215d15e\": container with ID starting with 848585143b032b747bf91ae324687abd2f2a6c1938bd4fefc10294d4c215d15e not found: ID does not exist" containerID="848585143b032b747bf91ae324687abd2f2a6c1938bd4fefc10294d4c215d15e"
Sep 18 17:22:09.451074 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:22:09.451031    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"848585143b032b747bf91ae324687abd2f2a6c1938bd4fefc10294d4c215d15e"} err="failed to get container status \"848585143b032b747bf91ae324687abd2f2a6c1938bd4fefc10294d4c215d15e\": rpc error: code = NotFound desc = could not find container \"848585143b032b747bf91ae324687abd2f2a6c1938bd4fefc10294d4c215d15e\": container with ID starting with 848585143b032b747bf91ae324687abd2f2a6c1938bd4fefc10294d4c215d15e not found: ID does not exist"
Sep 18 17:22:09.451074 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:22:09.451058    2881 scope.go:117] "RemoveContainer" containerID="141293d8c25660f8a5fd9dc6d80e0bb7ea423966cb904ac5d7010178d2809b98"
Sep 18 17:22:09.451393 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:22:09.451371    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"141293d8c25660f8a5fd9dc6d80e0bb7ea423966cb904ac5d7010178d2809b98\": container with ID starting with 141293d8c25660f8a5fd9dc6d80e0bb7ea423966cb904ac5d7010178d2809b98 not found: ID does not exist" containerID="141293d8c25660f8a5fd9dc6d80e0bb7ea423966cb904ac5d7010178d2809b98"
Sep 18 17:22:09.451487 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:22:09.451396    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"141293d8c25660f8a5fd9dc6d80e0bb7ea423966cb904ac5d7010178d2809b98"} err="failed to get container status \"141293d8c25660f8a5fd9dc6d80e0bb7ea423966cb904ac5d7010178d2809b98\": rpc error: code = NotFound desc = could not find container \"141293d8c25660f8a5fd9dc6d80e0bb7ea423966cb904ac5d7010178d2809b98\": container with ID starting with 141293d8c25660f8a5fd9dc6d80e0bb7ea423966cb904ac5d7010178d2809b98 not found: ID does not exist"
Sep 18 17:22:09.451487 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:22:09.451413    2881 scope.go:117] "RemoveContainer" containerID="a276abc6bf0837299eecc62a0d97d4b594952e52cea7b242216122fa7e5d685a"
Sep 18 17:22:09.451728 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:22:09.451703    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a276abc6bf0837299eecc62a0d97d4b594952e52cea7b242216122fa7e5d685a\": container with ID starting with a276abc6bf0837299eecc62a0d97d4b594952e52cea7b242216122fa7e5d685a not found: ID does not exist" containerID="a276abc6bf0837299eecc62a0d97d4b594952e52cea7b242216122fa7e5d685a"
Sep 18 17:22:09.451772 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:22:09.451734    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a276abc6bf0837299eecc62a0d97d4b594952e52cea7b242216122fa7e5d685a"} err="failed to get container status \"a276abc6bf0837299eecc62a0d97d4b594952e52cea7b242216122fa7e5d685a\": rpc error: code = NotFound desc = could not find container \"a276abc6bf0837299eecc62a0d97d4b594952e52cea7b242216122fa7e5d685a\": container with ID starting with a276abc6bf0837299eecc62a0d97d4b594952e52cea7b242216122fa7e5d685a not found: ID does not exist"
Sep 18 17:22:09.485589 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:22:09.485548    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-d448d4989-knt95"]
Sep 18 17:22:09.489462 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:22:09.489431    2881 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-d448d4989-knt95"]
Sep 18 17:22:09.900475 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:22:09.900437    2881 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4375b2a5-4110-4096-ac03-759325e95a54" path="/var/lib/kubelet/pods/4375b2a5-4110-4096-ac03-759325e95a54/volumes"
Sep 18 17:22:13.441549 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:22:13.441453    2881 generic.go:356] "Generic (PLEG): container finished" podID="ab69c5ba-e722-4484-b8d4-4bd1f26d86f1" containerID="1e300c3cb0318f61668d46777a33a5655c449e6eeb2d9afe2815bf65ca7df001" exitCode=0
Sep 18 17:22:13.441549 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:22:13.441508    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qwr9x" event={"ID":"ab69c5ba-e722-4484-b8d4-4bd1f26d86f1","Type":"ContainerDied","Data":"1e300c3cb0318f61668d46777a33a5655c449e6eeb2d9afe2815bf65ca7df001"}
Sep 18 17:24:16.125343 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:24:16.125303    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qwr9x" event={"ID":"ab69c5ba-e722-4484-b8d4-4bd1f26d86f1","Type":"ContainerStarted","Data":"66892f53955ad5cd8ccee6d76b15f6e90ab3c68635c1a96e8f1bdfcf2d2cd6c7"}
Sep 18 17:24:16.125768 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:24:16.125366    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qwr9x" event={"ID":"ab69c5ba-e722-4484-b8d4-4bd1f26d86f1","Type":"ContainerStarted","Data":"e3a163a6bf4e3d9c85d6adcab2aafd069f9ded6e7517c1cdc4fd72f82ae0135c"}
Sep 18 17:24:16.125768 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:24:16.125479    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qwr9x"
Sep 18 17:24:16.151600 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:24:16.151541    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qwr9x" podStartSLOduration=2.410177688 podStartE2EDuration="2m8.151522719s" totalImagesPullingTime="2m1.923183256s" totalInitContainerRuntime="3.818161775s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:22:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:24:16.148667754 +0000 UTC m=+1846.348523039" watchObservedRunningTime="2026-09-18 17:24:16.151522719 +0000 UTC m=+1846.351378090"
Sep 18 17:24:17.130522 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:24:17.130488    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qwr9x"
Sep 18 17:24:23.141657 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:24:23.141625    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qwr9x"
Sep 18 17:24:53.233285 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:24:53.233212    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qwr9x" podUID="ab69c5ba-e722-4484-b8d4-4bd1f26d86f1" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 18 17:25:03.144944 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:03.144905    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qwr9x"
Sep 18 17:25:08.676199 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:08.676156    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qwr9x"]
Sep 18 17:25:08.676645 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:08.676532    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qwr9x" podUID="ab69c5ba-e722-4484-b8d4-4bd1f26d86f1" containerName="kserve-container" containerID="cri-o://e3a163a6bf4e3d9c85d6adcab2aafd069f9ded6e7517c1cdc4fd72f82ae0135c" gracePeriod=30
Sep 18 17:25:08.676645 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:08.676578    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qwr9x" podUID="ab69c5ba-e722-4484-b8d4-4bd1f26d86f1" containerName="kube-rbac-proxy" containerID="cri-o://66892f53955ad5cd8ccee6d76b15f6e90ab3c68635c1a96e8f1bdfcf2d2cd6c7" gracePeriod=30
Sep 18 17:25:09.365593 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:09.365556    2881 generic.go:356] "Generic (PLEG): container finished" podID="ab69c5ba-e722-4484-b8d4-4bd1f26d86f1" containerID="66892f53955ad5cd8ccee6d76b15f6e90ab3c68635c1a96e8f1bdfcf2d2cd6c7" exitCode=2
Sep 18 17:25:09.365781 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:09.365631    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qwr9x" event={"ID":"ab69c5ba-e722-4484-b8d4-4bd1f26d86f1","Type":"ContainerDied","Data":"66892f53955ad5cd8ccee6d76b15f6e90ab3c68635c1a96e8f1bdfcf2d2cd6c7"}
Sep 18 17:25:13.136474 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:13.136422    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qwr9x" podUID="ab69c5ba-e722-4484-b8d4-4bd1f26d86f1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Sep 18 17:25:13.142707 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:13.142668    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qwr9x" podUID="ab69c5ba-e722-4484-b8d4-4bd1f26d86f1" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.37:8080/v2/models/sklearn-v2-mlserver/ready\": dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 18 17:25:13.919423 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:13.919387    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6dfdd9f4c7-m76sf"]
Sep 18 17:25:13.919823 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:13.919807    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4375b2a5-4110-4096-ac03-759325e95a54" containerName="kube-rbac-proxy"
Sep 18 17:25:13.919874 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:13.919825    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="4375b2a5-4110-4096-ac03-759325e95a54" containerName="kube-rbac-proxy"
Sep 18 17:25:13.919874 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:13.919834    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4375b2a5-4110-4096-ac03-759325e95a54" containerName="kserve-container"
Sep 18 17:25:13.919874 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:13.919840    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="4375b2a5-4110-4096-ac03-759325e95a54" containerName="kserve-container"
Sep 18 17:25:13.919874 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:13.919851    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4375b2a5-4110-4096-ac03-759325e95a54" containerName="storage-initializer"
Sep 18 17:25:13.919874 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:13.919858    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="4375b2a5-4110-4096-ac03-759325e95a54" containerName="storage-initializer"
Sep 18 17:25:13.920038 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:13.919919    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="4375b2a5-4110-4096-ac03-759325e95a54" containerName="kube-rbac-proxy"
Sep 18 17:25:13.920038 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:13.919928    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="4375b2a5-4110-4096-ac03-759325e95a54" containerName="kserve-container"
Sep 18 17:25:13.922411 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:13.922393    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6dfdd9f4c7-m76sf"
Sep 18 17:25:13.926268 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:13.926242    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-predictor-serving-cert\""
Sep 18 17:25:13.926268 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:13.926240    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\""
Sep 18 17:25:13.939822 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:13.939786    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6dfdd9f4c7-m76sf"]
Sep 18 17:25:14.002503 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:14.002454    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92t29\" (UniqueName: \"kubernetes.io/projected/994db0da-2014-4de8-88fb-3de7b6bac887-kube-api-access-92t29\") pod \"isvc-sklearn-runtime-predictor-6dfdd9f4c7-m76sf\" (UID: \"994db0da-2014-4de8-88fb-3de7b6bac887\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6dfdd9f4c7-m76sf"
Sep 18 17:25:14.002726 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:14.002540    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/994db0da-2014-4de8-88fb-3de7b6bac887-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-6dfdd9f4c7-m76sf\" (UID: \"994db0da-2014-4de8-88fb-3de7b6bac887\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6dfdd9f4c7-m76sf"
Sep 18 17:25:14.002726 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:14.002653    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/994db0da-2014-4de8-88fb-3de7b6bac887-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-6dfdd9f4c7-m76sf\" (UID: \"994db0da-2014-4de8-88fb-3de7b6bac887\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6dfdd9f4c7-m76sf"
Sep 18 17:25:14.003150 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:14.003124    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/994db0da-2014-4de8-88fb-3de7b6bac887-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-6dfdd9f4c7-m76sf\" (UID: \"994db0da-2014-4de8-88fb-3de7b6bac887\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6dfdd9f4c7-m76sf"
Sep 18 17:25:14.003239 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:14.003190    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/994db0da-2014-4de8-88fb-3de7b6bac887-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-6dfdd9f4c7-m76sf\" (UID: \"994db0da-2014-4de8-88fb-3de7b6bac887\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6dfdd9f4c7-m76sf"
Sep 18 17:25:14.104651 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:14.104610    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/994db0da-2014-4de8-88fb-3de7b6bac887-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-6dfdd9f4c7-m76sf\" (UID: \"994db0da-2014-4de8-88fb-3de7b6bac887\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6dfdd9f4c7-m76sf"
Sep 18 17:25:14.104651 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:14.104650    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/994db0da-2014-4de8-88fb-3de7b6bac887-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-6dfdd9f4c7-m76sf\" (UID: \"994db0da-2014-4de8-88fb-3de7b6bac887\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6dfdd9f4c7-m76sf"
Sep 18 17:25:14.104937 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:14.104686    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92t29\" (UniqueName: \"kubernetes.io/projected/994db0da-2014-4de8-88fb-3de7b6bac887-kube-api-access-92t29\") pod \"isvc-sklearn-runtime-predictor-6dfdd9f4c7-m76sf\" (UID: \"994db0da-2014-4de8-88fb-3de7b6bac887\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6dfdd9f4c7-m76sf"
Sep 18 17:25:14.104937 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:14.104709    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/994db0da-2014-4de8-88fb-3de7b6bac887-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-6dfdd9f4c7-m76sf\" (UID: \"994db0da-2014-4de8-88fb-3de7b6bac887\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6dfdd9f4c7-m76sf"
Sep 18 17:25:14.104937 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:14.104727    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/994db0da-2014-4de8-88fb-3de7b6bac887-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-6dfdd9f4c7-m76sf\" (UID: \"994db0da-2014-4de8-88fb-3de7b6bac887\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6dfdd9f4c7-m76sf"
Sep 18 17:25:14.105242 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:14.105195    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/994db0da-2014-4de8-88fb-3de7b6bac887-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-6dfdd9f4c7-m76sf\" (UID: \"994db0da-2014-4de8-88fb-3de7b6bac887\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6dfdd9f4c7-m76sf"
Sep 18 17:25:14.105536 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:14.105519    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/994db0da-2014-4de8-88fb-3de7b6bac887-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-6dfdd9f4c7-m76sf\" (UID: \"994db0da-2014-4de8-88fb-3de7b6bac887\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6dfdd9f4c7-m76sf"
Sep 18 17:25:14.105684 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:14.105661    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/994db0da-2014-4de8-88fb-3de7b6bac887-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-6dfdd9f4c7-m76sf\" (UID: \"994db0da-2014-4de8-88fb-3de7b6bac887\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6dfdd9f4c7-m76sf"
Sep 18 17:25:14.107321 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:14.107299    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/994db0da-2014-4de8-88fb-3de7b6bac887-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-6dfdd9f4c7-m76sf\" (UID: \"994db0da-2014-4de8-88fb-3de7b6bac887\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6dfdd9f4c7-m76sf"
Sep 18 17:25:14.115138 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:14.115101    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92t29\" (UniqueName: \"kubernetes.io/projected/994db0da-2014-4de8-88fb-3de7b6bac887-kube-api-access-92t29\") pod \"isvc-sklearn-runtime-predictor-6dfdd9f4c7-m76sf\" (UID: \"994db0da-2014-4de8-88fb-3de7b6bac887\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6dfdd9f4c7-m76sf"
Sep 18 17:25:14.234629 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:14.234526    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6dfdd9f4c7-m76sf"
Sep 18 17:25:14.517009 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:14.516971    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6dfdd9f4c7-m76sf"]
Sep 18 17:25:14.519680 ip-10-0-143-5 kubenswrapper[2881]: W0918 17:25:14.519650    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod994db0da_2014_4de8_88fb_3de7b6bac887.slice/crio-e6c8294c4d9c97e1575f02aaaf25488b754b7d78ff2cefa71af4c762acccdac7 WatchSource:0}: Error finding container e6c8294c4d9c97e1575f02aaaf25488b754b7d78ff2cefa71af4c762acccdac7: Status 404 returned error can't find the container with id e6c8294c4d9c97e1575f02aaaf25488b754b7d78ff2cefa71af4c762acccdac7
Sep 18 17:25:15.394215 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:15.394163    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6dfdd9f4c7-m76sf" event={"ID":"994db0da-2014-4de8-88fb-3de7b6bac887","Type":"ContainerStarted","Data":"6bb423ffee087b1bc42b94687a84e816297a4bcd0358a64b395712baa603a154"}
Sep 18 17:25:15.394643 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:15.394279    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6dfdd9f4c7-m76sf" event={"ID":"994db0da-2014-4de8-88fb-3de7b6bac887","Type":"ContainerStarted","Data":"e6c8294c4d9c97e1575f02aaaf25488b754b7d78ff2cefa71af4c762acccdac7"}
Sep 18 17:25:17.226661 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:17.226633    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qwr9x"
Sep 18 17:25:17.339460 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:17.339366    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab69c5ba-e722-4484-b8d4-4bd1f26d86f1-kserve-provision-location\") pod \"ab69c5ba-e722-4484-b8d4-4bd1f26d86f1\" (UID: \"ab69c5ba-e722-4484-b8d4-4bd1f26d86f1\") "
Sep 18 17:25:17.339460 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:17.339456    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l9mk8\" (UniqueName: \"kubernetes.io/projected/ab69c5ba-e722-4484-b8d4-4bd1f26d86f1-kube-api-access-l9mk8\") pod \"ab69c5ba-e722-4484-b8d4-4bd1f26d86f1\" (UID: \"ab69c5ba-e722-4484-b8d4-4bd1f26d86f1\") "
Sep 18 17:25:17.339668 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:17.339505    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab69c5ba-e722-4484-b8d4-4bd1f26d86f1-kube-rbac-proxy-sar-config\") pod \"ab69c5ba-e722-4484-b8d4-4bd1f26d86f1\" (UID: \"ab69c5ba-e722-4484-b8d4-4bd1f26d86f1\") "
Sep 18 17:25:17.339668 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:17.339539    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab69c5ba-e722-4484-b8d4-4bd1f26d86f1-cabundle-cert\") pod \"ab69c5ba-e722-4484-b8d4-4bd1f26d86f1\" (UID: \"ab69c5ba-e722-4484-b8d4-4bd1f26d86f1\") "
Sep 18 17:25:17.339746 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:17.339728    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab69c5ba-e722-4484-b8d4-4bd1f26d86f1-proxy-tls\") pod \"ab69c5ba-e722-4484-b8d4-4bd1f26d86f1\" (UID: \"ab69c5ba-e722-4484-b8d4-4bd1f26d86f1\") "
Sep 18 17:25:17.339888 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:17.339853    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab69c5ba-e722-4484-b8d4-4bd1f26d86f1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ab69c5ba-e722-4484-b8d4-4bd1f26d86f1" (UID: "ab69c5ba-e722-4484-b8d4-4bd1f26d86f1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:25:17.339948 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:17.339869    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab69c5ba-e722-4484-b8d4-4bd1f26d86f1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ab69c5ba-e722-4484-b8d4-4bd1f26d86f1" (UID: "ab69c5ba-e722-4484-b8d4-4bd1f26d86f1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:25:17.339991 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:17.339945    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab69c5ba-e722-4484-b8d4-4bd1f26d86f1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ab69c5ba-e722-4484-b8d4-4bd1f26d86f1" (UID: "ab69c5ba-e722-4484-b8d4-4bd1f26d86f1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:25:17.340136 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:17.340119    2881 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab69c5ba-e722-4484-b8d4-4bd1f26d86f1-kserve-provision-location\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:25:17.340190 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:17.340137    2881 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab69c5ba-e722-4484-b8d4-4bd1f26d86f1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:25:17.340190 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:17.340147    2881 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab69c5ba-e722-4484-b8d4-4bd1f26d86f1-cabundle-cert\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:25:17.342196 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:17.342175    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab69c5ba-e722-4484-b8d4-4bd1f26d86f1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ab69c5ba-e722-4484-b8d4-4bd1f26d86f1" (UID: "ab69c5ba-e722-4484-b8d4-4bd1f26d86f1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:25:17.342392 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:17.342374    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab69c5ba-e722-4484-b8d4-4bd1f26d86f1-kube-api-access-l9mk8" (OuterVolumeSpecName: "kube-api-access-l9mk8") pod "ab69c5ba-e722-4484-b8d4-4bd1f26d86f1" (UID: "ab69c5ba-e722-4484-b8d4-4bd1f26d86f1"). InnerVolumeSpecName "kube-api-access-l9mk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:25:17.407259 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:17.407210    2881 generic.go:356] "Generic (PLEG): container finished" podID="ab69c5ba-e722-4484-b8d4-4bd1f26d86f1" containerID="e3a163a6bf4e3d9c85d6adcab2aafd069f9ded6e7517c1cdc4fd72f82ae0135c" exitCode=0
Sep 18 17:25:17.407466 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:17.407281    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qwr9x" event={"ID":"ab69c5ba-e722-4484-b8d4-4bd1f26d86f1","Type":"ContainerDied","Data":"e3a163a6bf4e3d9c85d6adcab2aafd069f9ded6e7517c1cdc4fd72f82ae0135c"}
Sep 18 17:25:17.407466 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:17.407306    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qwr9x"
Sep 18 17:25:17.407466 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:17.407336    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qwr9x" event={"ID":"ab69c5ba-e722-4484-b8d4-4bd1f26d86f1","Type":"ContainerDied","Data":"54fff305cf373b94dc33b2db2ee541832429879278508dbdd8426252af7d333d"}
Sep 18 17:25:17.407466 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:17.407364    2881 scope.go:117] "RemoveContainer" containerID="66892f53955ad5cd8ccee6d76b15f6e90ab3c68635c1a96e8f1bdfcf2d2cd6c7"
Sep 18 17:25:17.416388 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:17.416372    2881 scope.go:117] "RemoveContainer" containerID="e3a163a6bf4e3d9c85d6adcab2aafd069f9ded6e7517c1cdc4fd72f82ae0135c"
Sep 18 17:25:17.426093 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:17.426070    2881 scope.go:117] "RemoveContainer" containerID="1e300c3cb0318f61668d46777a33a5655c449e6eeb2d9afe2815bf65ca7df001"
Sep 18 17:25:17.435981 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:17.435956    2881 scope.go:117] "RemoveContainer" containerID="66892f53955ad5cd8ccee6d76b15f6e90ab3c68635c1a96e8f1bdfcf2d2cd6c7"
Sep 18 17:25:17.436356 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:25:17.436331    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66892f53955ad5cd8ccee6d76b15f6e90ab3c68635c1a96e8f1bdfcf2d2cd6c7\": container with ID starting with 66892f53955ad5cd8ccee6d76b15f6e90ab3c68635c1a96e8f1bdfcf2d2cd6c7 not found: ID does not exist" containerID="66892f53955ad5cd8ccee6d76b15f6e90ab3c68635c1a96e8f1bdfcf2d2cd6c7"
Sep 18 17:25:17.436465 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:17.436372    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66892f53955ad5cd8ccee6d76b15f6e90ab3c68635c1a96e8f1bdfcf2d2cd6c7"} err="failed to get container status \"66892f53955ad5cd8ccee6d76b15f6e90ab3c68635c1a96e8f1bdfcf2d2cd6c7\": rpc error: code = NotFound desc = could not find container \"66892f53955ad5cd8ccee6d76b15f6e90ab3c68635c1a96e8f1bdfcf2d2cd6c7\": container with ID starting with 66892f53955ad5cd8ccee6d76b15f6e90ab3c68635c1a96e8f1bdfcf2d2cd6c7 not found: ID does not exist"
Sep 18 17:25:17.436465 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:17.436399    2881 scope.go:117] "RemoveContainer" containerID="e3a163a6bf4e3d9c85d6adcab2aafd069f9ded6e7517c1cdc4fd72f82ae0135c"
Sep 18 17:25:17.436722 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:25:17.436704    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3a163a6bf4e3d9c85d6adcab2aafd069f9ded6e7517c1cdc4fd72f82ae0135c\": container with ID starting with e3a163a6bf4e3d9c85d6adcab2aafd069f9ded6e7517c1cdc4fd72f82ae0135c not found: ID does not exist" containerID="e3a163a6bf4e3d9c85d6adcab2aafd069f9ded6e7517c1cdc4fd72f82ae0135c"
Sep 18 17:25:17.436803 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:17.436732    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3a163a6bf4e3d9c85d6adcab2aafd069f9ded6e7517c1cdc4fd72f82ae0135c"} err="failed to get container status \"e3a163a6bf4e3d9c85d6adcab2aafd069f9ded6e7517c1cdc4fd72f82ae0135c\": rpc error: code = NotFound desc = could not find container \"e3a163a6bf4e3d9c85d6adcab2aafd069f9ded6e7517c1cdc4fd72f82ae0135c\": container with ID starting with e3a163a6bf4e3d9c85d6adcab2aafd069f9ded6e7517c1cdc4fd72f82ae0135c not found: ID does not exist"
Sep 18 17:25:17.436803 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:17.436755    2881 scope.go:117] "RemoveContainer" containerID="1e300c3cb0318f61668d46777a33a5655c449e6eeb2d9afe2815bf65ca7df001"
Sep 18 17:25:17.437014 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:25:17.436993    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e300c3cb0318f61668d46777a33a5655c449e6eeb2d9afe2815bf65ca7df001\": container with ID starting with 1e300c3cb0318f61668d46777a33a5655c449e6eeb2d9afe2815bf65ca7df001 not found: ID does not exist" containerID="1e300c3cb0318f61668d46777a33a5655c449e6eeb2d9afe2815bf65ca7df001"
Sep 18 17:25:17.437061 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:17.437024    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e300c3cb0318f61668d46777a33a5655c449e6eeb2d9afe2815bf65ca7df001"} err="failed to get container status \"1e300c3cb0318f61668d46777a33a5655c449e6eeb2d9afe2815bf65ca7df001\": rpc error: code = NotFound desc = could not find container \"1e300c3cb0318f61668d46777a33a5655c449e6eeb2d9afe2815bf65ca7df001\": container with ID starting with 1e300c3cb0318f61668d46777a33a5655c449e6eeb2d9afe2815bf65ca7df001 not found: ID does not exist"
Sep 18 17:25:17.441717 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:17.441687    2881 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9mk8\" (UniqueName: \"kubernetes.io/projected/ab69c5ba-e722-4484-b8d4-4bd1f26d86f1-kube-api-access-l9mk8\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:25:17.441717 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:17.441718    2881 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab69c5ba-e722-4484-b8d4-4bd1f26d86f1-proxy-tls\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:25:17.444548 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:17.444520    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qwr9x"]
Sep 18 17:25:17.456956 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:17.456922    2881 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qwr9x"]
Sep 18 17:25:17.894447 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:17.894411    2881 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab69c5ba-e722-4484-b8d4-4bd1f26d86f1" path="/var/lib/kubelet/pods/ab69c5ba-e722-4484-b8d4-4bd1f26d86f1/volumes"
Sep 18 17:25:20.422843 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:20.422807    2881 generic.go:356] "Generic (PLEG): container finished" podID="994db0da-2014-4de8-88fb-3de7b6bac887" containerID="6bb423ffee087b1bc42b94687a84e816297a4bcd0358a64b395712baa603a154" exitCode=0
Sep 18 17:25:20.423310 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:20.422861    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6dfdd9f4c7-m76sf" event={"ID":"994db0da-2014-4de8-88fb-3de7b6bac887","Type":"ContainerDied","Data":"6bb423ffee087b1bc42b94687a84e816297a4bcd0358a64b395712baa603a154"}
Sep 18 17:25:21.430041 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:21.429993    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6dfdd9f4c7-m76sf" event={"ID":"994db0da-2014-4de8-88fb-3de7b6bac887","Type":"ContainerStarted","Data":"4f457c7858282a855a8dbc3e8ebf203de7ec02f266b9b064e8eefb5e71915e36"}
Sep 18 17:25:21.430494 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:21.430088    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6dfdd9f4c7-m76sf" event={"ID":"994db0da-2014-4de8-88fb-3de7b6bac887","Type":"ContainerStarted","Data":"b81920b6e20ec0200be592b11684ead17273ee881e762eac818f9ee1c8b00c2e"}
Sep 18 17:25:21.430494 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:21.430355    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6dfdd9f4c7-m76sf"
Sep 18 17:25:21.453619 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:21.453551    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6dfdd9f4c7-m76sf" podStartSLOduration=2.736358753 podStartE2EDuration="8.453529263s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.71717051s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:25:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:25:21.450767824 +0000 UTC m=+1911.650623112" watchObservedRunningTime="2026-09-18 17:25:21.453529263 +0000 UTC m=+1911.653384549"
Sep 18 17:25:22.435679 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:22.435634    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6dfdd9f4c7-m76sf"
Sep 18 17:25:22.437018 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:22.436988    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6dfdd9f4c7-m76sf" podUID="994db0da-2014-4de8-88fb-3de7b6bac887" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 18 17:25:23.441460 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:23.441415    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6dfdd9f4c7-m76sf" podUID="994db0da-2014-4de8-88fb-3de7b6bac887" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 18 17:25:28.448069 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:28.448032    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6dfdd9f4c7-m76sf"
Sep 18 17:25:28.448555 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:28.448521    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6dfdd9f4c7-m76sf" podUID="994db0da-2014-4de8-88fb-3de7b6bac887" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 18 17:25:38.449817 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:38.449774    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6dfdd9f4c7-m76sf"
Sep 18 17:25:46.932557 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:46.932520    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6dfdd9f4c7-m76sf"]
Sep 18 17:25:46.932979 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:46.932769    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6dfdd9f4c7-m76sf" podUID="994db0da-2014-4de8-88fb-3de7b6bac887" containerName="kserve-container" containerID="cri-o://b81920b6e20ec0200be592b11684ead17273ee881e762eac818f9ee1c8b00c2e" gracePeriod=30
Sep 18 17:25:46.932979 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:46.932828    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6dfdd9f4c7-m76sf" podUID="994db0da-2014-4de8-88fb-3de7b6bac887" containerName="kube-rbac-proxy" containerID="cri-o://4f457c7858282a855a8dbc3e8ebf203de7ec02f266b9b064e8eefb5e71915e36" gracePeriod=30
Sep 18 17:25:47.559758 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:47.559716    2881 generic.go:356] "Generic (PLEG): container finished" podID="994db0da-2014-4de8-88fb-3de7b6bac887" containerID="4f457c7858282a855a8dbc3e8ebf203de7ec02f266b9b064e8eefb5e71915e36" exitCode=2
Sep 18 17:25:47.559959 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:47.559786    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6dfdd9f4c7-m76sf" event={"ID":"994db0da-2014-4de8-88fb-3de7b6bac887","Type":"ContainerDied","Data":"4f457c7858282a855a8dbc3e8ebf203de7ec02f266b9b064e8eefb5e71915e36"}
Sep 18 17:25:48.176392 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:48.176356    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6dfdd9f4c7-m76sf"
Sep 18 17:25:48.220610 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:48.220520    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/994db0da-2014-4de8-88fb-3de7b6bac887-kserve-provision-location\") pod \"994db0da-2014-4de8-88fb-3de7b6bac887\" (UID: \"994db0da-2014-4de8-88fb-3de7b6bac887\") "
Sep 18 17:25:48.220768 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:48.220614    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/994db0da-2014-4de8-88fb-3de7b6bac887-kube-rbac-proxy-sar-config\") pod \"994db0da-2014-4de8-88fb-3de7b6bac887\" (UID: \"994db0da-2014-4de8-88fb-3de7b6bac887\") "
Sep 18 17:25:48.220768 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:48.220659    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/994db0da-2014-4de8-88fb-3de7b6bac887-proxy-tls\") pod \"994db0da-2014-4de8-88fb-3de7b6bac887\" (UID: \"994db0da-2014-4de8-88fb-3de7b6bac887\") "
Sep 18 17:25:48.220855 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:48.220774    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/994db0da-2014-4de8-88fb-3de7b6bac887-cabundle-cert\") pod \"994db0da-2014-4de8-88fb-3de7b6bac887\" (UID: \"994db0da-2014-4de8-88fb-3de7b6bac887\") "
Sep 18 17:25:48.220855 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:48.220843    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-92t29\" (UniqueName: \"kubernetes.io/projected/994db0da-2014-4de8-88fb-3de7b6bac887-kube-api-access-92t29\") pod \"994db0da-2014-4de8-88fb-3de7b6bac887\" (UID: \"994db0da-2014-4de8-88fb-3de7b6bac887\") "
Sep 18 17:25:48.221110 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:48.221084    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/994db0da-2014-4de8-88fb-3de7b6bac887-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "994db0da-2014-4de8-88fb-3de7b6bac887" (UID: "994db0da-2014-4de8-88fb-3de7b6bac887"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:25:48.221193 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:48.221117    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/994db0da-2014-4de8-88fb-3de7b6bac887-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "994db0da-2014-4de8-88fb-3de7b6bac887" (UID: "994db0da-2014-4de8-88fb-3de7b6bac887"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:25:48.221193 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:48.221170    2881 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/994db0da-2014-4de8-88fb-3de7b6bac887-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:25:48.222823 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:48.222796    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/994db0da-2014-4de8-88fb-3de7b6bac887-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "994db0da-2014-4de8-88fb-3de7b6bac887" (UID: "994db0da-2014-4de8-88fb-3de7b6bac887"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:25:48.222982 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:48.222963    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/994db0da-2014-4de8-88fb-3de7b6bac887-kube-api-access-92t29" (OuterVolumeSpecName: "kube-api-access-92t29") pod "994db0da-2014-4de8-88fb-3de7b6bac887" (UID: "994db0da-2014-4de8-88fb-3de7b6bac887"). InnerVolumeSpecName "kube-api-access-92t29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:25:48.250527 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:48.250487    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/994db0da-2014-4de8-88fb-3de7b6bac887-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "994db0da-2014-4de8-88fb-3de7b6bac887" (UID: "994db0da-2014-4de8-88fb-3de7b6bac887"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:25:48.322535 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:48.322495    2881 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/994db0da-2014-4de8-88fb-3de7b6bac887-kserve-provision-location\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:25:48.322535 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:48.322530    2881 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/994db0da-2014-4de8-88fb-3de7b6bac887-proxy-tls\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:25:48.322535 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:48.322540    2881 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/994db0da-2014-4de8-88fb-3de7b6bac887-cabundle-cert\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:25:48.322774 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:48.322551    2881 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92t29\" (UniqueName: \"kubernetes.io/projected/994db0da-2014-4de8-88fb-3de7b6bac887-kube-api-access-92t29\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:25:48.566362 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:48.566327    2881 generic.go:356] "Generic (PLEG): container finished" podID="994db0da-2014-4de8-88fb-3de7b6bac887" containerID="b81920b6e20ec0200be592b11684ead17273ee881e762eac818f9ee1c8b00c2e" exitCode=0
Sep 18 17:25:48.566570 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:48.566391    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6dfdd9f4c7-m76sf"
Sep 18 17:25:48.566570 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:48.566411    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6dfdd9f4c7-m76sf" event={"ID":"994db0da-2014-4de8-88fb-3de7b6bac887","Type":"ContainerDied","Data":"b81920b6e20ec0200be592b11684ead17273ee881e762eac818f9ee1c8b00c2e"}
Sep 18 17:25:48.566570 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:48.566497    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6dfdd9f4c7-m76sf" event={"ID":"994db0da-2014-4de8-88fb-3de7b6bac887","Type":"ContainerDied","Data":"e6c8294c4d9c97e1575f02aaaf25488b754b7d78ff2cefa71af4c762acccdac7"}
Sep 18 17:25:48.566570 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:48.566535    2881 scope.go:117] "RemoveContainer" containerID="4f457c7858282a855a8dbc3e8ebf203de7ec02f266b9b064e8eefb5e71915e36"
Sep 18 17:25:48.575853 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:48.575831    2881 scope.go:117] "RemoveContainer" containerID="b81920b6e20ec0200be592b11684ead17273ee881e762eac818f9ee1c8b00c2e"
Sep 18 17:25:48.586841 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:48.586388    2881 scope.go:117] "RemoveContainer" containerID="6bb423ffee087b1bc42b94687a84e816297a4bcd0358a64b395712baa603a154"
Sep 18 17:25:48.590125 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:48.590104    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6dfdd9f4c7-m76sf"]
Sep 18 17:25:48.595490 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:48.595459    2881 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6dfdd9f4c7-m76sf"]
Sep 18 17:25:48.597579 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:48.597557    2881 scope.go:117] "RemoveContainer" containerID="4f457c7858282a855a8dbc3e8ebf203de7ec02f266b9b064e8eefb5e71915e36"
Sep 18 17:25:48.597919 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:25:48.597897    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f457c7858282a855a8dbc3e8ebf203de7ec02f266b9b064e8eefb5e71915e36\": container with ID starting with 4f457c7858282a855a8dbc3e8ebf203de7ec02f266b9b064e8eefb5e71915e36 not found: ID does not exist" containerID="4f457c7858282a855a8dbc3e8ebf203de7ec02f266b9b064e8eefb5e71915e36"
Sep 18 17:25:48.597990 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:48.597929    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f457c7858282a855a8dbc3e8ebf203de7ec02f266b9b064e8eefb5e71915e36"} err="failed to get container status \"4f457c7858282a855a8dbc3e8ebf203de7ec02f266b9b064e8eefb5e71915e36\": rpc error: code = NotFound desc = could not find container \"4f457c7858282a855a8dbc3e8ebf203de7ec02f266b9b064e8eefb5e71915e36\": container with ID starting with 4f457c7858282a855a8dbc3e8ebf203de7ec02f266b9b064e8eefb5e71915e36 not found: ID does not exist"
Sep 18 17:25:48.597990 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:48.597950    2881 scope.go:117] "RemoveContainer" containerID="b81920b6e20ec0200be592b11684ead17273ee881e762eac818f9ee1c8b00c2e"
Sep 18 17:25:48.598261 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:25:48.598242    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b81920b6e20ec0200be592b11684ead17273ee881e762eac818f9ee1c8b00c2e\": container with ID starting with b81920b6e20ec0200be592b11684ead17273ee881e762eac818f9ee1c8b00c2e not found: ID does not exist" containerID="b81920b6e20ec0200be592b11684ead17273ee881e762eac818f9ee1c8b00c2e"
Sep 18 17:25:48.598336 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:48.598266    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b81920b6e20ec0200be592b11684ead17273ee881e762eac818f9ee1c8b00c2e"} err="failed to get container status \"b81920b6e20ec0200be592b11684ead17273ee881e762eac818f9ee1c8b00c2e\": rpc error: code = NotFound desc = could not find container \"b81920b6e20ec0200be592b11684ead17273ee881e762eac818f9ee1c8b00c2e\": container with ID starting with b81920b6e20ec0200be592b11684ead17273ee881e762eac818f9ee1c8b00c2e not found: ID does not exist"
Sep 18 17:25:48.598336 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:48.598283    2881 scope.go:117] "RemoveContainer" containerID="6bb423ffee087b1bc42b94687a84e816297a4bcd0358a64b395712baa603a154"
Sep 18 17:25:48.598552 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:25:48.598535    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bb423ffee087b1bc42b94687a84e816297a4bcd0358a64b395712baa603a154\": container with ID starting with 6bb423ffee087b1bc42b94687a84e816297a4bcd0358a64b395712baa603a154 not found: ID does not exist" containerID="6bb423ffee087b1bc42b94687a84e816297a4bcd0358a64b395712baa603a154"
Sep 18 17:25:48.598590 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:48.598555    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bb423ffee087b1bc42b94687a84e816297a4bcd0358a64b395712baa603a154"} err="failed to get container status \"6bb423ffee087b1bc42b94687a84e816297a4bcd0358a64b395712baa603a154\": rpc error: code = NotFound desc = could not find container \"6bb423ffee087b1bc42b94687a84e816297a4bcd0358a64b395712baa603a154\": container with ID starting with 6bb423ffee087b1bc42b94687a84e816297a4bcd0358a64b395712baa603a154 not found: ID does not exist"
Sep 18 17:25:49.899517 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:49.899478    2881 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="994db0da-2014-4de8-88fb-3de7b6bac887" path="/var/lib/kubelet/pods/994db0da-2014-4de8-88fb-3de7b6bac887/volumes"
Sep 18 17:25:52.191066 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:52.191027    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hd26h"]
Sep 18 17:25:52.191497 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:52.191388    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="994db0da-2014-4de8-88fb-3de7b6bac887" containerName="storage-initializer"
Sep 18 17:25:52.191497 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:52.191402    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="994db0da-2014-4de8-88fb-3de7b6bac887" containerName="storage-initializer"
Sep 18 17:25:52.191497 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:52.191420    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab69c5ba-e722-4484-b8d4-4bd1f26d86f1" containerName="storage-initializer"
Sep 18 17:25:52.191497 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:52.191425    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab69c5ba-e722-4484-b8d4-4bd1f26d86f1" containerName="storage-initializer"
Sep 18 17:25:52.191497 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:52.191432    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="994db0da-2014-4de8-88fb-3de7b6bac887" containerName="kserve-container"
Sep 18 17:25:52.191497 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:52.191438    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="994db0da-2014-4de8-88fb-3de7b6bac887" containerName="kserve-container"
Sep 18 17:25:52.191497 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:52.191444    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab69c5ba-e722-4484-b8d4-4bd1f26d86f1" containerName="kserve-container"
Sep 18 17:25:52.191497 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:52.191450    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab69c5ba-e722-4484-b8d4-4bd1f26d86f1" containerName="kserve-container"
Sep 18 17:25:52.191497 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:52.191455    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab69c5ba-e722-4484-b8d4-4bd1f26d86f1" containerName="kube-rbac-proxy"
Sep 18 17:25:52.191497 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:52.191464    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab69c5ba-e722-4484-b8d4-4bd1f26d86f1" containerName="kube-rbac-proxy"
Sep 18 17:25:52.191497 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:52.191480    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="994db0da-2014-4de8-88fb-3de7b6bac887" containerName="kube-rbac-proxy"
Sep 18 17:25:52.191497 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:52.191485    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="994db0da-2014-4de8-88fb-3de7b6bac887" containerName="kube-rbac-proxy"
Sep 18 17:25:52.191881 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:52.191539    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="994db0da-2014-4de8-88fb-3de7b6bac887" containerName="kserve-container"
Sep 18 17:25:52.191881 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:52.191548    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="994db0da-2014-4de8-88fb-3de7b6bac887" containerName="kube-rbac-proxy"
Sep 18 17:25:52.191881 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:52.191557    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab69c5ba-e722-4484-b8d4-4bd1f26d86f1" containerName="kserve-container"
Sep 18 17:25:52.191881 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:52.191564    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab69c5ba-e722-4484-b8d4-4bd1f26d86f1" containerName="kube-rbac-proxy"
Sep 18 17:25:52.194773 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:52.194755    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hd26h"
Sep 18 17:25:52.198167 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:52.198138    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-predictor-serving-cert\""
Sep 18 17:25:52.198330 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:52.198264    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5hv9v\""
Sep 18 17:25:52.199458 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:52.199439    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 17:25:52.199564 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:52.199513    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 17:25:52.199564 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:52.199532    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 18 17:25:52.199677 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:52.199636    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 17:25:52.200300 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:52.200285    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 17:25:52.207490 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:52.207463    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hd26h"]
Sep 18 17:25:52.257553 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:52.257514    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-hd26h\" (UID: \"f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hd26h"
Sep 18 17:25:52.257748 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:52.257562    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-hd26h\" (UID: \"f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hd26h"
Sep 18 17:25:52.257748 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:52.257593    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-hd26h\" (UID: \"f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hd26h"
Sep 18 17:25:52.257748 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:52.257611    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxzqj\" (UniqueName: \"kubernetes.io/projected/f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc-kube-api-access-gxzqj\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-hd26h\" (UID: \"f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hd26h"
Sep 18 17:25:52.257748 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:52.257658    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-hd26h\" (UID: \"f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hd26h"
Sep 18 17:25:52.358436 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:52.358398    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-hd26h\" (UID: \"f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hd26h"
Sep 18 17:25:52.358436 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:52.358442    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-hd26h\" (UID: \"f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hd26h"
Sep 18 17:25:52.358690 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:52.358565    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-hd26h\" (UID: \"f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hd26h"
Sep 18 17:25:52.358690 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:52.358601    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxzqj\" (UniqueName: \"kubernetes.io/projected/f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc-kube-api-access-gxzqj\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-hd26h\" (UID: \"f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hd26h"
Sep 18 17:25:52.358690 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:52.358660    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-hd26h\" (UID: \"f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hd26h"
Sep 18 17:25:52.358995 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:52.358969    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-hd26h\" (UID: \"f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hd26h"
Sep 18 17:25:52.359127 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:52.359110    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-hd26h\" (UID: \"f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hd26h"
Sep 18 17:25:52.359285 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:52.359219    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-hd26h\" (UID: \"f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hd26h"
Sep 18 17:25:52.361020 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:52.361003    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-hd26h\" (UID: \"f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hd26h"
Sep 18 17:25:52.373290 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:52.373262    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxzqj\" (UniqueName: \"kubernetes.io/projected/f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc-kube-api-access-gxzqj\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-hd26h\" (UID: \"f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hd26h"
Sep 18 17:25:52.506899 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:52.506805    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hd26h"
Sep 18 17:25:52.793662 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:52.793627    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hd26h"]
Sep 18 17:25:52.794184 ip-10-0-143-5 kubenswrapper[2881]: W0918 17:25:52.794154    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3f739ff_5b1a_4dcd_a6ca_15a66bcc97bc.slice/crio-7cacc73096307ca924ba7763aa43ecc0f2295eeaa73f69b5ef9436d47e7c8a6a WatchSource:0}: Error finding container 7cacc73096307ca924ba7763aa43ecc0f2295eeaa73f69b5ef9436d47e7c8a6a: Status 404 returned error can't find the container with id 7cacc73096307ca924ba7763aa43ecc0f2295eeaa73f69b5ef9436d47e7c8a6a
Sep 18 17:25:53.590784 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:53.590747    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hd26h" event={"ID":"f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc","Type":"ContainerStarted","Data":"cc23ade7d57408b3d3799ba24e37461e6430de708f31e0dd4c7587f2bbdfca20"}
Sep 18 17:25:53.591199 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:53.590810    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hd26h" event={"ID":"f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc","Type":"ContainerStarted","Data":"7cacc73096307ca924ba7763aa43ecc0f2295eeaa73f69b5ef9436d47e7c8a6a"}
Sep 18 17:25:57.612188 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:57.612144    2881 generic.go:356] "Generic (PLEG): container finished" podID="f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc" containerID="cc23ade7d57408b3d3799ba24e37461e6430de708f31e0dd4c7587f2bbdfca20" exitCode=0
Sep 18 17:25:57.612702 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:57.612235    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hd26h" event={"ID":"f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc","Type":"ContainerDied","Data":"cc23ade7d57408b3d3799ba24e37461e6430de708f31e0dd4c7587f2bbdfca20"}
Sep 18 17:25:58.618599 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:58.618562    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hd26h" event={"ID":"f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc","Type":"ContainerStarted","Data":"c60522823ad68d7461daa37f717770ac56242ebb2ec9a306ea7576551220a451"}
Sep 18 17:25:58.619055 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:58.618629    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hd26h" event={"ID":"f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc","Type":"ContainerStarted","Data":"d0c6d802d15a94f02de4939380e7a5654403c4f619a86fc52106dce0f9f2f8d9"}
Sep 18 17:25:58.619055 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:58.618830    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hd26h"
Sep 18 17:25:58.644554 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:58.644492    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hd26h" podStartSLOduration=2.81655876 podStartE2EDuration="6.644471568s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.827912808s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:25:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:25:58.642189044 +0000 UTC m=+1948.842044341" watchObservedRunningTime="2026-09-18 17:25:58.644471568 +0000 UTC m=+1948.844326854"
Sep 18 17:25:59.623163 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:25:59.623131    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hd26h"
Sep 18 17:26:05.633688 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:05.633655    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hd26h"
Sep 18 17:26:30.155646 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:30.155610    2881 scope.go:117] "RemoveContainer" containerID="e29141587efcfec8c1863b7167c5f4b5a6181de71b8c85e59797ed5ee2098476"
Sep 18 17:26:30.164255 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:30.164214    2881 scope.go:117] "RemoveContainer" containerID="151a2c0a0495bd3347e28ec7e86179f5c9b8ab85b22be81aaa94f3daedf3c01e"
Sep 18 17:26:32.534995 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:32.534951    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-vssvp"]
Sep 18 17:26:32.540863 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:32.540836    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-vssvp"
Sep 18 17:26:32.544253 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:32.544201    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Sep 18 17:26:32.544415 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:32.544362    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 18 17:26:32.554156 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:32.554125    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-vssvp"]
Sep 18 17:26:32.564210 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:32.564179    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-dqzdg"]
Sep 18 17:26:32.568138 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:32.568112    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-dqzdg"
Sep 18 17:26:32.570945 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:32.570914    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 18 17:26:32.581407 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:32.581378    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-dqzdg"]
Sep 18 17:26:32.611734 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:32.611689    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/101e16ed-d709-4547-85c6-7daa62eb5a88-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-b75cfc78d-dqzdg\" (UID: \"101e16ed-d709-4547-85c6-7daa62eb5a88\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-dqzdg"
Sep 18 17:26:32.611898 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:32.611765    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/101e16ed-d709-4547-85c6-7daa62eb5a88-proxy-tls\") pod \"isvc-canary-create-predictor-b75cfc78d-dqzdg\" (UID: \"101e16ed-d709-4547-85c6-7daa62eb5a88\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-dqzdg"
Sep 18 17:26:32.611898 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:32.611799    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/101e16ed-d709-4547-85c6-7daa62eb5a88-kserve-provision-location\") pod \"isvc-canary-create-predictor-b75cfc78d-dqzdg\" (UID: \"101e16ed-d709-4547-85c6-7daa62eb5a88\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-dqzdg"
Sep 18 17:26:32.611898 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:32.611823    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc-proxy-tls\") pod \"isvc-canary-create-v2-predictor-7b4d9977c7-vssvp\" (UID: \"bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-vssvp"
Sep 18 17:26:32.611898 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:32.611883    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h65d8\" (UniqueName: \"kubernetes.io/projected/bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc-kube-api-access-h65d8\") pod \"isvc-canary-create-v2-predictor-7b4d9977c7-vssvp\" (UID: \"bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-vssvp"
Sep 18 17:26:32.612057 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:32.611904    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-7b4d9977c7-vssvp\" (UID: \"bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-vssvp"
Sep 18 17:26:32.612057 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:32.611957    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-7b4d9977c7-vssvp\" (UID: \"bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-vssvp"
Sep 18 17:26:32.612057 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:32.611996    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcnwt\" (UniqueName: \"kubernetes.io/projected/101e16ed-d709-4547-85c6-7daa62eb5a88-kube-api-access-xcnwt\") pod \"isvc-canary-create-predictor-b75cfc78d-dqzdg\" (UID: \"101e16ed-d709-4547-85c6-7daa62eb5a88\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-dqzdg"
Sep 18 17:26:32.612057 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:32.612040    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-7b4d9977c7-vssvp\" (UID: \"bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-vssvp"
Sep 18 17:26:32.612183 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:32.612067    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/101e16ed-d709-4547-85c6-7daa62eb5a88-cabundle-cert\") pod \"isvc-canary-create-predictor-b75cfc78d-dqzdg\" (UID: \"101e16ed-d709-4547-85c6-7daa62eb5a88\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-dqzdg"
Sep 18 17:26:32.712529 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:32.712482    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/101e16ed-d709-4547-85c6-7daa62eb5a88-kserve-provision-location\") pod \"isvc-canary-create-predictor-b75cfc78d-dqzdg\" (UID: \"101e16ed-d709-4547-85c6-7daa62eb5a88\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-dqzdg"
Sep 18 17:26:32.712529 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:32.712531    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc-proxy-tls\") pod \"isvc-canary-create-v2-predictor-7b4d9977c7-vssvp\" (UID: \"bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-vssvp"
Sep 18 17:26:32.712792 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:32.712561    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h65d8\" (UniqueName: \"kubernetes.io/projected/bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc-kube-api-access-h65d8\") pod \"isvc-canary-create-v2-predictor-7b4d9977c7-vssvp\" (UID: \"bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-vssvp"
Sep 18 17:26:32.712792 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:32.712584    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-7b4d9977c7-vssvp\" (UID: \"bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-vssvp"
Sep 18 17:26:32.712792 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:32.712608    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-7b4d9977c7-vssvp\" (UID: \"bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-vssvp"
Sep 18 17:26:32.712792 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:32.712638    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcnwt\" (UniqueName: \"kubernetes.io/projected/101e16ed-d709-4547-85c6-7daa62eb5a88-kube-api-access-xcnwt\") pod \"isvc-canary-create-predictor-b75cfc78d-dqzdg\" (UID: \"101e16ed-d709-4547-85c6-7daa62eb5a88\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-dqzdg"
Sep 18 17:26:32.713010 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:32.712808    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-7b4d9977c7-vssvp\" (UID: \"bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-vssvp"
Sep 18 17:26:32.713010 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:32.712856    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/101e16ed-d709-4547-85c6-7daa62eb5a88-cabundle-cert\") pod \"isvc-canary-create-predictor-b75cfc78d-dqzdg\" (UID: \"101e16ed-d709-4547-85c6-7daa62eb5a88\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-dqzdg"
Sep 18 17:26:32.713010 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:32.712897    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/101e16ed-d709-4547-85c6-7daa62eb5a88-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-b75cfc78d-dqzdg\" (UID: \"101e16ed-d709-4547-85c6-7daa62eb5a88\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-dqzdg"
Sep 18 17:26:32.713010 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:32.712949    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/101e16ed-d709-4547-85c6-7daa62eb5a88-kserve-provision-location\") pod \"isvc-canary-create-predictor-b75cfc78d-dqzdg\" (UID: \"101e16ed-d709-4547-85c6-7daa62eb5a88\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-dqzdg"
Sep 18 17:26:32.713010 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:32.712968    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/101e16ed-d709-4547-85c6-7daa62eb5a88-proxy-tls\") pod \"isvc-canary-create-predictor-b75cfc78d-dqzdg\" (UID: \"101e16ed-d709-4547-85c6-7daa62eb5a88\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-dqzdg"
Sep 18 17:26:32.713299 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:32.713192    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-7b4d9977c7-vssvp\" (UID: \"bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-vssvp"
Sep 18 17:26:32.713501 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:32.713460    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-7b4d9977c7-vssvp\" (UID: \"bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-vssvp"
Sep 18 17:26:32.713596 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:32.713521    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-7b4d9977c7-vssvp\" (UID: \"bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-vssvp"
Sep 18 17:26:32.713670 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:32.713648    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/101e16ed-d709-4547-85c6-7daa62eb5a88-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-b75cfc78d-dqzdg\" (UID: \"101e16ed-d709-4547-85c6-7daa62eb5a88\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-dqzdg"
Sep 18 17:26:32.713712 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:32.713688    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/101e16ed-d709-4547-85c6-7daa62eb5a88-cabundle-cert\") pod \"isvc-canary-create-predictor-b75cfc78d-dqzdg\" (UID: \"101e16ed-d709-4547-85c6-7daa62eb5a88\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-dqzdg"
Sep 18 17:26:32.715213 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:32.715190    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc-proxy-tls\") pod \"isvc-canary-create-v2-predictor-7b4d9977c7-vssvp\" (UID: \"bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-vssvp"
Sep 18 17:26:32.715332 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:32.715319    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/101e16ed-d709-4547-85c6-7daa62eb5a88-proxy-tls\") pod \"isvc-canary-create-predictor-b75cfc78d-dqzdg\" (UID: \"101e16ed-d709-4547-85c6-7daa62eb5a88\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-dqzdg"
Sep 18 17:26:32.722084 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:32.722060    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcnwt\" (UniqueName: \"kubernetes.io/projected/101e16ed-d709-4547-85c6-7daa62eb5a88-kube-api-access-xcnwt\") pod \"isvc-canary-create-predictor-b75cfc78d-dqzdg\" (UID: \"101e16ed-d709-4547-85c6-7daa62eb5a88\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-dqzdg"
Sep 18 17:26:32.722323 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:32.722300    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h65d8\" (UniqueName: \"kubernetes.io/projected/bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc-kube-api-access-h65d8\") pod \"isvc-canary-create-v2-predictor-7b4d9977c7-vssvp\" (UID: \"bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-vssvp"
Sep 18 17:26:32.851516 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:32.851412    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-vssvp"
Sep 18 17:26:32.878242 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:32.878196    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-dqzdg"
Sep 18 17:26:33.129612 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:33.129589    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-vssvp"]
Sep 18 17:26:33.131751 ip-10-0-143-5 kubenswrapper[2881]: W0918 17:26:33.131716    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd3cb6e4_917a_4a79_afb4_2fe3ee6140bc.slice/crio-e07d8fb350e5b4ad6d3dec63071d10d79229c54472a30ceec49e1efe2e370738 WatchSource:0}: Error finding container e07d8fb350e5b4ad6d3dec63071d10d79229c54472a30ceec49e1efe2e370738: Status 404 returned error can't find the container with id e07d8fb350e5b4ad6d3dec63071d10d79229c54472a30ceec49e1efe2e370738
Sep 18 17:26:33.163677 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:33.163653    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-dqzdg"]
Sep 18 17:26:33.166749 ip-10-0-143-5 kubenswrapper[2881]: W0918 17:26:33.166720    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod101e16ed_d709_4547_85c6_7daa62eb5a88.slice/crio-5d212ae897bcfa59851f71cfd7079c56171e279a97c0bfacb8416abaa1d0bdfb WatchSource:0}: Error finding container 5d212ae897bcfa59851f71cfd7079c56171e279a97c0bfacb8416abaa1d0bdfb: Status 404 returned error can't find the container with id 5d212ae897bcfa59851f71cfd7079c56171e279a97c0bfacb8416abaa1d0bdfb
Sep 18 17:26:33.785393 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:33.785349    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-dqzdg" event={"ID":"101e16ed-d709-4547-85c6-7daa62eb5a88","Type":"ContainerStarted","Data":"d18fdad0ac2dbd7f5a801039455346043f5482e1141922429e982f31eedd2f66"}
Sep 18 17:26:33.785882 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:33.785423    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-dqzdg" event={"ID":"101e16ed-d709-4547-85c6-7daa62eb5a88","Type":"ContainerStarted","Data":"5d212ae897bcfa59851f71cfd7079c56171e279a97c0bfacb8416abaa1d0bdfb"}
Sep 18 17:26:33.787395 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:33.787360    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-vssvp" event={"ID":"bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc","Type":"ContainerStarted","Data":"f4a844f485a1324a31c8bb8e6510f137447f7661f9ef7d0a064f4851c92af7f4"}
Sep 18 17:26:33.787489 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:33.787436    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-vssvp" event={"ID":"bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc","Type":"ContainerStarted","Data":"e07d8fb350e5b4ad6d3dec63071d10d79229c54472a30ceec49e1efe2e370738"}
Sep 18 17:26:36.031018 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:36.030957    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hd26h" podUID="f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 18 17:26:37.810825 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:37.810790    2881 generic.go:356] "Generic (PLEG): container finished" podID="101e16ed-d709-4547-85c6-7daa62eb5a88" containerID="d18fdad0ac2dbd7f5a801039455346043f5482e1141922429e982f31eedd2f66" exitCode=0
Sep 18 17:26:37.811363 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:37.810862    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-dqzdg" event={"ID":"101e16ed-d709-4547-85c6-7daa62eb5a88","Type":"ContainerDied","Data":"d18fdad0ac2dbd7f5a801039455346043f5482e1141922429e982f31eedd2f66"}
Sep 18 17:26:37.812646 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:37.812621    2881 generic.go:356] "Generic (PLEG): container finished" podID="bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc" containerID="f4a844f485a1324a31c8bb8e6510f137447f7661f9ef7d0a064f4851c92af7f4" exitCode=0
Sep 18 17:26:37.812750 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:37.812697    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-vssvp" event={"ID":"bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc","Type":"ContainerDied","Data":"f4a844f485a1324a31c8bb8e6510f137447f7661f9ef7d0a064f4851c92af7f4"}
Sep 18 17:26:38.820455 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:38.820417    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-dqzdg" event={"ID":"101e16ed-d709-4547-85c6-7daa62eb5a88","Type":"ContainerStarted","Data":"82f968b8cae8925b0b81233ae79c50545b45858ff364ef3ffeb39d16e3a08404"}
Sep 18 17:26:38.820899 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:38.820485    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-dqzdg" event={"ID":"101e16ed-d709-4547-85c6-7daa62eb5a88","Type":"ContainerStarted","Data":"567b8294593c4be2d62db6c9ef967731be7323aa6d54df26bace1d422bf75b2b"}
Sep 18 17:26:38.820899 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:38.820843    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-dqzdg"
Sep 18 17:26:38.821010 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:38.820910    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-dqzdg"
Sep 18 17:26:38.822817 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:38.822788    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-dqzdg" podUID="101e16ed-d709-4547-85c6-7daa62eb5a88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 18 17:26:38.822957 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:38.822906    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-vssvp" event={"ID":"bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc","Type":"ContainerStarted","Data":"29e9307008ef93d99bedadf9c40b57b6a4fdc525790f00c99601cd52adf6648a"}
Sep 18 17:26:38.823002 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:38.822956    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-vssvp" event={"ID":"bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc","Type":"ContainerStarted","Data":"9684643651e2b24a34016de54f5094a8515901c39f027b632cb89a2cc0a94b7a"}
Sep 18 17:26:38.823180 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:38.823160    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-vssvp"
Sep 18 17:26:38.846147 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:38.846091    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-dqzdg" podStartSLOduration=2.833584563 podStartE2EDuration="6.846074205s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.012489642s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:26:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:26:38.844930645 +0000 UTC m=+1989.044785953" watchObservedRunningTime="2026-09-18 17:26:38.846074205 +0000 UTC m=+1989.045929490"
Sep 18 17:26:38.865415 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:38.865358    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-vssvp" podStartSLOduration=3.035604741 podStartE2EDuration="6.865340196s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.829735455s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:26:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:26:38.863796469 +0000 UTC m=+1989.063651756" watchObservedRunningTime="2026-09-18 17:26:38.865340196 +0000 UTC m=+1989.065195485"
Sep 18 17:26:39.827904 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:39.827858    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-dqzdg" podUID="101e16ed-d709-4547-85c6-7daa62eb5a88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 18 17:26:39.828363 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:39.828066    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-vssvp"
Sep 18 17:26:39.829493 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:39.829463    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-vssvp" podUID="bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 18 17:26:40.833346 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:40.833291    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-vssvp" podUID="bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 18 17:26:44.835569 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:44.835533    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-dqzdg"
Sep 18 17:26:44.837664 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:44.837628    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-dqzdg" podUID="101e16ed-d709-4547-85c6-7daa62eb5a88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 18 17:26:45.637218 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:45.637185    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hd26h"
Sep 18 17:26:45.838089 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:45.838055    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-vssvp"
Sep 18 17:26:45.838763 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:45.838728    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-vssvp" podUID="bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 18 17:26:52.206331 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:52.206297    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hd26h"]
Sep 18 17:26:52.206778 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:52.206563    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hd26h" podUID="f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc" containerName="kserve-container" containerID="cri-o://d0c6d802d15a94f02de4939380e7a5654403c4f619a86fc52106dce0f9f2f8d9" gracePeriod=30
Sep 18 17:26:52.206778 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:52.206605    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hd26h" podUID="f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc" containerName="kube-rbac-proxy" containerID="cri-o://c60522823ad68d7461daa37f717770ac56242ebb2ec9a306ea7576551220a451" gracePeriod=30
Sep 18 17:26:52.895281 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:52.895239    2881 generic.go:356] "Generic (PLEG): container finished" podID="f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc" containerID="c60522823ad68d7461daa37f717770ac56242ebb2ec9a306ea7576551220a451" exitCode=2
Sep 18 17:26:52.895281 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:52.895281    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hd26h" event={"ID":"f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc","Type":"ContainerDied","Data":"c60522823ad68d7461daa37f717770ac56242ebb2ec9a306ea7576551220a451"}
Sep 18 17:26:54.838611 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:54.838562    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-dqzdg" podUID="101e16ed-d709-4547-85c6-7daa62eb5a88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 18 17:26:55.628248 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:55.628183    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hd26h" podUID="f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Sep 18 17:26:55.838812 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:55.838770    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-vssvp" podUID="bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 18 17:26:57.399662 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:57.399624    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5c84876bbf-wmp4k"]
Sep 18 17:26:57.403552 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:57.403530    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5c84876bbf-wmp4k"
Sep 18 17:26:57.409238 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:57.409200    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-predictor-serving-cert\""
Sep 18 17:26:57.409381 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:57.409246    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-kube-rbac-proxy-sar-config\""
Sep 18 17:26:57.429805 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:57.429773    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5c84876bbf-wmp4k"]
Sep 18 17:26:57.541687 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:57.541645    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/361c37a4-5f9a-4f7c-b07c-5cc502425d75-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-5c84876bbf-wmp4k\" (UID: \"361c37a4-5f9a-4f7c-b07c-5cc502425d75\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5c84876bbf-wmp4k"
Sep 18 17:26:57.541886 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:57.541708    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/361c37a4-5f9a-4f7c-b07c-5cc502425d75-proxy-tls\") pod \"isvc-sklearn-v2-predictor-5c84876bbf-wmp4k\" (UID: \"361c37a4-5f9a-4f7c-b07c-5cc502425d75\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5c84876bbf-wmp4k"
Sep 18 17:26:57.541886 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:57.541740    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/361c37a4-5f9a-4f7c-b07c-5cc502425d75-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-5c84876bbf-wmp4k\" (UID: \"361c37a4-5f9a-4f7c-b07c-5cc502425d75\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5c84876bbf-wmp4k"
Sep 18 17:26:57.541886 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:57.541768    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/361c37a4-5f9a-4f7c-b07c-5cc502425d75-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-5c84876bbf-wmp4k\" (UID: \"361c37a4-5f9a-4f7c-b07c-5cc502425d75\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5c84876bbf-wmp4k"
Sep 18 17:26:57.541886 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:57.541787    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94t45\" (UniqueName: \"kubernetes.io/projected/361c37a4-5f9a-4f7c-b07c-5cc502425d75-kube-api-access-94t45\") pod \"isvc-sklearn-v2-predictor-5c84876bbf-wmp4k\" (UID: \"361c37a4-5f9a-4f7c-b07c-5cc502425d75\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5c84876bbf-wmp4k"
Sep 18 17:26:57.642638 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:57.642582    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/361c37a4-5f9a-4f7c-b07c-5cc502425d75-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-5c84876bbf-wmp4k\" (UID: \"361c37a4-5f9a-4f7c-b07c-5cc502425d75\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5c84876bbf-wmp4k"
Sep 18 17:26:57.642836 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:57.642668    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/361c37a4-5f9a-4f7c-b07c-5cc502425d75-proxy-tls\") pod \"isvc-sklearn-v2-predictor-5c84876bbf-wmp4k\" (UID: \"361c37a4-5f9a-4f7c-b07c-5cc502425d75\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5c84876bbf-wmp4k"
Sep 18 17:26:57.642836 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:57.642699    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/361c37a4-5f9a-4f7c-b07c-5cc502425d75-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-5c84876bbf-wmp4k\" (UID: \"361c37a4-5f9a-4f7c-b07c-5cc502425d75\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5c84876bbf-wmp4k"
Sep 18 17:26:57.642836 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:57.642732    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/361c37a4-5f9a-4f7c-b07c-5cc502425d75-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-5c84876bbf-wmp4k\" (UID: \"361c37a4-5f9a-4f7c-b07c-5cc502425d75\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5c84876bbf-wmp4k"
Sep 18 17:26:57.642836 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:57.642754    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94t45\" (UniqueName: \"kubernetes.io/projected/361c37a4-5f9a-4f7c-b07c-5cc502425d75-kube-api-access-94t45\") pod \"isvc-sklearn-v2-predictor-5c84876bbf-wmp4k\" (UID: \"361c37a4-5f9a-4f7c-b07c-5cc502425d75\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5c84876bbf-wmp4k"
Sep 18 17:26:57.643127 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:57.643100    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/361c37a4-5f9a-4f7c-b07c-5cc502425d75-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-5c84876bbf-wmp4k\" (UID: \"361c37a4-5f9a-4f7c-b07c-5cc502425d75\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5c84876bbf-wmp4k"
Sep 18 17:26:57.643331 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:57.643307    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/361c37a4-5f9a-4f7c-b07c-5cc502425d75-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-5c84876bbf-wmp4k\" (UID: \"361c37a4-5f9a-4f7c-b07c-5cc502425d75\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5c84876bbf-wmp4k"
Sep 18 17:26:57.643499 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:57.643477    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/361c37a4-5f9a-4f7c-b07c-5cc502425d75-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-5c84876bbf-wmp4k\" (UID: \"361c37a4-5f9a-4f7c-b07c-5cc502425d75\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5c84876bbf-wmp4k"
Sep 18 17:26:57.645293 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:57.645275    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/361c37a4-5f9a-4f7c-b07c-5cc502425d75-proxy-tls\") pod \"isvc-sklearn-v2-predictor-5c84876bbf-wmp4k\" (UID: \"361c37a4-5f9a-4f7c-b07c-5cc502425d75\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5c84876bbf-wmp4k"
Sep 18 17:26:57.653130 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:57.653066    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94t45\" (UniqueName: \"kubernetes.io/projected/361c37a4-5f9a-4f7c-b07c-5cc502425d75-kube-api-access-94t45\") pod \"isvc-sklearn-v2-predictor-5c84876bbf-wmp4k\" (UID: \"361c37a4-5f9a-4f7c-b07c-5cc502425d75\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5c84876bbf-wmp4k"
Sep 18 17:26:57.713768 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:57.713727    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5c84876bbf-wmp4k"
Sep 18 17:26:57.993303 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:57.993273    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5c84876bbf-wmp4k"]
Sep 18 17:26:57.995105 ip-10-0-143-5 kubenswrapper[2881]: W0918 17:26:57.995075    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod361c37a4_5f9a_4f7c_b07c_5cc502425d75.slice/crio-9be433b20eb86106f429c05b0c464620425423046adc65c508eee0acfe97c53b WatchSource:0}: Error finding container 9be433b20eb86106f429c05b0c464620425423046adc65c508eee0acfe97c53b: Status 404 returned error can't find the container with id 9be433b20eb86106f429c05b0c464620425423046adc65c508eee0acfe97c53b
Sep 18 17:26:58.924713 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:58.924671    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5c84876bbf-wmp4k" event={"ID":"361c37a4-5f9a-4f7c-b07c-5cc502425d75","Type":"ContainerStarted","Data":"9d30149f06afdc066e27a502abb013d32bb98922bab9b3e556de0275a10d134c"}
Sep 18 17:26:58.925169 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:26:58.924737    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5c84876bbf-wmp4k" event={"ID":"361c37a4-5f9a-4f7c-b07c-5cc502425d75","Type":"ContainerStarted","Data":"9be433b20eb86106f429c05b0c464620425423046adc65c508eee0acfe97c53b"}
Sep 18 17:27:00.264393 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:27:00.264368    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hd26h"
Sep 18 17:27:00.369437 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:27:00.369343    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc-kube-rbac-proxy-sar-config\") pod \"f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc\" (UID: \"f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc\") "
Sep 18 17:27:00.369437 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:27:00.369420    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc-proxy-tls\") pod \"f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc\" (UID: \"f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc\") "
Sep 18 17:27:00.369437 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:27:00.369443    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc-cabundle-cert\") pod \"f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc\" (UID: \"f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc\") "
Sep 18 17:27:00.369726 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:27:00.369555    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gxzqj\" (UniqueName: \"kubernetes.io/projected/f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc-kube-api-access-gxzqj\") pod \"f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc\" (UID: \"f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc\") "
Sep 18 17:27:00.369726 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:27:00.369609    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc-kserve-provision-location\") pod \"f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc\" (UID: \"f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc\") "
Sep 18 17:27:00.369830 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:27:00.369733    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc" (UID: "f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:27:00.369830 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:27:00.369797    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc" (UID: "f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:27:00.370025 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:27:00.370000    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc" (UID: "f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:27:00.370025 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:27:00.370019    2881 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:27:00.370152 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:27:00.370034    2881 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc-cabundle-cert\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:27:00.371606 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:27:00.371583    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc-kube-api-access-gxzqj" (OuterVolumeSpecName: "kube-api-access-gxzqj") pod "f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc" (UID: "f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc"). InnerVolumeSpecName "kube-api-access-gxzqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:27:00.371719 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:27:00.371646    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc" (UID: "f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:27:00.471424 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:27:00.471393    2881 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc-proxy-tls\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:27:00.471424 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:27:00.471417    2881 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxzqj\" (UniqueName: \"kubernetes.io/projected/f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc-kube-api-access-gxzqj\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:27:00.471424 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:27:00.471429    2881 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc-kserve-provision-location\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:27:00.939467 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:27:00.939424    2881 generic.go:356] "Generic (PLEG): container finished" podID="f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc" containerID="d0c6d802d15a94f02de4939380e7a5654403c4f619a86fc52106dce0f9f2f8d9" exitCode=0
Sep 18 17:27:00.939671 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:27:00.939488    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hd26h"
Sep 18 17:27:00.939671 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:27:00.939499    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hd26h" event={"ID":"f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc","Type":"ContainerDied","Data":"d0c6d802d15a94f02de4939380e7a5654403c4f619a86fc52106dce0f9f2f8d9"}
Sep 18 17:27:00.939671 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:27:00.939579    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hd26h" event={"ID":"f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc","Type":"ContainerDied","Data":"7cacc73096307ca924ba7763aa43ecc0f2295eeaa73f69b5ef9436d47e7c8a6a"}
Sep 18 17:27:00.939671 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:27:00.939618    2881 scope.go:117] "RemoveContainer" containerID="c60522823ad68d7461daa37f717770ac56242ebb2ec9a306ea7576551220a451"
Sep 18 17:27:00.957286 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:27:00.957265    2881 scope.go:117] "RemoveContainer" containerID="d0c6d802d15a94f02de4939380e7a5654403c4f619a86fc52106dce0f9f2f8d9"
Sep 18 17:27:00.966171 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:27:00.966145    2881 scope.go:117] "RemoveContainer" containerID="cc23ade7d57408b3d3799ba24e37461e6430de708f31e0dd4c7587f2bbdfca20"
Sep 18 17:27:00.972129 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:27:00.972094    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hd26h"]
Sep 18 17:27:00.975924 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:27:00.975762    2881 scope.go:117] "RemoveContainer" containerID="c60522823ad68d7461daa37f717770ac56242ebb2ec9a306ea7576551220a451"
Sep 18 17:27:00.976613 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:27:00.976579    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c60522823ad68d7461daa37f717770ac56242ebb2ec9a306ea7576551220a451\": container with ID starting with c60522823ad68d7461daa37f717770ac56242ebb2ec9a306ea7576551220a451 not found: ID does not exist" containerID="c60522823ad68d7461daa37f717770ac56242ebb2ec9a306ea7576551220a451"
Sep 18 17:27:00.976717 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:27:00.976619    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c60522823ad68d7461daa37f717770ac56242ebb2ec9a306ea7576551220a451"} err="failed to get container status \"c60522823ad68d7461daa37f717770ac56242ebb2ec9a306ea7576551220a451\": rpc error: code = NotFound desc = could not find container \"c60522823ad68d7461daa37f717770ac56242ebb2ec9a306ea7576551220a451\": container with ID starting with c60522823ad68d7461daa37f717770ac56242ebb2ec9a306ea7576551220a451 not found: ID does not exist"
Sep 18 17:27:00.976717 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:27:00.976646    2881 scope.go:117] "RemoveContainer" containerID="d0c6d802d15a94f02de4939380e7a5654403c4f619a86fc52106dce0f9f2f8d9"
Sep 18 17:27:00.976988 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:27:00.976970    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0c6d802d15a94f02de4939380e7a5654403c4f619a86fc52106dce0f9f2f8d9\": container with ID starting with d0c6d802d15a94f02de4939380e7a5654403c4f619a86fc52106dce0f9f2f8d9 not found: ID does not exist" containerID="d0c6d802d15a94f02de4939380e7a5654403c4f619a86fc52106dce0f9f2f8d9"
Sep 18 17:27:00.977025 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:27:00.976995    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0c6d802d15a94f02de4939380e7a5654403c4f619a86fc52106dce0f9f2f8d9"} err="failed to get container status \"d0c6d802d15a94f02de4939380e7a5654403c4f619a86fc52106dce0f9f2f8d9\": rpc error: code = NotFound desc = could not find container \"d0c6d802d15a94f02de4939380e7a5654403c4f619a86fc52106dce0f9f2f8d9\": container with ID starting with d0c6d802d15a94f02de4939380e7a5654403c4f619a86fc52106dce0f9f2f8d9 not found: ID does not exist"
Sep 18 17:27:00.977025 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:27:00.977011    2881 scope.go:117] "RemoveContainer" containerID="cc23ade7d57408b3d3799ba24e37461e6430de708f31e0dd4c7587f2bbdfca20"
Sep 18 17:27:00.977316 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:27:00.977295    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc23ade7d57408b3d3799ba24e37461e6430de708f31e0dd4c7587f2bbdfca20\": container with ID starting with cc23ade7d57408b3d3799ba24e37461e6430de708f31e0dd4c7587f2bbdfca20 not found: ID does not exist" containerID="cc23ade7d57408b3d3799ba24e37461e6430de708f31e0dd4c7587f2bbdfca20"
Sep 18 17:27:00.977361 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:27:00.977322    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc23ade7d57408b3d3799ba24e37461e6430de708f31e0dd4c7587f2bbdfca20"} err="failed to get container status \"cc23ade7d57408b3d3799ba24e37461e6430de708f31e0dd4c7587f2bbdfca20\": rpc error: code = NotFound desc = could not find container \"cc23ade7d57408b3d3799ba24e37461e6430de708f31e0dd4c7587f2bbdfca20\": container with ID starting with cc23ade7d57408b3d3799ba24e37461e6430de708f31e0dd4c7587f2bbdfca20 not found: ID does not exist"
Sep 18 17:27:00.978087 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:27:00.978066    2881 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hd26h"]
Sep 18 17:27:01.893889 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:27:01.893851    2881 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc" path="/var/lib/kubelet/pods/f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc/volumes"
Sep 18 17:27:01.947300 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:27:01.947269    2881 generic.go:356] "Generic (PLEG): container finished" podID="361c37a4-5f9a-4f7c-b07c-5cc502425d75" containerID="9d30149f06afdc066e27a502abb013d32bb98922bab9b3e556de0275a10d134c" exitCode=0
Sep 18 17:27:01.947489 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:27:01.947342    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5c84876bbf-wmp4k" event={"ID":"361c37a4-5f9a-4f7c-b07c-5cc502425d75","Type":"ContainerDied","Data":"9d30149f06afdc066e27a502abb013d32bb98922bab9b3e556de0275a10d134c"}
Sep 18 17:27:02.956093 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:27:02.956057    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5c84876bbf-wmp4k" event={"ID":"361c37a4-5f9a-4f7c-b07c-5cc502425d75","Type":"ContainerStarted","Data":"41e8b6bb7a30589215b54ac3e1bb2ba7851a8b928fefa09f3eb97c94a76e2d4f"}
Sep 18 17:27:02.956504 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:27:02.956122    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5c84876bbf-wmp4k" event={"ID":"361c37a4-5f9a-4f7c-b07c-5cc502425d75","Type":"ContainerStarted","Data":"4341e6c2b0f8ef05f8c404c662a79bada19de91248c84499856ae8d3539472db"}
Sep 18 17:27:02.956504 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:27:02.956338    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5c84876bbf-wmp4k"
Sep 18 17:27:02.983457 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:27:02.983405    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5c84876bbf-wmp4k" podStartSLOduration=2.124935905 podStartE2EDuration="5.983387346s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.858451441s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:26:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:27:02.983255991 +0000 UTC m=+2013.183111279" watchObservedRunningTime="2026-09-18 17:27:02.983387346 +0000 UTC m=+2013.183242632"
Sep 18 17:27:03.960767 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:27:03.960736    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5c84876bbf-wmp4k"
Sep 18 17:27:03.962056 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:27:03.962029    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5c84876bbf-wmp4k" podUID="361c37a4-5f9a-4f7c-b07c-5cc502425d75" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 18 17:27:04.838560 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:27:04.838514    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-dqzdg" podUID="101e16ed-d709-4547-85c6-7daa62eb5a88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 18 17:27:04.966009 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:27:04.965966    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5c84876bbf-wmp4k" podUID="361c37a4-5f9a-4f7c-b07c-5cc502425d75" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 18 17:27:05.839591 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:27:05.839543    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-vssvp" podUID="bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 18 17:27:09.970613 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:27:09.970587    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5c84876bbf-wmp4k"
Sep 18 17:27:09.971122 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:27:09.971096    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5c84876bbf-wmp4k" podUID="361c37a4-5f9a-4f7c-b07c-5cc502425d75" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 18 17:27:14.838156 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:27:14.838111    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-dqzdg" podUID="101e16ed-d709-4547-85c6-7daa62eb5a88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 18 17:27:15.838703 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:27:15.838664    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-vssvp" podUID="bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 18 17:27:19.971037 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:27:19.970993    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5c84876bbf-wmp4k" podUID="361c37a4-5f9a-4f7c-b07c-5cc502425d75" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 18 17:27:24.838654 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:27:24.838605    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-dqzdg" podUID="101e16ed-d709-4547-85c6-7daa62eb5a88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 18 17:27:25.838800 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:27:25.838758    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-vssvp" podUID="bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 18 17:27:29.971845 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:27:29.971799    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5c84876bbf-wmp4k" podUID="361c37a4-5f9a-4f7c-b07c-5cc502425d75" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 18 17:27:34.838319 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:27:34.838266    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-dqzdg" podUID="101e16ed-d709-4547-85c6-7daa62eb5a88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 18 17:27:35.838773 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:27:35.838727    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-vssvp" podUID="bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 18 17:27:39.971802 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:27:39.971758    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5c84876bbf-wmp4k" podUID="361c37a4-5f9a-4f7c-b07c-5cc502425d75" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 18 17:27:44.838152 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:27:44.838101    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-dqzdg" podUID="101e16ed-d709-4547-85c6-7daa62eb5a88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 18 17:27:45.838728 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:27:45.838681    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-vssvp" podUID="bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 18 17:27:47.888746 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:27:47.888694    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-vssvp" podUID="bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 18 17:27:49.971821 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:27:49.971781    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5c84876bbf-wmp4k" podUID="361c37a4-5f9a-4f7c-b07c-5cc502425d75" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 18 17:27:54.838682 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:27:54.838627    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-dqzdg" podUID="101e16ed-d709-4547-85c6-7daa62eb5a88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 18 17:27:57.889084 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:27:57.889046    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-vssvp" podUID="bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 18 17:27:59.971302 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:27:59.971267    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5c84876bbf-wmp4k" podUID="361c37a4-5f9a-4f7c-b07c-5cc502425d75" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 18 17:28:04.838475 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:04.838441    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-dqzdg"
Sep 18 17:28:07.893674 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:07.893642    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-vssvp"
Sep 18 17:28:09.971676 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:09.971636    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5c84876bbf-wmp4k" podUID="361c37a4-5f9a-4f7c-b07c-5cc502425d75" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 18 17:28:12.628730 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:12.628688    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-vssvp"]
Sep 18 17:28:12.629207 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:12.628966    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-vssvp" podUID="bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc" containerName="kserve-container" containerID="cri-o://9684643651e2b24a34016de54f5094a8515901c39f027b632cb89a2cc0a94b7a" gracePeriod=30
Sep 18 17:28:12.629207 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:12.629013    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-vssvp" podUID="bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc" containerName="kube-rbac-proxy" containerID="cri-o://29e9307008ef93d99bedadf9c40b57b6a4fdc525790f00c99601cd52adf6648a" gracePeriod=30
Sep 18 17:28:12.645798 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:12.645762    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-dqzdg"]
Sep 18 17:28:12.646630 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:12.646585    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-dqzdg" podUID="101e16ed-d709-4547-85c6-7daa62eb5a88" containerName="kserve-container" containerID="cri-o://567b8294593c4be2d62db6c9ef967731be7323aa6d54df26bace1d422bf75b2b" gracePeriod=30
Sep 18 17:28:12.646768 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:12.646586    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-dqzdg" podUID="101e16ed-d709-4547-85c6-7daa62eb5a88" containerName="kube-rbac-proxy" containerID="cri-o://82f968b8cae8925b0b81233ae79c50545b45858ff364ef3ffeb39d16e3a08404" gracePeriod=30
Sep 18 17:28:13.287420 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:13.287385    2881 generic.go:356] "Generic (PLEG): container finished" podID="101e16ed-d709-4547-85c6-7daa62eb5a88" containerID="82f968b8cae8925b0b81233ae79c50545b45858ff364ef3ffeb39d16e3a08404" exitCode=2
Sep 18 17:28:13.287615 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:13.287458    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-dqzdg" event={"ID":"101e16ed-d709-4547-85c6-7daa62eb5a88","Type":"ContainerDied","Data":"82f968b8cae8925b0b81233ae79c50545b45858ff364ef3ffeb39d16e3a08404"}
Sep 18 17:28:13.289513 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:13.289489    2881 generic.go:356] "Generic (PLEG): container finished" podID="bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc" containerID="29e9307008ef93d99bedadf9c40b57b6a4fdc525790f00c99601cd52adf6648a" exitCode=2
Sep 18 17:28:13.289636 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:13.289557    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-vssvp" event={"ID":"bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc","Type":"ContainerDied","Data":"29e9307008ef93d99bedadf9c40b57b6a4fdc525790f00c99601cd52adf6648a"}
Sep 18 17:28:14.828057 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:14.828011    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-dqzdg" podUID="101e16ed-d709-4547-85c6-7daa62eb5a88" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
Sep 18 17:28:14.838683 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:14.838641    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-dqzdg" podUID="101e16ed-d709-4547-85c6-7daa62eb5a88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 18 17:28:15.834040 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:15.833985    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-vssvp" podUID="bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.40:8643/healthz\": dial tcp 10.134.0.40:8643: connect: connection refused"
Sep 18 17:28:17.888616 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:17.888565    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-vssvp" podUID="bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 18 17:28:18.014924 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.014895    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-dqzdg"
Sep 18 17:28:18.080012 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.079981    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/101e16ed-d709-4547-85c6-7daa62eb5a88-kube-rbac-proxy-sar-config\") pod \"101e16ed-d709-4547-85c6-7daa62eb5a88\" (UID: \"101e16ed-d709-4547-85c6-7daa62eb5a88\") "
Sep 18 17:28:18.080201 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.080065    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/101e16ed-d709-4547-85c6-7daa62eb5a88-cabundle-cert\") pod \"101e16ed-d709-4547-85c6-7daa62eb5a88\" (UID: \"101e16ed-d709-4547-85c6-7daa62eb5a88\") "
Sep 18 17:28:18.080478 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.080332    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/101e16ed-d709-4547-85c6-7daa62eb5a88-kserve-provision-location\") pod \"101e16ed-d709-4547-85c6-7daa62eb5a88\" (UID: \"101e16ed-d709-4547-85c6-7daa62eb5a88\") "
Sep 18 17:28:18.080478 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.080387    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xcnwt\" (UniqueName: \"kubernetes.io/projected/101e16ed-d709-4547-85c6-7daa62eb5a88-kube-api-access-xcnwt\") pod \"101e16ed-d709-4547-85c6-7daa62eb5a88\" (UID: \"101e16ed-d709-4547-85c6-7daa62eb5a88\") "
Sep 18 17:28:18.080478 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.080418    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/101e16ed-d709-4547-85c6-7daa62eb5a88-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "101e16ed-d709-4547-85c6-7daa62eb5a88" (UID: "101e16ed-d709-4547-85c6-7daa62eb5a88"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:28:18.080717 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.080688    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/101e16ed-d709-4547-85c6-7daa62eb5a88-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "101e16ed-d709-4547-85c6-7daa62eb5a88" (UID: "101e16ed-d709-4547-85c6-7daa62eb5a88"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:28:18.080840 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.080769    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/101e16ed-d709-4547-85c6-7daa62eb5a88-proxy-tls\") pod \"101e16ed-d709-4547-85c6-7daa62eb5a88\" (UID: \"101e16ed-d709-4547-85c6-7daa62eb5a88\") "
Sep 18 17:28:18.080974 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.080943    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/101e16ed-d709-4547-85c6-7daa62eb5a88-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "101e16ed-d709-4547-85c6-7daa62eb5a88" (UID: "101e16ed-d709-4547-85c6-7daa62eb5a88"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:28:18.081127 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.081103    2881 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/101e16ed-d709-4547-85c6-7daa62eb5a88-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:28:18.081274 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.081130    2881 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/101e16ed-d709-4547-85c6-7daa62eb5a88-cabundle-cert\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:28:18.081274 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.081144    2881 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/101e16ed-d709-4547-85c6-7daa62eb5a88-kserve-provision-location\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:28:18.082907 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.082819    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/101e16ed-d709-4547-85c6-7daa62eb5a88-kube-api-access-xcnwt" (OuterVolumeSpecName: "kube-api-access-xcnwt") pod "101e16ed-d709-4547-85c6-7daa62eb5a88" (UID: "101e16ed-d709-4547-85c6-7daa62eb5a88"). InnerVolumeSpecName "kube-api-access-xcnwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:28:18.083089 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.083067    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/101e16ed-d709-4547-85c6-7daa62eb5a88-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "101e16ed-d709-4547-85c6-7daa62eb5a88" (UID: "101e16ed-d709-4547-85c6-7daa62eb5a88"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:28:18.084804 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.084783    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-vssvp"
Sep 18 17:28:18.181832 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.181787    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h65d8\" (UniqueName: \"kubernetes.io/projected/bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc-kube-api-access-h65d8\") pod \"bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc\" (UID: \"bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc\") "
Sep 18 17:28:18.182032 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.181843    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc-proxy-tls\") pod \"bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc\" (UID: \"bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc\") "
Sep 18 17:28:18.182032 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.181969    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc-cabundle-cert\") pod \"bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc\" (UID: \"bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc\") "
Sep 18 17:28:18.182032 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.181999    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc-kserve-provision-location\") pod \"bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc\" (UID: \"bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc\") "
Sep 18 17:28:18.182205 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.182032    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc-kube-rbac-proxy-sar-config\") pod \"bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc\" (UID: \"bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc\") "
Sep 18 17:28:18.182731 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.182353    2881 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcnwt\" (UniqueName: \"kubernetes.io/projected/101e16ed-d709-4547-85c6-7daa62eb5a88-kube-api-access-xcnwt\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:28:18.182731 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.182376    2881 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/101e16ed-d709-4547-85c6-7daa62eb5a88-proxy-tls\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:28:18.182731 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.182406    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc" (UID: "bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:28:18.182731 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.182431    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc" (UID: "bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:28:18.182731 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.182685    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc" (UID: "bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:28:18.184046 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.184014    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc" (UID: "bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:28:18.184212 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.184192    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc-kube-api-access-h65d8" (OuterVolumeSpecName: "kube-api-access-h65d8") pod "bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc" (UID: "bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc"). InnerVolumeSpecName "kube-api-access-h65d8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:28:18.221716 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.221677    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-m2lrf"]
Sep 18 17:28:18.222064 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.222051    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc" containerName="storage-initializer"
Sep 18 17:28:18.222110 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.222066    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc" containerName="storage-initializer"
Sep 18 17:28:18.222110 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.222076    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc" containerName="kserve-container"
Sep 18 17:28:18.222110 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.222082    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc" containerName="kserve-container"
Sep 18 17:28:18.222110 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.222097    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc" containerName="kserve-container"
Sep 18 17:28:18.222110 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.222103    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc" containerName="kserve-container"
Sep 18 17:28:18.222110 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.222110    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="101e16ed-d709-4547-85c6-7daa62eb5a88" containerName="storage-initializer"
Sep 18 17:28:18.222335 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.222116    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="101e16ed-d709-4547-85c6-7daa62eb5a88" containerName="storage-initializer"
Sep 18 17:28:18.222335 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.222124    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc" containerName="kube-rbac-proxy"
Sep 18 17:28:18.222335 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.222129    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc" containerName="kube-rbac-proxy"
Sep 18 17:28:18.222335 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.222135    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc" containerName="storage-initializer"
Sep 18 17:28:18.222335 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.222141    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc" containerName="storage-initializer"
Sep 18 17:28:18.222335 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.222146    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="101e16ed-d709-4547-85c6-7daa62eb5a88" containerName="kserve-container"
Sep 18 17:28:18.222335 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.222153    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="101e16ed-d709-4547-85c6-7daa62eb5a88" containerName="kserve-container"
Sep 18 17:28:18.222335 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.222169    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="101e16ed-d709-4547-85c6-7daa62eb5a88" containerName="kube-rbac-proxy"
Sep 18 17:28:18.222335 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.222175    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="101e16ed-d709-4547-85c6-7daa62eb5a88" containerName="kube-rbac-proxy"
Sep 18 17:28:18.222335 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.222187    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc" containerName="kube-rbac-proxy"
Sep 18 17:28:18.222335 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.222193    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc" containerName="kube-rbac-proxy"
Sep 18 17:28:18.222335 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.222270    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc" containerName="kserve-container"
Sep 18 17:28:18.222335 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.222279    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc" containerName="kube-rbac-proxy"
Sep 18 17:28:18.222335 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.222285    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3f739ff-5b1a-4dcd-a6ca-15a66bcc97bc" containerName="kserve-container"
Sep 18 17:28:18.222335 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.222292    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="101e16ed-d709-4547-85c6-7daa62eb5a88" containerName="kserve-container"
Sep 18 17:28:18.222335 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.222297    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="101e16ed-d709-4547-85c6-7daa62eb5a88" containerName="kube-rbac-proxy"
Sep 18 17:28:18.222335 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.222302    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc" containerName="kube-rbac-proxy"
Sep 18 17:28:18.225767 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.225749    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-m2lrf"
Sep 18 17:28:18.229911 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.229891    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 18 17:28:18.230516 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.230501    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 18 17:28:18.245864 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.245837    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-m2lrf"]
Sep 18 17:28:18.283331 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.283294    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1f555f6-f287-4856-871a-242de5e78bbf-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5f4f848c44-m2lrf\" (UID: \"a1f555f6-f287-4856-871a-242de5e78bbf\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-m2lrf"
Sep 18 17:28:18.283331 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.283332    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a1f555f6-f287-4856-871a-242de5e78bbf-cabundle-cert\") pod \"isvc-canary-promote-predictor-5f4f848c44-m2lrf\" (UID: \"a1f555f6-f287-4856-871a-242de5e78bbf\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-m2lrf"
Sep 18 17:28:18.283571 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.283402    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1f555f6-f287-4856-871a-242de5e78bbf-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5f4f848c44-m2lrf\" (UID: \"a1f555f6-f287-4856-871a-242de5e78bbf\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-m2lrf"
Sep 18 17:28:18.283571 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.283449    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cs4j\" (UniqueName: \"kubernetes.io/projected/a1f555f6-f287-4856-871a-242de5e78bbf-kube-api-access-4cs4j\") pod \"isvc-canary-promote-predictor-5f4f848c44-m2lrf\" (UID: \"a1f555f6-f287-4856-871a-242de5e78bbf\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-m2lrf"
Sep 18 17:28:18.283571 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.283505    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1f555f6-f287-4856-871a-242de5e78bbf-proxy-tls\") pod \"isvc-canary-promote-predictor-5f4f848c44-m2lrf\" (UID: \"a1f555f6-f287-4856-871a-242de5e78bbf\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-m2lrf"
Sep 18 17:28:18.283571 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.283570    2881 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc-cabundle-cert\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:28:18.283717 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.283580    2881 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc-kserve-provision-location\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:28:18.283717 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.283594    2881 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:28:18.283717 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.283608    2881 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h65d8\" (UniqueName: \"kubernetes.io/projected/bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc-kube-api-access-h65d8\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:28:18.283717 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.283619    2881 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc-proxy-tls\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:28:18.314590 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.314559    2881 generic.go:356] "Generic (PLEG): container finished" podID="101e16ed-d709-4547-85c6-7daa62eb5a88" containerID="567b8294593c4be2d62db6c9ef967731be7323aa6d54df26bace1d422bf75b2b" exitCode=0
Sep 18 17:28:18.314788 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.314614    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-dqzdg"
Sep 18 17:28:18.314788 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.314645    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-dqzdg" event={"ID":"101e16ed-d709-4547-85c6-7daa62eb5a88","Type":"ContainerDied","Data":"567b8294593c4be2d62db6c9ef967731be7323aa6d54df26bace1d422bf75b2b"}
Sep 18 17:28:18.314788 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.314702    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-dqzdg" event={"ID":"101e16ed-d709-4547-85c6-7daa62eb5a88","Type":"ContainerDied","Data":"5d212ae897bcfa59851f71cfd7079c56171e279a97c0bfacb8416abaa1d0bdfb"}
Sep 18 17:28:18.314788 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.314737    2881 scope.go:117] "RemoveContainer" containerID="82f968b8cae8925b0b81233ae79c50545b45858ff364ef3ffeb39d16e3a08404"
Sep 18 17:28:18.316753 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.316732    2881 generic.go:356] "Generic (PLEG): container finished" podID="bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc" containerID="9684643651e2b24a34016de54f5094a8515901c39f027b632cb89a2cc0a94b7a" exitCode=0
Sep 18 17:28:18.316884 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.316787    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-vssvp"
Sep 18 17:28:18.316884 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.316804    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-vssvp" event={"ID":"bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc","Type":"ContainerDied","Data":"9684643651e2b24a34016de54f5094a8515901c39f027b632cb89a2cc0a94b7a"}
Sep 18 17:28:18.316884 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.316851    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-vssvp" event={"ID":"bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc","Type":"ContainerDied","Data":"e07d8fb350e5b4ad6d3dec63071d10d79229c54472a30ceec49e1efe2e370738"}
Sep 18 17:28:18.324753 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.324735    2881 scope.go:117] "RemoveContainer" containerID="567b8294593c4be2d62db6c9ef967731be7323aa6d54df26bace1d422bf75b2b"
Sep 18 17:28:18.333087 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.333042    2881 scope.go:117] "RemoveContainer" containerID="d18fdad0ac2dbd7f5a801039455346043f5482e1141922429e982f31eedd2f66"
Sep 18 17:28:18.341566 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.341546    2881 scope.go:117] "RemoveContainer" containerID="82f968b8cae8925b0b81233ae79c50545b45858ff364ef3ffeb39d16e3a08404"
Sep 18 17:28:18.341894 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:28:18.341875    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82f968b8cae8925b0b81233ae79c50545b45858ff364ef3ffeb39d16e3a08404\": container with ID starting with 82f968b8cae8925b0b81233ae79c50545b45858ff364ef3ffeb39d16e3a08404 not found: ID does not exist" containerID="82f968b8cae8925b0b81233ae79c50545b45858ff364ef3ffeb39d16e3a08404"
Sep 18 17:28:18.341954 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.341904    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82f968b8cae8925b0b81233ae79c50545b45858ff364ef3ffeb39d16e3a08404"} err="failed to get container status \"82f968b8cae8925b0b81233ae79c50545b45858ff364ef3ffeb39d16e3a08404\": rpc error: code = NotFound desc = could not find container \"82f968b8cae8925b0b81233ae79c50545b45858ff364ef3ffeb39d16e3a08404\": container with ID starting with 82f968b8cae8925b0b81233ae79c50545b45858ff364ef3ffeb39d16e3a08404 not found: ID does not exist"
Sep 18 17:28:18.341954 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.341923    2881 scope.go:117] "RemoveContainer" containerID="567b8294593c4be2d62db6c9ef967731be7323aa6d54df26bace1d422bf75b2b"
Sep 18 17:28:18.342248 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:28:18.342199    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"567b8294593c4be2d62db6c9ef967731be7323aa6d54df26bace1d422bf75b2b\": container with ID starting with 567b8294593c4be2d62db6c9ef967731be7323aa6d54df26bace1d422bf75b2b not found: ID does not exist" containerID="567b8294593c4be2d62db6c9ef967731be7323aa6d54df26bace1d422bf75b2b"
Sep 18 17:28:18.342296 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.342272    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"567b8294593c4be2d62db6c9ef967731be7323aa6d54df26bace1d422bf75b2b"} err="failed to get container status \"567b8294593c4be2d62db6c9ef967731be7323aa6d54df26bace1d422bf75b2b\": rpc error: code = NotFound desc = could not find container \"567b8294593c4be2d62db6c9ef967731be7323aa6d54df26bace1d422bf75b2b\": container with ID starting with 567b8294593c4be2d62db6c9ef967731be7323aa6d54df26bace1d422bf75b2b not found: ID does not exist"
Sep 18 17:28:18.342296 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.342289    2881 scope.go:117] "RemoveContainer" containerID="d18fdad0ac2dbd7f5a801039455346043f5482e1141922429e982f31eedd2f66"
Sep 18 17:28:18.342536 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:28:18.342519    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d18fdad0ac2dbd7f5a801039455346043f5482e1141922429e982f31eedd2f66\": container with ID starting with d18fdad0ac2dbd7f5a801039455346043f5482e1141922429e982f31eedd2f66 not found: ID does not exist" containerID="d18fdad0ac2dbd7f5a801039455346043f5482e1141922429e982f31eedd2f66"
Sep 18 17:28:18.342595 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.342538    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d18fdad0ac2dbd7f5a801039455346043f5482e1141922429e982f31eedd2f66"} err="failed to get container status \"d18fdad0ac2dbd7f5a801039455346043f5482e1141922429e982f31eedd2f66\": rpc error: code = NotFound desc = could not find container \"d18fdad0ac2dbd7f5a801039455346043f5482e1141922429e982f31eedd2f66\": container with ID starting with d18fdad0ac2dbd7f5a801039455346043f5482e1141922429e982f31eedd2f66 not found: ID does not exist"
Sep 18 17:28:18.342595 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.342551    2881 scope.go:117] "RemoveContainer" containerID="29e9307008ef93d99bedadf9c40b57b6a4fdc525790f00c99601cd52adf6648a"
Sep 18 17:28:18.350208 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.350185    2881 scope.go:117] "RemoveContainer" containerID="9684643651e2b24a34016de54f5094a8515901c39f027b632cb89a2cc0a94b7a"
Sep 18 17:28:18.354405 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.354379    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-dqzdg"]
Sep 18 17:28:18.358136 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.358120    2881 scope.go:117] "RemoveContainer" containerID="f4a844f485a1324a31c8bb8e6510f137447f7661f9ef7d0a064f4851c92af7f4"
Sep 18 17:28:18.367540 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.367513    2881 scope.go:117] "RemoveContainer" containerID="29e9307008ef93d99bedadf9c40b57b6a4fdc525790f00c99601cd52adf6648a"
Sep 18 17:28:18.367863 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:28:18.367843    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29e9307008ef93d99bedadf9c40b57b6a4fdc525790f00c99601cd52adf6648a\": container with ID starting with 29e9307008ef93d99bedadf9c40b57b6a4fdc525790f00c99601cd52adf6648a not found: ID does not exist" containerID="29e9307008ef93d99bedadf9c40b57b6a4fdc525790f00c99601cd52adf6648a"
Sep 18 17:28:18.367953 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.367868    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29e9307008ef93d99bedadf9c40b57b6a4fdc525790f00c99601cd52adf6648a"} err="failed to get container status \"29e9307008ef93d99bedadf9c40b57b6a4fdc525790f00c99601cd52adf6648a\": rpc error: code = NotFound desc = could not find container \"29e9307008ef93d99bedadf9c40b57b6a4fdc525790f00c99601cd52adf6648a\": container with ID starting with 29e9307008ef93d99bedadf9c40b57b6a4fdc525790f00c99601cd52adf6648a not found: ID does not exist"
Sep 18 17:28:18.367953 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.367887    2881 scope.go:117] "RemoveContainer" containerID="9684643651e2b24a34016de54f5094a8515901c39f027b632cb89a2cc0a94b7a"
Sep 18 17:28:18.367953 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.367896    2881 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-b75cfc78d-dqzdg"]
Sep 18 17:28:18.368174 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:28:18.368156    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9684643651e2b24a34016de54f5094a8515901c39f027b632cb89a2cc0a94b7a\": container with ID starting with 9684643651e2b24a34016de54f5094a8515901c39f027b632cb89a2cc0a94b7a not found: ID does not exist" containerID="9684643651e2b24a34016de54f5094a8515901c39f027b632cb89a2cc0a94b7a"
Sep 18 17:28:18.368213 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.368180    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9684643651e2b24a34016de54f5094a8515901c39f027b632cb89a2cc0a94b7a"} err="failed to get container status \"9684643651e2b24a34016de54f5094a8515901c39f027b632cb89a2cc0a94b7a\": rpc error: code = NotFound desc = could not find container \"9684643651e2b24a34016de54f5094a8515901c39f027b632cb89a2cc0a94b7a\": container with ID starting with 9684643651e2b24a34016de54f5094a8515901c39f027b632cb89a2cc0a94b7a not found: ID does not exist"
Sep 18 17:28:18.368213 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.368202    2881 scope.go:117] "RemoveContainer" containerID="f4a844f485a1324a31c8bb8e6510f137447f7661f9ef7d0a064f4851c92af7f4"
Sep 18 17:28:18.368467 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:28:18.368449    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4a844f485a1324a31c8bb8e6510f137447f7661f9ef7d0a064f4851c92af7f4\": container with ID starting with f4a844f485a1324a31c8bb8e6510f137447f7661f9ef7d0a064f4851c92af7f4 not found: ID does not exist" containerID="f4a844f485a1324a31c8bb8e6510f137447f7661f9ef7d0a064f4851c92af7f4"
Sep 18 17:28:18.368513 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.368474    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4a844f485a1324a31c8bb8e6510f137447f7661f9ef7d0a064f4851c92af7f4"} err="failed to get container status \"f4a844f485a1324a31c8bb8e6510f137447f7661f9ef7d0a064f4851c92af7f4\": rpc error: code = NotFound desc = could not find container \"f4a844f485a1324a31c8bb8e6510f137447f7661f9ef7d0a064f4851c92af7f4\": container with ID starting with f4a844f485a1324a31c8bb8e6510f137447f7661f9ef7d0a064f4851c92af7f4 not found: ID does not exist"
Sep 18 17:28:18.384643 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.384612    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1f555f6-f287-4856-871a-242de5e78bbf-proxy-tls\") pod \"isvc-canary-promote-predictor-5f4f848c44-m2lrf\" (UID: \"a1f555f6-f287-4856-871a-242de5e78bbf\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-m2lrf"
Sep 18 17:28:18.384827 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.384668    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1f555f6-f287-4856-871a-242de5e78bbf-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5f4f848c44-m2lrf\" (UID: \"a1f555f6-f287-4856-871a-242de5e78bbf\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-m2lrf"
Sep 18 17:28:18.384827 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.384686    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a1f555f6-f287-4856-871a-242de5e78bbf-cabundle-cert\") pod \"isvc-canary-promote-predictor-5f4f848c44-m2lrf\" (UID: \"a1f555f6-f287-4856-871a-242de5e78bbf\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-m2lrf"
Sep 18 17:28:18.384827 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.384724    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1f555f6-f287-4856-871a-242de5e78bbf-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5f4f848c44-m2lrf\" (UID: \"a1f555f6-f287-4856-871a-242de5e78bbf\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-m2lrf"
Sep 18 17:28:18.384827 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.384751    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cs4j\" (UniqueName: \"kubernetes.io/projected/a1f555f6-f287-4856-871a-242de5e78bbf-kube-api-access-4cs4j\") pod \"isvc-canary-promote-predictor-5f4f848c44-m2lrf\" (UID: \"a1f555f6-f287-4856-871a-242de5e78bbf\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-m2lrf"
Sep 18 17:28:18.384827 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:28:18.384767    2881 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-predictor-serving-cert: secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 18 17:28:18.385133 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:28:18.384842    2881 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1f555f6-f287-4856-871a-242de5e78bbf-proxy-tls podName:a1f555f6-f287-4856-871a-242de5e78bbf nodeName:}" failed. No retries permitted until 2026-09-18 17:28:18.884819914 +0000 UTC m=+2089.084675183 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a1f555f6-f287-4856-871a-242de5e78bbf-proxy-tls") pod "isvc-canary-promote-predictor-5f4f848c44-m2lrf" (UID: "a1f555f6-f287-4856-871a-242de5e78bbf") : secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 18 17:28:18.385269 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.385152    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1f555f6-f287-4856-871a-242de5e78bbf-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5f4f848c44-m2lrf\" (UID: \"a1f555f6-f287-4856-871a-242de5e78bbf\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-m2lrf"
Sep 18 17:28:18.385447 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.385430    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a1f555f6-f287-4856-871a-242de5e78bbf-cabundle-cert\") pod \"isvc-canary-promote-predictor-5f4f848c44-m2lrf\" (UID: \"a1f555f6-f287-4856-871a-242de5e78bbf\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-m2lrf"
Sep 18 17:28:18.385447 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.385441    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1f555f6-f287-4856-871a-242de5e78bbf-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5f4f848c44-m2lrf\" (UID: \"a1f555f6-f287-4856-871a-242de5e78bbf\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-m2lrf"
Sep 18 17:28:18.388190 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.388165    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-vssvp"]
Sep 18 17:28:18.389985 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.389965    2881 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7b4d9977c7-vssvp"]
Sep 18 17:28:18.400912 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.400889    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cs4j\" (UniqueName: \"kubernetes.io/projected/a1f555f6-f287-4856-871a-242de5e78bbf-kube-api-access-4cs4j\") pod \"isvc-canary-promote-predictor-5f4f848c44-m2lrf\" (UID: \"a1f555f6-f287-4856-871a-242de5e78bbf\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-m2lrf"
Sep 18 17:28:18.890867 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.890819    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1f555f6-f287-4856-871a-242de5e78bbf-proxy-tls\") pod \"isvc-canary-promote-predictor-5f4f848c44-m2lrf\" (UID: \"a1f555f6-f287-4856-871a-242de5e78bbf\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-m2lrf"
Sep 18 17:28:18.893440 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:18.893411    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1f555f6-f287-4856-871a-242de5e78bbf-proxy-tls\") pod \"isvc-canary-promote-predictor-5f4f848c44-m2lrf\" (UID: \"a1f555f6-f287-4856-871a-242de5e78bbf\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-m2lrf"
Sep 18 17:28:19.138321 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:19.138185    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-m2lrf"
Sep 18 17:28:19.435259 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:19.435151    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-m2lrf"]
Sep 18 17:28:19.439545 ip-10-0-143-5 kubenswrapper[2881]: W0918 17:28:19.439518    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1f555f6_f287_4856_871a_242de5e78bbf.slice/crio-9b46f7cb86ddbb7a29cd8cd0687dff4d8db128468ceaaff11e1efb463c1c47cd WatchSource:0}: Error finding container 9b46f7cb86ddbb7a29cd8cd0687dff4d8db128468ceaaff11e1efb463c1c47cd: Status 404 returned error can't find the container with id 9b46f7cb86ddbb7a29cd8cd0687dff4d8db128468ceaaff11e1efb463c1c47cd
Sep 18 17:28:19.893133 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:19.893099    2881 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="101e16ed-d709-4547-85c6-7daa62eb5a88" path="/var/lib/kubelet/pods/101e16ed-d709-4547-85c6-7daa62eb5a88/volumes"
Sep 18 17:28:19.893646 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:19.893632    2881 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc" path="/var/lib/kubelet/pods/bd3cb6e4-917a-4a79-afb4-2fe3ee6140bc/volumes"
Sep 18 17:28:19.971787 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:19.971743    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5c84876bbf-wmp4k" podUID="361c37a4-5f9a-4f7c-b07c-5cc502425d75" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 18 17:28:20.329722 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:20.329692    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-m2lrf" event={"ID":"a1f555f6-f287-4856-871a-242de5e78bbf","Type":"ContainerStarted","Data":"ef41fa7d32c775a325d13483b1f6314a12afaa9d89741a88e1a06a26a6497d9e"}
Sep 18 17:28:20.329916 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:20.329749    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-m2lrf" event={"ID":"a1f555f6-f287-4856-871a-242de5e78bbf","Type":"ContainerStarted","Data":"9b46f7cb86ddbb7a29cd8cd0687dff4d8db128468ceaaff11e1efb463c1c47cd"}
Sep 18 17:28:23.345338 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:23.345305    2881 generic.go:356] "Generic (PLEG): container finished" podID="a1f555f6-f287-4856-871a-242de5e78bbf" containerID="ef41fa7d32c775a325d13483b1f6314a12afaa9d89741a88e1a06a26a6497d9e" exitCode=0
Sep 18 17:28:23.345695 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:23.345355    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-m2lrf" event={"ID":"a1f555f6-f287-4856-871a-242de5e78bbf","Type":"ContainerDied","Data":"ef41fa7d32c775a325d13483b1f6314a12afaa9d89741a88e1a06a26a6497d9e"}
Sep 18 17:28:24.353038 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:24.352999    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-m2lrf" event={"ID":"a1f555f6-f287-4856-871a-242de5e78bbf","Type":"ContainerStarted","Data":"d66ce08948e6672773b99b8631006cd02d8d55a4bf1935b2831e96ebe495d0fb"}
Sep 18 17:28:24.353486 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:24.353070    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-m2lrf" event={"ID":"a1f555f6-f287-4856-871a-242de5e78bbf","Type":"ContainerStarted","Data":"455b946182f5979a899eb7de885a27ed608754f25c8e50ca98c17645c3f095ad"}
Sep 18 17:28:24.353486 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:24.353349    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-m2lrf"
Sep 18 17:28:24.353486 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:24.353399    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-m2lrf"
Sep 18 17:28:24.354792 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:24.354767    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-m2lrf" podUID="a1f555f6-f287-4856-871a-242de5e78bbf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 18 17:28:24.377297 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:24.377245    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-m2lrf" podStartSLOduration=2.6869886640000002 podStartE2EDuration="6.377210983s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.690222319s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:28:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:28:24.376353202 +0000 UTC m=+2094.576208488" watchObservedRunningTime="2026-09-18 17:28:24.377210983 +0000 UTC m=+2094.577066270"
Sep 18 17:28:25.357669 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:25.357621    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-m2lrf" podUID="a1f555f6-f287-4856-871a-242de5e78bbf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 18 17:28:29.972398 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:29.972363    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5c84876bbf-wmp4k"
Sep 18 17:28:30.362355 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:30.362326    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-m2lrf"
Sep 18 17:28:30.362964 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:30.362940    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-m2lrf" podUID="a1f555f6-f287-4856-871a-242de5e78bbf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 18 17:28:38.135628 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:38.135593    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5c84876bbf-wmp4k"]
Sep 18 17:28:38.136153 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:38.135923    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5c84876bbf-wmp4k" podUID="361c37a4-5f9a-4f7c-b07c-5cc502425d75" containerName="kserve-container" containerID="cri-o://4341e6c2b0f8ef05f8c404c662a79bada19de91248c84499856ae8d3539472db" gracePeriod=30
Sep 18 17:28:38.136153 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:38.136009    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5c84876bbf-wmp4k" podUID="361c37a4-5f9a-4f7c-b07c-5cc502425d75" containerName="kube-rbac-proxy" containerID="cri-o://41e8b6bb7a30589215b54ac3e1bb2ba7851a8b928fefa09f3eb97c94a76e2d4f" gracePeriod=30
Sep 18 17:28:38.416296 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:38.416188    2881 generic.go:356] "Generic (PLEG): container finished" podID="361c37a4-5f9a-4f7c-b07c-5cc502425d75" containerID="41e8b6bb7a30589215b54ac3e1bb2ba7851a8b928fefa09f3eb97c94a76e2d4f" exitCode=2
Sep 18 17:28:38.416296 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:38.416250    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5c84876bbf-wmp4k" event={"ID":"361c37a4-5f9a-4f7c-b07c-5cc502425d75","Type":"ContainerDied","Data":"41e8b6bb7a30589215b54ac3e1bb2ba7851a8b928fefa09f3eb97c94a76e2d4f"}
Sep 18 17:28:39.966789 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:39.966742    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5c84876bbf-wmp4k" podUID="361c37a4-5f9a-4f7c-b07c-5cc502425d75" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.42:8643/healthz\": dial tcp 10.134.0.42:8643: connect: connection refused"
Sep 18 17:28:39.971149 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:39.971118    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5c84876bbf-wmp4k" podUID="361c37a4-5f9a-4f7c-b07c-5cc502425d75" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 18 17:28:40.363193 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:40.363157    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-m2lrf" podUID="a1f555f6-f287-4856-871a-242de5e78bbf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 18 17:28:43.274628 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:43.274586    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-767cd75768-dvqsl"]
Sep 18 17:28:43.278289 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:43.278268    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-767cd75768-dvqsl"
Sep 18 17:28:43.280966 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:43.280943    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-predictor-serving-cert\""
Sep 18 17:28:43.281084 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:43.280943    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\""
Sep 18 17:28:43.287703 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:43.287675    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-767cd75768-dvqsl"]
Sep 18 17:28:43.410212 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:43.410170    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7018e6f0-2468-4a6d-a6ec-8c7943569ea3-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-767cd75768-dvqsl\" (UID: \"7018e6f0-2468-4a6d-a6ec-8c7943569ea3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-767cd75768-dvqsl"
Sep 18 17:28:43.410212 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:43.410215    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7018e6f0-2468-4a6d-a6ec-8c7943569ea3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-767cd75768-dvqsl\" (UID: \"7018e6f0-2468-4a6d-a6ec-8c7943569ea3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-767cd75768-dvqsl"
Sep 18 17:28:43.410456 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:43.410294    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qkp9\" (UniqueName: \"kubernetes.io/projected/7018e6f0-2468-4a6d-a6ec-8c7943569ea3-kube-api-access-5qkp9\") pod \"isvc-sklearn-v2-mixed-predictor-767cd75768-dvqsl\" (UID: \"7018e6f0-2468-4a6d-a6ec-8c7943569ea3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-767cd75768-dvqsl"
Sep 18 17:28:43.410456 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:43.410393    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7018e6f0-2468-4a6d-a6ec-8c7943569ea3-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-767cd75768-dvqsl\" (UID: \"7018e6f0-2468-4a6d-a6ec-8c7943569ea3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-767cd75768-dvqsl"
Sep 18 17:28:43.410456 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:43.410418    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7018e6f0-2468-4a6d-a6ec-8c7943569ea3-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-767cd75768-dvqsl\" (UID: \"7018e6f0-2468-4a6d-a6ec-8c7943569ea3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-767cd75768-dvqsl"
Sep 18 17:28:43.511830 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:43.511799    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7018e6f0-2468-4a6d-a6ec-8c7943569ea3-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-767cd75768-dvqsl\" (UID: \"7018e6f0-2468-4a6d-a6ec-8c7943569ea3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-767cd75768-dvqsl"
Sep 18 17:28:43.511999 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:43.511843    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7018e6f0-2468-4a6d-a6ec-8c7943569ea3-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-767cd75768-dvqsl\" (UID: \"7018e6f0-2468-4a6d-a6ec-8c7943569ea3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-767cd75768-dvqsl"
Sep 18 17:28:43.511999 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:43.511871    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7018e6f0-2468-4a6d-a6ec-8c7943569ea3-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-767cd75768-dvqsl\" (UID: \"7018e6f0-2468-4a6d-a6ec-8c7943569ea3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-767cd75768-dvqsl"
Sep 18 17:28:43.511999 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:43.511911    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7018e6f0-2468-4a6d-a6ec-8c7943569ea3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-767cd75768-dvqsl\" (UID: \"7018e6f0-2468-4a6d-a6ec-8c7943569ea3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-767cd75768-dvqsl"
Sep 18 17:28:43.511999 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:43.511941    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qkp9\" (UniqueName: \"kubernetes.io/projected/7018e6f0-2468-4a6d-a6ec-8c7943569ea3-kube-api-access-5qkp9\") pod \"isvc-sklearn-v2-mixed-predictor-767cd75768-dvqsl\" (UID: \"7018e6f0-2468-4a6d-a6ec-8c7943569ea3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-767cd75768-dvqsl"
Sep 18 17:28:43.512358 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:43.512331    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7018e6f0-2468-4a6d-a6ec-8c7943569ea3-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-767cd75768-dvqsl\" (UID: \"7018e6f0-2468-4a6d-a6ec-8c7943569ea3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-767cd75768-dvqsl"
Sep 18 17:28:43.512648 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:43.512629    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7018e6f0-2468-4a6d-a6ec-8c7943569ea3-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-767cd75768-dvqsl\" (UID: \"7018e6f0-2468-4a6d-a6ec-8c7943569ea3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-767cd75768-dvqsl"
Sep 18 17:28:43.512754 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:43.512735    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7018e6f0-2468-4a6d-a6ec-8c7943569ea3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-767cd75768-dvqsl\" (UID: \"7018e6f0-2468-4a6d-a6ec-8c7943569ea3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-767cd75768-dvqsl"
Sep 18 17:28:43.514417 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:43.514398    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7018e6f0-2468-4a6d-a6ec-8c7943569ea3-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-767cd75768-dvqsl\" (UID: \"7018e6f0-2468-4a6d-a6ec-8c7943569ea3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-767cd75768-dvqsl"
Sep 18 17:28:43.521362 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:43.521338    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qkp9\" (UniqueName: \"kubernetes.io/projected/7018e6f0-2468-4a6d-a6ec-8c7943569ea3-kube-api-access-5qkp9\") pod \"isvc-sklearn-v2-mixed-predictor-767cd75768-dvqsl\" (UID: \"7018e6f0-2468-4a6d-a6ec-8c7943569ea3\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-767cd75768-dvqsl"
Sep 18 17:28:43.588344 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:43.588310    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5c84876bbf-wmp4k"
Sep 18 17:28:43.588589 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:43.588569    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-767cd75768-dvqsl"
Sep 18 17:28:43.713439 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:43.713397    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/361c37a4-5f9a-4f7c-b07c-5cc502425d75-kserve-provision-location\") pod \"361c37a4-5f9a-4f7c-b07c-5cc502425d75\" (UID: \"361c37a4-5f9a-4f7c-b07c-5cc502425d75\") "
Sep 18 17:28:43.713641 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:43.713494    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/361c37a4-5f9a-4f7c-b07c-5cc502425d75-cabundle-cert\") pod \"361c37a4-5f9a-4f7c-b07c-5cc502425d75\" (UID: \"361c37a4-5f9a-4f7c-b07c-5cc502425d75\") "
Sep 18 17:28:43.713641 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:43.713513    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-94t45\" (UniqueName: \"kubernetes.io/projected/361c37a4-5f9a-4f7c-b07c-5cc502425d75-kube-api-access-94t45\") pod \"361c37a4-5f9a-4f7c-b07c-5cc502425d75\" (UID: \"361c37a4-5f9a-4f7c-b07c-5cc502425d75\") "
Sep 18 17:28:43.713641 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:43.713537    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/361c37a4-5f9a-4f7c-b07c-5cc502425d75-kube-rbac-proxy-sar-config\") pod \"361c37a4-5f9a-4f7c-b07c-5cc502425d75\" (UID: \"361c37a4-5f9a-4f7c-b07c-5cc502425d75\") "
Sep 18 17:28:43.713825 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:43.713674    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/361c37a4-5f9a-4f7c-b07c-5cc502425d75-proxy-tls\") pod \"361c37a4-5f9a-4f7c-b07c-5cc502425d75\" (UID: \"361c37a4-5f9a-4f7c-b07c-5cc502425d75\") "
Sep 18 17:28:43.713825 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:43.713779    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/361c37a4-5f9a-4f7c-b07c-5cc502425d75-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "361c37a4-5f9a-4f7c-b07c-5cc502425d75" (UID: "361c37a4-5f9a-4f7c-b07c-5cc502425d75"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:28:43.713948 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:43.713921    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/361c37a4-5f9a-4f7c-b07c-5cc502425d75-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "361c37a4-5f9a-4f7c-b07c-5cc502425d75" (UID: "361c37a4-5f9a-4f7c-b07c-5cc502425d75"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:28:43.713998 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:43.713928    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/361c37a4-5f9a-4f7c-b07c-5cc502425d75-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "361c37a4-5f9a-4f7c-b07c-5cc502425d75" (UID: "361c37a4-5f9a-4f7c-b07c-5cc502425d75"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:28:43.714041 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:43.714002    2881 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/361c37a4-5f9a-4f7c-b07c-5cc502425d75-cabundle-cert\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:28:43.714436 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:43.714413    2881 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/361c37a4-5f9a-4f7c-b07c-5cc502425d75-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:28:43.714535 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:43.714444    2881 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/361c37a4-5f9a-4f7c-b07c-5cc502425d75-kserve-provision-location\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:28:43.720729 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:43.716194    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/361c37a4-5f9a-4f7c-b07c-5cc502425d75-kube-api-access-94t45" (OuterVolumeSpecName: "kube-api-access-94t45") pod "361c37a4-5f9a-4f7c-b07c-5cc502425d75" (UID: "361c37a4-5f9a-4f7c-b07c-5cc502425d75"). InnerVolumeSpecName "kube-api-access-94t45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:28:43.720851 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:43.720742    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/361c37a4-5f9a-4f7c-b07c-5cc502425d75-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "361c37a4-5f9a-4f7c-b07c-5cc502425d75" (UID: "361c37a4-5f9a-4f7c-b07c-5cc502425d75"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:28:43.815591 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:43.815554    2881 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94t45\" (UniqueName: \"kubernetes.io/projected/361c37a4-5f9a-4f7c-b07c-5cc502425d75-kube-api-access-94t45\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:28:43.815591 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:43.815591    2881 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/361c37a4-5f9a-4f7c-b07c-5cc502425d75-proxy-tls\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:28:43.863456 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:43.863433    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-767cd75768-dvqsl"]
Sep 18 17:28:43.866066 ip-10-0-143-5 kubenswrapper[2881]: W0918 17:28:43.866029    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7018e6f0_2468_4a6d_a6ec_8c7943569ea3.slice/crio-3473f853fc699c470fbe03781637fb4d46f67c20d01364be8a3fd37754233ffe WatchSource:0}: Error finding container 3473f853fc699c470fbe03781637fb4d46f67c20d01364be8a3fd37754233ffe: Status 404 returned error can't find the container with id 3473f853fc699c470fbe03781637fb4d46f67c20d01364be8a3fd37754233ffe
Sep 18 17:28:44.448374 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:44.448337    2881 generic.go:356] "Generic (PLEG): container finished" podID="361c37a4-5f9a-4f7c-b07c-5cc502425d75" containerID="4341e6c2b0f8ef05f8c404c662a79bada19de91248c84499856ae8d3539472db" exitCode=0
Sep 18 17:28:44.448768 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:44.448445    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5c84876bbf-wmp4k" event={"ID":"361c37a4-5f9a-4f7c-b07c-5cc502425d75","Type":"ContainerDied","Data":"4341e6c2b0f8ef05f8c404c662a79bada19de91248c84499856ae8d3539472db"}
Sep 18 17:28:44.448768 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:44.448456    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5c84876bbf-wmp4k"
Sep 18 17:28:44.448768 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:44.448516    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5c84876bbf-wmp4k" event={"ID":"361c37a4-5f9a-4f7c-b07c-5cc502425d75","Type":"ContainerDied","Data":"9be433b20eb86106f429c05b0c464620425423046adc65c508eee0acfe97c53b"}
Sep 18 17:28:44.448768 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:44.448559    2881 scope.go:117] "RemoveContainer" containerID="41e8b6bb7a30589215b54ac3e1bb2ba7851a8b928fefa09f3eb97c94a76e2d4f"
Sep 18 17:28:44.450292 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:44.450261    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-767cd75768-dvqsl" event={"ID":"7018e6f0-2468-4a6d-a6ec-8c7943569ea3","Type":"ContainerStarted","Data":"ae39eefa95e1bcfbf4051719560aa2709407a15e749e301281f2c57209187ec8"}
Sep 18 17:28:44.450434 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:44.450320    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-767cd75768-dvqsl" event={"ID":"7018e6f0-2468-4a6d-a6ec-8c7943569ea3","Type":"ContainerStarted","Data":"3473f853fc699c470fbe03781637fb4d46f67c20d01364be8a3fd37754233ffe"}
Sep 18 17:28:44.459265 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:44.459035    2881 scope.go:117] "RemoveContainer" containerID="4341e6c2b0f8ef05f8c404c662a79bada19de91248c84499856ae8d3539472db"
Sep 18 17:28:44.467169 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:44.467144    2881 scope.go:117] "RemoveContainer" containerID="9d30149f06afdc066e27a502abb013d32bb98922bab9b3e556de0275a10d134c"
Sep 18 17:28:44.470552 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:44.470525    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5c84876bbf-wmp4k"]
Sep 18 17:28:44.474809 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:44.474784    2881 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5c84876bbf-wmp4k"]
Sep 18 17:28:44.478389 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:44.478366    2881 scope.go:117] "RemoveContainer" containerID="41e8b6bb7a30589215b54ac3e1bb2ba7851a8b928fefa09f3eb97c94a76e2d4f"
Sep 18 17:28:44.478735 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:28:44.478711    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41e8b6bb7a30589215b54ac3e1bb2ba7851a8b928fefa09f3eb97c94a76e2d4f\": container with ID starting with 41e8b6bb7a30589215b54ac3e1bb2ba7851a8b928fefa09f3eb97c94a76e2d4f not found: ID does not exist" containerID="41e8b6bb7a30589215b54ac3e1bb2ba7851a8b928fefa09f3eb97c94a76e2d4f"
Sep 18 17:28:44.478809 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:44.478746    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41e8b6bb7a30589215b54ac3e1bb2ba7851a8b928fefa09f3eb97c94a76e2d4f"} err="failed to get container status \"41e8b6bb7a30589215b54ac3e1bb2ba7851a8b928fefa09f3eb97c94a76e2d4f\": rpc error: code = NotFound desc = could not find container \"41e8b6bb7a30589215b54ac3e1bb2ba7851a8b928fefa09f3eb97c94a76e2d4f\": container with ID starting with 41e8b6bb7a30589215b54ac3e1bb2ba7851a8b928fefa09f3eb97c94a76e2d4f not found: ID does not exist"
Sep 18 17:28:44.478809 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:44.478765    2881 scope.go:117] "RemoveContainer" containerID="4341e6c2b0f8ef05f8c404c662a79bada19de91248c84499856ae8d3539472db"
Sep 18 17:28:44.479084 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:28:44.479066    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4341e6c2b0f8ef05f8c404c662a79bada19de91248c84499856ae8d3539472db\": container with ID starting with 4341e6c2b0f8ef05f8c404c662a79bada19de91248c84499856ae8d3539472db not found: ID does not exist" containerID="4341e6c2b0f8ef05f8c404c662a79bada19de91248c84499856ae8d3539472db"
Sep 18 17:28:44.479125 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:44.479091    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4341e6c2b0f8ef05f8c404c662a79bada19de91248c84499856ae8d3539472db"} err="failed to get container status \"4341e6c2b0f8ef05f8c404c662a79bada19de91248c84499856ae8d3539472db\": rpc error: code = NotFound desc = could not find container \"4341e6c2b0f8ef05f8c404c662a79bada19de91248c84499856ae8d3539472db\": container with ID starting with 4341e6c2b0f8ef05f8c404c662a79bada19de91248c84499856ae8d3539472db not found: ID does not exist"
Sep 18 17:28:44.479125 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:44.479106    2881 scope.go:117] "RemoveContainer" containerID="9d30149f06afdc066e27a502abb013d32bb98922bab9b3e556de0275a10d134c"
Sep 18 17:28:44.479372 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:28:44.479353    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d30149f06afdc066e27a502abb013d32bb98922bab9b3e556de0275a10d134c\": container with ID starting with 9d30149f06afdc066e27a502abb013d32bb98922bab9b3e556de0275a10d134c not found: ID does not exist" containerID="9d30149f06afdc066e27a502abb013d32bb98922bab9b3e556de0275a10d134c"
Sep 18 17:28:44.479431 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:44.479375    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d30149f06afdc066e27a502abb013d32bb98922bab9b3e556de0275a10d134c"} err="failed to get container status \"9d30149f06afdc066e27a502abb013d32bb98922bab9b3e556de0275a10d134c\": rpc error: code = NotFound desc = could not find container \"9d30149f06afdc066e27a502abb013d32bb98922bab9b3e556de0275a10d134c\": container with ID starting with 9d30149f06afdc066e27a502abb013d32bb98922bab9b3e556de0275a10d134c not found: ID does not exist"
Sep 18 17:28:45.892897 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:45.892865    2881 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="361c37a4-5f9a-4f7c-b07c-5cc502425d75" path="/var/lib/kubelet/pods/361c37a4-5f9a-4f7c-b07c-5cc502425d75/volumes"
Sep 18 17:28:48.471751 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:48.471721    2881 generic.go:356] "Generic (PLEG): container finished" podID="7018e6f0-2468-4a6d-a6ec-8c7943569ea3" containerID="ae39eefa95e1bcfbf4051719560aa2709407a15e749e301281f2c57209187ec8" exitCode=0
Sep 18 17:28:48.472165 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:48.471809    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-767cd75768-dvqsl" event={"ID":"7018e6f0-2468-4a6d-a6ec-8c7943569ea3","Type":"ContainerDied","Data":"ae39eefa95e1bcfbf4051719560aa2709407a15e749e301281f2c57209187ec8"}
Sep 18 17:28:49.478433 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:49.478396    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-767cd75768-dvqsl" event={"ID":"7018e6f0-2468-4a6d-a6ec-8c7943569ea3","Type":"ContainerStarted","Data":"f25dff21000dc167ab6fe7a565c457432960ab5c42c06b64257d8ed9e42a93f0"}
Sep 18 17:28:49.478833 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:49.478494    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-767cd75768-dvqsl" event={"ID":"7018e6f0-2468-4a6d-a6ec-8c7943569ea3","Type":"ContainerStarted","Data":"b89b0253412bd9cad1ea2e736144ffdb6b463ef1c4b37e5d36c97bdcb5dbfe2a"}
Sep 18 17:28:49.478833 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:49.478687    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-767cd75768-dvqsl"
Sep 18 17:28:49.504338 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:49.504282    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-767cd75768-dvqsl" podStartSLOduration=2.772746515 podStartE2EDuration="6.504261533s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.731515018s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:28:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:28:49.502159752 +0000 UTC m=+2119.702015038" watchObservedRunningTime="2026-09-18 17:28:49.504261533 +0000 UTC m=+2119.704116819"
Sep 18 17:28:50.363372 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:50.363328    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-m2lrf" podUID="a1f555f6-f287-4856-871a-242de5e78bbf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 18 17:28:50.482530 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:50.482494    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-767cd75768-dvqsl"
Sep 18 17:28:50.484203 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:50.484172    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-767cd75768-dvqsl" podUID="7018e6f0-2468-4a6d-a6ec-8c7943569ea3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 18 17:28:51.487331 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:51.487296    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-767cd75768-dvqsl" podUID="7018e6f0-2468-4a6d-a6ec-8c7943569ea3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 18 17:28:56.491996 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:56.491963    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-767cd75768-dvqsl"
Sep 18 17:28:56.492475 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:28:56.492440    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-767cd75768-dvqsl" podUID="7018e6f0-2468-4a6d-a6ec-8c7943569ea3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 18 17:29:00.363828 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:29:00.363792    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-m2lrf" podUID="a1f555f6-f287-4856-871a-242de5e78bbf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 18 17:29:06.493218 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:29:06.493176    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-767cd75768-dvqsl" podUID="7018e6f0-2468-4a6d-a6ec-8c7943569ea3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 18 17:29:10.363520 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:29:10.363480    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-m2lrf" podUID="a1f555f6-f287-4856-871a-242de5e78bbf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 18 17:29:16.492681 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:29:16.492635    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-767cd75768-dvqsl" podUID="7018e6f0-2468-4a6d-a6ec-8c7943569ea3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 18 17:29:20.363645 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:29:20.363606    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-m2lrf" podUID="a1f555f6-f287-4856-871a-242de5e78bbf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 18 17:29:26.492941 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:29:26.492892    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-767cd75768-dvqsl" podUID="7018e6f0-2468-4a6d-a6ec-8c7943569ea3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 18 17:29:30.363196 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:29:30.363155    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-m2lrf" podUID="a1f555f6-f287-4856-871a-242de5e78bbf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 18 17:29:36.492896 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:29:36.492854    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-767cd75768-dvqsl" podUID="7018e6f0-2468-4a6d-a6ec-8c7943569ea3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 18 17:29:40.363666 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:29:40.363631    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-m2lrf" podUID="a1f555f6-f287-4856-871a-242de5e78bbf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 18 17:29:46.492549 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:29:46.492511    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-767cd75768-dvqsl" podUID="7018e6f0-2468-4a6d-a6ec-8c7943569ea3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 18 17:29:50.363391 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:29:50.363360    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-m2lrf"
Sep 18 17:29:56.492696 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:29:56.492654    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-767cd75768-dvqsl" podUID="7018e6f0-2468-4a6d-a6ec-8c7943569ea3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 18 17:29:57.870912 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:29:57.870872    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-m2lrf"]
Sep 18 17:29:57.871328 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:29:57.871120    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-m2lrf" podUID="a1f555f6-f287-4856-871a-242de5e78bbf" containerName="kserve-container" containerID="cri-o://455b946182f5979a899eb7de885a27ed608754f25c8e50ca98c17645c3f095ad" gracePeriod=30
Sep 18 17:29:57.871328 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:29:57.871133    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-m2lrf" podUID="a1f555f6-f287-4856-871a-242de5e78bbf" containerName="kube-rbac-proxy" containerID="cri-o://d66ce08948e6672773b99b8631006cd02d8d55a4bf1935b2831e96ebe495d0fb" gracePeriod=30
Sep 18 17:29:57.919826 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:29:57.919786    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-9kw4c"]
Sep 18 17:29:57.920146 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:29:57.920134    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="361c37a4-5f9a-4f7c-b07c-5cc502425d75" containerName="kube-rbac-proxy"
Sep 18 17:29:57.920191 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:29:57.920148    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="361c37a4-5f9a-4f7c-b07c-5cc502425d75" containerName="kube-rbac-proxy"
Sep 18 17:29:57.920191 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:29:57.920171    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="361c37a4-5f9a-4f7c-b07c-5cc502425d75" containerName="kserve-container"
Sep 18 17:29:57.920191 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:29:57.920177    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="361c37a4-5f9a-4f7c-b07c-5cc502425d75" containerName="kserve-container"
Sep 18 17:29:57.920191 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:29:57.920184    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="361c37a4-5f9a-4f7c-b07c-5cc502425d75" containerName="storage-initializer"
Sep 18 17:29:57.920191 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:29:57.920189    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="361c37a4-5f9a-4f7c-b07c-5cc502425d75" containerName="storage-initializer"
Sep 18 17:29:57.920393 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:29:57.920262    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="361c37a4-5f9a-4f7c-b07c-5cc502425d75" containerName="kserve-container"
Sep 18 17:29:57.920393 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:29:57.920273    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="361c37a4-5f9a-4f7c-b07c-5cc502425d75" containerName="kube-rbac-proxy"
Sep 18 17:29:57.923449 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:29:57.923433    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-9kw4c"
Sep 18 17:29:57.925901 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:29:57.925873    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 18 17:29:57.935105 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:29:57.935078    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-9kw4c"]
Sep 18 17:29:57.967492 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:29:57.967457    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d22cfc6-e80f-427a-aa7f-68fa0e780253-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6f7fd74c59-9kw4c\" (UID: \"4d22cfc6-e80f-427a-aa7f-68fa0e780253\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-9kw4c"
Sep 18 17:29:57.967492 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:29:57.967491    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d22cfc6-e80f-427a-aa7f-68fa0e780253-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6f7fd74c59-9kw4c\" (UID: \"4d22cfc6-e80f-427a-aa7f-68fa0e780253\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-9kw4c"
Sep 18 17:29:57.967724 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:29:57.967513    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp76z\" (UniqueName: \"kubernetes.io/projected/4d22cfc6-e80f-427a-aa7f-68fa0e780253-kube-api-access-rp76z\") pod \"isvc-canary-promote-v2-predictor-6f7fd74c59-9kw4c\" (UID: \"4d22cfc6-e80f-427a-aa7f-68fa0e780253\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-9kw4c"
Sep 18 17:29:57.967724 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:29:57.967614    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d22cfc6-e80f-427a-aa7f-68fa0e780253-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6f7fd74c59-9kw4c\" (UID: \"4d22cfc6-e80f-427a-aa7f-68fa0e780253\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-9kw4c"
Sep 18 17:29:57.967724 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:29:57.967678    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d22cfc6-e80f-427a-aa7f-68fa0e780253-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6f7fd74c59-9kw4c\" (UID: \"4d22cfc6-e80f-427a-aa7f-68fa0e780253\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-9kw4c"
Sep 18 17:29:58.068497 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:29:58.068466    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d22cfc6-e80f-427a-aa7f-68fa0e780253-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6f7fd74c59-9kw4c\" (UID: \"4d22cfc6-e80f-427a-aa7f-68fa0e780253\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-9kw4c"
Sep 18 17:29:58.068696 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:29:58.068503    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d22cfc6-e80f-427a-aa7f-68fa0e780253-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6f7fd74c59-9kw4c\" (UID: \"4d22cfc6-e80f-427a-aa7f-68fa0e780253\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-9kw4c"
Sep 18 17:29:58.068696 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:29:58.068523    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp76z\" (UniqueName: \"kubernetes.io/projected/4d22cfc6-e80f-427a-aa7f-68fa0e780253-kube-api-access-rp76z\") pod \"isvc-canary-promote-v2-predictor-6f7fd74c59-9kw4c\" (UID: \"4d22cfc6-e80f-427a-aa7f-68fa0e780253\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-9kw4c"
Sep 18 17:29:58.068696 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:29:58.068565    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d22cfc6-e80f-427a-aa7f-68fa0e780253-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6f7fd74c59-9kw4c\" (UID: \"4d22cfc6-e80f-427a-aa7f-68fa0e780253\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-9kw4c"
Sep 18 17:29:58.068696 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:29:58.068615    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d22cfc6-e80f-427a-aa7f-68fa0e780253-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6f7fd74c59-9kw4c\" (UID: \"4d22cfc6-e80f-427a-aa7f-68fa0e780253\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-9kw4c"
Sep 18 17:29:58.069047 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:29:58.069019    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d22cfc6-e80f-427a-aa7f-68fa0e780253-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6f7fd74c59-9kw4c\" (UID: \"4d22cfc6-e80f-427a-aa7f-68fa0e780253\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-9kw4c"
Sep 18 17:29:58.069199 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:29:58.069174    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d22cfc6-e80f-427a-aa7f-68fa0e780253-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6f7fd74c59-9kw4c\" (UID: \"4d22cfc6-e80f-427a-aa7f-68fa0e780253\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-9kw4c"
Sep 18 17:29:58.069297 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:29:58.069252    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d22cfc6-e80f-427a-aa7f-68fa0e780253-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6f7fd74c59-9kw4c\" (UID: \"4d22cfc6-e80f-427a-aa7f-68fa0e780253\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-9kw4c"
Sep 18 17:29:58.071057 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:29:58.071024    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d22cfc6-e80f-427a-aa7f-68fa0e780253-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6f7fd74c59-9kw4c\" (UID: \"4d22cfc6-e80f-427a-aa7f-68fa0e780253\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-9kw4c"
Sep 18 17:29:58.078208 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:29:58.078185    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp76z\" (UniqueName: \"kubernetes.io/projected/4d22cfc6-e80f-427a-aa7f-68fa0e780253-kube-api-access-rp76z\") pod \"isvc-canary-promote-v2-predictor-6f7fd74c59-9kw4c\" (UID: \"4d22cfc6-e80f-427a-aa7f-68fa0e780253\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-9kw4c"
Sep 18 17:29:58.234170 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:29:58.234072    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-9kw4c"
Sep 18 17:29:58.531683 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:29:58.531653    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-9kw4c"]
Sep 18 17:29:58.532921 ip-10-0-143-5 kubenswrapper[2881]: W0918 17:29:58.532891    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d22cfc6_e80f_427a_aa7f_68fa0e780253.slice/crio-acbf16a812ca61f1be5ee49c598fa6c83297be212ddc625164fec52fe284b48d WatchSource:0}: Error finding container acbf16a812ca61f1be5ee49c598fa6c83297be212ddc625164fec52fe284b48d: Status 404 returned error can't find the container with id acbf16a812ca61f1be5ee49c598fa6c83297be212ddc625164fec52fe284b48d
Sep 18 17:29:58.788276 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:29:58.788158    2881 generic.go:356] "Generic (PLEG): container finished" podID="a1f555f6-f287-4856-871a-242de5e78bbf" containerID="d66ce08948e6672773b99b8631006cd02d8d55a4bf1935b2831e96ebe495d0fb" exitCode=2
Sep 18 17:29:58.788276 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:29:58.788238    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-m2lrf" event={"ID":"a1f555f6-f287-4856-871a-242de5e78bbf","Type":"ContainerDied","Data":"d66ce08948e6672773b99b8631006cd02d8d55a4bf1935b2831e96ebe495d0fb"}
Sep 18 17:29:58.790053 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:29:58.790028    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-9kw4c" event={"ID":"4d22cfc6-e80f-427a-aa7f-68fa0e780253","Type":"ContainerStarted","Data":"65cf5a34a7389feb827ad725272b0a757789cdb04b712adc83be5f86b5f8ce14"}
Sep 18 17:29:58.790202 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:29:58.790081    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-9kw4c" event={"ID":"4d22cfc6-e80f-427a-aa7f-68fa0e780253","Type":"ContainerStarted","Data":"acbf16a812ca61f1be5ee49c598fa6c83297be212ddc625164fec52fe284b48d"}
Sep 18 17:30:00.358480 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:00.358388    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-m2lrf" podUID="a1f555f6-f287-4856-871a-242de5e78bbf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Sep 18 17:30:00.362949 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:00.362914    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-m2lrf" podUID="a1f555f6-f287-4856-871a-242de5e78bbf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 18 17:30:02.809845 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:02.809809    2881 generic.go:356] "Generic (PLEG): container finished" podID="4d22cfc6-e80f-427a-aa7f-68fa0e780253" containerID="65cf5a34a7389feb827ad725272b0a757789cdb04b712adc83be5f86b5f8ce14" exitCode=0
Sep 18 17:30:02.810262 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:02.809871    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-9kw4c" event={"ID":"4d22cfc6-e80f-427a-aa7f-68fa0e780253","Type":"ContainerDied","Data":"65cf5a34a7389feb827ad725272b0a757789cdb04b712adc83be5f86b5f8ce14"}
Sep 18 17:30:03.206500 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:03.206472    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-m2lrf"
Sep 18 17:30:03.314410 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:03.314375    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4cs4j\" (UniqueName: \"kubernetes.io/projected/a1f555f6-f287-4856-871a-242de5e78bbf-kube-api-access-4cs4j\") pod \"a1f555f6-f287-4856-871a-242de5e78bbf\" (UID: \"a1f555f6-f287-4856-871a-242de5e78bbf\") "
Sep 18 17:30:03.314598 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:03.314431    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a1f555f6-f287-4856-871a-242de5e78bbf-cabundle-cert\") pod \"a1f555f6-f287-4856-871a-242de5e78bbf\" (UID: \"a1f555f6-f287-4856-871a-242de5e78bbf\") "
Sep 18 17:30:03.314598 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:03.314470    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1f555f6-f287-4856-871a-242de5e78bbf-kserve-provision-location\") pod \"a1f555f6-f287-4856-871a-242de5e78bbf\" (UID: \"a1f555f6-f287-4856-871a-242de5e78bbf\") "
Sep 18 17:30:03.314598 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:03.314574    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1f555f6-f287-4856-871a-242de5e78bbf-proxy-tls\") pod \"a1f555f6-f287-4856-871a-242de5e78bbf\" (UID: \"a1f555f6-f287-4856-871a-242de5e78bbf\") "
Sep 18 17:30:03.314748 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:03.314629    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1f555f6-f287-4856-871a-242de5e78bbf-kube-rbac-proxy-sar-config\") pod \"a1f555f6-f287-4856-871a-242de5e78bbf\" (UID: \"a1f555f6-f287-4856-871a-242de5e78bbf\") "
Sep 18 17:30:03.314812 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:03.314763    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1f555f6-f287-4856-871a-242de5e78bbf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a1f555f6-f287-4856-871a-242de5e78bbf" (UID: "a1f555f6-f287-4856-871a-242de5e78bbf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:30:03.314894 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:03.314871    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1f555f6-f287-4856-871a-242de5e78bbf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a1f555f6-f287-4856-871a-242de5e78bbf" (UID: "a1f555f6-f287-4856-871a-242de5e78bbf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:30:03.315058 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:03.315037    2881 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a1f555f6-f287-4856-871a-242de5e78bbf-cabundle-cert\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:30:03.315058 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:03.315054    2881 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1f555f6-f287-4856-871a-242de5e78bbf-kserve-provision-location\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:30:03.315179 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:03.315055    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1f555f6-f287-4856-871a-242de5e78bbf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a1f555f6-f287-4856-871a-242de5e78bbf" (UID: "a1f555f6-f287-4856-871a-242de5e78bbf"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:30:03.316622 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:03.316601    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1f555f6-f287-4856-871a-242de5e78bbf-kube-api-access-4cs4j" (OuterVolumeSpecName: "kube-api-access-4cs4j") pod "a1f555f6-f287-4856-871a-242de5e78bbf" (UID: "a1f555f6-f287-4856-871a-242de5e78bbf"). InnerVolumeSpecName "kube-api-access-4cs4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:30:03.316666 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:03.316635    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1f555f6-f287-4856-871a-242de5e78bbf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a1f555f6-f287-4856-871a-242de5e78bbf" (UID: "a1f555f6-f287-4856-871a-242de5e78bbf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:30:03.416297 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:03.416186    2881 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1f555f6-f287-4856-871a-242de5e78bbf-proxy-tls\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:30:03.416297 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:03.416239    2881 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1f555f6-f287-4856-871a-242de5e78bbf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:30:03.416297 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:03.416252    2881 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cs4j\" (UniqueName: \"kubernetes.io/projected/a1f555f6-f287-4856-871a-242de5e78bbf-kube-api-access-4cs4j\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:30:03.817302 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:03.817261    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-9kw4c" event={"ID":"4d22cfc6-e80f-427a-aa7f-68fa0e780253","Type":"ContainerStarted","Data":"fbdadead941bee1afa6df133862d9beb683a4387812f160e8591100863ec8bc6"}
Sep 18 17:30:03.817785 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:03.817338    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-9kw4c" event={"ID":"4d22cfc6-e80f-427a-aa7f-68fa0e780253","Type":"ContainerStarted","Data":"fa50687c00c25988502caaf76eef2ba394c28140ce9c29c6b4f17854de17d938"}
Sep 18 17:30:03.817785 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:03.817565    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-9kw4c"
Sep 18 17:30:03.819277 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:03.819251    2881 generic.go:356] "Generic (PLEG): container finished" podID="a1f555f6-f287-4856-871a-242de5e78bbf" containerID="455b946182f5979a899eb7de885a27ed608754f25c8e50ca98c17645c3f095ad" exitCode=0
Sep 18 17:30:03.819428 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:03.819288    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-m2lrf" event={"ID":"a1f555f6-f287-4856-871a-242de5e78bbf","Type":"ContainerDied","Data":"455b946182f5979a899eb7de885a27ed608754f25c8e50ca98c17645c3f095ad"}
Sep 18 17:30:03.819428 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:03.819307    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-m2lrf"
Sep 18 17:30:03.819428 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:03.819330    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-m2lrf" event={"ID":"a1f555f6-f287-4856-871a-242de5e78bbf","Type":"ContainerDied","Data":"9b46f7cb86ddbb7a29cd8cd0687dff4d8db128468ceaaff11e1efb463c1c47cd"}
Sep 18 17:30:03.819428 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:03.819356    2881 scope.go:117] "RemoveContainer" containerID="d66ce08948e6672773b99b8631006cd02d8d55a4bf1935b2831e96ebe495d0fb"
Sep 18 17:30:03.828337 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:03.828319    2881 scope.go:117] "RemoveContainer" containerID="455b946182f5979a899eb7de885a27ed608754f25c8e50ca98c17645c3f095ad"
Sep 18 17:30:03.838053 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:03.838029    2881 scope.go:117] "RemoveContainer" containerID="ef41fa7d32c775a325d13483b1f6314a12afaa9d89741a88e1a06a26a6497d9e"
Sep 18 17:30:03.847192 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:03.847172    2881 scope.go:117] "RemoveContainer" containerID="d66ce08948e6672773b99b8631006cd02d8d55a4bf1935b2831e96ebe495d0fb"
Sep 18 17:30:03.847675 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:30:03.847655    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d66ce08948e6672773b99b8631006cd02d8d55a4bf1935b2831e96ebe495d0fb\": container with ID starting with d66ce08948e6672773b99b8631006cd02d8d55a4bf1935b2831e96ebe495d0fb not found: ID does not exist" containerID="d66ce08948e6672773b99b8631006cd02d8d55a4bf1935b2831e96ebe495d0fb"
Sep 18 17:30:03.847755 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:03.847685    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d66ce08948e6672773b99b8631006cd02d8d55a4bf1935b2831e96ebe495d0fb"} err="failed to get container status \"d66ce08948e6672773b99b8631006cd02d8d55a4bf1935b2831e96ebe495d0fb\": rpc error: code = NotFound desc = could not find container \"d66ce08948e6672773b99b8631006cd02d8d55a4bf1935b2831e96ebe495d0fb\": container with ID starting with d66ce08948e6672773b99b8631006cd02d8d55a4bf1935b2831e96ebe495d0fb not found: ID does not exist"
Sep 18 17:30:03.847755 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:03.847708    2881 scope.go:117] "RemoveContainer" containerID="455b946182f5979a899eb7de885a27ed608754f25c8e50ca98c17645c3f095ad"
Sep 18 17:30:03.847997 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:30:03.847981    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"455b946182f5979a899eb7de885a27ed608754f25c8e50ca98c17645c3f095ad\": container with ID starting with 455b946182f5979a899eb7de885a27ed608754f25c8e50ca98c17645c3f095ad not found: ID does not exist" containerID="455b946182f5979a899eb7de885a27ed608754f25c8e50ca98c17645c3f095ad"
Sep 18 17:30:03.848035 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:03.848004    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"455b946182f5979a899eb7de885a27ed608754f25c8e50ca98c17645c3f095ad"} err="failed to get container status \"455b946182f5979a899eb7de885a27ed608754f25c8e50ca98c17645c3f095ad\": rpc error: code = NotFound desc = could not find container \"455b946182f5979a899eb7de885a27ed608754f25c8e50ca98c17645c3f095ad\": container with ID starting with 455b946182f5979a899eb7de885a27ed608754f25c8e50ca98c17645c3f095ad not found: ID does not exist"
Sep 18 17:30:03.848035 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:03.848019    2881 scope.go:117] "RemoveContainer" containerID="ef41fa7d32c775a325d13483b1f6314a12afaa9d89741a88e1a06a26a6497d9e"
Sep 18 17:30:03.848300 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:30:03.848273    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef41fa7d32c775a325d13483b1f6314a12afaa9d89741a88e1a06a26a6497d9e\": container with ID starting with ef41fa7d32c775a325d13483b1f6314a12afaa9d89741a88e1a06a26a6497d9e not found: ID does not exist" containerID="ef41fa7d32c775a325d13483b1f6314a12afaa9d89741a88e1a06a26a6497d9e"
Sep 18 17:30:03.848348 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:03.848301    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef41fa7d32c775a325d13483b1f6314a12afaa9d89741a88e1a06a26a6497d9e"} err="failed to get container status \"ef41fa7d32c775a325d13483b1f6314a12afaa9d89741a88e1a06a26a6497d9e\": rpc error: code = NotFound desc = could not find container \"ef41fa7d32c775a325d13483b1f6314a12afaa9d89741a88e1a06a26a6497d9e\": container with ID starting with ef41fa7d32c775a325d13483b1f6314a12afaa9d89741a88e1a06a26a6497d9e not found: ID does not exist"
Sep 18 17:30:03.865082 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:03.865027    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-9kw4c" podStartSLOduration=3.137167311 podStartE2EDuration="6.865012313s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.727845002s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:29:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:30:03.863638245 +0000 UTC m=+2194.063493533" watchObservedRunningTime="2026-09-18 17:30:03.865012313 +0000 UTC m=+2194.064867598"
Sep 18 17:30:03.893234 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:03.893197    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-m2lrf"]
Sep 18 17:30:03.895630 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:03.895602    2881 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5f4f848c44-m2lrf"]
Sep 18 17:30:04.825325 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:04.825291    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-9kw4c"
Sep 18 17:30:04.826531 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:04.826503    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-9kw4c" podUID="4d22cfc6-e80f-427a-aa7f-68fa0e780253" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 18 17:30:05.830148 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:05.830108    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-9kw4c" podUID="4d22cfc6-e80f-427a-aa7f-68fa0e780253" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 18 17:30:05.893329 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:05.893295    2881 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1f555f6-f287-4856-871a-242de5e78bbf" path="/var/lib/kubelet/pods/a1f555f6-f287-4856-871a-242de5e78bbf/volumes"
Sep 18 17:30:06.492616 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:06.492577    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-767cd75768-dvqsl" podUID="7018e6f0-2468-4a6d-a6ec-8c7943569ea3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 18 17:30:10.835453 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:10.835422    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-9kw4c"
Sep 18 17:30:10.836128 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:10.836101    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-9kw4c" podUID="4d22cfc6-e80f-427a-aa7f-68fa0e780253" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 18 17:30:15.893540 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:15.893511    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-767cd75768-dvqsl"
Sep 18 17:30:20.837031 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:20.836977    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-9kw4c" podUID="4d22cfc6-e80f-427a-aa7f-68fa0e780253" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 18 17:30:24.216927 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:24.216894    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-767cd75768-dvqsl"]
Sep 18 17:30:24.217509 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:24.217186    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-767cd75768-dvqsl" podUID="7018e6f0-2468-4a6d-a6ec-8c7943569ea3" containerName="kserve-container" containerID="cri-o://b89b0253412bd9cad1ea2e736144ffdb6b463ef1c4b37e5d36c97bdcb5dbfe2a" gracePeriod=30
Sep 18 17:30:24.217509 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:24.217251    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-767cd75768-dvqsl" podUID="7018e6f0-2468-4a6d-a6ec-8c7943569ea3" containerName="kube-rbac-proxy" containerID="cri-o://f25dff21000dc167ab6fe7a565c457432960ab5c42c06b64257d8ed9e42a93f0" gracePeriod=30
Sep 18 17:30:24.926114 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:24.926080    2881 generic.go:356] "Generic (PLEG): container finished" podID="7018e6f0-2468-4a6d-a6ec-8c7943569ea3" containerID="f25dff21000dc167ab6fe7a565c457432960ab5c42c06b64257d8ed9e42a93f0" exitCode=2
Sep 18 17:30:24.926311 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:24.926127    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-767cd75768-dvqsl" event={"ID":"7018e6f0-2468-4a6d-a6ec-8c7943569ea3","Type":"ContainerDied","Data":"f25dff21000dc167ab6fe7a565c457432960ab5c42c06b64257d8ed9e42a93f0"}
Sep 18 17:30:25.888470 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:25.888428    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-767cd75768-dvqsl" podUID="7018e6f0-2468-4a6d-a6ec-8c7943569ea3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 18 17:30:26.487564 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:26.487523    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-767cd75768-dvqsl" podUID="7018e6f0-2468-4a6d-a6ec-8c7943569ea3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Sep 18 17:30:29.766178 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:29.766149    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-767cd75768-dvqsl"
Sep 18 17:30:29.858577 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:29.858480    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7018e6f0-2468-4a6d-a6ec-8c7943569ea3-cabundle-cert\") pod \"7018e6f0-2468-4a6d-a6ec-8c7943569ea3\" (UID: \"7018e6f0-2468-4a6d-a6ec-8c7943569ea3\") "
Sep 18 17:30:29.858577 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:29.858556    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7018e6f0-2468-4a6d-a6ec-8c7943569ea3-kserve-provision-location\") pod \"7018e6f0-2468-4a6d-a6ec-8c7943569ea3\" (UID: \"7018e6f0-2468-4a6d-a6ec-8c7943569ea3\") "
Sep 18 17:30:29.858813 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:29.858585    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7018e6f0-2468-4a6d-a6ec-8c7943569ea3-kube-rbac-proxy-sar-config\") pod \"7018e6f0-2468-4a6d-a6ec-8c7943569ea3\" (UID: \"7018e6f0-2468-4a6d-a6ec-8c7943569ea3\") "
Sep 18 17:30:29.858813 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:29.858641    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7018e6f0-2468-4a6d-a6ec-8c7943569ea3-proxy-tls\") pod \"7018e6f0-2468-4a6d-a6ec-8c7943569ea3\" (UID: \"7018e6f0-2468-4a6d-a6ec-8c7943569ea3\") "
Sep 18 17:30:29.858813 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:29.858676    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5qkp9\" (UniqueName: \"kubernetes.io/projected/7018e6f0-2468-4a6d-a6ec-8c7943569ea3-kube-api-access-5qkp9\") pod \"7018e6f0-2468-4a6d-a6ec-8c7943569ea3\" (UID: \"7018e6f0-2468-4a6d-a6ec-8c7943569ea3\") "
Sep 18 17:30:29.859012 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:29.858917    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7018e6f0-2468-4a6d-a6ec-8c7943569ea3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7018e6f0-2468-4a6d-a6ec-8c7943569ea3" (UID: "7018e6f0-2468-4a6d-a6ec-8c7943569ea3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:30:29.859012 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:29.858927    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7018e6f0-2468-4a6d-a6ec-8c7943569ea3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7018e6f0-2468-4a6d-a6ec-8c7943569ea3" (UID: "7018e6f0-2468-4a6d-a6ec-8c7943569ea3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:30:29.859012 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:29.858962    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7018e6f0-2468-4a6d-a6ec-8c7943569ea3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7018e6f0-2468-4a6d-a6ec-8c7943569ea3" (UID: "7018e6f0-2468-4a6d-a6ec-8c7943569ea3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:30:29.860849 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:29.860823    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7018e6f0-2468-4a6d-a6ec-8c7943569ea3-kube-api-access-5qkp9" (OuterVolumeSpecName: "kube-api-access-5qkp9") pod "7018e6f0-2468-4a6d-a6ec-8c7943569ea3" (UID: "7018e6f0-2468-4a6d-a6ec-8c7943569ea3"). InnerVolumeSpecName "kube-api-access-5qkp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:30:29.860849 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:29.860832    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7018e6f0-2468-4a6d-a6ec-8c7943569ea3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7018e6f0-2468-4a6d-a6ec-8c7943569ea3" (UID: "7018e6f0-2468-4a6d-a6ec-8c7943569ea3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:30:29.951375 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:29.951334    2881 generic.go:356] "Generic (PLEG): container finished" podID="7018e6f0-2468-4a6d-a6ec-8c7943569ea3" containerID="b89b0253412bd9cad1ea2e736144ffdb6b463ef1c4b37e5d36c97bdcb5dbfe2a" exitCode=0
Sep 18 17:30:29.951557 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:29.951395    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-767cd75768-dvqsl"
Sep 18 17:30:29.951557 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:29.951396    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-767cd75768-dvqsl" event={"ID":"7018e6f0-2468-4a6d-a6ec-8c7943569ea3","Type":"ContainerDied","Data":"b89b0253412bd9cad1ea2e736144ffdb6b463ef1c4b37e5d36c97bdcb5dbfe2a"}
Sep 18 17:30:29.951557 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:29.951459    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-767cd75768-dvqsl" event={"ID":"7018e6f0-2468-4a6d-a6ec-8c7943569ea3","Type":"ContainerDied","Data":"3473f853fc699c470fbe03781637fb4d46f67c20d01364be8a3fd37754233ffe"}
Sep 18 17:30:29.951557 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:29.951497    2881 scope.go:117] "RemoveContainer" containerID="f25dff21000dc167ab6fe7a565c457432960ab5c42c06b64257d8ed9e42a93f0"
Sep 18 17:30:29.959809 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:29.959782    2881 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7018e6f0-2468-4a6d-a6ec-8c7943569ea3-cabundle-cert\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:30:29.959809 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:29.959807    2881 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7018e6f0-2468-4a6d-a6ec-8c7943569ea3-kserve-provision-location\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:30:29.960033 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:29.959817    2881 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7018e6f0-2468-4a6d-a6ec-8c7943569ea3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:30:29.960033 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:29.959827    2881 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7018e6f0-2468-4a6d-a6ec-8c7943569ea3-proxy-tls\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:30:29.960033 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:29.959836    2881 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qkp9\" (UniqueName: \"kubernetes.io/projected/7018e6f0-2468-4a6d-a6ec-8c7943569ea3-kube-api-access-5qkp9\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:30:29.960279 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:29.960092    2881 scope.go:117] "RemoveContainer" containerID="b89b0253412bd9cad1ea2e736144ffdb6b463ef1c4b37e5d36c97bdcb5dbfe2a"
Sep 18 17:30:29.968128 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:29.968108    2881 scope.go:117] "RemoveContainer" containerID="ae39eefa95e1bcfbf4051719560aa2709407a15e749e301281f2c57209187ec8"
Sep 18 17:30:29.976177 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:29.976152    2881 scope.go:117] "RemoveContainer" containerID="f25dff21000dc167ab6fe7a565c457432960ab5c42c06b64257d8ed9e42a93f0"
Sep 18 17:30:29.976618 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:30:29.976592    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f25dff21000dc167ab6fe7a565c457432960ab5c42c06b64257d8ed9e42a93f0\": container with ID starting with f25dff21000dc167ab6fe7a565c457432960ab5c42c06b64257d8ed9e42a93f0 not found: ID does not exist" containerID="f25dff21000dc167ab6fe7a565c457432960ab5c42c06b64257d8ed9e42a93f0"
Sep 18 17:30:29.976721 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:29.976631    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f25dff21000dc167ab6fe7a565c457432960ab5c42c06b64257d8ed9e42a93f0"} err="failed to get container status \"f25dff21000dc167ab6fe7a565c457432960ab5c42c06b64257d8ed9e42a93f0\": rpc error: code = NotFound desc = could not find container \"f25dff21000dc167ab6fe7a565c457432960ab5c42c06b64257d8ed9e42a93f0\": container with ID starting with f25dff21000dc167ab6fe7a565c457432960ab5c42c06b64257d8ed9e42a93f0 not found: ID does not exist"
Sep 18 17:30:29.976721 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:29.976657    2881 scope.go:117] "RemoveContainer" containerID="b89b0253412bd9cad1ea2e736144ffdb6b463ef1c4b37e5d36c97bdcb5dbfe2a"
Sep 18 17:30:29.977021 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:30:29.977000    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b89b0253412bd9cad1ea2e736144ffdb6b463ef1c4b37e5d36c97bdcb5dbfe2a\": container with ID starting with b89b0253412bd9cad1ea2e736144ffdb6b463ef1c4b37e5d36c97bdcb5dbfe2a not found: ID does not exist" containerID="b89b0253412bd9cad1ea2e736144ffdb6b463ef1c4b37e5d36c97bdcb5dbfe2a"
Sep 18 17:30:29.977107 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:29.977024    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b89b0253412bd9cad1ea2e736144ffdb6b463ef1c4b37e5d36c97bdcb5dbfe2a"} err="failed to get container status \"b89b0253412bd9cad1ea2e736144ffdb6b463ef1c4b37e5d36c97bdcb5dbfe2a\": rpc error: code = NotFound desc = could not find container \"b89b0253412bd9cad1ea2e736144ffdb6b463ef1c4b37e5d36c97bdcb5dbfe2a\": container with ID starting with b89b0253412bd9cad1ea2e736144ffdb6b463ef1c4b37e5d36c97bdcb5dbfe2a not found: ID does not exist"
Sep 18 17:30:29.977107 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:29.977042    2881 scope.go:117] "RemoveContainer" containerID="ae39eefa95e1bcfbf4051719560aa2709407a15e749e301281f2c57209187ec8"
Sep 18 17:30:29.977445 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:30:29.977425    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae39eefa95e1bcfbf4051719560aa2709407a15e749e301281f2c57209187ec8\": container with ID starting with ae39eefa95e1bcfbf4051719560aa2709407a15e749e301281f2c57209187ec8 not found: ID does not exist" containerID="ae39eefa95e1bcfbf4051719560aa2709407a15e749e301281f2c57209187ec8"
Sep 18 17:30:29.977533 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:29.977455    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae39eefa95e1bcfbf4051719560aa2709407a15e749e301281f2c57209187ec8"} err="failed to get container status \"ae39eefa95e1bcfbf4051719560aa2709407a15e749e301281f2c57209187ec8\": rpc error: code = NotFound desc = could not find container \"ae39eefa95e1bcfbf4051719560aa2709407a15e749e301281f2c57209187ec8\": container with ID starting with ae39eefa95e1bcfbf4051719560aa2709407a15e749e301281f2c57209187ec8 not found: ID does not exist"
Sep 18 17:30:29.979093 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:29.979072    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-767cd75768-dvqsl"]
Sep 18 17:30:29.992317 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:29.992287    2881 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-767cd75768-dvqsl"]
Sep 18 17:30:30.837096 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:30.837058    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-9kw4c" podUID="4d22cfc6-e80f-427a-aa7f-68fa0e780253" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 18 17:30:31.892548 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:31.892517    2881 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7018e6f0-2468-4a6d-a6ec-8c7943569ea3" path="/var/lib/kubelet/pods/7018e6f0-2468-4a6d-a6ec-8c7943569ea3/volumes"
Sep 18 17:30:40.836500 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:40.836459    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-9kw4c" podUID="4d22cfc6-e80f-427a-aa7f-68fa0e780253" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 18 17:30:50.836418 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:30:50.836369    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-9kw4c" podUID="4d22cfc6-e80f-427a-aa7f-68fa0e780253" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 18 17:31:00.836841 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:00.836795    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-9kw4c" podUID="4d22cfc6-e80f-427a-aa7f-68fa0e780253" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 18 17:31:10.836844 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:10.836793    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-9kw4c" podUID="4d22cfc6-e80f-427a-aa7f-68fa0e780253" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 18 17:31:15.322033 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:15.321992    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wt4z2"]
Sep 18 17:31:15.322479 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:15.322370    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7018e6f0-2468-4a6d-a6ec-8c7943569ea3" containerName="kserve-container"
Sep 18 17:31:15.322479 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:15.322382    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="7018e6f0-2468-4a6d-a6ec-8c7943569ea3" containerName="kserve-container"
Sep 18 17:31:15.322479 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:15.322396    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1f555f6-f287-4856-871a-242de5e78bbf" containerName="kube-rbac-proxy"
Sep 18 17:31:15.322479 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:15.322401    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1f555f6-f287-4856-871a-242de5e78bbf" containerName="kube-rbac-proxy"
Sep 18 17:31:15.322479 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:15.322408    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1f555f6-f287-4856-871a-242de5e78bbf" containerName="kserve-container"
Sep 18 17:31:15.322479 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:15.322414    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1f555f6-f287-4856-871a-242de5e78bbf" containerName="kserve-container"
Sep 18 17:31:15.322479 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:15.322420    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7018e6f0-2468-4a6d-a6ec-8c7943569ea3" containerName="storage-initializer"
Sep 18 17:31:15.322479 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:15.322425    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="7018e6f0-2468-4a6d-a6ec-8c7943569ea3" containerName="storage-initializer"
Sep 18 17:31:15.322479 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:15.322443    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1f555f6-f287-4856-871a-242de5e78bbf" containerName="storage-initializer"
Sep 18 17:31:15.322479 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:15.322448    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1f555f6-f287-4856-871a-242de5e78bbf" containerName="storage-initializer"
Sep 18 17:31:15.322479 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:15.322457    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7018e6f0-2468-4a6d-a6ec-8c7943569ea3" containerName="kube-rbac-proxy"
Sep 18 17:31:15.322479 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:15.322462    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="7018e6f0-2468-4a6d-a6ec-8c7943569ea3" containerName="kube-rbac-proxy"
Sep 18 17:31:15.322849 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:15.322515    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="7018e6f0-2468-4a6d-a6ec-8c7943569ea3" containerName="kube-rbac-proxy"
Sep 18 17:31:15.322849 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:15.322525    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="7018e6f0-2468-4a6d-a6ec-8c7943569ea3" containerName="kserve-container"
Sep 18 17:31:15.322849 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:15.322533    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1f555f6-f287-4856-871a-242de5e78bbf" containerName="kube-rbac-proxy"
Sep 18 17:31:15.322849 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:15.322540    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1f555f6-f287-4856-871a-242de5e78bbf" containerName="kserve-container"
Sep 18 17:31:15.325950 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:15.325932    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wt4z2"
Sep 18 17:31:15.328549 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:15.328524    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-predictor-serving-cert\""
Sep 18 17:31:15.328655 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:15.328572    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\""
Sep 18 17:31:15.337431 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:15.337396    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wt4z2"]
Sep 18 17:31:15.458783 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:15.458741    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4872485b-3724-4fb1-ae6b-c8efb41670f5-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-wt4z2\" (UID: \"4872485b-3724-4fb1-ae6b-c8efb41670f5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wt4z2"
Sep 18 17:31:15.458783 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:15.458789    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rst4l\" (UniqueName: \"kubernetes.io/projected/4872485b-3724-4fb1-ae6b-c8efb41670f5-kube-api-access-rst4l\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-wt4z2\" (UID: \"4872485b-3724-4fb1-ae6b-c8efb41670f5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wt4z2"
Sep 18 17:31:15.459043 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:15.458844    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4872485b-3724-4fb1-ae6b-c8efb41670f5-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-wt4z2\" (UID: \"4872485b-3724-4fb1-ae6b-c8efb41670f5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wt4z2"
Sep 18 17:31:15.459043 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:15.458910    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4872485b-3724-4fb1-ae6b-c8efb41670f5-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-wt4z2\" (UID: \"4872485b-3724-4fb1-ae6b-c8efb41670f5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wt4z2"
Sep 18 17:31:15.459043 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:15.458939    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4872485b-3724-4fb1-ae6b-c8efb41670f5-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-wt4z2\" (UID: \"4872485b-3724-4fb1-ae6b-c8efb41670f5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wt4z2"
Sep 18 17:31:15.560205 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:15.560161    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rst4l\" (UniqueName: \"kubernetes.io/projected/4872485b-3724-4fb1-ae6b-c8efb41670f5-kube-api-access-rst4l\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-wt4z2\" (UID: \"4872485b-3724-4fb1-ae6b-c8efb41670f5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wt4z2"
Sep 18 17:31:15.560205 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:15.560209    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4872485b-3724-4fb1-ae6b-c8efb41670f5-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-wt4z2\" (UID: \"4872485b-3724-4fb1-ae6b-c8efb41670f5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wt4z2"
Sep 18 17:31:15.560517 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:15.560281    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4872485b-3724-4fb1-ae6b-c8efb41670f5-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-wt4z2\" (UID: \"4872485b-3724-4fb1-ae6b-c8efb41670f5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wt4z2"
Sep 18 17:31:15.560517 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:15.560307    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4872485b-3724-4fb1-ae6b-c8efb41670f5-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-wt4z2\" (UID: \"4872485b-3724-4fb1-ae6b-c8efb41670f5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wt4z2"
Sep 18 17:31:15.560517 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:15.560354    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4872485b-3724-4fb1-ae6b-c8efb41670f5-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-wt4z2\" (UID: \"4872485b-3724-4fb1-ae6b-c8efb41670f5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wt4z2"
Sep 18 17:31:15.560847 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:15.560821    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4872485b-3724-4fb1-ae6b-c8efb41670f5-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-wt4z2\" (UID: \"4872485b-3724-4fb1-ae6b-c8efb41670f5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wt4z2"
Sep 18 17:31:15.561017 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:15.560996    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4872485b-3724-4fb1-ae6b-c8efb41670f5-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-wt4z2\" (UID: \"4872485b-3724-4fb1-ae6b-c8efb41670f5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wt4z2"
Sep 18 17:31:15.561112 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:15.561090    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4872485b-3724-4fb1-ae6b-c8efb41670f5-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-wt4z2\" (UID: \"4872485b-3724-4fb1-ae6b-c8efb41670f5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wt4z2"
Sep 18 17:31:15.562776 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:15.562743    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4872485b-3724-4fb1-ae6b-c8efb41670f5-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-wt4z2\" (UID: \"4872485b-3724-4fb1-ae6b-c8efb41670f5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wt4z2"
Sep 18 17:31:15.570985 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:15.570963    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rst4l\" (UniqueName: \"kubernetes.io/projected/4872485b-3724-4fb1-ae6b-c8efb41670f5-kube-api-access-rst4l\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-wt4z2\" (UID: \"4872485b-3724-4fb1-ae6b-c8efb41670f5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wt4z2"
Sep 18 17:31:15.637142 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:15.637052    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wt4z2"
Sep 18 17:31:15.913344 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:15.913241    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wt4z2"]
Sep 18 17:31:15.916295 ip-10-0-143-5 kubenswrapper[2881]: W0918 17:31:15.916259    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4872485b_3724_4fb1_ae6b_c8efb41670f5.slice/crio-1499dd2a66ef9d55125dd28ec67bd44b1de35310dcbc18aa6b3b79daf963b43b WatchSource:0}: Error finding container 1499dd2a66ef9d55125dd28ec67bd44b1de35310dcbc18aa6b3b79daf963b43b: Status 404 returned error can't find the container with id 1499dd2a66ef9d55125dd28ec67bd44b1de35310dcbc18aa6b3b79daf963b43b
Sep 18 17:31:16.152417 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:16.152370    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wt4z2" event={"ID":"4872485b-3724-4fb1-ae6b-c8efb41670f5","Type":"ContainerStarted","Data":"895eaef05d14ba2b78086332443a4bfa86b1475009d5908946c7a7b1d7b4b3fe"}
Sep 18 17:31:16.152596 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:16.152458    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wt4z2" event={"ID":"4872485b-3724-4fb1-ae6b-c8efb41670f5","Type":"ContainerStarted","Data":"1499dd2a66ef9d55125dd28ec67bd44b1de35310dcbc18aa6b3b79daf963b43b"}
Sep 18 17:31:18.888148 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:18.888094    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-9kw4c" podUID="4d22cfc6-e80f-427a-aa7f-68fa0e780253" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 18 17:31:21.179724 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:21.179686    2881 generic.go:356] "Generic (PLEG): container finished" podID="4872485b-3724-4fb1-ae6b-c8efb41670f5" containerID="895eaef05d14ba2b78086332443a4bfa86b1475009d5908946c7a7b1d7b4b3fe" exitCode=0
Sep 18 17:31:21.180272 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:21.179734    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wt4z2" event={"ID":"4872485b-3724-4fb1-ae6b-c8efb41670f5","Type":"ContainerDied","Data":"895eaef05d14ba2b78086332443a4bfa86b1475009d5908946c7a7b1d7b4b3fe"}
Sep 18 17:31:21.181201 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:21.181185    2881 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 17:31:25.202736 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:25.202648    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wt4z2" event={"ID":"4872485b-3724-4fb1-ae6b-c8efb41670f5","Type":"ContainerStarted","Data":"24d77394403cf7d29572c296a516c2e6f1d3b483e024a4092376a63f9613f599"}
Sep 18 17:31:25.202736 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:25.202714    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wt4z2" event={"ID":"4872485b-3724-4fb1-ae6b-c8efb41670f5","Type":"ContainerStarted","Data":"72fd0a8b0e646ccce063d67948453df96521994f390ce115dbad8173fb839094"}
Sep 18 17:31:25.203199 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:25.202917    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wt4z2"
Sep 18 17:31:25.222881 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:25.222823    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wt4z2" podStartSLOduration=1.870006115 podStartE2EDuration="10.222806436s" totalImagesPullingTime="3.741006257s" totalInitContainerRuntime="4.611794064s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:31:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:31:25.221579437 +0000 UTC m=+2275.421434723" watchObservedRunningTime="2026-09-18 17:31:25.222806436 +0000 UTC m=+2275.422661721"
Sep 18 17:31:26.207304 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:26.207259    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wt4z2"
Sep 18 17:31:26.208694 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:26.208661    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wt4z2" podUID="4872485b-3724-4fb1-ae6b-c8efb41670f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 18 17:31:27.211650 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:27.211603    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wt4z2" podUID="4872485b-3724-4fb1-ae6b-c8efb41670f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 18 17:31:28.889200 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:28.889163    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-9kw4c"
Sep 18 17:31:32.216506 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:32.216433    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wt4z2"
Sep 18 17:31:32.217027 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:32.217002    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wt4z2" podUID="4872485b-3724-4fb1-ae6b-c8efb41670f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 18 17:31:38.039602 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:38.039558    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-9kw4c"]
Sep 18 17:31:38.040075 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:38.039816    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-9kw4c" podUID="4d22cfc6-e80f-427a-aa7f-68fa0e780253" containerName="kserve-container" containerID="cri-o://fa50687c00c25988502caaf76eef2ba394c28140ce9c29c6b4f17854de17d938" gracePeriod=30
Sep 18 17:31:38.040075 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:38.039876    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-9kw4c" podUID="4d22cfc6-e80f-427a-aa7f-68fa0e780253" containerName="kube-rbac-proxy" containerID="cri-o://fbdadead941bee1afa6df133862d9beb683a4387812f160e8591100863ec8bc6" gracePeriod=30
Sep 18 17:31:38.264482 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:38.264450    2881 generic.go:356] "Generic (PLEG): container finished" podID="4d22cfc6-e80f-427a-aa7f-68fa0e780253" containerID="fbdadead941bee1afa6df133862d9beb683a4387812f160e8591100863ec8bc6" exitCode=2
Sep 18 17:31:38.264668 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:38.264507    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-9kw4c" event={"ID":"4d22cfc6-e80f-427a-aa7f-68fa0e780253","Type":"ContainerDied","Data":"fbdadead941bee1afa6df133862d9beb683a4387812f160e8591100863ec8bc6"}
Sep 18 17:31:38.888839 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:38.888797    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-9kw4c" podUID="4d22cfc6-e80f-427a-aa7f-68fa0e780253" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 18 17:31:40.831116 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:40.831069    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-9kw4c" podUID="4d22cfc6-e80f-427a-aa7f-68fa0e780253" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.45:8643/healthz\": dial tcp 10.134.0.45:8643: connect: connection refused"
Sep 18 17:31:42.217108 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:42.217077    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wt4z2"
Sep 18 17:31:43.261806 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:43.261776    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-sgvlv"]
Sep 18 17:31:43.264605 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:43.264585    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-sgvlv"
Sep 18 17:31:43.267372 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:43.267351    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 18 17:31:43.267540 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:43.267522    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 18 17:31:43.278181 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:43.278146    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-sgvlv"]
Sep 18 17:31:43.290196 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:43.290161    2881 generic.go:356] "Generic (PLEG): container finished" podID="4d22cfc6-e80f-427a-aa7f-68fa0e780253" containerID="fa50687c00c25988502caaf76eef2ba394c28140ce9c29c6b4f17854de17d938" exitCode=0
Sep 18 17:31:43.290395 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:43.290276    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-9kw4c" event={"ID":"4d22cfc6-e80f-427a-aa7f-68fa0e780253","Type":"ContainerDied","Data":"fa50687c00c25988502caaf76eef2ba394c28140ce9c29c6b4f17854de17d938"}
Sep 18 17:31:43.391602 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:43.391574    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-9kw4c"
Sep 18 17:31:43.407561 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:43.407534    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0bb3352b-d853-44aa-a02d-f0a0fe8bc270-cabundle-cert\") pod \"isvc-canary-rollback-predictor-69978998f6-sgvlv\" (UID: \"0bb3352b-d853-44aa-a02d-f0a0fe8bc270\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-sgvlv"
Sep 18 17:31:43.407662 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:43.407599    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0bb3352b-d853-44aa-a02d-f0a0fe8bc270-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-69978998f6-sgvlv\" (UID: \"0bb3352b-d853-44aa-a02d-f0a0fe8bc270\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-sgvlv"
Sep 18 17:31:43.407662 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:43.407624    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcnhn\" (UniqueName: \"kubernetes.io/projected/0bb3352b-d853-44aa-a02d-f0a0fe8bc270-kube-api-access-qcnhn\") pod \"isvc-canary-rollback-predictor-69978998f6-sgvlv\" (UID: \"0bb3352b-d853-44aa-a02d-f0a0fe8bc270\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-sgvlv"
Sep 18 17:31:43.407662 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:43.407654    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0bb3352b-d853-44aa-a02d-f0a0fe8bc270-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-69978998f6-sgvlv\" (UID: \"0bb3352b-d853-44aa-a02d-f0a0fe8bc270\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-sgvlv"
Sep 18 17:31:43.407795 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:43.407713    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0bb3352b-d853-44aa-a02d-f0a0fe8bc270-proxy-tls\") pod \"isvc-canary-rollback-predictor-69978998f6-sgvlv\" (UID: \"0bb3352b-d853-44aa-a02d-f0a0fe8bc270\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-sgvlv"
Sep 18 17:31:43.508285 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:43.508166    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d22cfc6-e80f-427a-aa7f-68fa0e780253-kserve-provision-location\") pod \"4d22cfc6-e80f-427a-aa7f-68fa0e780253\" (UID: \"4d22cfc6-e80f-427a-aa7f-68fa0e780253\") "
Sep 18 17:31:43.508285 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:43.508271    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d22cfc6-e80f-427a-aa7f-68fa0e780253-proxy-tls\") pod \"4d22cfc6-e80f-427a-aa7f-68fa0e780253\" (UID: \"4d22cfc6-e80f-427a-aa7f-68fa0e780253\") "
Sep 18 17:31:43.508533 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:43.508300    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rp76z\" (UniqueName: \"kubernetes.io/projected/4d22cfc6-e80f-427a-aa7f-68fa0e780253-kube-api-access-rp76z\") pod \"4d22cfc6-e80f-427a-aa7f-68fa0e780253\" (UID: \"4d22cfc6-e80f-427a-aa7f-68fa0e780253\") "
Sep 18 17:31:43.508533 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:43.508325    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d22cfc6-e80f-427a-aa7f-68fa0e780253-kube-rbac-proxy-sar-config\") pod \"4d22cfc6-e80f-427a-aa7f-68fa0e780253\" (UID: \"4d22cfc6-e80f-427a-aa7f-68fa0e780253\") "
Sep 18 17:31:43.508533 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:43.508370    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d22cfc6-e80f-427a-aa7f-68fa0e780253-cabundle-cert\") pod \"4d22cfc6-e80f-427a-aa7f-68fa0e780253\" (UID: \"4d22cfc6-e80f-427a-aa7f-68fa0e780253\") "
Sep 18 17:31:43.508533 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:43.508495    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0bb3352b-d853-44aa-a02d-f0a0fe8bc270-cabundle-cert\") pod \"isvc-canary-rollback-predictor-69978998f6-sgvlv\" (UID: \"0bb3352b-d853-44aa-a02d-f0a0fe8bc270\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-sgvlv"
Sep 18 17:31:43.508758 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:43.508545    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0bb3352b-d853-44aa-a02d-f0a0fe8bc270-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-69978998f6-sgvlv\" (UID: \"0bb3352b-d853-44aa-a02d-f0a0fe8bc270\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-sgvlv"
Sep 18 17:31:43.508758 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:43.508536    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d22cfc6-e80f-427a-aa7f-68fa0e780253-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4d22cfc6-e80f-427a-aa7f-68fa0e780253" (UID: "4d22cfc6-e80f-427a-aa7f-68fa0e780253"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:31:43.508758 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:43.508741    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcnhn\" (UniqueName: \"kubernetes.io/projected/0bb3352b-d853-44aa-a02d-f0a0fe8bc270-kube-api-access-qcnhn\") pod \"isvc-canary-rollback-predictor-69978998f6-sgvlv\" (UID: \"0bb3352b-d853-44aa-a02d-f0a0fe8bc270\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-sgvlv"
Sep 18 17:31:43.508918 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:43.508778    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d22cfc6-e80f-427a-aa7f-68fa0e780253-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4d22cfc6-e80f-427a-aa7f-68fa0e780253" (UID: "4d22cfc6-e80f-427a-aa7f-68fa0e780253"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:31:43.508918 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:43.508808    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0bb3352b-d853-44aa-a02d-f0a0fe8bc270-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-69978998f6-sgvlv\" (UID: \"0bb3352b-d853-44aa-a02d-f0a0fe8bc270\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-sgvlv"
Sep 18 17:31:43.508918 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:43.508856    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0bb3352b-d853-44aa-a02d-f0a0fe8bc270-proxy-tls\") pod \"isvc-canary-rollback-predictor-69978998f6-sgvlv\" (UID: \"0bb3352b-d853-44aa-a02d-f0a0fe8bc270\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-sgvlv"
Sep 18 17:31:43.509080 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:43.508947    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d22cfc6-e80f-427a-aa7f-68fa0e780253-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4d22cfc6-e80f-427a-aa7f-68fa0e780253" (UID: "4d22cfc6-e80f-427a-aa7f-68fa0e780253"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:31:43.509080 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:43.508952    2881 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d22cfc6-e80f-427a-aa7f-68fa0e780253-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:31:43.509080 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:43.508987    2881 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d22cfc6-e80f-427a-aa7f-68fa0e780253-kserve-provision-location\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:31:43.509080 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:31:43.509035    2881 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 18 17:31:43.509322 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:31:43.509094    2881 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bb3352b-d853-44aa-a02d-f0a0fe8bc270-proxy-tls podName:0bb3352b-d853-44aa-a02d-f0a0fe8bc270 nodeName:}" failed. No retries permitted until 2026-09-18 17:31:44.009076136 +0000 UTC m=+2294.208931403 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0bb3352b-d853-44aa-a02d-f0a0fe8bc270-proxy-tls") pod "isvc-canary-rollback-predictor-69978998f6-sgvlv" (UID: "0bb3352b-d853-44aa-a02d-f0a0fe8bc270") : secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 18 17:31:43.509397 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:43.509326    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0bb3352b-d853-44aa-a02d-f0a0fe8bc270-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-69978998f6-sgvlv\" (UID: \"0bb3352b-d853-44aa-a02d-f0a0fe8bc270\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-sgvlv"
Sep 18 17:31:43.509397 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:43.509348    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0bb3352b-d853-44aa-a02d-f0a0fe8bc270-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-69978998f6-sgvlv\" (UID: \"0bb3352b-d853-44aa-a02d-f0a0fe8bc270\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-sgvlv"
Sep 18 17:31:43.509397 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:43.509348    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0bb3352b-d853-44aa-a02d-f0a0fe8bc270-cabundle-cert\") pod \"isvc-canary-rollback-predictor-69978998f6-sgvlv\" (UID: \"0bb3352b-d853-44aa-a02d-f0a0fe8bc270\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-sgvlv"
Sep 18 17:31:43.510596 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:43.510574    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d22cfc6-e80f-427a-aa7f-68fa0e780253-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4d22cfc6-e80f-427a-aa7f-68fa0e780253" (UID: "4d22cfc6-e80f-427a-aa7f-68fa0e780253"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:31:43.510704 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:43.510625    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d22cfc6-e80f-427a-aa7f-68fa0e780253-kube-api-access-rp76z" (OuterVolumeSpecName: "kube-api-access-rp76z") pod "4d22cfc6-e80f-427a-aa7f-68fa0e780253" (UID: "4d22cfc6-e80f-427a-aa7f-68fa0e780253"). InnerVolumeSpecName "kube-api-access-rp76z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:31:43.517724 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:43.517701    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcnhn\" (UniqueName: \"kubernetes.io/projected/0bb3352b-d853-44aa-a02d-f0a0fe8bc270-kube-api-access-qcnhn\") pod \"isvc-canary-rollback-predictor-69978998f6-sgvlv\" (UID: \"0bb3352b-d853-44aa-a02d-f0a0fe8bc270\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-sgvlv"
Sep 18 17:31:43.610216 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:43.610179    2881 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d22cfc6-e80f-427a-aa7f-68fa0e780253-proxy-tls\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:31:43.610216 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:43.610207    2881 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp76z\" (UniqueName: \"kubernetes.io/projected/4d22cfc6-e80f-427a-aa7f-68fa0e780253-kube-api-access-rp76z\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:31:43.610216 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:43.610217    2881 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d22cfc6-e80f-427a-aa7f-68fa0e780253-cabundle-cert\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:31:44.014327 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:44.014293    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0bb3352b-d853-44aa-a02d-f0a0fe8bc270-proxy-tls\") pod \"isvc-canary-rollback-predictor-69978998f6-sgvlv\" (UID: \"0bb3352b-d853-44aa-a02d-f0a0fe8bc270\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-sgvlv"
Sep 18 17:31:44.016713 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:44.016682    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0bb3352b-d853-44aa-a02d-f0a0fe8bc270-proxy-tls\") pod \"isvc-canary-rollback-predictor-69978998f6-sgvlv\" (UID: \"0bb3352b-d853-44aa-a02d-f0a0fe8bc270\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-sgvlv"
Sep 18 17:31:44.195719 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:44.195629    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-sgvlv"
Sep 18 17:31:44.296751 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:44.296724    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-9kw4c"
Sep 18 17:31:44.296751 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:44.296739    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-9kw4c" event={"ID":"4d22cfc6-e80f-427a-aa7f-68fa0e780253","Type":"ContainerDied","Data":"acbf16a812ca61f1be5ee49c598fa6c83297be212ddc625164fec52fe284b48d"}
Sep 18 17:31:44.297243 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:44.296803    2881 scope.go:117] "RemoveContainer" containerID="fbdadead941bee1afa6df133862d9beb683a4387812f160e8591100863ec8bc6"
Sep 18 17:31:44.306157 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:44.306138    2881 scope.go:117] "RemoveContainer" containerID="fa50687c00c25988502caaf76eef2ba394c28140ce9c29c6b4f17854de17d938"
Sep 18 17:31:44.315140 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:44.315106    2881 scope.go:117] "RemoveContainer" containerID="65cf5a34a7389feb827ad725272b0a757789cdb04b712adc83be5f86b5f8ce14"
Sep 18 17:31:44.316843 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:44.316822    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-9kw4c"]
Sep 18 17:31:44.320867 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:44.320844    2881 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7fd74c59-9kw4c"]
Sep 18 17:31:44.472769 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:44.472691    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-sgvlv"]
Sep 18 17:31:44.476285 ip-10-0-143-5 kubenswrapper[2881]: W0918 17:31:44.476254    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bb3352b_d853_44aa_a02d_f0a0fe8bc270.slice/crio-64935df2a1ddbda04e8e3ac69cde1fb4c5200a32d1bf49bf86e044333d316df8 WatchSource:0}: Error finding container 64935df2a1ddbda04e8e3ac69cde1fb4c5200a32d1bf49bf86e044333d316df8: Status 404 returned error can't find the container with id 64935df2a1ddbda04e8e3ac69cde1fb4c5200a32d1bf49bf86e044333d316df8
Sep 18 17:31:45.302735 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:45.302703    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-sgvlv" event={"ID":"0bb3352b-d853-44aa-a02d-f0a0fe8bc270","Type":"ContainerStarted","Data":"d4221aee007f2c83da83d012520b848cb1d310dc81c6c9030a2a962be95f6c8a"}
Sep 18 17:31:45.303239 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:45.302766    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-sgvlv" event={"ID":"0bb3352b-d853-44aa-a02d-f0a0fe8bc270","Type":"ContainerStarted","Data":"64935df2a1ddbda04e8e3ac69cde1fb4c5200a32d1bf49bf86e044333d316df8"}
Sep 18 17:31:45.894589 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:45.894548    2881 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d22cfc6-e80f-427a-aa7f-68fa0e780253" path="/var/lib/kubelet/pods/4d22cfc6-e80f-427a-aa7f-68fa0e780253/volumes"
Sep 18 17:31:48.319734 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:48.319698    2881 generic.go:356] "Generic (PLEG): container finished" podID="0bb3352b-d853-44aa-a02d-f0a0fe8bc270" containerID="d4221aee007f2c83da83d012520b848cb1d310dc81c6c9030a2a962be95f6c8a" exitCode=0
Sep 18 17:31:48.320172 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:48.319767    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-sgvlv" event={"ID":"0bb3352b-d853-44aa-a02d-f0a0fe8bc270","Type":"ContainerDied","Data":"d4221aee007f2c83da83d012520b848cb1d310dc81c6c9030a2a962be95f6c8a"}
Sep 18 17:31:49.327185 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:49.327152    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-sgvlv" event={"ID":"0bb3352b-d853-44aa-a02d-f0a0fe8bc270","Type":"ContainerStarted","Data":"0ac67ce23db47ca9c5e2cd5794d9c0ea7eacc8a99c8a8d6cb5558a32fd7ccbd2"}
Sep 18 17:31:49.327646 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:49.327217    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-sgvlv" event={"ID":"0bb3352b-d853-44aa-a02d-f0a0fe8bc270","Type":"ContainerStarted","Data":"5273b60fef4e540b67fc6d12cf5e9bfef8e90458e7315bd4ada7da88c80fb184"}
Sep 18 17:31:49.327646 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:49.327504    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-sgvlv"
Sep 18 17:31:49.327736 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:49.327666    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-sgvlv"
Sep 18 17:31:49.328819 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:49.328793    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-sgvlv" podUID="0bb3352b-d853-44aa-a02d-f0a0fe8bc270" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 18 17:31:49.349424 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:49.349365    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-sgvlv" podStartSLOduration=2.6369370930000002 podStartE2EDuration="6.349345538s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.712408445s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:31:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:31:49.348277413 +0000 UTC m=+2299.548132696" watchObservedRunningTime="2026-09-18 17:31:49.349345538 +0000 UTC m=+2299.549200824"
Sep 18 17:31:50.332722 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:50.332678    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-sgvlv" podUID="0bb3352b-d853-44aa-a02d-f0a0fe8bc270" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 18 17:31:55.337399 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:55.337364    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-sgvlv"
Sep 18 17:31:55.338202 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:55.338170    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-sgvlv" podUID="0bb3352b-d853-44aa-a02d-f0a0fe8bc270" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 18 17:31:57.364379 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:57.364347    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wt4z2"]
Sep 18 17:31:57.364862 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:57.364585    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wt4z2" podUID="4872485b-3724-4fb1-ae6b-c8efb41670f5" containerName="kserve-container" containerID="cri-o://72fd0a8b0e646ccce063d67948453df96521994f390ce115dbad8173fb839094" gracePeriod=30
Sep 18 17:31:57.364862 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:57.364612    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wt4z2" podUID="4872485b-3724-4fb1-ae6b-c8efb41670f5" containerName="kube-rbac-proxy" containerID="cri-o://24d77394403cf7d29572c296a516c2e6f1d3b483e024a4092376a63f9613f599" gracePeriod=30
Sep 18 17:31:58.377906 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:58.377872    2881 generic.go:356] "Generic (PLEG): container finished" podID="4872485b-3724-4fb1-ae6b-c8efb41670f5" containerID="24d77394403cf7d29572c296a516c2e6f1d3b483e024a4092376a63f9613f599" exitCode=2
Sep 18 17:31:58.378303 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:31:58.377922    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wt4z2" event={"ID":"4872485b-3724-4fb1-ae6b-c8efb41670f5","Type":"ContainerDied","Data":"24d77394403cf7d29572c296a516c2e6f1d3b483e024a4092376a63f9613f599"}
Sep 18 17:32:02.212812 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:32:02.212766    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wt4z2" podUID="4872485b-3724-4fb1-ae6b-c8efb41670f5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.46:8643/healthz\": dial tcp 10.134.0.46:8643: connect: connection refused"
Sep 18 17:32:05.339180 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:32:05.339139    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-sgvlv" podUID="0bb3352b-d853-44aa-a02d-f0a0fe8bc270" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 18 17:32:07.212549 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:32:07.212502    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wt4z2" podUID="4872485b-3724-4fb1-ae6b-c8efb41670f5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.46:8643/healthz\": dial tcp 10.134.0.46:8643: connect: connection refused"
Sep 18 17:32:12.212191 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:32:12.212150    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wt4z2" podUID="4872485b-3724-4fb1-ae6b-c8efb41670f5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.46:8643/healthz\": dial tcp 10.134.0.46:8643: connect: connection refused"
Sep 18 17:32:12.212633 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:32:12.212295    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wt4z2"
Sep 18 17:32:15.338535 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:32:15.338497    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-sgvlv" podUID="0bb3352b-d853-44aa-a02d-f0a0fe8bc270" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 18 17:32:17.212768 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:32:17.212725    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wt4z2" podUID="4872485b-3724-4fb1-ae6b-c8efb41670f5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.46:8643/healthz\": dial tcp 10.134.0.46:8643: connect: connection refused"
Sep 18 17:32:22.212082 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:32:22.212029    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wt4z2" podUID="4872485b-3724-4fb1-ae6b-c8efb41670f5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.46:8643/healthz\": dial tcp 10.134.0.46:8643: connect: connection refused"
Sep 18 17:32:25.338420 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:32:25.338381    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-sgvlv" podUID="0bb3352b-d853-44aa-a02d-f0a0fe8bc270" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 18 17:32:27.212370 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:32:27.212321    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wt4z2" podUID="4872485b-3724-4fb1-ae6b-c8efb41670f5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.46:8643/healthz\": dial tcp 10.134.0.46:8643: connect: connection refused"
Sep 18 17:32:27.516920 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:32:27.516890    2881 generic.go:356] "Generic (PLEG): container finished" podID="4872485b-3724-4fb1-ae6b-c8efb41670f5" containerID="72fd0a8b0e646ccce063d67948453df96521994f390ce115dbad8173fb839094" exitCode=137
Sep 18 17:32:27.517075 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:32:27.516957    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wt4z2" event={"ID":"4872485b-3724-4fb1-ae6b-c8efb41670f5","Type":"ContainerDied","Data":"72fd0a8b0e646ccce063d67948453df96521994f390ce115dbad8173fb839094"}
Sep 18 17:32:28.002811 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:32:28.002784    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wt4z2"
Sep 18 17:32:28.110582 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:32:28.110541    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4872485b-3724-4fb1-ae6b-c8efb41670f5-kube-rbac-proxy-sar-config\") pod \"4872485b-3724-4fb1-ae6b-c8efb41670f5\" (UID: \"4872485b-3724-4fb1-ae6b-c8efb41670f5\") "
Sep 18 17:32:28.110784 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:32:28.110600    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4872485b-3724-4fb1-ae6b-c8efb41670f5-proxy-tls\") pod \"4872485b-3724-4fb1-ae6b-c8efb41670f5\" (UID: \"4872485b-3724-4fb1-ae6b-c8efb41670f5\") "
Sep 18 17:32:28.110862 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:32:28.110786    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rst4l\" (UniqueName: \"kubernetes.io/projected/4872485b-3724-4fb1-ae6b-c8efb41670f5-kube-api-access-rst4l\") pod \"4872485b-3724-4fb1-ae6b-c8efb41670f5\" (UID: \"4872485b-3724-4fb1-ae6b-c8efb41670f5\") "
Sep 18 17:32:28.110921 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:32:28.110859    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4872485b-3724-4fb1-ae6b-c8efb41670f5-cabundle-cert\") pod \"4872485b-3724-4fb1-ae6b-c8efb41670f5\" (UID: \"4872485b-3724-4fb1-ae6b-c8efb41670f5\") "
Sep 18 17:32:28.110921 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:32:28.110901    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4872485b-3724-4fb1-ae6b-c8efb41670f5-kserve-provision-location\") pod \"4872485b-3724-4fb1-ae6b-c8efb41670f5\" (UID: \"4872485b-3724-4fb1-ae6b-c8efb41670f5\") "
Sep 18 17:32:28.111017 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:32:28.110958    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4872485b-3724-4fb1-ae6b-c8efb41670f5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4872485b-3724-4fb1-ae6b-c8efb41670f5" (UID: "4872485b-3724-4fb1-ae6b-c8efb41670f5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:32:28.111194 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:32:28.111177    2881 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4872485b-3724-4fb1-ae6b-c8efb41670f5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:32:28.111306 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:32:28.111238    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4872485b-3724-4fb1-ae6b-c8efb41670f5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4872485b-3724-4fb1-ae6b-c8efb41670f5" (UID: "4872485b-3724-4fb1-ae6b-c8efb41670f5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:32:28.112798 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:32:28.112780    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4872485b-3724-4fb1-ae6b-c8efb41670f5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4872485b-3724-4fb1-ae6b-c8efb41670f5" (UID: "4872485b-3724-4fb1-ae6b-c8efb41670f5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:32:28.113114 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:32:28.113099    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4872485b-3724-4fb1-ae6b-c8efb41670f5-kube-api-access-rst4l" (OuterVolumeSpecName: "kube-api-access-rst4l") pod "4872485b-3724-4fb1-ae6b-c8efb41670f5" (UID: "4872485b-3724-4fb1-ae6b-c8efb41670f5"). InnerVolumeSpecName "kube-api-access-rst4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:32:28.122023 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:32:28.121990    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4872485b-3724-4fb1-ae6b-c8efb41670f5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4872485b-3724-4fb1-ae6b-c8efb41670f5" (UID: "4872485b-3724-4fb1-ae6b-c8efb41670f5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:32:28.212490 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:32:28.212397    2881 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4872485b-3724-4fb1-ae6b-c8efb41670f5-proxy-tls\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:32:28.212490 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:32:28.212431    2881 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rst4l\" (UniqueName: \"kubernetes.io/projected/4872485b-3724-4fb1-ae6b-c8efb41670f5-kube-api-access-rst4l\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:32:28.212490 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:32:28.212446    2881 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4872485b-3724-4fb1-ae6b-c8efb41670f5-cabundle-cert\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:32:28.212490 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:32:28.212460    2881 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4872485b-3724-4fb1-ae6b-c8efb41670f5-kserve-provision-location\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:32:28.524380 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:32:28.524356    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wt4z2"
Sep 18 17:32:28.524569 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:32:28.524355    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wt4z2" event={"ID":"4872485b-3724-4fb1-ae6b-c8efb41670f5","Type":"ContainerDied","Data":"1499dd2a66ef9d55125dd28ec67bd44b1de35310dcbc18aa6b3b79daf963b43b"}
Sep 18 17:32:28.524569 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:32:28.524515    2881 scope.go:117] "RemoveContainer" containerID="24d77394403cf7d29572c296a516c2e6f1d3b483e024a4092376a63f9613f599"
Sep 18 17:32:28.533740 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:32:28.533718    2881 scope.go:117] "RemoveContainer" containerID="72fd0a8b0e646ccce063d67948453df96521994f390ce115dbad8173fb839094"
Sep 18 17:32:28.542613 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:32:28.542591    2881 scope.go:117] "RemoveContainer" containerID="895eaef05d14ba2b78086332443a4bfa86b1475009d5908946c7a7b1d7b4b3fe"
Sep 18 17:32:28.548289 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:32:28.548261    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wt4z2"]
Sep 18 17:32:28.553147 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:32:28.553121    2881 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wt4z2"]
Sep 18 17:32:29.895401 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:32:29.895366    2881 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4872485b-3724-4fb1-ae6b-c8efb41670f5" path="/var/lib/kubelet/pods/4872485b-3724-4fb1-ae6b-c8efb41670f5/volumes"
Sep 18 17:32:35.339186 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:32:35.339145    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-sgvlv" podUID="0bb3352b-d853-44aa-a02d-f0a0fe8bc270" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 18 17:32:45.338731 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:32:45.338685    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-sgvlv" podUID="0bb3352b-d853-44aa-a02d-f0a0fe8bc270" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 18 17:32:55.338750 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:32:55.338708    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-sgvlv" podUID="0bb3352b-d853-44aa-a02d-f0a0fe8bc270" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 18 17:33:05.338417 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:33:05.338327    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-sgvlv" podUID="0bb3352b-d853-44aa-a02d-f0a0fe8bc270" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 18 17:33:15.339438 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:33:15.339404    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-sgvlv"
Sep 18 17:34:31.353861 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:34:31.353762    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-kb5h6"]
Sep 18 17:34:31.354371 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:34:31.354334    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4872485b-3724-4fb1-ae6b-c8efb41670f5" containerName="storage-initializer"
Sep 18 17:34:31.354371 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:34:31.354351    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="4872485b-3724-4fb1-ae6b-c8efb41670f5" containerName="storage-initializer"
Sep 18 17:34:31.354371 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:34:31.354364    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d22cfc6-e80f-427a-aa7f-68fa0e780253" containerName="kube-rbac-proxy"
Sep 18 17:34:31.354371 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:34:31.354370    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d22cfc6-e80f-427a-aa7f-68fa0e780253" containerName="kube-rbac-proxy"
Sep 18 17:34:31.354510 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:34:31.354379    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4872485b-3724-4fb1-ae6b-c8efb41670f5" containerName="kserve-container"
Sep 18 17:34:31.354510 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:34:31.354387    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="4872485b-3724-4fb1-ae6b-c8efb41670f5" containerName="kserve-container"
Sep 18 17:34:31.354510 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:34:31.354396    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d22cfc6-e80f-427a-aa7f-68fa0e780253" containerName="storage-initializer"
Sep 18 17:34:31.354510 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:34:31.354403    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d22cfc6-e80f-427a-aa7f-68fa0e780253" containerName="storage-initializer"
Sep 18 17:34:31.354510 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:34:31.354420    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4872485b-3724-4fb1-ae6b-c8efb41670f5" containerName="kube-rbac-proxy"
Sep 18 17:34:31.354510 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:34:31.354425    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="4872485b-3724-4fb1-ae6b-c8efb41670f5" containerName="kube-rbac-proxy"
Sep 18 17:34:31.354510 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:34:31.354431    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d22cfc6-e80f-427a-aa7f-68fa0e780253" containerName="kserve-container"
Sep 18 17:34:31.354510 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:34:31.354436    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d22cfc6-e80f-427a-aa7f-68fa0e780253" containerName="kserve-container"
Sep 18 17:34:31.354510 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:34:31.354491    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d22cfc6-e80f-427a-aa7f-68fa0e780253" containerName="kube-rbac-proxy"
Sep 18 17:34:31.354510 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:34:31.354500    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="4872485b-3724-4fb1-ae6b-c8efb41670f5" containerName="kube-rbac-proxy"
Sep 18 17:34:31.354510 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:34:31.354507    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="4872485b-3724-4fb1-ae6b-c8efb41670f5" containerName="kserve-container"
Sep 18 17:34:31.354841 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:34:31.354517    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d22cfc6-e80f-427a-aa7f-68fa0e780253" containerName="kserve-container"
Sep 18 17:34:31.356722 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:34:31.356705    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-kb5h6"
Sep 18 17:34:31.359435 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:34:31.359413    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-kube-rbac-proxy-sar-config\""
Sep 18 17:34:31.359562 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:34:31.359423    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-predictor-serving-cert\""
Sep 18 17:34:31.374767 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:34:31.374730    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-kb5h6"]
Sep 18 17:34:31.437088 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:34:31.437047    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c047619-6f56-4e5e-b697-23be03a5f440-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-kb5h6\" (UID: \"3c047619-6f56-4e5e-b697-23be03a5f440\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-kb5h6"
Sep 18 17:34:31.437088 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:34:31.437105    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c047619-6f56-4e5e-b697-23be03a5f440-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-kb5h6\" (UID: \"3c047619-6f56-4e5e-b697-23be03a5f440\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-kb5h6"
Sep 18 17:34:31.437407 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:34:31.437131    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3c047619-6f56-4e5e-b697-23be03a5f440-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-kb5h6\" (UID: \"3c047619-6f56-4e5e-b697-23be03a5f440\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-kb5h6"
Sep 18 17:34:31.437407 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:34:31.437248    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3c047619-6f56-4e5e-b697-23be03a5f440-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-kb5h6\" (UID: \"3c047619-6f56-4e5e-b697-23be03a5f440\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-kb5h6"
Sep 18 17:34:31.437407 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:34:31.437346    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ccfh\" (UniqueName: \"kubernetes.io/projected/3c047619-6f56-4e5e-b697-23be03a5f440-kube-api-access-5ccfh\") pod \"isvc-xgboost-predictor-7566d65bf7-kb5h6\" (UID: \"3c047619-6f56-4e5e-b697-23be03a5f440\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-kb5h6"
Sep 18 17:34:31.538468 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:34:31.538428    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c047619-6f56-4e5e-b697-23be03a5f440-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-kb5h6\" (UID: \"3c047619-6f56-4e5e-b697-23be03a5f440\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-kb5h6"
Sep 18 17:34:31.538468 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:34:31.538476    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c047619-6f56-4e5e-b697-23be03a5f440-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-kb5h6\" (UID: \"3c047619-6f56-4e5e-b697-23be03a5f440\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-kb5h6"
Sep 18 17:34:31.538713 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:34:31.538599    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3c047619-6f56-4e5e-b697-23be03a5f440-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-kb5h6\" (UID: \"3c047619-6f56-4e5e-b697-23be03a5f440\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-kb5h6"
Sep 18 17:34:31.538713 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:34:31.538671    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3c047619-6f56-4e5e-b697-23be03a5f440-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-kb5h6\" (UID: \"3c047619-6f56-4e5e-b697-23be03a5f440\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-kb5h6"
Sep 18 17:34:31.538791 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:34:31.538761    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ccfh\" (UniqueName: \"kubernetes.io/projected/3c047619-6f56-4e5e-b697-23be03a5f440-kube-api-access-5ccfh\") pod \"isvc-xgboost-predictor-7566d65bf7-kb5h6\" (UID: \"3c047619-6f56-4e5e-b697-23be03a5f440\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-kb5h6"
Sep 18 17:34:31.538894 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:34:31.538874    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c047619-6f56-4e5e-b697-23be03a5f440-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-kb5h6\" (UID: \"3c047619-6f56-4e5e-b697-23be03a5f440\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-kb5h6"
Sep 18 17:34:31.539327 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:34:31.539305    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3c047619-6f56-4e5e-b697-23be03a5f440-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-kb5h6\" (UID: \"3c047619-6f56-4e5e-b697-23be03a5f440\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-kb5h6"
Sep 18 17:34:31.539327 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:34:31.539319    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3c047619-6f56-4e5e-b697-23be03a5f440-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-kb5h6\" (UID: \"3c047619-6f56-4e5e-b697-23be03a5f440\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-kb5h6"
Sep 18 17:34:31.541114 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:34:31.541091    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c047619-6f56-4e5e-b697-23be03a5f440-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-kb5h6\" (UID: \"3c047619-6f56-4e5e-b697-23be03a5f440\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-kb5h6"
Sep 18 17:34:31.548032 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:34:31.548003    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ccfh\" (UniqueName: \"kubernetes.io/projected/3c047619-6f56-4e5e-b697-23be03a5f440-kube-api-access-5ccfh\") pod \"isvc-xgboost-predictor-7566d65bf7-kb5h6\" (UID: \"3c047619-6f56-4e5e-b697-23be03a5f440\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-kb5h6"
Sep 18 17:34:31.666875 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:34:31.666783    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-kb5h6"
Sep 18 17:34:31.948455 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:34:31.948430    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-kb5h6"]
Sep 18 17:34:31.951056 ip-10-0-143-5 kubenswrapper[2881]: W0918 17:34:31.951026    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c047619_6f56_4e5e_b697_23be03a5f440.slice/crio-ffe5eebd9f8e2f0e8ae2afb34e621d860d0386b9dfb90af00ed135ae1a2b7e3c WatchSource:0}: Error finding container ffe5eebd9f8e2f0e8ae2afb34e621d860d0386b9dfb90af00ed135ae1a2b7e3c: Status 404 returned error can't find the container with id ffe5eebd9f8e2f0e8ae2afb34e621d860d0386b9dfb90af00ed135ae1a2b7e3c
Sep 18 17:34:32.069770 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:34:32.069725    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-kb5h6" event={"ID":"3c047619-6f56-4e5e-b697-23be03a5f440","Type":"ContainerStarted","Data":"ebaa4d9b69e057e1f767495e42d26a00e83f498bc32327d0eeefec4a240b220e"}
Sep 18 17:34:32.069979 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:34:32.069808    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-kb5h6" event={"ID":"3c047619-6f56-4e5e-b697-23be03a5f440","Type":"ContainerStarted","Data":"ffe5eebd9f8e2f0e8ae2afb34e621d860d0386b9dfb90af00ed135ae1a2b7e3c"}
Sep 18 17:34:36.092199 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:34:36.092166    2881 generic.go:356] "Generic (PLEG): container finished" podID="3c047619-6f56-4e5e-b697-23be03a5f440" containerID="ebaa4d9b69e057e1f767495e42d26a00e83f498bc32327d0eeefec4a240b220e" exitCode=0
Sep 18 17:34:36.092603 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:34:36.092242    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-kb5h6" event={"ID":"3c047619-6f56-4e5e-b697-23be03a5f440","Type":"ContainerDied","Data":"ebaa4d9b69e057e1f767495e42d26a00e83f498bc32327d0eeefec4a240b220e"}
Sep 18 17:34:56.196669 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:34:56.196630    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-kb5h6" event={"ID":"3c047619-6f56-4e5e-b697-23be03a5f440","Type":"ContainerStarted","Data":"2d1d30f8b6031a2440a90efbada46d9e395f1773aee2116e653ea4b8d2cd16ab"}
Sep 18 17:34:57.203363 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:34:57.203322    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-kb5h6" event={"ID":"3c047619-6f56-4e5e-b697-23be03a5f440","Type":"ContainerStarted","Data":"f49f76cb328a0fe10df457372e4cfb771c8ba4fd5211b939907ceb549ca11323"}
Sep 18 17:34:57.204832 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:34:57.203512    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-kb5h6"
Sep 18 17:34:57.228265 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:34:57.228194    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-kb5h6" podStartSLOduration=2.506178991 podStartE2EDuration="26.228175534s" totalImagesPullingTime="20.023884645s" totalInitContainerRuntime="3.698111898s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:34:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:34:57.226303862 +0000 UTC m=+2487.426159149" watchObservedRunningTime="2026-09-18 17:34:57.228175534 +0000 UTC m=+2487.428030820"
Sep 18 17:34:58.207526 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:34:58.207491    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-kb5h6"
Sep 18 17:34:58.208828 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:34:58.208797    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-kb5h6" podUID="3c047619-6f56-4e5e-b697-23be03a5f440" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 18 17:34:59.212116 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:34:59.212073    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-kb5h6" podUID="3c047619-6f56-4e5e-b697-23be03a5f440" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 18 17:35:04.216262 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:04.216214    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-kb5h6"
Sep 18 17:35:04.216706 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:04.216682    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-kb5h6" podUID="3c047619-6f56-4e5e-b697-23be03a5f440" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 18 17:35:14.216887 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:14.216843    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-kb5h6" podUID="3c047619-6f56-4e5e-b697-23be03a5f440" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 18 17:35:24.217268 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:24.217204    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-kb5h6" podUID="3c047619-6f56-4e5e-b697-23be03a5f440" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 18 17:35:34.216682 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:34.216644    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-kb5h6" podUID="3c047619-6f56-4e5e-b697-23be03a5f440" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 18 17:35:38.506492 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:38.506457    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-sgvlv"]
Sep 18 17:35:38.506887 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:38.506740    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-sgvlv" podUID="0bb3352b-d853-44aa-a02d-f0a0fe8bc270" containerName="kserve-container" containerID="cri-o://5273b60fef4e540b67fc6d12cf5e9bfef8e90458e7315bd4ada7da88c80fb184" gracePeriod=30
Sep 18 17:35:38.506887 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:38.506812    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-sgvlv" podUID="0bb3352b-d853-44aa-a02d-f0a0fe8bc270" containerName="kube-rbac-proxy" containerID="cri-o://0ac67ce23db47ca9c5e2cd5794d9c0ea7eacc8a99c8a8d6cb5558a32fd7ccbd2" gracePeriod=30
Sep 18 17:35:39.391070 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:39.391033    2881 generic.go:356] "Generic (PLEG): container finished" podID="0bb3352b-d853-44aa-a02d-f0a0fe8bc270" containerID="0ac67ce23db47ca9c5e2cd5794d9c0ea7eacc8a99c8a8d6cb5558a32fd7ccbd2" exitCode=2
Sep 18 17:35:39.391272 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:39.391087    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-sgvlv" event={"ID":"0bb3352b-d853-44aa-a02d-f0a0fe8bc270","Type":"ContainerDied","Data":"0ac67ce23db47ca9c5e2cd5794d9c0ea7eacc8a99c8a8d6cb5558a32fd7ccbd2"}
Sep 18 17:35:40.333423 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:40.333379    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-sgvlv" podUID="0bb3352b-d853-44aa-a02d-f0a0fe8bc270" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.47:8643/healthz\": dial tcp 10.134.0.47:8643: connect: connection refused"
Sep 18 17:35:43.713901 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:43.713864    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-nzh9s"]
Sep 18 17:35:43.716688 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:43.716670    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-nzh9s"
Sep 18 17:35:43.719298 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:43.719277    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Sep 18 17:35:43.719298 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:43.719286    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 18 17:35:43.729684 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:43.729658    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-nzh9s"]
Sep 18 17:35:43.748312 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:43.748285    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-88zvb"]
Sep 18 17:35:43.752057 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:43.752038    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-88zvb"
Sep 18 17:35:43.754807 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:43.754789    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 18 17:35:43.763712 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:43.763685    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-88zvb"]
Sep 18 17:35:43.799643 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:43.799614    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq49n\" (UniqueName: \"kubernetes.io/projected/303d1a24-b976-481f-9e07-bdf80d8abd68-kube-api-access-kq49n\") pod \"isvc-canary-stop-v2-predictor-c697b4d69-nzh9s\" (UID: \"303d1a24-b976-481f-9e07-bdf80d8abd68\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-nzh9s"
Sep 18 17:35:43.799823 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:43.799645    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/303d1a24-b976-481f-9e07-bdf80d8abd68-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-c697b4d69-nzh9s\" (UID: \"303d1a24-b976-481f-9e07-bdf80d8abd68\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-nzh9s"
Sep 18 17:35:43.799823 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:43.799664    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2n4l\" (UniqueName: \"kubernetes.io/projected/df0424ee-7af5-46a2-a907-30c65f68206b-kube-api-access-x2n4l\") pod \"isvc-canary-stop-predictor-6958d6c864-88zvb\" (UID: \"df0424ee-7af5-46a2-a907-30c65f68206b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-88zvb"
Sep 18 17:35:43.799823 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:43.799778    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df0424ee-7af5-46a2-a907-30c65f68206b-cabundle-cert\") pod \"isvc-canary-stop-predictor-6958d6c864-88zvb\" (UID: \"df0424ee-7af5-46a2-a907-30c65f68206b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-88zvb"
Sep 18 17:35:43.800006 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:43.799823    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df0424ee-7af5-46a2-a907-30c65f68206b-kserve-provision-location\") pod \"isvc-canary-stop-predictor-6958d6c864-88zvb\" (UID: \"df0424ee-7af5-46a2-a907-30c65f68206b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-88zvb"
Sep 18 17:35:43.800006 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:43.799895    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/303d1a24-b976-481f-9e07-bdf80d8abd68-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-c697b4d69-nzh9s\" (UID: \"303d1a24-b976-481f-9e07-bdf80d8abd68\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-nzh9s"
Sep 18 17:35:43.800006 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:43.799937    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/303d1a24-b976-481f-9e07-bdf80d8abd68-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-c697b4d69-nzh9s\" (UID: \"303d1a24-b976-481f-9e07-bdf80d8abd68\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-nzh9s"
Sep 18 17:35:43.800006 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:43.799960    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df0424ee-7af5-46a2-a907-30c65f68206b-proxy-tls\") pod \"isvc-canary-stop-predictor-6958d6c864-88zvb\" (UID: \"df0424ee-7af5-46a2-a907-30c65f68206b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-88zvb"
Sep 18 17:35:43.800207 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:43.800020    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df0424ee-7af5-46a2-a907-30c65f68206b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-6958d6c864-88zvb\" (UID: \"df0424ee-7af5-46a2-a907-30c65f68206b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-88zvb"
Sep 18 17:35:43.800207 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:43.800102    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/303d1a24-b976-481f-9e07-bdf80d8abd68-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-c697b4d69-nzh9s\" (UID: \"303d1a24-b976-481f-9e07-bdf80d8abd68\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-nzh9s"
Sep 18 17:35:43.901291 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:43.901257    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/303d1a24-b976-481f-9e07-bdf80d8abd68-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-c697b4d69-nzh9s\" (UID: \"303d1a24-b976-481f-9e07-bdf80d8abd68\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-nzh9s"
Sep 18 17:35:43.901474 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:43.901304    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/303d1a24-b976-481f-9e07-bdf80d8abd68-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-c697b4d69-nzh9s\" (UID: \"303d1a24-b976-481f-9e07-bdf80d8abd68\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-nzh9s"
Sep 18 17:35:43.901474 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:43.901320    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df0424ee-7af5-46a2-a907-30c65f68206b-proxy-tls\") pod \"isvc-canary-stop-predictor-6958d6c864-88zvb\" (UID: \"df0424ee-7af5-46a2-a907-30c65f68206b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-88zvb"
Sep 18 17:35:43.901474 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:43.901351    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df0424ee-7af5-46a2-a907-30c65f68206b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-6958d6c864-88zvb\" (UID: \"df0424ee-7af5-46a2-a907-30c65f68206b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-88zvb"
Sep 18 17:35:43.901474 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:43.901387    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/303d1a24-b976-481f-9e07-bdf80d8abd68-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-c697b4d69-nzh9s\" (UID: \"303d1a24-b976-481f-9e07-bdf80d8abd68\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-nzh9s"
Sep 18 17:35:43.901474 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:43.901427    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq49n\" (UniqueName: \"kubernetes.io/projected/303d1a24-b976-481f-9e07-bdf80d8abd68-kube-api-access-kq49n\") pod \"isvc-canary-stop-v2-predictor-c697b4d69-nzh9s\" (UID: \"303d1a24-b976-481f-9e07-bdf80d8abd68\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-nzh9s"
Sep 18 17:35:43.901474 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:43.901451    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/303d1a24-b976-481f-9e07-bdf80d8abd68-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-c697b4d69-nzh9s\" (UID: \"303d1a24-b976-481f-9e07-bdf80d8abd68\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-nzh9s"
Sep 18 17:35:43.901474 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:43.901475    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2n4l\" (UniqueName: \"kubernetes.io/projected/df0424ee-7af5-46a2-a907-30c65f68206b-kube-api-access-x2n4l\") pod \"isvc-canary-stop-predictor-6958d6c864-88zvb\" (UID: \"df0424ee-7af5-46a2-a907-30c65f68206b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-88zvb"
Sep 18 17:35:43.901847 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:35:43.901497    2881 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-predictor-serving-cert: secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 18 17:35:43.901847 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:43.901530    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df0424ee-7af5-46a2-a907-30c65f68206b-cabundle-cert\") pod \"isvc-canary-stop-predictor-6958d6c864-88zvb\" (UID: \"df0424ee-7af5-46a2-a907-30c65f68206b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-88zvb"
Sep 18 17:35:43.901847 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:35:43.901573    2881 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df0424ee-7af5-46a2-a907-30c65f68206b-proxy-tls podName:df0424ee-7af5-46a2-a907-30c65f68206b nodeName:}" failed. No retries permitted until 2026-09-18 17:35:44.401551645 +0000 UTC m=+2534.601406927 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/df0424ee-7af5-46a2-a907-30c65f68206b-proxy-tls") pod "isvc-canary-stop-predictor-6958d6c864-88zvb" (UID: "df0424ee-7af5-46a2-a907-30c65f68206b") : secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 18 17:35:43.901847 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:43.901591    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df0424ee-7af5-46a2-a907-30c65f68206b-kserve-provision-location\") pod \"isvc-canary-stop-predictor-6958d6c864-88zvb\" (UID: \"df0424ee-7af5-46a2-a907-30c65f68206b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-88zvb"
Sep 18 17:35:43.902081 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:43.901889    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/303d1a24-b976-481f-9e07-bdf80d8abd68-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-c697b4d69-nzh9s\" (UID: \"303d1a24-b976-481f-9e07-bdf80d8abd68\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-nzh9s"
Sep 18 17:35:43.902169 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:43.902137    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/303d1a24-b976-481f-9e07-bdf80d8abd68-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-c697b4d69-nzh9s\" (UID: \"303d1a24-b976-481f-9e07-bdf80d8abd68\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-nzh9s"
Sep 18 17:35:43.902259 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:43.902183    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df0424ee-7af5-46a2-a907-30c65f68206b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-6958d6c864-88zvb\" (UID: \"df0424ee-7af5-46a2-a907-30c65f68206b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-88zvb"
Sep 18 17:35:43.902363 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:43.902342    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df0424ee-7af5-46a2-a907-30c65f68206b-cabundle-cert\") pod \"isvc-canary-stop-predictor-6958d6c864-88zvb\" (UID: \"df0424ee-7af5-46a2-a907-30c65f68206b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-88zvb"
Sep 18 17:35:43.902436 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:43.902380    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/303d1a24-b976-481f-9e07-bdf80d8abd68-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-c697b4d69-nzh9s\" (UID: \"303d1a24-b976-481f-9e07-bdf80d8abd68\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-nzh9s"
Sep 18 17:35:43.902436 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:43.902416    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df0424ee-7af5-46a2-a907-30c65f68206b-kserve-provision-location\") pod \"isvc-canary-stop-predictor-6958d6c864-88zvb\" (UID: \"df0424ee-7af5-46a2-a907-30c65f68206b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-88zvb"
Sep 18 17:35:43.904414 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:43.904394    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/303d1a24-b976-481f-9e07-bdf80d8abd68-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-c697b4d69-nzh9s\" (UID: \"303d1a24-b976-481f-9e07-bdf80d8abd68\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-nzh9s"
Sep 18 17:35:43.910960 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:43.910934    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq49n\" (UniqueName: \"kubernetes.io/projected/303d1a24-b976-481f-9e07-bdf80d8abd68-kube-api-access-kq49n\") pod \"isvc-canary-stop-v2-predictor-c697b4d69-nzh9s\" (UID: \"303d1a24-b976-481f-9e07-bdf80d8abd68\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-nzh9s"
Sep 18 17:35:43.911377 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:43.911358    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2n4l\" (UniqueName: \"kubernetes.io/projected/df0424ee-7af5-46a2-a907-30c65f68206b-kube-api-access-x2n4l\") pod \"isvc-canary-stop-predictor-6958d6c864-88zvb\" (UID: \"df0424ee-7af5-46a2-a907-30c65f68206b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-88zvb"
Sep 18 17:35:43.952828 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:43.952803    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-sgvlv"
Sep 18 17:35:44.002891 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:44.002804    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0bb3352b-d853-44aa-a02d-f0a0fe8bc270-cabundle-cert\") pod \"0bb3352b-d853-44aa-a02d-f0a0fe8bc270\" (UID: \"0bb3352b-d853-44aa-a02d-f0a0fe8bc270\") "
Sep 18 17:35:44.002891 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:44.002843    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0bb3352b-d853-44aa-a02d-f0a0fe8bc270-kserve-provision-location\") pod \"0bb3352b-d853-44aa-a02d-f0a0fe8bc270\" (UID: \"0bb3352b-d853-44aa-a02d-f0a0fe8bc270\") "
Sep 18 17:35:44.003086 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:44.002920    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0bb3352b-d853-44aa-a02d-f0a0fe8bc270-kube-rbac-proxy-sar-config\") pod \"0bb3352b-d853-44aa-a02d-f0a0fe8bc270\" (UID: \"0bb3352b-d853-44aa-a02d-f0a0fe8bc270\") "
Sep 18 17:35:44.003086 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:44.002979    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qcnhn\" (UniqueName: \"kubernetes.io/projected/0bb3352b-d853-44aa-a02d-f0a0fe8bc270-kube-api-access-qcnhn\") pod \"0bb3352b-d853-44aa-a02d-f0a0fe8bc270\" (UID: \"0bb3352b-d853-44aa-a02d-f0a0fe8bc270\") "
Sep 18 17:35:44.003086 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:44.003004    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0bb3352b-d853-44aa-a02d-f0a0fe8bc270-proxy-tls\") pod \"0bb3352b-d853-44aa-a02d-f0a0fe8bc270\" (UID: \"0bb3352b-d853-44aa-a02d-f0a0fe8bc270\") "
Sep 18 17:35:44.003276 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:44.003253    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bb3352b-d853-44aa-a02d-f0a0fe8bc270-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0bb3352b-d853-44aa-a02d-f0a0fe8bc270" (UID: "0bb3352b-d853-44aa-a02d-f0a0fe8bc270"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:35:44.003349 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:44.003297    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bb3352b-d853-44aa-a02d-f0a0fe8bc270-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0bb3352b-d853-44aa-a02d-f0a0fe8bc270" (UID: "0bb3352b-d853-44aa-a02d-f0a0fe8bc270"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:35:44.003349 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:44.003315    2881 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0bb3352b-d853-44aa-a02d-f0a0fe8bc270-kserve-provision-location\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:35:44.003349 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:44.003323    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bb3352b-d853-44aa-a02d-f0a0fe8bc270-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0bb3352b-d853-44aa-a02d-f0a0fe8bc270" (UID: "0bb3352b-d853-44aa-a02d-f0a0fe8bc270"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:35:44.005078 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:44.005059    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bb3352b-d853-44aa-a02d-f0a0fe8bc270-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0bb3352b-d853-44aa-a02d-f0a0fe8bc270" (UID: "0bb3352b-d853-44aa-a02d-f0a0fe8bc270"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:35:44.005133 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:44.005090    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bb3352b-d853-44aa-a02d-f0a0fe8bc270-kube-api-access-qcnhn" (OuterVolumeSpecName: "kube-api-access-qcnhn") pod "0bb3352b-d853-44aa-a02d-f0a0fe8bc270" (UID: "0bb3352b-d853-44aa-a02d-f0a0fe8bc270"). InnerVolumeSpecName "kube-api-access-qcnhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:35:44.027352 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:44.027320    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-nzh9s"
Sep 18 17:35:44.104009 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:44.103977    2881 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcnhn\" (UniqueName: \"kubernetes.io/projected/0bb3352b-d853-44aa-a02d-f0a0fe8bc270-kube-api-access-qcnhn\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:35:44.104009 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:44.104005    2881 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0bb3352b-d853-44aa-a02d-f0a0fe8bc270-proxy-tls\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:35:44.104189 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:44.104017    2881 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0bb3352b-d853-44aa-a02d-f0a0fe8bc270-cabundle-cert\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:35:44.104189 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:44.104027    2881 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0bb3352b-d853-44aa-a02d-f0a0fe8bc270-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:35:44.217167 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:44.217127    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-kb5h6" podUID="3c047619-6f56-4e5e-b697-23be03a5f440" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 18 17:35:44.303850 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:44.303825    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-nzh9s"]
Sep 18 17:35:44.306354 ip-10-0-143-5 kubenswrapper[2881]: W0918 17:35:44.306326    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod303d1a24_b976_481f_9e07_bdf80d8abd68.slice/crio-7e7806f77596d23a254bf6ee5876ae24c6fef34eab17106615717485664c6876 WatchSource:0}: Error finding container 7e7806f77596d23a254bf6ee5876ae24c6fef34eab17106615717485664c6876: Status 404 returned error can't find the container with id 7e7806f77596d23a254bf6ee5876ae24c6fef34eab17106615717485664c6876
Sep 18 17:35:44.406097 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:44.406049    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df0424ee-7af5-46a2-a907-30c65f68206b-proxy-tls\") pod \"isvc-canary-stop-predictor-6958d6c864-88zvb\" (UID: \"df0424ee-7af5-46a2-a907-30c65f68206b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-88zvb"
Sep 18 17:35:44.408419 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:44.408392    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df0424ee-7af5-46a2-a907-30c65f68206b-proxy-tls\") pod \"isvc-canary-stop-predictor-6958d6c864-88zvb\" (UID: \"df0424ee-7af5-46a2-a907-30c65f68206b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-88zvb"
Sep 18 17:35:44.413958 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:44.413934    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-nzh9s" event={"ID":"303d1a24-b976-481f-9e07-bdf80d8abd68","Type":"ContainerStarted","Data":"c9ce3d2d128d0ee5bcb1883e855d0390e1154f9dfa8f92f199ae5fa350ae776d"}
Sep 18 17:35:44.414091 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:44.413997    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-nzh9s" event={"ID":"303d1a24-b976-481f-9e07-bdf80d8abd68","Type":"ContainerStarted","Data":"7e7806f77596d23a254bf6ee5876ae24c6fef34eab17106615717485664c6876"}
Sep 18 17:35:44.415958 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:44.415926    2881 generic.go:356] "Generic (PLEG): container finished" podID="0bb3352b-d853-44aa-a02d-f0a0fe8bc270" containerID="5273b60fef4e540b67fc6d12cf5e9bfef8e90458e7315bd4ada7da88c80fb184" exitCode=0
Sep 18 17:35:44.416074 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:44.415996    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-sgvlv"
Sep 18 17:35:44.416146 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:44.415997    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-sgvlv" event={"ID":"0bb3352b-d853-44aa-a02d-f0a0fe8bc270","Type":"ContainerDied","Data":"5273b60fef4e540b67fc6d12cf5e9bfef8e90458e7315bd4ada7da88c80fb184"}
Sep 18 17:35:44.416146 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:44.416124    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-sgvlv" event={"ID":"0bb3352b-d853-44aa-a02d-f0a0fe8bc270","Type":"ContainerDied","Data":"64935df2a1ddbda04e8e3ac69cde1fb4c5200a32d1bf49bf86e044333d316df8"}
Sep 18 17:35:44.416287 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:44.416154    2881 scope.go:117] "RemoveContainer" containerID="0ac67ce23db47ca9c5e2cd5794d9c0ea7eacc8a99c8a8d6cb5558a32fd7ccbd2"
Sep 18 17:35:44.425289 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:44.425269    2881 scope.go:117] "RemoveContainer" containerID="5273b60fef4e540b67fc6d12cf5e9bfef8e90458e7315bd4ada7da88c80fb184"
Sep 18 17:35:44.433116 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:44.433092    2881 scope.go:117] "RemoveContainer" containerID="d4221aee007f2c83da83d012520b848cb1d310dc81c6c9030a2a962be95f6c8a"
Sep 18 17:35:44.442593 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:44.442571    2881 scope.go:117] "RemoveContainer" containerID="0ac67ce23db47ca9c5e2cd5794d9c0ea7eacc8a99c8a8d6cb5558a32fd7ccbd2"
Sep 18 17:35:44.442938 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:35:44.442915    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ac67ce23db47ca9c5e2cd5794d9c0ea7eacc8a99c8a8d6cb5558a32fd7ccbd2\": container with ID starting with 0ac67ce23db47ca9c5e2cd5794d9c0ea7eacc8a99c8a8d6cb5558a32fd7ccbd2 not found: ID does not exist" containerID="0ac67ce23db47ca9c5e2cd5794d9c0ea7eacc8a99c8a8d6cb5558a32fd7ccbd2"
Sep 18 17:35:44.443014 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:44.442945    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ac67ce23db47ca9c5e2cd5794d9c0ea7eacc8a99c8a8d6cb5558a32fd7ccbd2"} err="failed to get container status \"0ac67ce23db47ca9c5e2cd5794d9c0ea7eacc8a99c8a8d6cb5558a32fd7ccbd2\": rpc error: code = NotFound desc = could not find container \"0ac67ce23db47ca9c5e2cd5794d9c0ea7eacc8a99c8a8d6cb5558a32fd7ccbd2\": container with ID starting with 0ac67ce23db47ca9c5e2cd5794d9c0ea7eacc8a99c8a8d6cb5558a32fd7ccbd2 not found: ID does not exist"
Sep 18 17:35:44.443014 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:44.442962    2881 scope.go:117] "RemoveContainer" containerID="5273b60fef4e540b67fc6d12cf5e9bfef8e90458e7315bd4ada7da88c80fb184"
Sep 18 17:35:44.443335 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:35:44.443307    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5273b60fef4e540b67fc6d12cf5e9bfef8e90458e7315bd4ada7da88c80fb184\": container with ID starting with 5273b60fef4e540b67fc6d12cf5e9bfef8e90458e7315bd4ada7da88c80fb184 not found: ID does not exist" containerID="5273b60fef4e540b67fc6d12cf5e9bfef8e90458e7315bd4ada7da88c80fb184"
Sep 18 17:35:44.443410 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:44.443344    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5273b60fef4e540b67fc6d12cf5e9bfef8e90458e7315bd4ada7da88c80fb184"} err="failed to get container status \"5273b60fef4e540b67fc6d12cf5e9bfef8e90458e7315bd4ada7da88c80fb184\": rpc error: code = NotFound desc = could not find container \"5273b60fef4e540b67fc6d12cf5e9bfef8e90458e7315bd4ada7da88c80fb184\": container with ID starting with 5273b60fef4e540b67fc6d12cf5e9bfef8e90458e7315bd4ada7da88c80fb184 not found: ID does not exist"
Sep 18 17:35:44.443410 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:44.443363    2881 scope.go:117] "RemoveContainer" containerID="d4221aee007f2c83da83d012520b848cb1d310dc81c6c9030a2a962be95f6c8a"
Sep 18 17:35:44.443632 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:35:44.443608    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4221aee007f2c83da83d012520b848cb1d310dc81c6c9030a2a962be95f6c8a\": container with ID starting with d4221aee007f2c83da83d012520b848cb1d310dc81c6c9030a2a962be95f6c8a not found: ID does not exist" containerID="d4221aee007f2c83da83d012520b848cb1d310dc81c6c9030a2a962be95f6c8a"
Sep 18 17:35:44.443705 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:44.443635    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4221aee007f2c83da83d012520b848cb1d310dc81c6c9030a2a962be95f6c8a"} err="failed to get container status \"d4221aee007f2c83da83d012520b848cb1d310dc81c6c9030a2a962be95f6c8a\": rpc error: code = NotFound desc = could not find container \"d4221aee007f2c83da83d012520b848cb1d310dc81c6c9030a2a962be95f6c8a\": container with ID starting with d4221aee007f2c83da83d012520b848cb1d310dc81c6c9030a2a962be95f6c8a not found: ID does not exist"
Sep 18 17:35:44.460874 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:44.460841    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-sgvlv"]
Sep 18 17:35:44.468568 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:44.468538    2881 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-69978998f6-sgvlv"]
Sep 18 17:35:44.689307 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:44.689194    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-88zvb"
Sep 18 17:35:44.975401 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:44.975319    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-88zvb"]
Sep 18 17:35:44.978673 ip-10-0-143-5 kubenswrapper[2881]: W0918 17:35:44.978643    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf0424ee_7af5_46a2_a907_30c65f68206b.slice/crio-dcf73d5e77bdf1f23c0a28edfcdf6d4acffb52f76a872eb3615f6f9f87abee99 WatchSource:0}: Error finding container dcf73d5e77bdf1f23c0a28edfcdf6d4acffb52f76a872eb3615f6f9f87abee99: Status 404 returned error can't find the container with id dcf73d5e77bdf1f23c0a28edfcdf6d4acffb52f76a872eb3615f6f9f87abee99
Sep 18 17:35:45.423456 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:45.423415    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-88zvb" event={"ID":"df0424ee-7af5-46a2-a907-30c65f68206b","Type":"ContainerStarted","Data":"7de054bae6686448eac9afd6ef2d5ca4edb8ff00546f2f6ad902f371b78f6673"}
Sep 18 17:35:45.423658 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:45.423477    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-88zvb" event={"ID":"df0424ee-7af5-46a2-a907-30c65f68206b","Type":"ContainerStarted","Data":"dcf73d5e77bdf1f23c0a28edfcdf6d4acffb52f76a872eb3615f6f9f87abee99"}
Sep 18 17:35:45.893118 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:45.893083    2881 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bb3352b-d853-44aa-a02d-f0a0fe8bc270" path="/var/lib/kubelet/pods/0bb3352b-d853-44aa-a02d-f0a0fe8bc270/volumes"
Sep 18 17:35:48.440932 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:48.440837    2881 generic.go:356] "Generic (PLEG): container finished" podID="303d1a24-b976-481f-9e07-bdf80d8abd68" containerID="c9ce3d2d128d0ee5bcb1883e855d0390e1154f9dfa8f92f199ae5fa350ae776d" exitCode=0
Sep 18 17:35:48.440932 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:48.440907    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-nzh9s" event={"ID":"303d1a24-b976-481f-9e07-bdf80d8abd68","Type":"ContainerDied","Data":"c9ce3d2d128d0ee5bcb1883e855d0390e1154f9dfa8f92f199ae5fa350ae776d"}
Sep 18 17:35:49.447562 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:49.447529    2881 generic.go:356] "Generic (PLEG): container finished" podID="df0424ee-7af5-46a2-a907-30c65f68206b" containerID="7de054bae6686448eac9afd6ef2d5ca4edb8ff00546f2f6ad902f371b78f6673" exitCode=0
Sep 18 17:35:49.447971 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:49.447597    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-88zvb" event={"ID":"df0424ee-7af5-46a2-a907-30c65f68206b","Type":"ContainerDied","Data":"7de054bae6686448eac9afd6ef2d5ca4edb8ff00546f2f6ad902f371b78f6673"}
Sep 18 17:35:49.450130 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:49.450101    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-nzh9s" event={"ID":"303d1a24-b976-481f-9e07-bdf80d8abd68","Type":"ContainerStarted","Data":"6fab586dacae6434b3bb34f1135d259f4e60b7d7b3dafb5e90df40cfcc1c1459"}
Sep 18 17:35:49.450289 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:49.450168    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-nzh9s" event={"ID":"303d1a24-b976-481f-9e07-bdf80d8abd68","Type":"ContainerStarted","Data":"e53c96b6e393386facde7770945f5d126994a200b23142ccc43a4aef2a6e34ce"}
Sep 18 17:35:49.450499 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:49.450477    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-nzh9s"
Sep 18 17:35:49.450556 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:49.450538    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-nzh9s"
Sep 18 17:35:49.451936 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:49.451911    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-nzh9s" podUID="303d1a24-b976-481f-9e07-bdf80d8abd68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 18 17:35:49.491174 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:49.491107    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-nzh9s" podStartSLOduration=2.806158767 podStartE2EDuration="6.491088467s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.6849297s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:35:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:35:49.490146874 +0000 UTC m=+2539.690002181" watchObservedRunningTime="2026-09-18 17:35:49.491088467 +0000 UTC m=+2539.690943799"
Sep 18 17:35:50.457846 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:50.457803    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-88zvb" event={"ID":"df0424ee-7af5-46a2-a907-30c65f68206b","Type":"ContainerStarted","Data":"ffc4086aa1658abfacbc1db07dbeb2968d676ab0d880929373104b5593f9481e"}
Sep 18 17:35:50.458351 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:50.457865    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-88zvb" event={"ID":"df0424ee-7af5-46a2-a907-30c65f68206b","Type":"ContainerStarted","Data":"d2133e9eee0b804bdcd9e68658fbf1e5b116df91dbd5c6ca23897f5ff1c24c04"}
Sep 18 17:35:50.458351 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:50.458325    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-nzh9s" podUID="303d1a24-b976-481f-9e07-bdf80d8abd68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 18 17:35:50.458489 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:50.458411    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-88zvb"
Sep 18 17:35:50.481186 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:50.481137    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-88zvb" podStartSLOduration=3.564288469 podStartE2EDuration="7.481123382s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.916834913s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:35:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:35:50.479251246 +0000 UTC m=+2540.679106529" watchObservedRunningTime="2026-09-18 17:35:50.481123382 +0000 UTC m=+2540.680978668"
Sep 18 17:35:51.463664 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:51.463628    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-88zvb"
Sep 18 17:35:51.465251 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:51.465210    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-88zvb" podUID="df0424ee-7af5-46a2-a907-30c65f68206b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 18 17:35:52.468716 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:52.468676    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-88zvb" podUID="df0424ee-7af5-46a2-a907-30c65f68206b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 18 17:35:54.217059 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:54.217021    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-kb5h6" podUID="3c047619-6f56-4e5e-b697-23be03a5f440" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 18 17:35:55.463317 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:55.463285    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-nzh9s"
Sep 18 17:35:55.463788 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:55.463764    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-nzh9s" podUID="303d1a24-b976-481f-9e07-bdf80d8abd68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 18 17:35:57.473791 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:57.473762    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-88zvb"
Sep 18 17:35:57.474348 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:35:57.474321    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-88zvb" podUID="df0424ee-7af5-46a2-a907-30c65f68206b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 18 17:36:04.217352 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:04.217261    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-kb5h6" podUID="3c047619-6f56-4e5e-b697-23be03a5f440" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 18 17:36:05.464414 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:05.464367    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-nzh9s" podUID="303d1a24-b976-481f-9e07-bdf80d8abd68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 18 17:36:07.474621 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:07.474580    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-88zvb" podUID="df0424ee-7af5-46a2-a907-30c65f68206b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 18 17:36:09.894000 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:09.893974    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-kb5h6"
Sep 18 17:36:11.879615 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:11.879575    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-kb5h6"]
Sep 18 17:36:11.880114 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:11.879944    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-kb5h6" podUID="3c047619-6f56-4e5e-b697-23be03a5f440" containerName="kserve-container" containerID="cri-o://2d1d30f8b6031a2440a90efbada46d9e395f1773aee2116e653ea4b8d2cd16ab" gracePeriod=30
Sep 18 17:36:11.880114 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:11.879956    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-kb5h6" podUID="3c047619-6f56-4e5e-b697-23be03a5f440" containerName="kube-rbac-proxy" containerID="cri-o://f49f76cb328a0fe10df457372e4cfb771c8ba4fd5211b939907ceb549ca11323" gracePeriod=30
Sep 18 17:36:12.563960 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:12.563927    2881 generic.go:356] "Generic (PLEG): container finished" podID="3c047619-6f56-4e5e-b697-23be03a5f440" containerID="f49f76cb328a0fe10df457372e4cfb771c8ba4fd5211b939907ceb549ca11323" exitCode=2
Sep 18 17:36:12.564153 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:12.564007    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-kb5h6" event={"ID":"3c047619-6f56-4e5e-b697-23be03a5f440","Type":"ContainerDied","Data":"f49f76cb328a0fe10df457372e4cfb771c8ba4fd5211b939907ceb549ca11323"}
Sep 18 17:36:14.213175 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:14.213124    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-kb5h6" podUID="3c047619-6f56-4e5e-b697-23be03a5f440" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.48:8643/healthz\": dial tcp 10.134.0.48:8643: connect: connection refused"
Sep 18 17:36:15.464333 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:15.464289    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-nzh9s" podUID="303d1a24-b976-481f-9e07-bdf80d8abd68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 18 17:36:16.424137 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:16.424111    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-kb5h6"
Sep 18 17:36:16.503339 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:16.503244    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3c047619-6f56-4e5e-b697-23be03a5f440-kube-rbac-proxy-sar-config\") pod \"3c047619-6f56-4e5e-b697-23be03a5f440\" (UID: \"3c047619-6f56-4e5e-b697-23be03a5f440\") "
Sep 18 17:36:16.503339 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:16.503312    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c047619-6f56-4e5e-b697-23be03a5f440-kserve-provision-location\") pod \"3c047619-6f56-4e5e-b697-23be03a5f440\" (UID: \"3c047619-6f56-4e5e-b697-23be03a5f440\") "
Sep 18 17:36:16.503866 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:16.503359    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3c047619-6f56-4e5e-b697-23be03a5f440-cabundle-cert\") pod \"3c047619-6f56-4e5e-b697-23be03a5f440\" (UID: \"3c047619-6f56-4e5e-b697-23be03a5f440\") "
Sep 18 17:36:16.503866 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:16.503384    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c047619-6f56-4e5e-b697-23be03a5f440-proxy-tls\") pod \"3c047619-6f56-4e5e-b697-23be03a5f440\" (UID: \"3c047619-6f56-4e5e-b697-23be03a5f440\") "
Sep 18 17:36:16.503866 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:16.503402    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5ccfh\" (UniqueName: \"kubernetes.io/projected/3c047619-6f56-4e5e-b697-23be03a5f440-kube-api-access-5ccfh\") pod \"3c047619-6f56-4e5e-b697-23be03a5f440\" (UID: \"3c047619-6f56-4e5e-b697-23be03a5f440\") "
Sep 18 17:36:16.503866 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:16.503681    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c047619-6f56-4e5e-b697-23be03a5f440-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3c047619-6f56-4e5e-b697-23be03a5f440" (UID: "3c047619-6f56-4e5e-b697-23be03a5f440"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:36:16.503866 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:16.503722    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c047619-6f56-4e5e-b697-23be03a5f440-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3c047619-6f56-4e5e-b697-23be03a5f440" (UID: "3c047619-6f56-4e5e-b697-23be03a5f440"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:36:16.503866 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:16.503781    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c047619-6f56-4e5e-b697-23be03a5f440-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3c047619-6f56-4e5e-b697-23be03a5f440" (UID: "3c047619-6f56-4e5e-b697-23be03a5f440"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:36:16.505523 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:16.505502    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c047619-6f56-4e5e-b697-23be03a5f440-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3c047619-6f56-4e5e-b697-23be03a5f440" (UID: "3c047619-6f56-4e5e-b697-23be03a5f440"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:36:16.505614 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:16.505589    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c047619-6f56-4e5e-b697-23be03a5f440-kube-api-access-5ccfh" (OuterVolumeSpecName: "kube-api-access-5ccfh") pod "3c047619-6f56-4e5e-b697-23be03a5f440" (UID: "3c047619-6f56-4e5e-b697-23be03a5f440"). InnerVolumeSpecName "kube-api-access-5ccfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:36:16.584164 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:16.584128    2881 generic.go:356] "Generic (PLEG): container finished" podID="3c047619-6f56-4e5e-b697-23be03a5f440" containerID="2d1d30f8b6031a2440a90efbada46d9e395f1773aee2116e653ea4b8d2cd16ab" exitCode=0
Sep 18 17:36:16.584356 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:16.584188    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-kb5h6"
Sep 18 17:36:16.584356 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:16.584211    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-kb5h6" event={"ID":"3c047619-6f56-4e5e-b697-23be03a5f440","Type":"ContainerDied","Data":"2d1d30f8b6031a2440a90efbada46d9e395f1773aee2116e653ea4b8d2cd16ab"}
Sep 18 17:36:16.584356 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:16.584308    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-kb5h6" event={"ID":"3c047619-6f56-4e5e-b697-23be03a5f440","Type":"ContainerDied","Data":"ffe5eebd9f8e2f0e8ae2afb34e621d860d0386b9dfb90af00ed135ae1a2b7e3c"}
Sep 18 17:36:16.584356 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:16.584337    2881 scope.go:117] "RemoveContainer" containerID="f49f76cb328a0fe10df457372e4cfb771c8ba4fd5211b939907ceb549ca11323"
Sep 18 17:36:16.593577 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:16.593561    2881 scope.go:117] "RemoveContainer" containerID="2d1d30f8b6031a2440a90efbada46d9e395f1773aee2116e653ea4b8d2cd16ab"
Sep 18 17:36:16.601356 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:16.601331    2881 scope.go:117] "RemoveContainer" containerID="ebaa4d9b69e057e1f767495e42d26a00e83f498bc32327d0eeefec4a240b220e"
Sep 18 17:36:16.604825 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:16.604800    2881 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3c047619-6f56-4e5e-b697-23be03a5f440-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:36:16.604925 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:16.604831    2881 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c047619-6f56-4e5e-b697-23be03a5f440-kserve-provision-location\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:36:16.604925 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:16.604851    2881 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3c047619-6f56-4e5e-b697-23be03a5f440-cabundle-cert\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:36:16.604925 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:16.604864    2881 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c047619-6f56-4e5e-b697-23be03a5f440-proxy-tls\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:36:16.604925 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:16.604877    2881 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ccfh\" (UniqueName: \"kubernetes.io/projected/3c047619-6f56-4e5e-b697-23be03a5f440-kube-api-access-5ccfh\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:36:16.613875 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:16.610563    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-kb5h6"]
Sep 18 17:36:16.613875 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:16.612736    2881 scope.go:117] "RemoveContainer" containerID="f49f76cb328a0fe10df457372e4cfb771c8ba4fd5211b939907ceb549ca11323"
Sep 18 17:36:16.614183 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:36:16.613866    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f49f76cb328a0fe10df457372e4cfb771c8ba4fd5211b939907ceb549ca11323\": container with ID starting with f49f76cb328a0fe10df457372e4cfb771c8ba4fd5211b939907ceb549ca11323 not found: ID does not exist" containerID="f49f76cb328a0fe10df457372e4cfb771c8ba4fd5211b939907ceb549ca11323"
Sep 18 17:36:16.614183 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:16.613916    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f49f76cb328a0fe10df457372e4cfb771c8ba4fd5211b939907ceb549ca11323"} err="failed to get container status \"f49f76cb328a0fe10df457372e4cfb771c8ba4fd5211b939907ceb549ca11323\": rpc error: code = NotFound desc = could not find container \"f49f76cb328a0fe10df457372e4cfb771c8ba4fd5211b939907ceb549ca11323\": container with ID starting with f49f76cb328a0fe10df457372e4cfb771c8ba4fd5211b939907ceb549ca11323 not found: ID does not exist"
Sep 18 17:36:16.614183 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:16.613942    2881 scope.go:117] "RemoveContainer" containerID="2d1d30f8b6031a2440a90efbada46d9e395f1773aee2116e653ea4b8d2cd16ab"
Sep 18 17:36:16.614754 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:36:16.614712    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d1d30f8b6031a2440a90efbada46d9e395f1773aee2116e653ea4b8d2cd16ab\": container with ID starting with 2d1d30f8b6031a2440a90efbada46d9e395f1773aee2116e653ea4b8d2cd16ab not found: ID does not exist" containerID="2d1d30f8b6031a2440a90efbada46d9e395f1773aee2116e653ea4b8d2cd16ab"
Sep 18 17:36:16.614866 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:16.614761    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d1d30f8b6031a2440a90efbada46d9e395f1773aee2116e653ea4b8d2cd16ab"} err="failed to get container status \"2d1d30f8b6031a2440a90efbada46d9e395f1773aee2116e653ea4b8d2cd16ab\": rpc error: code = NotFound desc = could not find container \"2d1d30f8b6031a2440a90efbada46d9e395f1773aee2116e653ea4b8d2cd16ab\": container with ID starting with 2d1d30f8b6031a2440a90efbada46d9e395f1773aee2116e653ea4b8d2cd16ab not found: ID does not exist"
Sep 18 17:36:16.614866 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:16.614784    2881 scope.go:117] "RemoveContainer" containerID="ebaa4d9b69e057e1f767495e42d26a00e83f498bc32327d0eeefec4a240b220e"
Sep 18 17:36:16.614998 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:16.614975    2881 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-kb5h6"]
Sep 18 17:36:16.615856 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:36:16.615831    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebaa4d9b69e057e1f767495e42d26a00e83f498bc32327d0eeefec4a240b220e\": container with ID starting with ebaa4d9b69e057e1f767495e42d26a00e83f498bc32327d0eeefec4a240b220e not found: ID does not exist" containerID="ebaa4d9b69e057e1f767495e42d26a00e83f498bc32327d0eeefec4a240b220e"
Sep 18 17:36:16.616009 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:16.615858    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebaa4d9b69e057e1f767495e42d26a00e83f498bc32327d0eeefec4a240b220e"} err="failed to get container status \"ebaa4d9b69e057e1f767495e42d26a00e83f498bc32327d0eeefec4a240b220e\": rpc error: code = NotFound desc = could not find container \"ebaa4d9b69e057e1f767495e42d26a00e83f498bc32327d0eeefec4a240b220e\": container with ID starting with ebaa4d9b69e057e1f767495e42d26a00e83f498bc32327d0eeefec4a240b220e not found: ID does not exist"
Sep 18 17:36:17.146487 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:17.146454    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-8s6wz"]
Sep 18 17:36:17.146927 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:17.146907    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bb3352b-d853-44aa-a02d-f0a0fe8bc270" containerName="kube-rbac-proxy"
Sep 18 17:36:17.146998 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:17.146930    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bb3352b-d853-44aa-a02d-f0a0fe8bc270" containerName="kube-rbac-proxy"
Sep 18 17:36:17.146998 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:17.146938    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c047619-6f56-4e5e-b697-23be03a5f440" containerName="kube-rbac-proxy"
Sep 18 17:36:17.146998 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:17.146943    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c047619-6f56-4e5e-b697-23be03a5f440" containerName="kube-rbac-proxy"
Sep 18 17:36:17.146998 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:17.146961    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c047619-6f56-4e5e-b697-23be03a5f440" containerName="storage-initializer"
Sep 18 17:36:17.146998 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:17.146967    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c047619-6f56-4e5e-b697-23be03a5f440" containerName="storage-initializer"
Sep 18 17:36:17.146998 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:17.146978    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c047619-6f56-4e5e-b697-23be03a5f440" containerName="kserve-container"
Sep 18 17:36:17.146998 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:17.146984    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c047619-6f56-4e5e-b697-23be03a5f440" containerName="kserve-container"
Sep 18 17:36:17.146998 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:17.147000    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bb3352b-d853-44aa-a02d-f0a0fe8bc270" containerName="storage-initializer"
Sep 18 17:36:17.147256 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:17.147006    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bb3352b-d853-44aa-a02d-f0a0fe8bc270" containerName="storage-initializer"
Sep 18 17:36:17.147256 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:17.147012    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bb3352b-d853-44aa-a02d-f0a0fe8bc270" containerName="kserve-container"
Sep 18 17:36:17.147256 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:17.147018    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bb3352b-d853-44aa-a02d-f0a0fe8bc270" containerName="kserve-container"
Sep 18 17:36:17.147256 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:17.147073    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bb3352b-d853-44aa-a02d-f0a0fe8bc270" containerName="kube-rbac-proxy"
Sep 18 17:36:17.147256 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:17.147079    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c047619-6f56-4e5e-b697-23be03a5f440" containerName="kube-rbac-proxy"
Sep 18 17:36:17.147256 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:17.147086    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bb3352b-d853-44aa-a02d-f0a0fe8bc270" containerName="kserve-container"
Sep 18 17:36:17.147256 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:17.147092    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c047619-6f56-4e5e-b697-23be03a5f440" containerName="kserve-container"
Sep 18 17:36:17.150638 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:17.150612    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-8s6wz"
Sep 18 17:36:17.153859 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:17.153838    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-predictor-serving-cert\""
Sep 18 17:36:17.154308 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:17.154290    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 18 17:36:17.170478 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:17.170452    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-8s6wz"]
Sep 18 17:36:17.208399 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:17.208361    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/72f5373a-ce4f-476b-a835-3b7a4cb9122e-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-8s6wz\" (UID: \"72f5373a-ce4f-476b-a835-3b7a4cb9122e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-8s6wz"
Sep 18 17:36:17.208571 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:17.208412    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwbmx\" (UniqueName: \"kubernetes.io/projected/72f5373a-ce4f-476b-a835-3b7a4cb9122e-kube-api-access-gwbmx\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-8s6wz\" (UID: \"72f5373a-ce4f-476b-a835-3b7a4cb9122e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-8s6wz"
Sep 18 17:36:17.208571 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:17.208440    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72f5373a-ce4f-476b-a835-3b7a4cb9122e-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-8s6wz\" (UID: \"72f5373a-ce4f-476b-a835-3b7a4cb9122e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-8s6wz"
Sep 18 17:36:17.208571 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:17.208470    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/72f5373a-ce4f-476b-a835-3b7a4cb9122e-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-8s6wz\" (UID: \"72f5373a-ce4f-476b-a835-3b7a4cb9122e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-8s6wz"
Sep 18 17:36:17.208571 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:17.208488    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/72f5373a-ce4f-476b-a835-3b7a4cb9122e-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-8s6wz\" (UID: \"72f5373a-ce4f-476b-a835-3b7a4cb9122e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-8s6wz"
Sep 18 17:36:17.309692 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:17.309649    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/72f5373a-ce4f-476b-a835-3b7a4cb9122e-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-8s6wz\" (UID: \"72f5373a-ce4f-476b-a835-3b7a4cb9122e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-8s6wz"
Sep 18 17:36:17.309884 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:17.309736    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/72f5373a-ce4f-476b-a835-3b7a4cb9122e-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-8s6wz\" (UID: \"72f5373a-ce4f-476b-a835-3b7a4cb9122e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-8s6wz"
Sep 18 17:36:17.309884 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:17.309774    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwbmx\" (UniqueName: \"kubernetes.io/projected/72f5373a-ce4f-476b-a835-3b7a4cb9122e-kube-api-access-gwbmx\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-8s6wz\" (UID: \"72f5373a-ce4f-476b-a835-3b7a4cb9122e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-8s6wz"
Sep 18 17:36:17.309884 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:17.309791    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72f5373a-ce4f-476b-a835-3b7a4cb9122e-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-8s6wz\" (UID: \"72f5373a-ce4f-476b-a835-3b7a4cb9122e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-8s6wz"
Sep 18 17:36:17.309884 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:17.309829    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/72f5373a-ce4f-476b-a835-3b7a4cb9122e-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-8s6wz\" (UID: \"72f5373a-ce4f-476b-a835-3b7a4cb9122e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-8s6wz"
Sep 18 17:36:17.310272 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:17.310217    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/72f5373a-ce4f-476b-a835-3b7a4cb9122e-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-8s6wz\" (UID: \"72f5373a-ce4f-476b-a835-3b7a4cb9122e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-8s6wz"
Sep 18 17:36:17.310502 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:17.310483    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/72f5373a-ce4f-476b-a835-3b7a4cb9122e-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-8s6wz\" (UID: \"72f5373a-ce4f-476b-a835-3b7a4cb9122e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-8s6wz"
Sep 18 17:36:17.310571 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:17.310519    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/72f5373a-ce4f-476b-a835-3b7a4cb9122e-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-8s6wz\" (UID: \"72f5373a-ce4f-476b-a835-3b7a4cb9122e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-8s6wz"
Sep 18 17:36:17.312423 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:17.312404    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72f5373a-ce4f-476b-a835-3b7a4cb9122e-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-8s6wz\" (UID: \"72f5373a-ce4f-476b-a835-3b7a4cb9122e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-8s6wz"
Sep 18 17:36:17.327580 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:17.327547    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwbmx\" (UniqueName: \"kubernetes.io/projected/72f5373a-ce4f-476b-a835-3b7a4cb9122e-kube-api-access-gwbmx\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-8s6wz\" (UID: \"72f5373a-ce4f-476b-a835-3b7a4cb9122e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-8s6wz"
Sep 18 17:36:17.461440 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:17.461335    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-8s6wz"
Sep 18 17:36:17.475095 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:17.475055    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-88zvb" podUID="df0424ee-7af5-46a2-a907-30c65f68206b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 18 17:36:17.748858 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:17.748834    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-8s6wz"]
Sep 18 17:36:17.751558 ip-10-0-143-5 kubenswrapper[2881]: W0918 17:36:17.751527    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72f5373a_ce4f_476b_a835_3b7a4cb9122e.slice/crio-b57685c627e829ecb4c9c166f70c97f494e7bbd57d35aeb09f5c554418974bfe WatchSource:0}: Error finding container b57685c627e829ecb4c9c166f70c97f494e7bbd57d35aeb09f5c554418974bfe: Status 404 returned error can't find the container with id b57685c627e829ecb4c9c166f70c97f494e7bbd57d35aeb09f5c554418974bfe
Sep 18 17:36:17.896568 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:17.896539    2881 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c047619-6f56-4e5e-b697-23be03a5f440" path="/var/lib/kubelet/pods/3c047619-6f56-4e5e-b697-23be03a5f440/volumes"
Sep 18 17:36:18.596771 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:18.596739    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-8s6wz" event={"ID":"72f5373a-ce4f-476b-a835-3b7a4cb9122e","Type":"ContainerStarted","Data":"92c7f2694b37ac383d85e60a6bc4fe1cd0a982ef2318088bbbc559c3588959b2"}
Sep 18 17:36:18.596966 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:18.596797    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-8s6wz" event={"ID":"72f5373a-ce4f-476b-a835-3b7a4cb9122e","Type":"ContainerStarted","Data":"b57685c627e829ecb4c9c166f70c97f494e7bbd57d35aeb09f5c554418974bfe"}
Sep 18 17:36:22.615974 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:22.615944    2881 generic.go:356] "Generic (PLEG): container finished" podID="72f5373a-ce4f-476b-a835-3b7a4cb9122e" containerID="92c7f2694b37ac383d85e60a6bc4fe1cd0a982ef2318088bbbc559c3588959b2" exitCode=0
Sep 18 17:36:22.616387 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:22.616029    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-8s6wz" event={"ID":"72f5373a-ce4f-476b-a835-3b7a4cb9122e","Type":"ContainerDied","Data":"92c7f2694b37ac383d85e60a6bc4fe1cd0a982ef2318088bbbc559c3588959b2"}
Sep 18 17:36:23.623957 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:23.623916    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-8s6wz" event={"ID":"72f5373a-ce4f-476b-a835-3b7a4cb9122e","Type":"ContainerStarted","Data":"cdc109f1c9573a300d408be33faecf7221c07b81e0a010034b10caf8b2e82b76"}
Sep 18 17:36:23.624468 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:23.624027    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-8s6wz" event={"ID":"72f5373a-ce4f-476b-a835-3b7a4cb9122e","Type":"ContainerStarted","Data":"0d33271475ace6c446169d901f568ef9170f37b00ffbde443bfdc52c8d29b50e"}
Sep 18 17:36:23.624468 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:23.624257    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-8s6wz"
Sep 18 17:36:23.650180 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:23.650125    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-8s6wz" podStartSLOduration=2.771242625 podStartE2EDuration="6.650109918s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.878867293s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:36:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:36:23.648579374 +0000 UTC m=+2573.848434659" watchObservedRunningTime="2026-09-18 17:36:23.650109918 +0000 UTC m=+2573.849965202"
Sep 18 17:36:24.629934 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:24.629899    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-8s6wz"
Sep 18 17:36:25.463980 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:25.463943    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-nzh9s" podUID="303d1a24-b976-481f-9e07-bdf80d8abd68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 18 17:36:27.474884 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:27.474850    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-88zvb" podUID="df0424ee-7af5-46a2-a907-30c65f68206b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 18 17:36:30.642644 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:30.642560    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-8s6wz"
Sep 18 17:36:35.464407 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:35.464359    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-nzh9s" podUID="303d1a24-b976-481f-9e07-bdf80d8abd68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 18 17:36:37.475010 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:37.474974    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-88zvb" podUID="df0424ee-7af5-46a2-a907-30c65f68206b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 18 17:36:45.464325 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:45.464287    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-nzh9s" podUID="303d1a24-b976-481f-9e07-bdf80d8abd68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 18 17:36:47.474962 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:47.474923    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-88zvb" podUID="df0424ee-7af5-46a2-a907-30c65f68206b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 18 17:36:55.464707 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:55.464667    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-nzh9s" podUID="303d1a24-b976-481f-9e07-bdf80d8abd68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 18 17:36:57.474574 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:36:57.474529    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-88zvb" podUID="df0424ee-7af5-46a2-a907-30c65f68206b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 18 17:37:00.733332 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:00.733294    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-8s6wz"
Sep 18 17:37:05.464556 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:05.464519    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-nzh9s" podUID="303d1a24-b976-481f-9e07-bdf80d8abd68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 18 17:37:07.144861 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:07.144825    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-8s6wz"]
Sep 18 17:37:07.145369 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:07.145073    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-8s6wz" podUID="72f5373a-ce4f-476b-a835-3b7a4cb9122e" containerName="kserve-container" containerID="cri-o://0d33271475ace6c446169d901f568ef9170f37b00ffbde443bfdc52c8d29b50e" gracePeriod=30
Sep 18 17:37:07.145369 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:07.145128    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-8s6wz" podUID="72f5373a-ce4f-476b-a835-3b7a4cb9122e" containerName="kube-rbac-proxy" containerID="cri-o://cdc109f1c9573a300d408be33faecf7221c07b81e0a010034b10caf8b2e82b76" gracePeriod=30
Sep 18 17:37:07.474945 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:07.474858    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-88zvb" podUID="df0424ee-7af5-46a2-a907-30c65f68206b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 18 17:37:07.840857 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:07.840826    2881 generic.go:356] "Generic (PLEG): container finished" podID="72f5373a-ce4f-476b-a835-3b7a4cb9122e" containerID="cdc109f1c9573a300d408be33faecf7221c07b81e0a010034b10caf8b2e82b76" exitCode=2
Sep 18 17:37:07.841027 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:07.840900    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-8s6wz" event={"ID":"72f5373a-ce4f-476b-a835-3b7a4cb9122e","Type":"ContainerDied","Data":"cdc109f1c9573a300d408be33faecf7221c07b81e0a010034b10caf8b2e82b76"}
Sep 18 17:37:10.636932 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:10.636893    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-8s6wz" podUID="72f5373a-ce4f-476b-a835-3b7a4cb9122e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.51:8643/healthz\": dial tcp 10.134.0.51:8643: connect: connection refused"
Sep 18 17:37:10.644076 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:10.644046    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-8s6wz" podUID="72f5373a-ce4f-476b-a835-3b7a4cb9122e" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.51:8080/v2/models/isvc-xgboost-v2-mlserver/ready\": dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 18 17:37:12.419167 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:12.419129    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-k56jw"]
Sep 18 17:37:12.421910 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:12.421891    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-k56jw"
Sep 18 17:37:12.427012 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:12.426992    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-predictor-serving-cert\""
Sep 18 17:37:12.427103 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:12.427064    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 18 17:37:12.448106 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:12.448076    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-k56jw"]
Sep 18 17:37:12.512136 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:12.512096    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1759a40-c771-4a1e-b515-cfecd57732fd-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-k56jw\" (UID: \"a1759a40-c771-4a1e-b515-cfecd57732fd\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-k56jw"
Sep 18 17:37:12.512347 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:12.512157    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1759a40-c771-4a1e-b515-cfecd57732fd-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-k56jw\" (UID: \"a1759a40-c771-4a1e-b515-cfecd57732fd\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-k56jw"
Sep 18 17:37:12.512347 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:12.512182    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzwnb\" (UniqueName: \"kubernetes.io/projected/a1759a40-c771-4a1e-b515-cfecd57732fd-kube-api-access-fzwnb\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-k56jw\" (UID: \"a1759a40-c771-4a1e-b515-cfecd57732fd\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-k56jw"
Sep 18 17:37:12.512347 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:12.512201    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a1759a40-c771-4a1e-b515-cfecd57732fd-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-k56jw\" (UID: \"a1759a40-c771-4a1e-b515-cfecd57732fd\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-k56jw"
Sep 18 17:37:12.512347 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:12.512318    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1759a40-c771-4a1e-b515-cfecd57732fd-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-k56jw\" (UID: \"a1759a40-c771-4a1e-b515-cfecd57732fd\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-k56jw"
Sep 18 17:37:12.613658 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:12.613620    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1759a40-c771-4a1e-b515-cfecd57732fd-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-k56jw\" (UID: \"a1759a40-c771-4a1e-b515-cfecd57732fd\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-k56jw"
Sep 18 17:37:12.613834 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:12.613674    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzwnb\" (UniqueName: \"kubernetes.io/projected/a1759a40-c771-4a1e-b515-cfecd57732fd-kube-api-access-fzwnb\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-k56jw\" (UID: \"a1759a40-c771-4a1e-b515-cfecd57732fd\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-k56jw"
Sep 18 17:37:12.613834 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:12.613786    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a1759a40-c771-4a1e-b515-cfecd57732fd-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-k56jw\" (UID: \"a1759a40-c771-4a1e-b515-cfecd57732fd\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-k56jw"
Sep 18 17:37:12.613915 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:12.613847    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1759a40-c771-4a1e-b515-cfecd57732fd-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-k56jw\" (UID: \"a1759a40-c771-4a1e-b515-cfecd57732fd\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-k56jw"
Sep 18 17:37:12.613915 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:12.613892    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1759a40-c771-4a1e-b515-cfecd57732fd-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-k56jw\" (UID: \"a1759a40-c771-4a1e-b515-cfecd57732fd\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-k56jw"
Sep 18 17:37:12.614345 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:12.614317    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1759a40-c771-4a1e-b515-cfecd57732fd-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-k56jw\" (UID: \"a1759a40-c771-4a1e-b515-cfecd57732fd\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-k56jw"
Sep 18 17:37:12.614520 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:12.614495    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a1759a40-c771-4a1e-b515-cfecd57732fd-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-k56jw\" (UID: \"a1759a40-c771-4a1e-b515-cfecd57732fd\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-k56jw"
Sep 18 17:37:12.614592 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:12.614530    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1759a40-c771-4a1e-b515-cfecd57732fd-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-k56jw\" (UID: \"a1759a40-c771-4a1e-b515-cfecd57732fd\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-k56jw"
Sep 18 17:37:12.616253 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:12.616217    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1759a40-c771-4a1e-b515-cfecd57732fd-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-k56jw\" (UID: \"a1759a40-c771-4a1e-b515-cfecd57732fd\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-k56jw"
Sep 18 17:37:12.624560 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:12.624528    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzwnb\" (UniqueName: \"kubernetes.io/projected/a1759a40-c771-4a1e-b515-cfecd57732fd-kube-api-access-fzwnb\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-k56jw\" (UID: \"a1759a40-c771-4a1e-b515-cfecd57732fd\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-k56jw"
Sep 18 17:37:12.733489 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:12.733386    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-k56jw"
Sep 18 17:37:13.016398 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:13.016361    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-k56jw"]
Sep 18 17:37:13.018475 ip-10-0-143-5 kubenswrapper[2881]: W0918 17:37:13.018448    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1759a40_c771_4a1e_b515_cfecd57732fd.slice/crio-c7a63efc5064af717a12bb6fc737c26e2c01357442fe89fac1093b0a7b242975 WatchSource:0}: Error finding container c7a63efc5064af717a12bb6fc737c26e2c01357442fe89fac1093b0a7b242975: Status 404 returned error can't find the container with id c7a63efc5064af717a12bb6fc737c26e2c01357442fe89fac1093b0a7b242975
Sep 18 17:37:13.869896 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:13.869860    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-k56jw" event={"ID":"a1759a40-c771-4a1e-b515-cfecd57732fd","Type":"ContainerStarted","Data":"b4aa4d6839e560dbb72ca8ef05d20cf4a76ab33b83f37640a08abe435be1f88f"}
Sep 18 17:37:13.870259 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:13.869939    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-k56jw" event={"ID":"a1759a40-c771-4a1e-b515-cfecd57732fd","Type":"ContainerStarted","Data":"c7a63efc5064af717a12bb6fc737c26e2c01357442fe89fac1093b0a7b242975"}
Sep 18 17:37:13.872147 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:13.872125    2881 generic.go:356] "Generic (PLEG): container finished" podID="72f5373a-ce4f-476b-a835-3b7a4cb9122e" containerID="0d33271475ace6c446169d901f568ef9170f37b00ffbde443bfdc52c8d29b50e" exitCode=0
Sep 18 17:37:13.872268 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:13.872198    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-8s6wz" event={"ID":"72f5373a-ce4f-476b-a835-3b7a4cb9122e","Type":"ContainerDied","Data":"0d33271475ace6c446169d901f568ef9170f37b00ffbde443bfdc52c8d29b50e"}
Sep 18 17:37:14.000594 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:14.000565    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-8s6wz"
Sep 18 17:37:14.127072 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:14.126982    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/72f5373a-ce4f-476b-a835-3b7a4cb9122e-kube-rbac-proxy-sar-config\") pod \"72f5373a-ce4f-476b-a835-3b7a4cb9122e\" (UID: \"72f5373a-ce4f-476b-a835-3b7a4cb9122e\") "
Sep 18 17:37:14.127072 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:14.127032    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/72f5373a-ce4f-476b-a835-3b7a4cb9122e-cabundle-cert\") pod \"72f5373a-ce4f-476b-a835-3b7a4cb9122e\" (UID: \"72f5373a-ce4f-476b-a835-3b7a4cb9122e\") "
Sep 18 17:37:14.127343 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:14.127113    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/72f5373a-ce4f-476b-a835-3b7a4cb9122e-kserve-provision-location\") pod \"72f5373a-ce4f-476b-a835-3b7a4cb9122e\" (UID: \"72f5373a-ce4f-476b-a835-3b7a4cb9122e\") "
Sep 18 17:37:14.127343 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:14.127142    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72f5373a-ce4f-476b-a835-3b7a4cb9122e-proxy-tls\") pod \"72f5373a-ce4f-476b-a835-3b7a4cb9122e\" (UID: \"72f5373a-ce4f-476b-a835-3b7a4cb9122e\") "
Sep 18 17:37:14.127343 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:14.127200    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gwbmx\" (UniqueName: \"kubernetes.io/projected/72f5373a-ce4f-476b-a835-3b7a4cb9122e-kube-api-access-gwbmx\") pod \"72f5373a-ce4f-476b-a835-3b7a4cb9122e\" (UID: \"72f5373a-ce4f-476b-a835-3b7a4cb9122e\") "
Sep 18 17:37:14.127505 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:14.127461    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72f5373a-ce4f-476b-a835-3b7a4cb9122e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "72f5373a-ce4f-476b-a835-3b7a4cb9122e" (UID: "72f5373a-ce4f-476b-a835-3b7a4cb9122e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:37:14.127568 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:14.127535    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72f5373a-ce4f-476b-a835-3b7a4cb9122e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "72f5373a-ce4f-476b-a835-3b7a4cb9122e" (UID: "72f5373a-ce4f-476b-a835-3b7a4cb9122e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:37:14.127568 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:14.127532    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f5373a-ce4f-476b-a835-3b7a4cb9122e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "72f5373a-ce4f-476b-a835-3b7a4cb9122e" (UID: "72f5373a-ce4f-476b-a835-3b7a4cb9122e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:37:14.129444 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:14.129415    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72f5373a-ce4f-476b-a835-3b7a4cb9122e-kube-api-access-gwbmx" (OuterVolumeSpecName: "kube-api-access-gwbmx") pod "72f5373a-ce4f-476b-a835-3b7a4cb9122e" (UID: "72f5373a-ce4f-476b-a835-3b7a4cb9122e"). InnerVolumeSpecName "kube-api-access-gwbmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:37:14.129444 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:14.129436    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72f5373a-ce4f-476b-a835-3b7a4cb9122e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "72f5373a-ce4f-476b-a835-3b7a4cb9122e" (UID: "72f5373a-ce4f-476b-a835-3b7a4cb9122e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:37:14.227972 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:14.227918    2881 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72f5373a-ce4f-476b-a835-3b7a4cb9122e-proxy-tls\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:37:14.227972 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:14.227967    2881 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwbmx\" (UniqueName: \"kubernetes.io/projected/72f5373a-ce4f-476b-a835-3b7a4cb9122e-kube-api-access-gwbmx\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:37:14.227972 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:14.227979    2881 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/72f5373a-ce4f-476b-a835-3b7a4cb9122e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:37:14.227972 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:14.227988    2881 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/72f5373a-ce4f-476b-a835-3b7a4cb9122e-cabundle-cert\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:37:14.228299 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:14.227996    2881 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/72f5373a-ce4f-476b-a835-3b7a4cb9122e-kserve-provision-location\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:37:14.880545 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:14.880515    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-8s6wz"
Sep 18 17:37:14.881036 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:14.880513    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-8s6wz" event={"ID":"72f5373a-ce4f-476b-a835-3b7a4cb9122e","Type":"ContainerDied","Data":"b57685c627e829ecb4c9c166f70c97f494e7bbd57d35aeb09f5c554418974bfe"}
Sep 18 17:37:14.881036 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:14.880683    2881 scope.go:117] "RemoveContainer" containerID="cdc109f1c9573a300d408be33faecf7221c07b81e0a010034b10caf8b2e82b76"
Sep 18 17:37:14.889832 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:14.889810    2881 scope.go:117] "RemoveContainer" containerID="0d33271475ace6c446169d901f568ef9170f37b00ffbde443bfdc52c8d29b50e"
Sep 18 17:37:14.897862 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:14.897838    2881 scope.go:117] "RemoveContainer" containerID="92c7f2694b37ac383d85e60a6bc4fe1cd0a982ef2318088bbbc559c3588959b2"
Sep 18 17:37:14.904085 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:14.904058    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-8s6wz"]
Sep 18 17:37:14.908368 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:14.908346    2881 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-8s6wz"]
Sep 18 17:37:15.465203 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:15.465175    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-nzh9s"
Sep 18 17:37:15.597971 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:15.597938    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-88zvb"]
Sep 18 17:37:15.598241 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:15.598189    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-88zvb" podUID="df0424ee-7af5-46a2-a907-30c65f68206b" containerName="kserve-container" containerID="cri-o://d2133e9eee0b804bdcd9e68658fbf1e5b116df91dbd5c6ca23897f5ff1c24c04" gracePeriod=30
Sep 18 17:37:15.598409 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:15.598259    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-88zvb" podUID="df0424ee-7af5-46a2-a907-30c65f68206b" containerName="kube-rbac-proxy" containerID="cri-o://ffc4086aa1658abfacbc1db07dbeb2968d676ab0d880929373104b5593f9481e" gracePeriod=30
Sep 18 17:37:15.599152 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:15.599134    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-88zvb"
Sep 18 17:37:15.888293 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:15.888261    2881 generic.go:356] "Generic (PLEG): container finished" podID="df0424ee-7af5-46a2-a907-30c65f68206b" containerID="ffc4086aa1658abfacbc1db07dbeb2968d676ab0d880929373104b5593f9481e" exitCode=2
Sep 18 17:37:15.892841 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:15.892808    2881 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72f5373a-ce4f-476b-a835-3b7a4cb9122e" path="/var/lib/kubelet/pods/72f5373a-ce4f-476b-a835-3b7a4cb9122e/volumes"
Sep 18 17:37:15.893159 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:15.893142    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-88zvb" event={"ID":"df0424ee-7af5-46a2-a907-30c65f68206b","Type":"ContainerDied","Data":"ffc4086aa1658abfacbc1db07dbeb2968d676ab0d880929373104b5593f9481e"}
Sep 18 17:37:16.895795 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:16.895758    2881 generic.go:356] "Generic (PLEG): container finished" podID="a1759a40-c771-4a1e-b515-cfecd57732fd" containerID="b4aa4d6839e560dbb72ca8ef05d20cf4a76ab33b83f37640a08abe435be1f88f" exitCode=0
Sep 18 17:37:16.896354 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:16.895837    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-k56jw" event={"ID":"a1759a40-c771-4a1e-b515-cfecd57732fd","Type":"ContainerDied","Data":"b4aa4d6839e560dbb72ca8ef05d20cf4a76ab33b83f37640a08abe435be1f88f"}
Sep 18 17:37:17.469219 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:17.469115    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-88zvb" podUID="df0424ee-7af5-46a2-a907-30c65f68206b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.50:8643/healthz\": dial tcp 10.134.0.50:8643: connect: connection refused"
Sep 18 17:37:17.903080 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:17.903047    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-k56jw" event={"ID":"a1759a40-c771-4a1e-b515-cfecd57732fd","Type":"ContainerStarted","Data":"bf93eaed71ae1dc25204509be64c2e5dfecd1d38b27401fc0e542971c59e7fcb"}
Sep 18 17:37:17.903528 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:17.903109    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-k56jw" event={"ID":"a1759a40-c771-4a1e-b515-cfecd57732fd","Type":"ContainerStarted","Data":"9456c24d9d3d1cabd8d5bded119204f082d79855ac4a7ce48487fda5197fa306"}
Sep 18 17:37:17.903528 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:17.903401    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-k56jw"
Sep 18 17:37:17.903528 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:17.903461    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-k56jw"
Sep 18 17:37:17.929006 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:17.928954    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-k56jw" podStartSLOduration=2.251699846 podStartE2EDuration="5.928936184s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.677236338s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:37:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:37:17.927191745 +0000 UTC m=+2628.127047031" watchObservedRunningTime="2026-09-18 17:37:17.928936184 +0000 UTC m=+2628.128791550"
Sep 18 17:37:20.920983 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:20.920948    2881 generic.go:356] "Generic (PLEG): container finished" podID="df0424ee-7af5-46a2-a907-30c65f68206b" containerID="d2133e9eee0b804bdcd9e68658fbf1e5b116df91dbd5c6ca23897f5ff1c24c04" exitCode=0
Sep 18 17:37:20.921390 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:20.921019    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-88zvb" event={"ID":"df0424ee-7af5-46a2-a907-30c65f68206b","Type":"ContainerDied","Data":"d2133e9eee0b804bdcd9e68658fbf1e5b116df91dbd5c6ca23897f5ff1c24c04"}
Sep 18 17:37:20.942974 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:20.942948    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-88zvb"
Sep 18 17:37:21.099899 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:21.099861    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df0424ee-7af5-46a2-a907-30c65f68206b-cabundle-cert\") pod \"df0424ee-7af5-46a2-a907-30c65f68206b\" (UID: \"df0424ee-7af5-46a2-a907-30c65f68206b\") "
Sep 18 17:37:21.100121 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:21.099959    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df0424ee-7af5-46a2-a907-30c65f68206b-proxy-tls\") pod \"df0424ee-7af5-46a2-a907-30c65f68206b\" (UID: \"df0424ee-7af5-46a2-a907-30c65f68206b\") "
Sep 18 17:37:21.100121 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:21.099988    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2n4l\" (UniqueName: \"kubernetes.io/projected/df0424ee-7af5-46a2-a907-30c65f68206b-kube-api-access-x2n4l\") pod \"df0424ee-7af5-46a2-a907-30c65f68206b\" (UID: \"df0424ee-7af5-46a2-a907-30c65f68206b\") "
Sep 18 17:37:21.100270 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:21.100140    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df0424ee-7af5-46a2-a907-30c65f68206b-kserve-provision-location\") pod \"df0424ee-7af5-46a2-a907-30c65f68206b\" (UID: \"df0424ee-7af5-46a2-a907-30c65f68206b\") "
Sep 18 17:37:21.100270 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:21.100188    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df0424ee-7af5-46a2-a907-30c65f68206b-kube-rbac-proxy-sar-config\") pod \"df0424ee-7af5-46a2-a907-30c65f68206b\" (UID: \"df0424ee-7af5-46a2-a907-30c65f68206b\") "
Sep 18 17:37:21.100387 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:21.100322    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df0424ee-7af5-46a2-a907-30c65f68206b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "df0424ee-7af5-46a2-a907-30c65f68206b" (UID: "df0424ee-7af5-46a2-a907-30c65f68206b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:37:21.100517 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:21.100489    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df0424ee-7af5-46a2-a907-30c65f68206b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "df0424ee-7af5-46a2-a907-30c65f68206b" (UID: "df0424ee-7af5-46a2-a907-30c65f68206b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:37:21.100578 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:21.100525    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df0424ee-7af5-46a2-a907-30c65f68206b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "df0424ee-7af5-46a2-a907-30c65f68206b" (UID: "df0424ee-7af5-46a2-a907-30c65f68206b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:37:21.100627 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:21.100596    2881 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df0424ee-7af5-46a2-a907-30c65f68206b-kserve-provision-location\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:37:21.100627 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:21.100614    2881 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df0424ee-7af5-46a2-a907-30c65f68206b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:37:21.100738 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:21.100628    2881 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df0424ee-7af5-46a2-a907-30c65f68206b-cabundle-cert\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:37:21.102192 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:21.102167    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df0424ee-7af5-46a2-a907-30c65f68206b-kube-api-access-x2n4l" (OuterVolumeSpecName: "kube-api-access-x2n4l") pod "df0424ee-7af5-46a2-a907-30c65f68206b" (UID: "df0424ee-7af5-46a2-a907-30c65f68206b"). InnerVolumeSpecName "kube-api-access-x2n4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:37:21.102192 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:21.102175    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df0424ee-7af5-46a2-a907-30c65f68206b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "df0424ee-7af5-46a2-a907-30c65f68206b" (UID: "df0424ee-7af5-46a2-a907-30c65f68206b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:37:21.201532 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:21.201494    2881 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df0424ee-7af5-46a2-a907-30c65f68206b-proxy-tls\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:37:21.201532 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:21.201528    2881 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2n4l\" (UniqueName: \"kubernetes.io/projected/df0424ee-7af5-46a2-a907-30c65f68206b-kube-api-access-x2n4l\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:37:21.928259 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:21.928212    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-88zvb"
Sep 18 17:37:21.928671 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:21.928232    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-88zvb" event={"ID":"df0424ee-7af5-46a2-a907-30c65f68206b","Type":"ContainerDied","Data":"dcf73d5e77bdf1f23c0a28edfcdf6d4acffb52f76a872eb3615f6f9f87abee99"}
Sep 18 17:37:21.928671 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:21.928370    2881 scope.go:117] "RemoveContainer" containerID="ffc4086aa1658abfacbc1db07dbeb2968d676ab0d880929373104b5593f9481e"
Sep 18 17:37:21.938209 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:21.938187    2881 scope.go:117] "RemoveContainer" containerID="d2133e9eee0b804bdcd9e68658fbf1e5b116df91dbd5c6ca23897f5ff1c24c04"
Sep 18 17:37:21.946436 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:21.946410    2881 scope.go:117] "RemoveContainer" containerID="7de054bae6686448eac9afd6ef2d5ca4edb8ff00546f2f6ad902f371b78f6673"
Sep 18 17:37:21.949808 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:21.949786    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-88zvb"]
Sep 18 17:37:21.953464 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:21.953441    2881 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6958d6c864-88zvb"]
Sep 18 17:37:23.701684 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:23.701645    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-nzh9s"]
Sep 18 17:37:23.702153 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:23.701946    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-nzh9s" podUID="303d1a24-b976-481f-9e07-bdf80d8abd68" containerName="kserve-container" containerID="cri-o://e53c96b6e393386facde7770945f5d126994a200b23142ccc43a4aef2a6e34ce" gracePeriod=30
Sep 18 17:37:23.702153 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:23.701979    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-nzh9s" podUID="303d1a24-b976-481f-9e07-bdf80d8abd68" containerName="kube-rbac-proxy" containerID="cri-o://6fab586dacae6434b3bb34f1135d259f4e60b7d7b3dafb5e90df40cfcc1c1459" gracePeriod=30
Sep 18 17:37:23.892988 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:23.892955    2881 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df0424ee-7af5-46a2-a907-30c65f68206b" path="/var/lib/kubelet/pods/df0424ee-7af5-46a2-a907-30c65f68206b/volumes"
Sep 18 17:37:23.914057 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:23.914031    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-k56jw"
Sep 18 17:37:23.942043 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:23.942013    2881 generic.go:356] "Generic (PLEG): container finished" podID="303d1a24-b976-481f-9e07-bdf80d8abd68" containerID="6fab586dacae6434b3bb34f1135d259f4e60b7d7b3dafb5e90df40cfcc1c1459" exitCode=2
Sep 18 17:37:23.942242 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:23.942087    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-nzh9s" event={"ID":"303d1a24-b976-481f-9e07-bdf80d8abd68","Type":"ContainerDied","Data":"6fab586dacae6434b3bb34f1135d259f4e60b7d7b3dafb5e90df40cfcc1c1459"}
Sep 18 17:37:25.459057 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:25.459015    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-nzh9s" podUID="303d1a24-b976-481f-9e07-bdf80d8abd68" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.49:8643/healthz\": dial tcp 10.134.0.49:8643: connect: connection refused"
Sep 18 17:37:25.464485 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:25.464456    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-nzh9s" podUID="303d1a24-b976-481f-9e07-bdf80d8abd68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 18 17:37:29.253049 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:29.253021    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-nzh9s"
Sep 18 17:37:29.380835 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:29.380732    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/303d1a24-b976-481f-9e07-bdf80d8abd68-proxy-tls\") pod \"303d1a24-b976-481f-9e07-bdf80d8abd68\" (UID: \"303d1a24-b976-481f-9e07-bdf80d8abd68\") "
Sep 18 17:37:29.380835 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:29.380777    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/303d1a24-b976-481f-9e07-bdf80d8abd68-kube-rbac-proxy-sar-config\") pod \"303d1a24-b976-481f-9e07-bdf80d8abd68\" (UID: \"303d1a24-b976-481f-9e07-bdf80d8abd68\") "
Sep 18 17:37:29.381075 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:29.380845    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/303d1a24-b976-481f-9e07-bdf80d8abd68-kserve-provision-location\") pod \"303d1a24-b976-481f-9e07-bdf80d8abd68\" (UID: \"303d1a24-b976-481f-9e07-bdf80d8abd68\") "
Sep 18 17:37:29.381075 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:29.380878    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/303d1a24-b976-481f-9e07-bdf80d8abd68-cabundle-cert\") pod \"303d1a24-b976-481f-9e07-bdf80d8abd68\" (UID: \"303d1a24-b976-481f-9e07-bdf80d8abd68\") "
Sep 18 17:37:29.381075 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:29.380903    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kq49n\" (UniqueName: \"kubernetes.io/projected/303d1a24-b976-481f-9e07-bdf80d8abd68-kube-api-access-kq49n\") pod \"303d1a24-b976-481f-9e07-bdf80d8abd68\" (UID: \"303d1a24-b976-481f-9e07-bdf80d8abd68\") "
Sep 18 17:37:29.381274 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:29.381210    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/303d1a24-b976-481f-9e07-bdf80d8abd68-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "303d1a24-b976-481f-9e07-bdf80d8abd68" (UID: "303d1a24-b976-481f-9e07-bdf80d8abd68"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:37:29.381274 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:29.381269    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/303d1a24-b976-481f-9e07-bdf80d8abd68-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "303d1a24-b976-481f-9e07-bdf80d8abd68" (UID: "303d1a24-b976-481f-9e07-bdf80d8abd68"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:37:29.381451 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:29.381276    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/303d1a24-b976-481f-9e07-bdf80d8abd68-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "303d1a24-b976-481f-9e07-bdf80d8abd68" (UID: "303d1a24-b976-481f-9e07-bdf80d8abd68"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:37:29.382935 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:29.382917    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/303d1a24-b976-481f-9e07-bdf80d8abd68-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "303d1a24-b976-481f-9e07-bdf80d8abd68" (UID: "303d1a24-b976-481f-9e07-bdf80d8abd68"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:37:29.383030 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:29.382999    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/303d1a24-b976-481f-9e07-bdf80d8abd68-kube-api-access-kq49n" (OuterVolumeSpecName: "kube-api-access-kq49n") pod "303d1a24-b976-481f-9e07-bdf80d8abd68" (UID: "303d1a24-b976-481f-9e07-bdf80d8abd68"). InnerVolumeSpecName "kube-api-access-kq49n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:37:29.482459 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:29.482421    2881 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/303d1a24-b976-481f-9e07-bdf80d8abd68-proxy-tls\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:37:29.482459 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:29.482452    2881 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/303d1a24-b976-481f-9e07-bdf80d8abd68-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:37:29.482459 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:29.482464    2881 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/303d1a24-b976-481f-9e07-bdf80d8abd68-kserve-provision-location\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:37:29.482707 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:29.482478    2881 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/303d1a24-b976-481f-9e07-bdf80d8abd68-cabundle-cert\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:37:29.482707 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:29.482489    2881 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kq49n\" (UniqueName: \"kubernetes.io/projected/303d1a24-b976-481f-9e07-bdf80d8abd68-kube-api-access-kq49n\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:37:29.973769 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:29.973734    2881 generic.go:356] "Generic (PLEG): container finished" podID="303d1a24-b976-481f-9e07-bdf80d8abd68" containerID="e53c96b6e393386facde7770945f5d126994a200b23142ccc43a4aef2a6e34ce" exitCode=0
Sep 18 17:37:29.973947 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:29.973801    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-nzh9s"
Sep 18 17:37:29.973947 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:29.973817    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-nzh9s" event={"ID":"303d1a24-b976-481f-9e07-bdf80d8abd68","Type":"ContainerDied","Data":"e53c96b6e393386facde7770945f5d126994a200b23142ccc43a4aef2a6e34ce"}
Sep 18 17:37:29.973947 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:29.973906    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-nzh9s" event={"ID":"303d1a24-b976-481f-9e07-bdf80d8abd68","Type":"ContainerDied","Data":"7e7806f77596d23a254bf6ee5876ae24c6fef34eab17106615717485664c6876"}
Sep 18 17:37:29.973947 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:29.973934    2881 scope.go:117] "RemoveContainer" containerID="6fab586dacae6434b3bb34f1135d259f4e60b7d7b3dafb5e90df40cfcc1c1459"
Sep 18 17:37:29.982406 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:29.982384    2881 scope.go:117] "RemoveContainer" containerID="e53c96b6e393386facde7770945f5d126994a200b23142ccc43a4aef2a6e34ce"
Sep 18 17:37:29.989947 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:29.989927    2881 scope.go:117] "RemoveContainer" containerID="c9ce3d2d128d0ee5bcb1883e855d0390e1154f9dfa8f92f199ae5fa350ae776d"
Sep 18 17:37:29.993551 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:29.993527    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-nzh9s"]
Sep 18 17:37:29.998489 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:29.998463    2881 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c697b4d69-nzh9s"]
Sep 18 17:37:29.998963 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:29.998945    2881 scope.go:117] "RemoveContainer" containerID="6fab586dacae6434b3bb34f1135d259f4e60b7d7b3dafb5e90df40cfcc1c1459"
Sep 18 17:37:29.999242 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:37:29.999211    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fab586dacae6434b3bb34f1135d259f4e60b7d7b3dafb5e90df40cfcc1c1459\": container with ID starting with 6fab586dacae6434b3bb34f1135d259f4e60b7d7b3dafb5e90df40cfcc1c1459 not found: ID does not exist" containerID="6fab586dacae6434b3bb34f1135d259f4e60b7d7b3dafb5e90df40cfcc1c1459"
Sep 18 17:37:29.999331 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:29.999254    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fab586dacae6434b3bb34f1135d259f4e60b7d7b3dafb5e90df40cfcc1c1459"} err="failed to get container status \"6fab586dacae6434b3bb34f1135d259f4e60b7d7b3dafb5e90df40cfcc1c1459\": rpc error: code = NotFound desc = could not find container \"6fab586dacae6434b3bb34f1135d259f4e60b7d7b3dafb5e90df40cfcc1c1459\": container with ID starting with 6fab586dacae6434b3bb34f1135d259f4e60b7d7b3dafb5e90df40cfcc1c1459 not found: ID does not exist"
Sep 18 17:37:29.999331 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:29.999279    2881 scope.go:117] "RemoveContainer" containerID="e53c96b6e393386facde7770945f5d126994a200b23142ccc43a4aef2a6e34ce"
Sep 18 17:37:29.999571 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:37:29.999554    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e53c96b6e393386facde7770945f5d126994a200b23142ccc43a4aef2a6e34ce\": container with ID starting with e53c96b6e393386facde7770945f5d126994a200b23142ccc43a4aef2a6e34ce not found: ID does not exist" containerID="e53c96b6e393386facde7770945f5d126994a200b23142ccc43a4aef2a6e34ce"
Sep 18 17:37:29.999639 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:29.999579    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e53c96b6e393386facde7770945f5d126994a200b23142ccc43a4aef2a6e34ce"} err="failed to get container status \"e53c96b6e393386facde7770945f5d126994a200b23142ccc43a4aef2a6e34ce\": rpc error: code = NotFound desc = could not find container \"e53c96b6e393386facde7770945f5d126994a200b23142ccc43a4aef2a6e34ce\": container with ID starting with e53c96b6e393386facde7770945f5d126994a200b23142ccc43a4aef2a6e34ce not found: ID does not exist"
Sep 18 17:37:29.999639 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:29.999601    2881 scope.go:117] "RemoveContainer" containerID="c9ce3d2d128d0ee5bcb1883e855d0390e1154f9dfa8f92f199ae5fa350ae776d"
Sep 18 17:37:29.999858 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:37:29.999840    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9ce3d2d128d0ee5bcb1883e855d0390e1154f9dfa8f92f199ae5fa350ae776d\": container with ID starting with c9ce3d2d128d0ee5bcb1883e855d0390e1154f9dfa8f92f199ae5fa350ae776d not found: ID does not exist" containerID="c9ce3d2d128d0ee5bcb1883e855d0390e1154f9dfa8f92f199ae5fa350ae776d"
Sep 18 17:37:29.999903 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:29.999866    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9ce3d2d128d0ee5bcb1883e855d0390e1154f9dfa8f92f199ae5fa350ae776d"} err="failed to get container status \"c9ce3d2d128d0ee5bcb1883e855d0390e1154f9dfa8f92f199ae5fa350ae776d\": rpc error: code = NotFound desc = could not find container \"c9ce3d2d128d0ee5bcb1883e855d0390e1154f9dfa8f92f199ae5fa350ae776d\": container with ID starting with c9ce3d2d128d0ee5bcb1883e855d0390e1154f9dfa8f92f199ae5fa350ae776d not found: ID does not exist"
Sep 18 17:37:31.893765 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:31.893725    2881 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="303d1a24-b976-481f-9e07-bdf80d8abd68" path="/var/lib/kubelet/pods/303d1a24-b976-481f-9e07-bdf80d8abd68/volumes"
Sep 18 17:37:33.967394 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:33.967358    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6w9rp"]
Sep 18 17:37:33.967819 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:33.967701    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72f5373a-ce4f-476b-a835-3b7a4cb9122e" containerName="storage-initializer"
Sep 18 17:37:33.967819 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:33.967712    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="72f5373a-ce4f-476b-a835-3b7a4cb9122e" containerName="storage-initializer"
Sep 18 17:37:33.967819 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:33.967721    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df0424ee-7af5-46a2-a907-30c65f68206b" containerName="kserve-container"
Sep 18 17:37:33.967819 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:33.967727    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="df0424ee-7af5-46a2-a907-30c65f68206b" containerName="kserve-container"
Sep 18 17:37:33.967819 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:33.967739    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="303d1a24-b976-481f-9e07-bdf80d8abd68" containerName="kube-rbac-proxy"
Sep 18 17:37:33.967819 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:33.967745    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="303d1a24-b976-481f-9e07-bdf80d8abd68" containerName="kube-rbac-proxy"
Sep 18 17:37:33.967819 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:33.967755    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="303d1a24-b976-481f-9e07-bdf80d8abd68" containerName="storage-initializer"
Sep 18 17:37:33.967819 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:33.967769    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="303d1a24-b976-481f-9e07-bdf80d8abd68" containerName="storage-initializer"
Sep 18 17:37:33.967819 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:33.967779    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="303d1a24-b976-481f-9e07-bdf80d8abd68" containerName="kserve-container"
Sep 18 17:37:33.967819 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:33.967784    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="303d1a24-b976-481f-9e07-bdf80d8abd68" containerName="kserve-container"
Sep 18 17:37:33.967819 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:33.967789    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72f5373a-ce4f-476b-a835-3b7a4cb9122e" containerName="kube-rbac-proxy"
Sep 18 17:37:33.967819 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:33.967794    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="72f5373a-ce4f-476b-a835-3b7a4cb9122e" containerName="kube-rbac-proxy"
Sep 18 17:37:33.967819 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:33.967803    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72f5373a-ce4f-476b-a835-3b7a4cb9122e" containerName="kserve-container"
Sep 18 17:37:33.967819 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:33.967807    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="72f5373a-ce4f-476b-a835-3b7a4cb9122e" containerName="kserve-container"
Sep 18 17:37:33.967819 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:33.967814    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df0424ee-7af5-46a2-a907-30c65f68206b" containerName="storage-initializer"
Sep 18 17:37:33.967819 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:33.967820    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="df0424ee-7af5-46a2-a907-30c65f68206b" containerName="storage-initializer"
Sep 18 17:37:33.967819 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:33.967828    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df0424ee-7af5-46a2-a907-30c65f68206b" containerName="kube-rbac-proxy"
Sep 18 17:37:33.968433 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:33.967833    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="df0424ee-7af5-46a2-a907-30c65f68206b" containerName="kube-rbac-proxy"
Sep 18 17:37:33.968433 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:33.967879    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="303d1a24-b976-481f-9e07-bdf80d8abd68" containerName="kserve-container"
Sep 18 17:37:33.968433 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:33.967888    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="df0424ee-7af5-46a2-a907-30c65f68206b" containerName="kserve-container"
Sep 18 17:37:33.968433 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:33.967894    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="72f5373a-ce4f-476b-a835-3b7a4cb9122e" containerName="kube-rbac-proxy"
Sep 18 17:37:33.968433 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:33.967900    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="df0424ee-7af5-46a2-a907-30c65f68206b" containerName="kube-rbac-proxy"
Sep 18 17:37:33.968433 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:33.967906    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="303d1a24-b976-481f-9e07-bdf80d8abd68" containerName="kube-rbac-proxy"
Sep 18 17:37:33.968433 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:33.967913    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="72f5373a-ce4f-476b-a835-3b7a4cb9122e" containerName="kserve-container"
Sep 18 17:37:33.972773 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:33.972752    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6w9rp"
Sep 18 17:37:33.975175 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:33.975142    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-predictor-serving-cert\""
Sep 18 17:37:33.975346 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:33.975144    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-kube-rbac-proxy-sar-config\""
Sep 18 17:37:33.986301 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:33.986271    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6w9rp"]
Sep 18 17:37:34.124577 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:34.124536    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57e56b44-1893-4e11-88d8-46f01379ee03-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-6w9rp\" (UID: \"57e56b44-1893-4e11-88d8-46f01379ee03\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6w9rp"
Sep 18 17:37:34.124577 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:34.124585    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57e56b44-1893-4e11-88d8-46f01379ee03-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-6w9rp\" (UID: \"57e56b44-1893-4e11-88d8-46f01379ee03\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6w9rp"
Sep 18 17:37:34.124804 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:34.124617    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57e56b44-1893-4e11-88d8-46f01379ee03-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-6w9rp\" (UID: \"57e56b44-1893-4e11-88d8-46f01379ee03\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6w9rp"
Sep 18 17:37:34.124804 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:34.124684    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb5mq\" (UniqueName: \"kubernetes.io/projected/57e56b44-1893-4e11-88d8-46f01379ee03-kube-api-access-cb5mq\") pod \"isvc-lightgbm-predictor-54f85ddf88-6w9rp\" (UID: \"57e56b44-1893-4e11-88d8-46f01379ee03\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6w9rp"
Sep 18 17:37:34.124804 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:34.124705    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57e56b44-1893-4e11-88d8-46f01379ee03-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-6w9rp\" (UID: \"57e56b44-1893-4e11-88d8-46f01379ee03\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6w9rp"
Sep 18 17:37:34.225670 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:34.225577    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57e56b44-1893-4e11-88d8-46f01379ee03-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-6w9rp\" (UID: \"57e56b44-1893-4e11-88d8-46f01379ee03\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6w9rp"
Sep 18 17:37:34.225670 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:34.225615    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57e56b44-1893-4e11-88d8-46f01379ee03-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-6w9rp\" (UID: \"57e56b44-1893-4e11-88d8-46f01379ee03\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6w9rp"
Sep 18 17:37:34.225670 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:34.225665    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb5mq\" (UniqueName: \"kubernetes.io/projected/57e56b44-1893-4e11-88d8-46f01379ee03-kube-api-access-cb5mq\") pod \"isvc-lightgbm-predictor-54f85ddf88-6w9rp\" (UID: \"57e56b44-1893-4e11-88d8-46f01379ee03\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6w9rp"
Sep 18 17:37:34.225909 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:34.225689    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57e56b44-1893-4e11-88d8-46f01379ee03-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-6w9rp\" (UID: \"57e56b44-1893-4e11-88d8-46f01379ee03\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6w9rp"
Sep 18 17:37:34.225909 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:34.225739    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57e56b44-1893-4e11-88d8-46f01379ee03-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-6w9rp\" (UID: \"57e56b44-1893-4e11-88d8-46f01379ee03\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6w9rp"
Sep 18 17:37:34.226181 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:34.226155    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57e56b44-1893-4e11-88d8-46f01379ee03-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-6w9rp\" (UID: \"57e56b44-1893-4e11-88d8-46f01379ee03\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6w9rp"
Sep 18 17:37:34.226509 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:34.226487    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57e56b44-1893-4e11-88d8-46f01379ee03-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-6w9rp\" (UID: \"57e56b44-1893-4e11-88d8-46f01379ee03\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6w9rp"
Sep 18 17:37:34.226561 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:34.226487    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57e56b44-1893-4e11-88d8-46f01379ee03-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-6w9rp\" (UID: \"57e56b44-1893-4e11-88d8-46f01379ee03\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6w9rp"
Sep 18 17:37:34.228292 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:34.228270    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57e56b44-1893-4e11-88d8-46f01379ee03-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-6w9rp\" (UID: \"57e56b44-1893-4e11-88d8-46f01379ee03\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6w9rp"
Sep 18 17:37:34.235094 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:34.235063    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb5mq\" (UniqueName: \"kubernetes.io/projected/57e56b44-1893-4e11-88d8-46f01379ee03-kube-api-access-cb5mq\") pod \"isvc-lightgbm-predictor-54f85ddf88-6w9rp\" (UID: \"57e56b44-1893-4e11-88d8-46f01379ee03\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6w9rp"
Sep 18 17:37:34.283137 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:34.283096    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6w9rp"
Sep 18 17:37:34.563832 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:34.563804    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6w9rp"]
Sep 18 17:37:34.565764 ip-10-0-143-5 kubenswrapper[2881]: W0918 17:37:34.565735    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57e56b44_1893_4e11_88d8_46f01379ee03.slice/crio-df73e342f9733d57bd62612f42cd5cad914e92c4e4d9e0df39dca0ccabca04a6 WatchSource:0}: Error finding container df73e342f9733d57bd62612f42cd5cad914e92c4e4d9e0df39dca0ccabca04a6: Status 404 returned error can't find the container with id df73e342f9733d57bd62612f42cd5cad914e92c4e4d9e0df39dca0ccabca04a6
Sep 18 17:37:34.997424 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:34.997344    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6w9rp" event={"ID":"57e56b44-1893-4e11-88d8-46f01379ee03","Type":"ContainerStarted","Data":"a753e78d3593f336c702d581cb34ac45a1540507de8d48dd6d3fd0e52d053348"}
Sep 18 17:37:34.997424 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:34.997400    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6w9rp" event={"ID":"57e56b44-1893-4e11-88d8-46f01379ee03","Type":"ContainerStarted","Data":"df73e342f9733d57bd62612f42cd5cad914e92c4e4d9e0df39dca0ccabca04a6"}
Sep 18 17:37:37.355388 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:37:37.355338    2881 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod303d1a24_b976_481f_9e07_bdf80d8abd68.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod303d1a24_b976_481f_9e07_bdf80d8abd68.slice/crio-7e7806f77596d23a254bf6ee5876ae24c6fef34eab17106615717485664c6876\": RecentStats: unable to find data in memory cache]"
Sep 18 17:37:39.015716 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:39.015687    2881 generic.go:356] "Generic (PLEG): container finished" podID="57e56b44-1893-4e11-88d8-46f01379ee03" containerID="a753e78d3593f336c702d581cb34ac45a1540507de8d48dd6d3fd0e52d053348" exitCode=0
Sep 18 17:37:39.016108 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:39.015738    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6w9rp" event={"ID":"57e56b44-1893-4e11-88d8-46f01379ee03","Type":"ContainerDied","Data":"a753e78d3593f336c702d581cb34ac45a1540507de8d48dd6d3fd0e52d053348"}
Sep 18 17:37:39.017067 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:39.017042    2881 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 17:37:40.698249 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:37:40.698189    2881 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod303d1a24_b976_481f_9e07_bdf80d8abd68.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod303d1a24_b976_481f_9e07_bdf80d8abd68.slice/crio-7e7806f77596d23a254bf6ee5876ae24c6fef34eab17106615717485664c6876\": RecentStats: unable to find data in memory cache]"
Sep 18 17:37:46.055065 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:46.055026    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6w9rp" event={"ID":"57e56b44-1893-4e11-88d8-46f01379ee03","Type":"ContainerStarted","Data":"a0f7c243d46db58c7bb9dbbf85f507325b920e8f67f56af0a3f70086c978cf3e"}
Sep 18 17:37:46.055524 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:46.055097    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6w9rp" event={"ID":"57e56b44-1893-4e11-88d8-46f01379ee03","Type":"ContainerStarted","Data":"2b2014d8a35977c700cd250601dd97c3dd6e9207b444e250f7c11c95ebe7fccc"}
Sep 18 17:37:46.055524 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:46.055308    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6w9rp"
Sep 18 17:37:46.085542 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:46.085492    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6w9rp" podStartSLOduration=2.424490923 podStartE2EDuration="13.085477511s" totalImagesPullingTime="6.628578177s" totalInitContainerRuntime="4.032408411s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:37:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:37:46.08434153 +0000 UTC m=+2656.284196816" watchObservedRunningTime="2026-09-18 17:37:46.085477511 +0000 UTC m=+2656.285332797"
Sep 18 17:37:47.060147 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:47.060115    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6w9rp"
Sep 18 17:37:47.061187 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:47.061162    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6w9rp" podUID="57e56b44-1893-4e11-88d8-46f01379ee03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 18 17:37:47.398130 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:37:47.398032    2881 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod303d1a24_b976_481f_9e07_bdf80d8abd68.slice/crio-7e7806f77596d23a254bf6ee5876ae24c6fef34eab17106615717485664c6876\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod303d1a24_b976_481f_9e07_bdf80d8abd68.slice\": RecentStats: unable to find data in memory cache]"
Sep 18 17:37:48.066023 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:48.065980    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6w9rp" podUID="57e56b44-1893-4e11-88d8-46f01379ee03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 18 17:37:53.070922 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:53.070888    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6w9rp"
Sep 18 17:37:53.071605 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:53.071577    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6w9rp" podUID="57e56b44-1893-4e11-88d8-46f01379ee03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 18 17:37:53.933304 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:37:53.933254    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-k56jw"
Sep 18 17:37:55.731026 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:37:55.730984    2881 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod303d1a24_b976_481f_9e07_bdf80d8abd68.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod303d1a24_b976_481f_9e07_bdf80d8abd68.slice/crio-7e7806f77596d23a254bf6ee5876ae24c6fef34eab17106615717485664c6876\": RecentStats: unable to find data in memory cache]"
Sep 18 17:37:57.407328 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:37:57.407276    2881 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod303d1a24_b976_481f_9e07_bdf80d8abd68.slice/crio-7e7806f77596d23a254bf6ee5876ae24c6fef34eab17106615717485664c6876\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod303d1a24_b976_481f_9e07_bdf80d8abd68.slice\": RecentStats: unable to find data in memory cache]"
Sep 18 17:38:02.402442 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:02.402404    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-k56jw"]
Sep 18 17:38:02.402895 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:02.402685    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-k56jw" podUID="a1759a40-c771-4a1e-b515-cfecd57732fd" containerName="kserve-container" containerID="cri-o://9456c24d9d3d1cabd8d5bded119204f082d79855ac4a7ce48487fda5197fa306" gracePeriod=30
Sep 18 17:38:02.402895 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:02.402727    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-k56jw" podUID="a1759a40-c771-4a1e-b515-cfecd57732fd" containerName="kube-rbac-proxy" containerID="cri-o://bf93eaed71ae1dc25204509be64c2e5dfecd1d38b27401fc0e542971c59e7fcb" gracePeriod=30
Sep 18 17:38:03.071713 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:03.071664    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6w9rp" podUID="57e56b44-1893-4e11-88d8-46f01379ee03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 18 17:38:03.139990 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:03.139955    2881 generic.go:356] "Generic (PLEG): container finished" podID="a1759a40-c771-4a1e-b515-cfecd57732fd" containerID="bf93eaed71ae1dc25204509be64c2e5dfecd1d38b27401fc0e542971c59e7fcb" exitCode=2
Sep 18 17:38:03.140173 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:03.140011    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-k56jw" event={"ID":"a1759a40-c771-4a1e-b515-cfecd57732fd","Type":"ContainerDied","Data":"bf93eaed71ae1dc25204509be64c2e5dfecd1d38b27401fc0e542971c59e7fcb"}
Sep 18 17:38:03.909072 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:03.909020    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-k56jw" podUID="a1759a40-c771-4a1e-b515-cfecd57732fd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.52:8643/healthz\": dial tcp 10.134.0.52:8643: connect: connection refused"
Sep 18 17:38:04.955449 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:04.955395    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-k56jw" podUID="a1759a40-c771-4a1e-b515-cfecd57732fd" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.52:8080/v2/models/xgboost-v2-mlserver/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 18 17:38:07.450696 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:38:07.450650    2881 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod303d1a24_b976_481f_9e07_bdf80d8abd68.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod303d1a24_b976_481f_9e07_bdf80d8abd68.slice/crio-7e7806f77596d23a254bf6ee5876ae24c6fef34eab17106615717485664c6876\": RecentStats: unable to find data in memory cache]"
Sep 18 17:38:07.803680 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:07.803643    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p8hjd"]
Sep 18 17:38:07.807427 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:07.807406    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p8hjd"
Sep 18 17:38:07.810727 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:07.810702    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-predictor-serving-cert\""
Sep 18 17:38:07.811116 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:07.811098    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\""
Sep 18 17:38:07.827161 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:07.827129    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p8hjd"]
Sep 18 17:38:07.914024 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:07.913979    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86543b82-494a-4d6f-8e38-0da899e79681-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-p8hjd\" (UID: \"86543b82-494a-4d6f-8e38-0da899e79681\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p8hjd"
Sep 18 17:38:07.914263 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:07.914042    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86543b82-494a-4d6f-8e38-0da899e79681-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-p8hjd\" (UID: \"86543b82-494a-4d6f-8e38-0da899e79681\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p8hjd"
Sep 18 17:38:07.914263 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:07.914098    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9flpl\" (UniqueName: \"kubernetes.io/projected/86543b82-494a-4d6f-8e38-0da899e79681-kube-api-access-9flpl\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-p8hjd\" (UID: \"86543b82-494a-4d6f-8e38-0da899e79681\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p8hjd"
Sep 18 17:38:07.914263 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:07.914124    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86543b82-494a-4d6f-8e38-0da899e79681-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-p8hjd\" (UID: \"86543b82-494a-4d6f-8e38-0da899e79681\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p8hjd"
Sep 18 17:38:07.914263 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:07.914145    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86543b82-494a-4d6f-8e38-0da899e79681-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-p8hjd\" (UID: \"86543b82-494a-4d6f-8e38-0da899e79681\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p8hjd"
Sep 18 17:38:08.014735 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:08.014699    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9flpl\" (UniqueName: \"kubernetes.io/projected/86543b82-494a-4d6f-8e38-0da899e79681-kube-api-access-9flpl\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-p8hjd\" (UID: \"86543b82-494a-4d6f-8e38-0da899e79681\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p8hjd"
Sep 18 17:38:08.014940 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:08.014741    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86543b82-494a-4d6f-8e38-0da899e79681-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-p8hjd\" (UID: \"86543b82-494a-4d6f-8e38-0da899e79681\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p8hjd"
Sep 18 17:38:08.015016 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:08.014995    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86543b82-494a-4d6f-8e38-0da899e79681-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-p8hjd\" (UID: \"86543b82-494a-4d6f-8e38-0da899e79681\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p8hjd"
Sep 18 17:38:08.015141 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:08.015121    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86543b82-494a-4d6f-8e38-0da899e79681-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-p8hjd\" (UID: \"86543b82-494a-4d6f-8e38-0da899e79681\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p8hjd"
Sep 18 17:38:08.015216 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:08.015192    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86543b82-494a-4d6f-8e38-0da899e79681-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-p8hjd\" (UID: \"86543b82-494a-4d6f-8e38-0da899e79681\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p8hjd"
Sep 18 17:38:08.015325 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:08.015249    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86543b82-494a-4d6f-8e38-0da899e79681-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-p8hjd\" (UID: \"86543b82-494a-4d6f-8e38-0da899e79681\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p8hjd"
Sep 18 17:38:08.015325 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:38:08.015313    2881 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-serving-cert: secret "isvc-xgboost-runtime-predictor-serving-cert" not found
Sep 18 17:38:08.015440 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:38:08.015376    2881 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86543b82-494a-4d6f-8e38-0da899e79681-proxy-tls podName:86543b82-494a-4d6f-8e38-0da899e79681 nodeName:}" failed. No retries permitted until 2026-09-18 17:38:08.515357601 +0000 UTC m=+2678.715212871 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/86543b82-494a-4d6f-8e38-0da899e79681-proxy-tls") pod "isvc-xgboost-runtime-predictor-54dd488c85-p8hjd" (UID: "86543b82-494a-4d6f-8e38-0da899e79681") : secret "isvc-xgboost-runtime-predictor-serving-cert" not found
Sep 18 17:38:08.015725 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:08.015703    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86543b82-494a-4d6f-8e38-0da899e79681-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-p8hjd\" (UID: \"86543b82-494a-4d6f-8e38-0da899e79681\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p8hjd"
Sep 18 17:38:08.015829 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:08.015775    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86543b82-494a-4d6f-8e38-0da899e79681-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-p8hjd\" (UID: \"86543b82-494a-4d6f-8e38-0da899e79681\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p8hjd"
Sep 18 17:38:08.024818 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:08.024783    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9flpl\" (UniqueName: \"kubernetes.io/projected/86543b82-494a-4d6f-8e38-0da899e79681-kube-api-access-9flpl\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-p8hjd\" (UID: \"86543b82-494a-4d6f-8e38-0da899e79681\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p8hjd"
Sep 18 17:38:08.520332 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:08.520292    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86543b82-494a-4d6f-8e38-0da899e79681-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-p8hjd\" (UID: \"86543b82-494a-4d6f-8e38-0da899e79681\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p8hjd"
Sep 18 17:38:08.523005 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:08.522968    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86543b82-494a-4d6f-8e38-0da899e79681-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-p8hjd\" (UID: \"86543b82-494a-4d6f-8e38-0da899e79681\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p8hjd"
Sep 18 17:38:08.717808 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:08.717755    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p8hjd"
Sep 18 17:38:08.909260 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:08.909139    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-k56jw" podUID="a1759a40-c771-4a1e-b515-cfecd57732fd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.52:8643/healthz\": dial tcp 10.134.0.52:8643: connect: connection refused"
Sep 18 17:38:09.000061 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:09.000022    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p8hjd"]
Sep 18 17:38:09.003942 ip-10-0-143-5 kubenswrapper[2881]: W0918 17:38:09.003916    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86543b82_494a_4d6f_8e38_0da899e79681.slice/crio-ca7d3940e5936767bb3410a5c2df382ceec24a42b69a9f3189cb5b038e8ddd77 WatchSource:0}: Error finding container ca7d3940e5936767bb3410a5c2df382ceec24a42b69a9f3189cb5b038e8ddd77: Status 404 returned error can't find the container with id ca7d3940e5936767bb3410a5c2df382ceec24a42b69a9f3189cb5b038e8ddd77
Sep 18 17:38:09.173364 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:09.173271    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p8hjd" event={"ID":"86543b82-494a-4d6f-8e38-0da899e79681","Type":"ContainerStarted","Data":"1ad5d8badec24383d8a68275d82ec08db447a21b8fe0deb9e235e9bb1fe7b871"}
Sep 18 17:38:09.173364 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:09.173335    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p8hjd" event={"ID":"86543b82-494a-4d6f-8e38-0da899e79681","Type":"ContainerStarted","Data":"ca7d3940e5936767bb3410a5c2df382ceec24a42b69a9f3189cb5b038e8ddd77"}
Sep 18 17:38:09.558652 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:09.558623    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-k56jw"
Sep 18 17:38:09.630893 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:09.630853    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1759a40-c771-4a1e-b515-cfecd57732fd-kserve-provision-location\") pod \"a1759a40-c771-4a1e-b515-cfecd57732fd\" (UID: \"a1759a40-c771-4a1e-b515-cfecd57732fd\") "
Sep 18 17:38:09.631098 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:09.630924    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a1759a40-c771-4a1e-b515-cfecd57732fd-cabundle-cert\") pod \"a1759a40-c771-4a1e-b515-cfecd57732fd\" (UID: \"a1759a40-c771-4a1e-b515-cfecd57732fd\") "
Sep 18 17:38:09.631098 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:09.630973    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1759a40-c771-4a1e-b515-cfecd57732fd-kube-rbac-proxy-sar-config\") pod \"a1759a40-c771-4a1e-b515-cfecd57732fd\" (UID: \"a1759a40-c771-4a1e-b515-cfecd57732fd\") "
Sep 18 17:38:09.631098 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:09.630996    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1759a40-c771-4a1e-b515-cfecd57732fd-proxy-tls\") pod \"a1759a40-c771-4a1e-b515-cfecd57732fd\" (UID: \"a1759a40-c771-4a1e-b515-cfecd57732fd\") "
Sep 18 17:38:09.631098 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:09.631091    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fzwnb\" (UniqueName: \"kubernetes.io/projected/a1759a40-c771-4a1e-b515-cfecd57732fd-kube-api-access-fzwnb\") pod \"a1759a40-c771-4a1e-b515-cfecd57732fd\" (UID: \"a1759a40-c771-4a1e-b515-cfecd57732fd\") "
Sep 18 17:38:09.631368 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:09.631337    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1759a40-c771-4a1e-b515-cfecd57732fd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a1759a40-c771-4a1e-b515-cfecd57732fd" (UID: "a1759a40-c771-4a1e-b515-cfecd57732fd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:38:09.631431 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:09.631390    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1759a40-c771-4a1e-b515-cfecd57732fd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a1759a40-c771-4a1e-b515-cfecd57732fd" (UID: "a1759a40-c771-4a1e-b515-cfecd57732fd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:38:09.631431 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:09.631404    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1759a40-c771-4a1e-b515-cfecd57732fd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a1759a40-c771-4a1e-b515-cfecd57732fd" (UID: "a1759a40-c771-4a1e-b515-cfecd57732fd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:38:09.633352 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:09.633301    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1759a40-c771-4a1e-b515-cfecd57732fd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a1759a40-c771-4a1e-b515-cfecd57732fd" (UID: "a1759a40-c771-4a1e-b515-cfecd57732fd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:38:09.633510 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:09.633410    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1759a40-c771-4a1e-b515-cfecd57732fd-kube-api-access-fzwnb" (OuterVolumeSpecName: "kube-api-access-fzwnb") pod "a1759a40-c771-4a1e-b515-cfecd57732fd" (UID: "a1759a40-c771-4a1e-b515-cfecd57732fd"). InnerVolumeSpecName "kube-api-access-fzwnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:38:09.732667 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:09.732559    2881 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1759a40-c771-4a1e-b515-cfecd57732fd-kserve-provision-location\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:38:09.732667 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:09.732592    2881 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a1759a40-c771-4a1e-b515-cfecd57732fd-cabundle-cert\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:38:09.732667 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:09.732602    2881 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1759a40-c771-4a1e-b515-cfecd57732fd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:38:09.732667 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:09.732614    2881 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1759a40-c771-4a1e-b515-cfecd57732fd-proxy-tls\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:38:09.732667 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:09.732623    2881 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzwnb\" (UniqueName: \"kubernetes.io/projected/a1759a40-c771-4a1e-b515-cfecd57732fd-kube-api-access-fzwnb\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:38:10.179646 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:10.179599    2881 generic.go:356] "Generic (PLEG): container finished" podID="a1759a40-c771-4a1e-b515-cfecd57732fd" containerID="9456c24d9d3d1cabd8d5bded119204f082d79855ac4a7ce48487fda5197fa306" exitCode=0
Sep 18 17:38:10.179844 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:10.179672    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-k56jw"
Sep 18 17:38:10.179844 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:10.179668    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-k56jw" event={"ID":"a1759a40-c771-4a1e-b515-cfecd57732fd","Type":"ContainerDied","Data":"9456c24d9d3d1cabd8d5bded119204f082d79855ac4a7ce48487fda5197fa306"}
Sep 18 17:38:10.179844 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:10.179810    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-k56jw" event={"ID":"a1759a40-c771-4a1e-b515-cfecd57732fd","Type":"ContainerDied","Data":"c7a63efc5064af717a12bb6fc737c26e2c01357442fe89fac1093b0a7b242975"}
Sep 18 17:38:10.179844 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:10.179840    2881 scope.go:117] "RemoveContainer" containerID="bf93eaed71ae1dc25204509be64c2e5dfecd1d38b27401fc0e542971c59e7fcb"
Sep 18 17:38:10.188629 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:10.188603    2881 scope.go:117] "RemoveContainer" containerID="9456c24d9d3d1cabd8d5bded119204f082d79855ac4a7ce48487fda5197fa306"
Sep 18 17:38:10.197610 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:10.197589    2881 scope.go:117] "RemoveContainer" containerID="b4aa4d6839e560dbb72ca8ef05d20cf4a76ab33b83f37640a08abe435be1f88f"
Sep 18 17:38:10.200688 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:10.200659    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-k56jw"]
Sep 18 17:38:10.205956 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:10.205925    2881 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-k56jw"]
Sep 18 17:38:10.207761 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:10.207740    2881 scope.go:117] "RemoveContainer" containerID="bf93eaed71ae1dc25204509be64c2e5dfecd1d38b27401fc0e542971c59e7fcb"
Sep 18 17:38:10.208101 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:38:10.208080    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf93eaed71ae1dc25204509be64c2e5dfecd1d38b27401fc0e542971c59e7fcb\": container with ID starting with bf93eaed71ae1dc25204509be64c2e5dfecd1d38b27401fc0e542971c59e7fcb not found: ID does not exist" containerID="bf93eaed71ae1dc25204509be64c2e5dfecd1d38b27401fc0e542971c59e7fcb"
Sep 18 17:38:10.208181 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:10.208116    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf93eaed71ae1dc25204509be64c2e5dfecd1d38b27401fc0e542971c59e7fcb"} err="failed to get container status \"bf93eaed71ae1dc25204509be64c2e5dfecd1d38b27401fc0e542971c59e7fcb\": rpc error: code = NotFound desc = could not find container \"bf93eaed71ae1dc25204509be64c2e5dfecd1d38b27401fc0e542971c59e7fcb\": container with ID starting with bf93eaed71ae1dc25204509be64c2e5dfecd1d38b27401fc0e542971c59e7fcb not found: ID does not exist"
Sep 18 17:38:10.208181 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:10.208143    2881 scope.go:117] "RemoveContainer" containerID="9456c24d9d3d1cabd8d5bded119204f082d79855ac4a7ce48487fda5197fa306"
Sep 18 17:38:10.208496 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:38:10.208478    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9456c24d9d3d1cabd8d5bded119204f082d79855ac4a7ce48487fda5197fa306\": container with ID starting with 9456c24d9d3d1cabd8d5bded119204f082d79855ac4a7ce48487fda5197fa306 not found: ID does not exist" containerID="9456c24d9d3d1cabd8d5bded119204f082d79855ac4a7ce48487fda5197fa306"
Sep 18 17:38:10.208542 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:10.208502    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9456c24d9d3d1cabd8d5bded119204f082d79855ac4a7ce48487fda5197fa306"} err="failed to get container status \"9456c24d9d3d1cabd8d5bded119204f082d79855ac4a7ce48487fda5197fa306\": rpc error: code = NotFound desc = could not find container \"9456c24d9d3d1cabd8d5bded119204f082d79855ac4a7ce48487fda5197fa306\": container with ID starting with 9456c24d9d3d1cabd8d5bded119204f082d79855ac4a7ce48487fda5197fa306 not found: ID does not exist"
Sep 18 17:38:10.208542 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:10.208517    2881 scope.go:117] "RemoveContainer" containerID="b4aa4d6839e560dbb72ca8ef05d20cf4a76ab33b83f37640a08abe435be1f88f"
Sep 18 17:38:10.208767 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:38:10.208749    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4aa4d6839e560dbb72ca8ef05d20cf4a76ab33b83f37640a08abe435be1f88f\": container with ID starting with b4aa4d6839e560dbb72ca8ef05d20cf4a76ab33b83f37640a08abe435be1f88f not found: ID does not exist" containerID="b4aa4d6839e560dbb72ca8ef05d20cf4a76ab33b83f37640a08abe435be1f88f"
Sep 18 17:38:10.208824 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:10.208777    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4aa4d6839e560dbb72ca8ef05d20cf4a76ab33b83f37640a08abe435be1f88f"} err="failed to get container status \"b4aa4d6839e560dbb72ca8ef05d20cf4a76ab33b83f37640a08abe435be1f88f\": rpc error: code = NotFound desc = could not find container \"b4aa4d6839e560dbb72ca8ef05d20cf4a76ab33b83f37640a08abe435be1f88f\": container with ID starting with b4aa4d6839e560dbb72ca8ef05d20cf4a76ab33b83f37640a08abe435be1f88f not found: ID does not exist"
Sep 18 17:38:10.699683 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:38:10.699643    2881 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod303d1a24_b976_481f_9e07_bdf80d8abd68.slice/crio-7e7806f77596d23a254bf6ee5876ae24c6fef34eab17106615717485664c6876\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod303d1a24_b976_481f_9e07_bdf80d8abd68.slice\": RecentStats: unable to find data in memory cache]"
Sep 18 17:38:11.893773 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:11.893740    2881 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1759a40-c771-4a1e-b515-cfecd57732fd" path="/var/lib/kubelet/pods/a1759a40-c771-4a1e-b515-cfecd57732fd/volumes"
Sep 18 17:38:13.072256 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:13.072186    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6w9rp" podUID="57e56b44-1893-4e11-88d8-46f01379ee03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 18 17:38:13.197643 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:13.197548    2881 generic.go:356] "Generic (PLEG): container finished" podID="86543b82-494a-4d6f-8e38-0da899e79681" containerID="1ad5d8badec24383d8a68275d82ec08db447a21b8fe0deb9e235e9bb1fe7b871" exitCode=0
Sep 18 17:38:13.197826 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:13.197636    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p8hjd" event={"ID":"86543b82-494a-4d6f-8e38-0da899e79681","Type":"ContainerDied","Data":"1ad5d8badec24383d8a68275d82ec08db447a21b8fe0deb9e235e9bb1fe7b871"}
Sep 18 17:38:14.205549 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:14.205505    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p8hjd" event={"ID":"86543b82-494a-4d6f-8e38-0da899e79681","Type":"ContainerStarted","Data":"b79a125a4b94ab55afd906e5136acda6d6ea71588b8c546492bf89a086c1d612"}
Sep 18 17:38:14.205983 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:14.205623    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p8hjd" event={"ID":"86543b82-494a-4d6f-8e38-0da899e79681","Type":"ContainerStarted","Data":"df9ce4c741d420f9c3f20e3660842d4b5f3e430a2734ce2f60c0f4e665e01798"}
Sep 18 17:38:14.205983 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:14.205836    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p8hjd"
Sep 18 17:38:14.227506 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:14.227451    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p8hjd" podStartSLOduration=3.368160435 podStartE2EDuration="7.227434932s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.859274497s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:38:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:38:14.226764169 +0000 UTC m=+2684.426619455" watchObservedRunningTime="2026-09-18 17:38:14.227434932 +0000 UTC m=+2684.427290217"
Sep 18 17:38:15.211777 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:15.211687    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p8hjd"
Sep 18 17:38:15.213248 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:15.213202    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p8hjd" podUID="86543b82-494a-4d6f-8e38-0da899e79681" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 18 17:38:16.184774 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:38:16.184721    2881 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod303d1a24_b976_481f_9e07_bdf80d8abd68.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod303d1a24_b976_481f_9e07_bdf80d8abd68.slice/crio-7e7806f77596d23a254bf6ee5876ae24c6fef34eab17106615717485664c6876\": RecentStats: unable to find data in memory cache]"
Sep 18 17:38:16.184981 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:38:16.184721    2881 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod303d1a24_b976_481f_9e07_bdf80d8abd68.slice/crio-7e7806f77596d23a254bf6ee5876ae24c6fef34eab17106615717485664c6876\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod303d1a24_b976_481f_9e07_bdf80d8abd68.slice\": RecentStats: unable to find data in memory cache]"
Sep 18 17:38:16.222796 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:16.222746    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p8hjd" podUID="86543b82-494a-4d6f-8e38-0da899e79681" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 18 17:38:17.460237 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:38:17.460177    2881 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod303d1a24_b976_481f_9e07_bdf80d8abd68.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod303d1a24_b976_481f_9e07_bdf80d8abd68.slice/crio-7e7806f77596d23a254bf6ee5876ae24c6fef34eab17106615717485664c6876\": RecentStats: unable to find data in memory cache]"
Sep 18 17:38:21.227419 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:21.227382    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p8hjd"
Sep 18 17:38:21.228007 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:21.227979    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p8hjd" podUID="86543b82-494a-4d6f-8e38-0da899e79681" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 18 17:38:23.071796 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:23.071755    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6w9rp" podUID="57e56b44-1893-4e11-88d8-46f01379ee03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 18 17:38:25.725637 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:38:25.725592    2881 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod303d1a24_b976_481f_9e07_bdf80d8abd68.slice/crio-7e7806f77596d23a254bf6ee5876ae24c6fef34eab17106615717485664c6876\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod303d1a24_b976_481f_9e07_bdf80d8abd68.slice\": RecentStats: unable to find data in memory cache]"
Sep 18 17:38:27.469674 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:38:27.469640    2881 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod303d1a24_b976_481f_9e07_bdf80d8abd68.slice/crio-7e7806f77596d23a254bf6ee5876ae24c6fef34eab17106615717485664c6876\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod303d1a24_b976_481f_9e07_bdf80d8abd68.slice\": RecentStats: unable to find data in memory cache]"
Sep 18 17:38:31.227949 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:31.227906    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p8hjd" podUID="86543b82-494a-4d6f-8e38-0da899e79681" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 18 17:38:33.071731 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:33.071695    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6w9rp" podUID="57e56b44-1893-4e11-88d8-46f01379ee03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 18 17:38:41.227869 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:41.227822    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p8hjd" podUID="86543b82-494a-4d6f-8e38-0da899e79681" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 18 17:38:43.071585 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:43.071540    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6w9rp" podUID="57e56b44-1893-4e11-88d8-46f01379ee03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 18 17:38:51.228037 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:51.227985    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p8hjd" podUID="86543b82-494a-4d6f-8e38-0da899e79681" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 18 17:38:53.072543 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:53.072496    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6w9rp" podUID="57e56b44-1893-4e11-88d8-46f01379ee03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 18 17:38:54.888484 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:38:54.888438    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6w9rp" podUID="57e56b44-1893-4e11-88d8-46f01379ee03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 18 17:39:01.228503 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:01.228413    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p8hjd" podUID="86543b82-494a-4d6f-8e38-0da899e79681" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 18 17:39:04.888498 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:04.888457    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6w9rp" podUID="57e56b44-1893-4e11-88d8-46f01379ee03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 18 17:39:11.228581 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:11.228519    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p8hjd" podUID="86543b82-494a-4d6f-8e38-0da899e79681" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 18 17:39:14.888589 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:14.888536    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6w9rp" podUID="57e56b44-1893-4e11-88d8-46f01379ee03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 18 17:39:21.228713 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:21.228667    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p8hjd" podUID="86543b82-494a-4d6f-8e38-0da899e79681" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 18 17:39:24.889375 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:24.889341    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6w9rp"
Sep 18 17:39:31.229424 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:31.229387    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p8hjd"
Sep 18 17:39:34.656552 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:34.656508    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6w9rp"]
Sep 18 17:39:34.657525 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:34.657467    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6w9rp" podUID="57e56b44-1893-4e11-88d8-46f01379ee03" containerName="kserve-container" containerID="cri-o://2b2014d8a35977c700cd250601dd97c3dd6e9207b444e250f7c11c95ebe7fccc" gracePeriod=30
Sep 18 17:39:34.657952 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:34.657906    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6w9rp" podUID="57e56b44-1893-4e11-88d8-46f01379ee03" containerName="kube-rbac-proxy" containerID="cri-o://a0f7c243d46db58c7bb9dbbf85f507325b920e8f67f56af0a3f70086c978cf3e" gracePeriod=30
Sep 18 17:39:34.888898 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:34.888851    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6w9rp" podUID="57e56b44-1893-4e11-88d8-46f01379ee03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 18 17:39:35.588025 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:35.587983    2881 generic.go:356] "Generic (PLEG): container finished" podID="57e56b44-1893-4e11-88d8-46f01379ee03" containerID="a0f7c243d46db58c7bb9dbbf85f507325b920e8f67f56af0a3f70086c978cf3e" exitCode=2
Sep 18 17:39:35.588216 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:35.588053    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6w9rp" event={"ID":"57e56b44-1893-4e11-88d8-46f01379ee03","Type":"ContainerDied","Data":"a0f7c243d46db58c7bb9dbbf85f507325b920e8f67f56af0a3f70086c978cf3e"}
Sep 18 17:39:38.067309 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:38.067267    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6w9rp" podUID="57e56b44-1893-4e11-88d8-46f01379ee03" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.53:8643/healthz\": dial tcp 10.134.0.53:8643: connect: connection refused"
Sep 18 17:39:38.277710 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:38.277674    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p8hjd"]
Sep 18 17:39:38.277942 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:38.277914    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p8hjd" podUID="86543b82-494a-4d6f-8e38-0da899e79681" containerName="kserve-container" containerID="cri-o://df9ce4c741d420f9c3f20e3660842d4b5f3e430a2734ce2f60c0f4e665e01798" gracePeriod=30
Sep 18 17:39:38.278083 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:38.277983    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p8hjd" podUID="86543b82-494a-4d6f-8e38-0da899e79681" containerName="kube-rbac-proxy" containerID="cri-o://b79a125a4b94ab55afd906e5136acda6d6ea71588b8c546492bf89a086c1d612" gracePeriod=30
Sep 18 17:39:38.603298 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:38.603268    2881 generic.go:356] "Generic (PLEG): container finished" podID="86543b82-494a-4d6f-8e38-0da899e79681" containerID="b79a125a4b94ab55afd906e5136acda6d6ea71588b8c546492bf89a086c1d612" exitCode=2
Sep 18 17:39:38.603478 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:38.603340    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p8hjd" event={"ID":"86543b82-494a-4d6f-8e38-0da899e79681","Type":"ContainerDied","Data":"b79a125a4b94ab55afd906e5136acda6d6ea71588b8c546492bf89a086c1d612"}
Sep 18 17:39:39.857410 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:39.857367    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nskpc"]
Sep 18 17:39:39.857920 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:39.857898    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1759a40-c771-4a1e-b515-cfecd57732fd" containerName="kserve-container"
Sep 18 17:39:39.858012 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:39.857922    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1759a40-c771-4a1e-b515-cfecd57732fd" containerName="kserve-container"
Sep 18 17:39:39.858012 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:39.857936    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1759a40-c771-4a1e-b515-cfecd57732fd" containerName="kube-rbac-proxy"
Sep 18 17:39:39.858012 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:39.857945    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1759a40-c771-4a1e-b515-cfecd57732fd" containerName="kube-rbac-proxy"
Sep 18 17:39:39.858012 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:39.857973    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1759a40-c771-4a1e-b515-cfecd57732fd" containerName="storage-initializer"
Sep 18 17:39:39.858012 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:39.857983    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1759a40-c771-4a1e-b515-cfecd57732fd" containerName="storage-initializer"
Sep 18 17:39:39.858320 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:39.858086    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1759a40-c771-4a1e-b515-cfecd57732fd" containerName="kube-rbac-proxy"
Sep 18 17:39:39.858320 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:39.858106    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1759a40-c771-4a1e-b515-cfecd57732fd" containerName="kserve-container"
Sep 18 17:39:39.861301 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:39.861277    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nskpc"
Sep 18 17:39:39.863759 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:39.863730    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-predictor-serving-cert\""
Sep 18 17:39:39.864006 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:39.863989    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\""
Sep 18 17:39:39.872019 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:39.871994    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nskpc"]
Sep 18 17:39:39.934962 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:39.934910    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f1bf9a13-4bdf-46f6-a80a-ea425450c0b3-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-nskpc\" (UID: \"f1bf9a13-4bdf-46f6-a80a-ea425450c0b3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nskpc"
Sep 18 17:39:39.935150 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:39.934974    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1bf9a13-4bdf-46f6-a80a-ea425450c0b3-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-nskpc\" (UID: \"f1bf9a13-4bdf-46f6-a80a-ea425450c0b3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nskpc"
Sep 18 17:39:39.935150 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:39.935012    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f1bf9a13-4bdf-46f6-a80a-ea425450c0b3-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-nskpc\" (UID: \"f1bf9a13-4bdf-46f6-a80a-ea425450c0b3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nskpc"
Sep 18 17:39:39.935150 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:39.935049    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f1bf9a13-4bdf-46f6-a80a-ea425450c0b3-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-nskpc\" (UID: \"f1bf9a13-4bdf-46f6-a80a-ea425450c0b3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nskpc"
Sep 18 17:39:39.935150 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:39.935098    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69xdg\" (UniqueName: \"kubernetes.io/projected/f1bf9a13-4bdf-46f6-a80a-ea425450c0b3-kube-api-access-69xdg\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-nskpc\" (UID: \"f1bf9a13-4bdf-46f6-a80a-ea425450c0b3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nskpc"
Sep 18 17:39:40.036611 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:40.036551    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f1bf9a13-4bdf-46f6-a80a-ea425450c0b3-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-nskpc\" (UID: \"f1bf9a13-4bdf-46f6-a80a-ea425450c0b3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nskpc"
Sep 18 17:39:40.036828 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:40.036626    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1bf9a13-4bdf-46f6-a80a-ea425450c0b3-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-nskpc\" (UID: \"f1bf9a13-4bdf-46f6-a80a-ea425450c0b3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nskpc"
Sep 18 17:39:40.036828 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:40.036667    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f1bf9a13-4bdf-46f6-a80a-ea425450c0b3-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-nskpc\" (UID: \"f1bf9a13-4bdf-46f6-a80a-ea425450c0b3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nskpc"
Sep 18 17:39:40.036828 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:40.036709    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f1bf9a13-4bdf-46f6-a80a-ea425450c0b3-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-nskpc\" (UID: \"f1bf9a13-4bdf-46f6-a80a-ea425450c0b3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nskpc"
Sep 18 17:39:40.036828 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:39:40.036738    2881 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-serving-cert: secret "isvc-lightgbm-runtime-predictor-serving-cert" not found
Sep 18 17:39:40.036828 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:40.036770    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69xdg\" (UniqueName: \"kubernetes.io/projected/f1bf9a13-4bdf-46f6-a80a-ea425450c0b3-kube-api-access-69xdg\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-nskpc\" (UID: \"f1bf9a13-4bdf-46f6-a80a-ea425450c0b3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nskpc"
Sep 18 17:39:40.036828 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:39:40.036826    2881 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1bf9a13-4bdf-46f6-a80a-ea425450c0b3-proxy-tls podName:f1bf9a13-4bdf-46f6-a80a-ea425450c0b3 nodeName:}" failed. No retries permitted until 2026-09-18 17:39:40.536803195 +0000 UTC m=+2770.736658459 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f1bf9a13-4bdf-46f6-a80a-ea425450c0b3-proxy-tls") pod "isvc-lightgbm-runtime-predictor-577f7db7cb-nskpc" (UID: "f1bf9a13-4bdf-46f6-a80a-ea425450c0b3") : secret "isvc-lightgbm-runtime-predictor-serving-cert" not found
Sep 18 17:39:40.037160 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:40.037135    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1bf9a13-4bdf-46f6-a80a-ea425450c0b3-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-nskpc\" (UID: \"f1bf9a13-4bdf-46f6-a80a-ea425450c0b3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nskpc"
Sep 18 17:39:40.037580 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:40.037555    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f1bf9a13-4bdf-46f6-a80a-ea425450c0b3-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-nskpc\" (UID: \"f1bf9a13-4bdf-46f6-a80a-ea425450c0b3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nskpc"
Sep 18 17:39:40.037729 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:40.037713    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f1bf9a13-4bdf-46f6-a80a-ea425450c0b3-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-nskpc\" (UID: \"f1bf9a13-4bdf-46f6-a80a-ea425450c0b3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nskpc"
Sep 18 17:39:40.046030 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:40.046001    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69xdg\" (UniqueName: \"kubernetes.io/projected/f1bf9a13-4bdf-46f6-a80a-ea425450c0b3-kube-api-access-69xdg\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-nskpc\" (UID: \"f1bf9a13-4bdf-46f6-a80a-ea425450c0b3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nskpc"
Sep 18 17:39:40.542075 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:40.542017    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f1bf9a13-4bdf-46f6-a80a-ea425450c0b3-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-nskpc\" (UID: \"f1bf9a13-4bdf-46f6-a80a-ea425450c0b3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nskpc"
Sep 18 17:39:40.544581 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:40.544548    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f1bf9a13-4bdf-46f6-a80a-ea425450c0b3-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-nskpc\" (UID: \"f1bf9a13-4bdf-46f6-a80a-ea425450c0b3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nskpc"
Sep 18 17:39:40.772541 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:40.772502    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nskpc"
Sep 18 17:39:41.050860 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:41.050827    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nskpc"]
Sep 18 17:39:41.053083 ip-10-0-143-5 kubenswrapper[2881]: W0918 17:39:41.053048    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1bf9a13_4bdf_46f6_a80a_ea425450c0b3.slice/crio-19318f0cc4ee749231e4c2ac0696b8ba2408f40ffb7ab32d4dbfdad269e58954 WatchSource:0}: Error finding container 19318f0cc4ee749231e4c2ac0696b8ba2408f40ffb7ab32d4dbfdad269e58954: Status 404 returned error can't find the container with id 19318f0cc4ee749231e4c2ac0696b8ba2408f40ffb7ab32d4dbfdad269e58954
Sep 18 17:39:41.119880 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:41.119854    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6w9rp"
Sep 18 17:39:41.223368 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:41.223322    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p8hjd" podUID="86543b82-494a-4d6f-8e38-0da899e79681" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.54:8643/healthz\": dial tcp 10.134.0.54:8643: connect: connection refused"
Sep 18 17:39:41.228787 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:41.228751    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p8hjd" podUID="86543b82-494a-4d6f-8e38-0da899e79681" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 18 17:39:41.248527 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:41.248491    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57e56b44-1893-4e11-88d8-46f01379ee03-proxy-tls\") pod \"57e56b44-1893-4e11-88d8-46f01379ee03\" (UID: \"57e56b44-1893-4e11-88d8-46f01379ee03\") "
Sep 18 17:39:41.248727 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:41.248553    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57e56b44-1893-4e11-88d8-46f01379ee03-cabundle-cert\") pod \"57e56b44-1893-4e11-88d8-46f01379ee03\" (UID: \"57e56b44-1893-4e11-88d8-46f01379ee03\") "
Sep 18 17:39:41.248727 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:41.248580    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57e56b44-1893-4e11-88d8-46f01379ee03-kube-rbac-proxy-sar-config\") pod \"57e56b44-1893-4e11-88d8-46f01379ee03\" (UID: \"57e56b44-1893-4e11-88d8-46f01379ee03\") "
Sep 18 17:39:41.248727 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:41.248657    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cb5mq\" (UniqueName: \"kubernetes.io/projected/57e56b44-1893-4e11-88d8-46f01379ee03-kube-api-access-cb5mq\") pod \"57e56b44-1893-4e11-88d8-46f01379ee03\" (UID: \"57e56b44-1893-4e11-88d8-46f01379ee03\") "
Sep 18 17:39:41.248727 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:41.248688    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57e56b44-1893-4e11-88d8-46f01379ee03-kserve-provision-location\") pod \"57e56b44-1893-4e11-88d8-46f01379ee03\" (UID: \"57e56b44-1893-4e11-88d8-46f01379ee03\") "
Sep 18 17:39:41.248992 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:41.248967    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57e56b44-1893-4e11-88d8-46f01379ee03-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "57e56b44-1893-4e11-88d8-46f01379ee03" (UID: "57e56b44-1893-4e11-88d8-46f01379ee03"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:39:41.249058 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:41.249008    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57e56b44-1893-4e11-88d8-46f01379ee03-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "57e56b44-1893-4e11-88d8-46f01379ee03" (UID: "57e56b44-1893-4e11-88d8-46f01379ee03"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:39:41.249115 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:41.249059    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57e56b44-1893-4e11-88d8-46f01379ee03-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "57e56b44-1893-4e11-88d8-46f01379ee03" (UID: "57e56b44-1893-4e11-88d8-46f01379ee03"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:39:41.250825 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:41.250797    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57e56b44-1893-4e11-88d8-46f01379ee03-kube-api-access-cb5mq" (OuterVolumeSpecName: "kube-api-access-cb5mq") pod "57e56b44-1893-4e11-88d8-46f01379ee03" (UID: "57e56b44-1893-4e11-88d8-46f01379ee03"). InnerVolumeSpecName "kube-api-access-cb5mq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:39:41.250825 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:41.250808    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57e56b44-1893-4e11-88d8-46f01379ee03-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "57e56b44-1893-4e11-88d8-46f01379ee03" (UID: "57e56b44-1893-4e11-88d8-46f01379ee03"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:39:41.350242 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:41.350142    2881 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57e56b44-1893-4e11-88d8-46f01379ee03-cabundle-cert\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:39:41.350242 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:41.350177    2881 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57e56b44-1893-4e11-88d8-46f01379ee03-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:39:41.350242 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:41.350187    2881 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb5mq\" (UniqueName: \"kubernetes.io/projected/57e56b44-1893-4e11-88d8-46f01379ee03-kube-api-access-cb5mq\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:39:41.350242 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:41.350196    2881 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57e56b44-1893-4e11-88d8-46f01379ee03-kserve-provision-location\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:39:41.350242 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:41.350206    2881 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57e56b44-1893-4e11-88d8-46f01379ee03-proxy-tls\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:39:41.619595 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:41.619498    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nskpc" event={"ID":"f1bf9a13-4bdf-46f6-a80a-ea425450c0b3","Type":"ContainerStarted","Data":"0e83ce0ada936975aeff43e9c0eea879e5a6a44c315698812e951b305e72e514"}
Sep 18 17:39:41.619595 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:41.619575    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nskpc" event={"ID":"f1bf9a13-4bdf-46f6-a80a-ea425450c0b3","Type":"ContainerStarted","Data":"19318f0cc4ee749231e4c2ac0696b8ba2408f40ffb7ab32d4dbfdad269e58954"}
Sep 18 17:39:41.621531 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:41.621496    2881 generic.go:356] "Generic (PLEG): container finished" podID="57e56b44-1893-4e11-88d8-46f01379ee03" containerID="2b2014d8a35977c700cd250601dd97c3dd6e9207b444e250f7c11c95ebe7fccc" exitCode=0
Sep 18 17:39:41.621650 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:41.621553    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6w9rp"
Sep 18 17:39:41.621650 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:41.621551    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6w9rp" event={"ID":"57e56b44-1893-4e11-88d8-46f01379ee03","Type":"ContainerDied","Data":"2b2014d8a35977c700cd250601dd97c3dd6e9207b444e250f7c11c95ebe7fccc"}
Sep 18 17:39:41.621750 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:41.621680    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6w9rp" event={"ID":"57e56b44-1893-4e11-88d8-46f01379ee03","Type":"ContainerDied","Data":"df73e342f9733d57bd62612f42cd5cad914e92c4e4d9e0df39dca0ccabca04a6"}
Sep 18 17:39:41.621750 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:41.621723    2881 scope.go:117] "RemoveContainer" containerID="a0f7c243d46db58c7bb9dbbf85f507325b920e8f67f56af0a3f70086c978cf3e"
Sep 18 17:39:41.631522 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:41.631496    2881 scope.go:117] "RemoveContainer" containerID="2b2014d8a35977c700cd250601dd97c3dd6e9207b444e250f7c11c95ebe7fccc"
Sep 18 17:39:41.641145 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:41.641109    2881 scope.go:117] "RemoveContainer" containerID="a753e78d3593f336c702d581cb34ac45a1540507de8d48dd6d3fd0e52d053348"
Sep 18 17:39:41.652193 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:41.652166    2881 scope.go:117] "RemoveContainer" containerID="a0f7c243d46db58c7bb9dbbf85f507325b920e8f67f56af0a3f70086c978cf3e"
Sep 18 17:39:41.652659 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:39:41.652615    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0f7c243d46db58c7bb9dbbf85f507325b920e8f67f56af0a3f70086c978cf3e\": container with ID starting with a0f7c243d46db58c7bb9dbbf85f507325b920e8f67f56af0a3f70086c978cf3e not found: ID does not exist" containerID="a0f7c243d46db58c7bb9dbbf85f507325b920e8f67f56af0a3f70086c978cf3e"
Sep 18 17:39:41.652791 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:41.652659    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0f7c243d46db58c7bb9dbbf85f507325b920e8f67f56af0a3f70086c978cf3e"} err="failed to get container status \"a0f7c243d46db58c7bb9dbbf85f507325b920e8f67f56af0a3f70086c978cf3e\": rpc error: code = NotFound desc = could not find container \"a0f7c243d46db58c7bb9dbbf85f507325b920e8f67f56af0a3f70086c978cf3e\": container with ID starting with a0f7c243d46db58c7bb9dbbf85f507325b920e8f67f56af0a3f70086c978cf3e not found: ID does not exist"
Sep 18 17:39:41.652791 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:41.652684    2881 scope.go:117] "RemoveContainer" containerID="2b2014d8a35977c700cd250601dd97c3dd6e9207b444e250f7c11c95ebe7fccc"
Sep 18 17:39:41.653446 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:39:41.653295    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b2014d8a35977c700cd250601dd97c3dd6e9207b444e250f7c11c95ebe7fccc\": container with ID starting with 2b2014d8a35977c700cd250601dd97c3dd6e9207b444e250f7c11c95ebe7fccc not found: ID does not exist" containerID="2b2014d8a35977c700cd250601dd97c3dd6e9207b444e250f7c11c95ebe7fccc"
Sep 18 17:39:41.653446 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:41.653331    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b2014d8a35977c700cd250601dd97c3dd6e9207b444e250f7c11c95ebe7fccc"} err="failed to get container status \"2b2014d8a35977c700cd250601dd97c3dd6e9207b444e250f7c11c95ebe7fccc\": rpc error: code = NotFound desc = could not find container \"2b2014d8a35977c700cd250601dd97c3dd6e9207b444e250f7c11c95ebe7fccc\": container with ID starting with 2b2014d8a35977c700cd250601dd97c3dd6e9207b444e250f7c11c95ebe7fccc not found: ID does not exist"
Sep 18 17:39:41.653446 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:41.653356    2881 scope.go:117] "RemoveContainer" containerID="a753e78d3593f336c702d581cb34ac45a1540507de8d48dd6d3fd0e52d053348"
Sep 18 17:39:41.653704 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:39:41.653682    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a753e78d3593f336c702d581cb34ac45a1540507de8d48dd6d3fd0e52d053348\": container with ID starting with a753e78d3593f336c702d581cb34ac45a1540507de8d48dd6d3fd0e52d053348 not found: ID does not exist" containerID="a753e78d3593f336c702d581cb34ac45a1540507de8d48dd6d3fd0e52d053348"
Sep 18 17:39:41.653747 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:41.653712    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a753e78d3593f336c702d581cb34ac45a1540507de8d48dd6d3fd0e52d053348"} err="failed to get container status \"a753e78d3593f336c702d581cb34ac45a1540507de8d48dd6d3fd0e52d053348\": rpc error: code = NotFound desc = could not find container \"a753e78d3593f336c702d581cb34ac45a1540507de8d48dd6d3fd0e52d053348\": container with ID starting with a753e78d3593f336c702d581cb34ac45a1540507de8d48dd6d3fd0e52d053348 not found: ID does not exist"
Sep 18 17:39:41.655497 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:41.655476    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6w9rp"]
Sep 18 17:39:41.660673 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:41.660646    2881 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6w9rp"]
Sep 18 17:39:41.893304 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:41.893194    2881 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57e56b44-1893-4e11-88d8-46f01379ee03" path="/var/lib/kubelet/pods/57e56b44-1893-4e11-88d8-46f01379ee03/volumes"
Sep 18 17:39:43.017505 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:43.017480    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p8hjd"
Sep 18 17:39:43.066511 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:43.066476    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9flpl\" (UniqueName: \"kubernetes.io/projected/86543b82-494a-4d6f-8e38-0da899e79681-kube-api-access-9flpl\") pod \"86543b82-494a-4d6f-8e38-0da899e79681\" (UID: \"86543b82-494a-4d6f-8e38-0da899e79681\") "
Sep 18 17:39:43.066715 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:43.066573    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86543b82-494a-4d6f-8e38-0da899e79681-kserve-provision-location\") pod \"86543b82-494a-4d6f-8e38-0da899e79681\" (UID: \"86543b82-494a-4d6f-8e38-0da899e79681\") "
Sep 18 17:39:43.066715 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:43.066612    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86543b82-494a-4d6f-8e38-0da899e79681-kube-rbac-proxy-sar-config\") pod \"86543b82-494a-4d6f-8e38-0da899e79681\" (UID: \"86543b82-494a-4d6f-8e38-0da899e79681\") "
Sep 18 17:39:43.066715 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:43.066663    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86543b82-494a-4d6f-8e38-0da899e79681-cabundle-cert\") pod \"86543b82-494a-4d6f-8e38-0da899e79681\" (UID: \"86543b82-494a-4d6f-8e38-0da899e79681\") "
Sep 18 17:39:43.066715 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:43.066693    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86543b82-494a-4d6f-8e38-0da899e79681-proxy-tls\") pod \"86543b82-494a-4d6f-8e38-0da899e79681\" (UID: \"86543b82-494a-4d6f-8e38-0da899e79681\") "
Sep 18 17:39:43.066970 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:43.066947    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86543b82-494a-4d6f-8e38-0da899e79681-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "86543b82-494a-4d6f-8e38-0da899e79681" (UID: "86543b82-494a-4d6f-8e38-0da899e79681"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:39:43.067074 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:43.067052    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86543b82-494a-4d6f-8e38-0da899e79681-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "86543b82-494a-4d6f-8e38-0da899e79681" (UID: "86543b82-494a-4d6f-8e38-0da899e79681"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:39:43.067114 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:43.067057    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86543b82-494a-4d6f-8e38-0da899e79681-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "86543b82-494a-4d6f-8e38-0da899e79681" (UID: "86543b82-494a-4d6f-8e38-0da899e79681"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:39:43.068776 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:43.068754    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86543b82-494a-4d6f-8e38-0da899e79681-kube-api-access-9flpl" (OuterVolumeSpecName: "kube-api-access-9flpl") pod "86543b82-494a-4d6f-8e38-0da899e79681" (UID: "86543b82-494a-4d6f-8e38-0da899e79681"). InnerVolumeSpecName "kube-api-access-9flpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:39:43.068861 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:43.068852    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86543b82-494a-4d6f-8e38-0da899e79681-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "86543b82-494a-4d6f-8e38-0da899e79681" (UID: "86543b82-494a-4d6f-8e38-0da899e79681"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:39:43.167556 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:43.167454    2881 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86543b82-494a-4d6f-8e38-0da899e79681-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:39:43.167556 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:43.167495    2881 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86543b82-494a-4d6f-8e38-0da899e79681-cabundle-cert\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:39:43.167556 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:43.167505    2881 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86543b82-494a-4d6f-8e38-0da899e79681-proxy-tls\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:39:43.167556 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:43.167513    2881 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9flpl\" (UniqueName: \"kubernetes.io/projected/86543b82-494a-4d6f-8e38-0da899e79681-kube-api-access-9flpl\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:39:43.167556 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:43.167523    2881 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86543b82-494a-4d6f-8e38-0da899e79681-kserve-provision-location\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:39:43.481203 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:43.481125    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pznx8"]
Sep 18 17:39:43.481505 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:43.481492    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86543b82-494a-4d6f-8e38-0da899e79681" containerName="kserve-container"
Sep 18 17:39:43.481551 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:43.481507    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="86543b82-494a-4d6f-8e38-0da899e79681" containerName="kserve-container"
Sep 18 17:39:43.481551 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:43.481519    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57e56b44-1893-4e11-88d8-46f01379ee03" containerName="storage-initializer"
Sep 18 17:39:43.481551 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:43.481526    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="57e56b44-1893-4e11-88d8-46f01379ee03" containerName="storage-initializer"
Sep 18 17:39:43.481551 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:43.481542    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86543b82-494a-4d6f-8e38-0da899e79681" containerName="storage-initializer"
Sep 18 17:39:43.481551 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:43.481548    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="86543b82-494a-4d6f-8e38-0da899e79681" containerName="storage-initializer"
Sep 18 17:39:43.481699 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:43.481557    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86543b82-494a-4d6f-8e38-0da899e79681" containerName="kube-rbac-proxy"
Sep 18 17:39:43.481699 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:43.481563    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="86543b82-494a-4d6f-8e38-0da899e79681" containerName="kube-rbac-proxy"
Sep 18 17:39:43.481699 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:43.481571    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57e56b44-1893-4e11-88d8-46f01379ee03" containerName="kserve-container"
Sep 18 17:39:43.481699 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:43.481577    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="57e56b44-1893-4e11-88d8-46f01379ee03" containerName="kserve-container"
Sep 18 17:39:43.481699 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:43.481588    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57e56b44-1893-4e11-88d8-46f01379ee03" containerName="kube-rbac-proxy"
Sep 18 17:39:43.481699 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:43.481593    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="57e56b44-1893-4e11-88d8-46f01379ee03" containerName="kube-rbac-proxy"
Sep 18 17:39:43.481699 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:43.481640    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="86543b82-494a-4d6f-8e38-0da899e79681" containerName="kserve-container"
Sep 18 17:39:43.481699 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:43.481648    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="57e56b44-1893-4e11-88d8-46f01379ee03" containerName="kserve-container"
Sep 18 17:39:43.481699 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:43.481655    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="57e56b44-1893-4e11-88d8-46f01379ee03" containerName="kube-rbac-proxy"
Sep 18 17:39:43.481699 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:43.481660    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="86543b82-494a-4d6f-8e38-0da899e79681" containerName="kube-rbac-proxy"
Sep 18 17:39:43.485062 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:43.485047    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pznx8"
Sep 18 17:39:43.489344 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:43.489321    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-predictor-serving-cert\""
Sep 18 17:39:43.489450 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:43.489349    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 18 17:39:43.520241 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:43.520200    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pznx8"]
Sep 18 17:39:43.571380 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:43.571338    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb8f3308-393a-4d9f-a57a-cfa8e31bff8b-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-pznx8\" (UID: \"eb8f3308-393a-4d9f-a57a-cfa8e31bff8b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pznx8"
Sep 18 17:39:43.571563 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:43.571439    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb8f3308-393a-4d9f-a57a-cfa8e31bff8b-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-pznx8\" (UID: \"eb8f3308-393a-4d9f-a57a-cfa8e31bff8b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pznx8"
Sep 18 17:39:43.571563 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:43.571474    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb8f3308-393a-4d9f-a57a-cfa8e31bff8b-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-pznx8\" (UID: \"eb8f3308-393a-4d9f-a57a-cfa8e31bff8b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pznx8"
Sep 18 17:39:43.571563 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:43.571538    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df674\" (UniqueName: \"kubernetes.io/projected/eb8f3308-393a-4d9f-a57a-cfa8e31bff8b-kube-api-access-df674\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-pznx8\" (UID: \"eb8f3308-393a-4d9f-a57a-cfa8e31bff8b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pznx8"
Sep 18 17:39:43.571563 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:43.571559    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb8f3308-393a-4d9f-a57a-cfa8e31bff8b-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-pznx8\" (UID: \"eb8f3308-393a-4d9f-a57a-cfa8e31bff8b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pznx8"
Sep 18 17:39:43.634140 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:43.634104    2881 generic.go:356] "Generic (PLEG): container finished" podID="86543b82-494a-4d6f-8e38-0da899e79681" containerID="df9ce4c741d420f9c3f20e3660842d4b5f3e430a2734ce2f60c0f4e665e01798" exitCode=0
Sep 18 17:39:43.634346 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:43.634157    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p8hjd" event={"ID":"86543b82-494a-4d6f-8e38-0da899e79681","Type":"ContainerDied","Data":"df9ce4c741d420f9c3f20e3660842d4b5f3e430a2734ce2f60c0f4e665e01798"}
Sep 18 17:39:43.634346 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:43.634168    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p8hjd"
Sep 18 17:39:43.634346 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:43.634207    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p8hjd" event={"ID":"86543b82-494a-4d6f-8e38-0da899e79681","Type":"ContainerDied","Data":"ca7d3940e5936767bb3410a5c2df382ceec24a42b69a9f3189cb5b038e8ddd77"}
Sep 18 17:39:43.634346 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:43.634256    2881 scope.go:117] "RemoveContainer" containerID="b79a125a4b94ab55afd906e5136acda6d6ea71588b8c546492bf89a086c1d612"
Sep 18 17:39:43.643196 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:43.643177    2881 scope.go:117] "RemoveContainer" containerID="df9ce4c741d420f9c3f20e3660842d4b5f3e430a2734ce2f60c0f4e665e01798"
Sep 18 17:39:43.651393 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:43.651374    2881 scope.go:117] "RemoveContainer" containerID="1ad5d8badec24383d8a68275d82ec08db447a21b8fe0deb9e235e9bb1fe7b871"
Sep 18 17:39:43.660025 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:43.659989    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p8hjd"]
Sep 18 17:39:43.660516 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:43.660495    2881 scope.go:117] "RemoveContainer" containerID="b79a125a4b94ab55afd906e5136acda6d6ea71588b8c546492bf89a086c1d612"
Sep 18 17:39:43.660846 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:39:43.660827    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b79a125a4b94ab55afd906e5136acda6d6ea71588b8c546492bf89a086c1d612\": container with ID starting with b79a125a4b94ab55afd906e5136acda6d6ea71588b8c546492bf89a086c1d612 not found: ID does not exist" containerID="b79a125a4b94ab55afd906e5136acda6d6ea71588b8c546492bf89a086c1d612"
Sep 18 17:39:43.660919 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:43.660859    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b79a125a4b94ab55afd906e5136acda6d6ea71588b8c546492bf89a086c1d612"} err="failed to get container status \"b79a125a4b94ab55afd906e5136acda6d6ea71588b8c546492bf89a086c1d612\": rpc error: code = NotFound desc = could not find container \"b79a125a4b94ab55afd906e5136acda6d6ea71588b8c546492bf89a086c1d612\": container with ID starting with b79a125a4b94ab55afd906e5136acda6d6ea71588b8c546492bf89a086c1d612 not found: ID does not exist"
Sep 18 17:39:43.660919 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:43.660903    2881 scope.go:117] "RemoveContainer" containerID="df9ce4c741d420f9c3f20e3660842d4b5f3e430a2734ce2f60c0f4e665e01798"
Sep 18 17:39:43.661244 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:39:43.661207    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df9ce4c741d420f9c3f20e3660842d4b5f3e430a2734ce2f60c0f4e665e01798\": container with ID starting with df9ce4c741d420f9c3f20e3660842d4b5f3e430a2734ce2f60c0f4e665e01798 not found: ID does not exist" containerID="df9ce4c741d420f9c3f20e3660842d4b5f3e430a2734ce2f60c0f4e665e01798"
Sep 18 17:39:43.661303 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:43.661250    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df9ce4c741d420f9c3f20e3660842d4b5f3e430a2734ce2f60c0f4e665e01798"} err="failed to get container status \"df9ce4c741d420f9c3f20e3660842d4b5f3e430a2734ce2f60c0f4e665e01798\": rpc error: code = NotFound desc = could not find container \"df9ce4c741d420f9c3f20e3660842d4b5f3e430a2734ce2f60c0f4e665e01798\": container with ID starting with df9ce4c741d420f9c3f20e3660842d4b5f3e430a2734ce2f60c0f4e665e01798 not found: ID does not exist"
Sep 18 17:39:43.661303 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:43.661271    2881 scope.go:117] "RemoveContainer" containerID="1ad5d8badec24383d8a68275d82ec08db447a21b8fe0deb9e235e9bb1fe7b871"
Sep 18 17:39:43.661662 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:39:43.661637    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ad5d8badec24383d8a68275d82ec08db447a21b8fe0deb9e235e9bb1fe7b871\": container with ID starting with 1ad5d8badec24383d8a68275d82ec08db447a21b8fe0deb9e235e9bb1fe7b871 not found: ID does not exist" containerID="1ad5d8badec24383d8a68275d82ec08db447a21b8fe0deb9e235e9bb1fe7b871"
Sep 18 17:39:43.661804 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:43.661662    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ad5d8badec24383d8a68275d82ec08db447a21b8fe0deb9e235e9bb1fe7b871"} err="failed to get container status \"1ad5d8badec24383d8a68275d82ec08db447a21b8fe0deb9e235e9bb1fe7b871\": rpc error: code = NotFound desc = could not find container \"1ad5d8badec24383d8a68275d82ec08db447a21b8fe0deb9e235e9bb1fe7b871\": container with ID starting with 1ad5d8badec24383d8a68275d82ec08db447a21b8fe0deb9e235e9bb1fe7b871 not found: ID does not exist"
Sep 18 17:39:43.663992 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:43.663967    2881 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-p8hjd"]
Sep 18 17:39:43.672414 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:43.672393    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb8f3308-393a-4d9f-a57a-cfa8e31bff8b-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-pznx8\" (UID: \"eb8f3308-393a-4d9f-a57a-cfa8e31bff8b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pznx8"
Sep 18 17:39:43.672494 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:43.672431    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb8f3308-393a-4d9f-a57a-cfa8e31bff8b-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-pznx8\" (UID: \"eb8f3308-393a-4d9f-a57a-cfa8e31bff8b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pznx8"
Sep 18 17:39:43.672541 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:43.672530    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df674\" (UniqueName: \"kubernetes.io/projected/eb8f3308-393a-4d9f-a57a-cfa8e31bff8b-kube-api-access-df674\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-pznx8\" (UID: \"eb8f3308-393a-4d9f-a57a-cfa8e31bff8b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pznx8"
Sep 18 17:39:43.672594 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:43.672555    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb8f3308-393a-4d9f-a57a-cfa8e31bff8b-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-pznx8\" (UID: \"eb8f3308-393a-4d9f-a57a-cfa8e31bff8b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pznx8"
Sep 18 17:39:43.672646 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:43.672606    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb8f3308-393a-4d9f-a57a-cfa8e31bff8b-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-pznx8\" (UID: \"eb8f3308-393a-4d9f-a57a-cfa8e31bff8b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pznx8"
Sep 18 17:39:43.672954 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:43.672936    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb8f3308-393a-4d9f-a57a-cfa8e31bff8b-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-pznx8\" (UID: \"eb8f3308-393a-4d9f-a57a-cfa8e31bff8b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pznx8"
Sep 18 17:39:43.673246 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:43.673212    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb8f3308-393a-4d9f-a57a-cfa8e31bff8b-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-pznx8\" (UID: \"eb8f3308-393a-4d9f-a57a-cfa8e31bff8b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pznx8"
Sep 18 17:39:43.673296 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:43.673211    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb8f3308-393a-4d9f-a57a-cfa8e31bff8b-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-pznx8\" (UID: \"eb8f3308-393a-4d9f-a57a-cfa8e31bff8b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pznx8"
Sep 18 17:39:43.675068 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:43.675048    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb8f3308-393a-4d9f-a57a-cfa8e31bff8b-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-pznx8\" (UID: \"eb8f3308-393a-4d9f-a57a-cfa8e31bff8b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pznx8"
Sep 18 17:39:43.684502 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:43.684475    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-df674\" (UniqueName: \"kubernetes.io/projected/eb8f3308-393a-4d9f-a57a-cfa8e31bff8b-kube-api-access-df674\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-pznx8\" (UID: \"eb8f3308-393a-4d9f-a57a-cfa8e31bff8b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pznx8"
Sep 18 17:39:43.795591 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:43.795539    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pznx8"
Sep 18 17:39:43.893155 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:43.893116    2881 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86543b82-494a-4d6f-8e38-0da899e79681" path="/var/lib/kubelet/pods/86543b82-494a-4d6f-8e38-0da899e79681/volumes"
Sep 18 17:39:44.074416 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:44.074390    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pznx8"]
Sep 18 17:39:44.076732 ip-10-0-143-5 kubenswrapper[2881]: W0918 17:39:44.076703    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb8f3308_393a_4d9f_a57a_cfa8e31bff8b.slice/crio-4f9e7296efd2e083102b754deba3bc251c4dc00b18487ad2758addc6add29b6c WatchSource:0}: Error finding container 4f9e7296efd2e083102b754deba3bc251c4dc00b18487ad2758addc6add29b6c: Status 404 returned error can't find the container with id 4f9e7296efd2e083102b754deba3bc251c4dc00b18487ad2758addc6add29b6c
Sep 18 17:39:44.642858 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:44.642818    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pznx8" event={"ID":"eb8f3308-393a-4d9f-a57a-cfa8e31bff8b","Type":"ContainerStarted","Data":"dd13034843bb65e0736f1c21176aeb914c88c0cd4eecda01743787290dda7d54"}
Sep 18 17:39:44.643062 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:44.642914    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pznx8" event={"ID":"eb8f3308-393a-4d9f-a57a-cfa8e31bff8b","Type":"ContainerStarted","Data":"4f9e7296efd2e083102b754deba3bc251c4dc00b18487ad2758addc6add29b6c"}
Sep 18 17:39:45.650144 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:45.650106    2881 generic.go:356] "Generic (PLEG): container finished" podID="f1bf9a13-4bdf-46f6-a80a-ea425450c0b3" containerID="0e83ce0ada936975aeff43e9c0eea879e5a6a44c315698812e951b305e72e514" exitCode=0
Sep 18 17:39:45.650637 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:45.650179    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nskpc" event={"ID":"f1bf9a13-4bdf-46f6-a80a-ea425450c0b3","Type":"ContainerDied","Data":"0e83ce0ada936975aeff43e9c0eea879e5a6a44c315698812e951b305e72e514"}
Sep 18 17:39:46.657005 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:46.656963    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nskpc" event={"ID":"f1bf9a13-4bdf-46f6-a80a-ea425450c0b3","Type":"ContainerStarted","Data":"fe2ef96e29a2fdedf10149e95251a9fa6ec18ddc2819038a408bdac8a95a4635"}
Sep 18 17:39:46.657757 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:46.657036    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nskpc" event={"ID":"f1bf9a13-4bdf-46f6-a80a-ea425450c0b3","Type":"ContainerStarted","Data":"ac66f71a29f79e2ea01f9d75294e0c406d9f55677f25515f285e9c23064723d5"}
Sep 18 17:39:46.657757 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:46.657264    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nskpc"
Sep 18 17:39:46.677869 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:46.677802    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nskpc" podStartSLOduration=4.007160596 podStartE2EDuration="7.677785199s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.670624603s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:39:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:39:46.677189727 +0000 UTC m=+2776.877045015" watchObservedRunningTime="2026-09-18 17:39:46.677785199 +0000 UTC m=+2776.877640484"
Sep 18 17:39:47.663441 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:47.663411    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nskpc"
Sep 18 17:39:47.664731 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:47.664701    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nskpc" podUID="f1bf9a13-4bdf-46f6-a80a-ea425450c0b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 18 17:39:48.668986 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:48.668953    2881 generic.go:356] "Generic (PLEG): container finished" podID="eb8f3308-393a-4d9f-a57a-cfa8e31bff8b" containerID="dd13034843bb65e0736f1c21176aeb914c88c0cd4eecda01743787290dda7d54" exitCode=0
Sep 18 17:39:48.669422 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:48.669031    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pznx8" event={"ID":"eb8f3308-393a-4d9f-a57a-cfa8e31bff8b","Type":"ContainerDied","Data":"dd13034843bb65e0736f1c21176aeb914c88c0cd4eecda01743787290dda7d54"}
Sep 18 17:39:48.669546 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:48.669526    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nskpc" podUID="f1bf9a13-4bdf-46f6-a80a-ea425450c0b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 18 17:39:49.675917 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:49.675880    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pznx8" event={"ID":"eb8f3308-393a-4d9f-a57a-cfa8e31bff8b","Type":"ContainerStarted","Data":"c65290186624e8c687b261a5bad528acddc1f92a3f43a3a63b6baa280fd81167"}
Sep 18 17:39:49.676342 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:49.675971    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pznx8" event={"ID":"eb8f3308-393a-4d9f-a57a-cfa8e31bff8b","Type":"ContainerStarted","Data":"d3baa7b815a220dc9de41db8912106b56f3873b26329ab839bd0f138d6245f8c"}
Sep 18 17:39:49.676342 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:49.676173    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pznx8"
Sep 18 17:39:49.700290 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:49.700218    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pznx8" podStartSLOduration=2.628553762 podStartE2EDuration="6.700197112s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.07164335s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:39:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:39:49.698905436 +0000 UTC m=+2779.898760723" watchObservedRunningTime="2026-09-18 17:39:49.700197112 +0000 UTC m=+2779.900052398"
Sep 18 17:39:50.680527 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:50.680496    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pznx8"
Sep 18 17:39:53.673928 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:53.673890    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nskpc"
Sep 18 17:39:53.674469 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:53.674442    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nskpc" podUID="f1bf9a13-4bdf-46f6-a80a-ea425450c0b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 18 17:39:56.690630 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:39:56.690601    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pznx8"
Sep 18 17:40:03.675096 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:03.675058    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nskpc" podUID="f1bf9a13-4bdf-46f6-a80a-ea425450c0b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 18 17:40:13.675396 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:13.675355    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nskpc" podUID="f1bf9a13-4bdf-46f6-a80a-ea425450c0b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 18 17:40:23.674868 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:23.674823    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nskpc" podUID="f1bf9a13-4bdf-46f6-a80a-ea425450c0b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 18 17:40:26.734438 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:26.734390    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pznx8" podUID="eb8f3308-393a-4d9f-a57a-cfa8e31bff8b" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 18 17:40:33.674804 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:33.674761    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nskpc" podUID="f1bf9a13-4bdf-46f6-a80a-ea425450c0b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 18 17:40:36.694150 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:36.694119    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pznx8"
Sep 18 17:40:43.555867 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:43.555832    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pznx8"]
Sep 18 17:40:43.556320 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:43.556065    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pznx8" podUID="eb8f3308-393a-4d9f-a57a-cfa8e31bff8b" containerName="kserve-container" containerID="cri-o://d3baa7b815a220dc9de41db8912106b56f3873b26329ab839bd0f138d6245f8c" gracePeriod=30
Sep 18 17:40:43.556320 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:43.556117    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pznx8" podUID="eb8f3308-393a-4d9f-a57a-cfa8e31bff8b" containerName="kube-rbac-proxy" containerID="cri-o://c65290186624e8c687b261a5bad528acddc1f92a3f43a3a63b6baa280fd81167" gracePeriod=30
Sep 18 17:40:43.674667 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:43.674534    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nskpc" podUID="f1bf9a13-4bdf-46f6-a80a-ea425450c0b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 18 17:40:43.930062 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:43.929969    2881 generic.go:356] "Generic (PLEG): container finished" podID="eb8f3308-393a-4d9f-a57a-cfa8e31bff8b" containerID="c65290186624e8c687b261a5bad528acddc1f92a3f43a3a63b6baa280fd81167" exitCode=2
Sep 18 17:40:43.930062 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:43.930041    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pznx8" event={"ID":"eb8f3308-393a-4d9f-a57a-cfa8e31bff8b","Type":"ContainerDied","Data":"c65290186624e8c687b261a5bad528acddc1f92a3f43a3a63b6baa280fd81167"}
Sep 18 17:40:46.685889 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:46.685843    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pznx8" podUID="eb8f3308-393a-4d9f-a57a-cfa8e31bff8b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.56:8643/healthz\": dial tcp 10.134.0.56:8643: connect: connection refused"
Sep 18 17:40:48.726201 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:48.726164    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-7rjg7"]
Sep 18 17:40:48.730438 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:48.730407    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-7rjg7"
Sep 18 17:40:48.733602 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:48.733581    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-predictor-serving-cert\""
Sep 18 17:40:48.733825 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:48.733808    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-kube-rbac-proxy-sar-config\""
Sep 18 17:40:48.742621 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:48.742597    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-7rjg7"]
Sep 18 17:40:48.868001 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:48.867964    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55v9n\" (UniqueName: \"kubernetes.io/projected/75e97599-fc29-4789-8a43-c26c78a3eaa0-kube-api-access-55v9n\") pod \"isvc-xgboost-v2-predictor-fcc668bc-7rjg7\" (UID: \"75e97599-fc29-4789-8a43-c26c78a3eaa0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-7rjg7"
Sep 18 17:40:48.868001 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:48.868012    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75e97599-fc29-4789-8a43-c26c78a3eaa0-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-7rjg7\" (UID: \"75e97599-fc29-4789-8a43-c26c78a3eaa0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-7rjg7"
Sep 18 17:40:48.868297 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:48.868084    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75e97599-fc29-4789-8a43-c26c78a3eaa0-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-7rjg7\" (UID: \"75e97599-fc29-4789-8a43-c26c78a3eaa0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-7rjg7"
Sep 18 17:40:48.868297 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:48.868158    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75e97599-fc29-4789-8a43-c26c78a3eaa0-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-7rjg7\" (UID: \"75e97599-fc29-4789-8a43-c26c78a3eaa0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-7rjg7"
Sep 18 17:40:48.868297 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:48.868217    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75e97599-fc29-4789-8a43-c26c78a3eaa0-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-7rjg7\" (UID: \"75e97599-fc29-4789-8a43-c26c78a3eaa0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-7rjg7"
Sep 18 17:40:48.969378 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:48.969332    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75e97599-fc29-4789-8a43-c26c78a3eaa0-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-7rjg7\" (UID: \"75e97599-fc29-4789-8a43-c26c78a3eaa0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-7rjg7"
Sep 18 17:40:48.969573 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:48.969395    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75e97599-fc29-4789-8a43-c26c78a3eaa0-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-7rjg7\" (UID: \"75e97599-fc29-4789-8a43-c26c78a3eaa0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-7rjg7"
Sep 18 17:40:48.969573 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:48.969423    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75e97599-fc29-4789-8a43-c26c78a3eaa0-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-7rjg7\" (UID: \"75e97599-fc29-4789-8a43-c26c78a3eaa0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-7rjg7"
Sep 18 17:40:48.969573 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:48.969461    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55v9n\" (UniqueName: \"kubernetes.io/projected/75e97599-fc29-4789-8a43-c26c78a3eaa0-kube-api-access-55v9n\") pod \"isvc-xgboost-v2-predictor-fcc668bc-7rjg7\" (UID: \"75e97599-fc29-4789-8a43-c26c78a3eaa0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-7rjg7"
Sep 18 17:40:48.969573 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:48.969497    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75e97599-fc29-4789-8a43-c26c78a3eaa0-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-7rjg7\" (UID: \"75e97599-fc29-4789-8a43-c26c78a3eaa0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-7rjg7"
Sep 18 17:40:48.969802 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:48.969766    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75e97599-fc29-4789-8a43-c26c78a3eaa0-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-7rjg7\" (UID: \"75e97599-fc29-4789-8a43-c26c78a3eaa0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-7rjg7"
Sep 18 17:40:48.970276 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:48.970244    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75e97599-fc29-4789-8a43-c26c78a3eaa0-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-7rjg7\" (UID: \"75e97599-fc29-4789-8a43-c26c78a3eaa0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-7rjg7"
Sep 18 17:40:48.970336 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:48.970252    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75e97599-fc29-4789-8a43-c26c78a3eaa0-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-7rjg7\" (UID: \"75e97599-fc29-4789-8a43-c26c78a3eaa0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-7rjg7"
Sep 18 17:40:48.971980 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:48.971961    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75e97599-fc29-4789-8a43-c26c78a3eaa0-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-7rjg7\" (UID: \"75e97599-fc29-4789-8a43-c26c78a3eaa0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-7rjg7"
Sep 18 17:40:48.979104 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:48.979046    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55v9n\" (UniqueName: \"kubernetes.io/projected/75e97599-fc29-4789-8a43-c26c78a3eaa0-kube-api-access-55v9n\") pod \"isvc-xgboost-v2-predictor-fcc668bc-7rjg7\" (UID: \"75e97599-fc29-4789-8a43-c26c78a3eaa0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-7rjg7"
Sep 18 17:40:49.043980 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:49.043938    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-7rjg7"
Sep 18 17:40:49.319016 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:49.318916    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-7rjg7"]
Sep 18 17:40:49.321457 ip-10-0-143-5 kubenswrapper[2881]: W0918 17:40:49.321425    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75e97599_fc29_4789_8a43_c26c78a3eaa0.slice/crio-cc7f42a96699b7001bd0bbcb6528e10c2f084c59b69787441c7e4fdbf774902f WatchSource:0}: Error finding container cc7f42a96699b7001bd0bbcb6528e10c2f084c59b69787441c7e4fdbf774902f: Status 404 returned error can't find the container with id cc7f42a96699b7001bd0bbcb6528e10c2f084c59b69787441c7e4fdbf774902f
Sep 18 17:40:49.959157 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:49.959122    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-7rjg7" event={"ID":"75e97599-fc29-4789-8a43-c26c78a3eaa0","Type":"ContainerStarted","Data":"82da5af8442bd36baf7ba3961edf02dae0401a0d5c8aeeeeac247b7f478d0cae"}
Sep 18 17:40:49.959556 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:49.959187    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-7rjg7" event={"ID":"75e97599-fc29-4789-8a43-c26c78a3eaa0","Type":"ContainerStarted","Data":"cc7f42a96699b7001bd0bbcb6528e10c2f084c59b69787441c7e4fdbf774902f"}
Sep 18 17:40:51.686450 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:51.686405    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pznx8" podUID="eb8f3308-393a-4d9f-a57a-cfa8e31bff8b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.56:8643/healthz\": dial tcp 10.134.0.56:8643: connect: connection refused"
Sep 18 17:40:51.801669 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:51.801645    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pznx8"
Sep 18 17:40:51.893458 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:51.893371    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb8f3308-393a-4d9f-a57a-cfa8e31bff8b-proxy-tls\") pod \"eb8f3308-393a-4d9f-a57a-cfa8e31bff8b\" (UID: \"eb8f3308-393a-4d9f-a57a-cfa8e31bff8b\") "
Sep 18 17:40:51.893458 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:51.893450    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb8f3308-393a-4d9f-a57a-cfa8e31bff8b-kube-rbac-proxy-sar-config\") pod \"eb8f3308-393a-4d9f-a57a-cfa8e31bff8b\" (UID: \"eb8f3308-393a-4d9f-a57a-cfa8e31bff8b\") "
Sep 18 17:40:51.893687 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:51.893604    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb8f3308-393a-4d9f-a57a-cfa8e31bff8b-cabundle-cert\") pod \"eb8f3308-393a-4d9f-a57a-cfa8e31bff8b\" (UID: \"eb8f3308-393a-4d9f-a57a-cfa8e31bff8b\") "
Sep 18 17:40:51.893748 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:51.893684    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb8f3308-393a-4d9f-a57a-cfa8e31bff8b-kserve-provision-location\") pod \"eb8f3308-393a-4d9f-a57a-cfa8e31bff8b\" (UID: \"eb8f3308-393a-4d9f-a57a-cfa8e31bff8b\") "
Sep 18 17:40:51.893748 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:51.893733    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-df674\" (UniqueName: \"kubernetes.io/projected/eb8f3308-393a-4d9f-a57a-cfa8e31bff8b-kube-api-access-df674\") pod \"eb8f3308-393a-4d9f-a57a-cfa8e31bff8b\" (UID: \"eb8f3308-393a-4d9f-a57a-cfa8e31bff8b\") "
Sep 18 17:40:51.893748 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:51.893739    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb8f3308-393a-4d9f-a57a-cfa8e31bff8b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "eb8f3308-393a-4d9f-a57a-cfa8e31bff8b" (UID: "eb8f3308-393a-4d9f-a57a-cfa8e31bff8b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:40:51.893970 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:51.893954    2881 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb8f3308-393a-4d9f-a57a-cfa8e31bff8b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:40:51.894024 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:51.894005    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb8f3308-393a-4d9f-a57a-cfa8e31bff8b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "eb8f3308-393a-4d9f-a57a-cfa8e31bff8b" (UID: "eb8f3308-393a-4d9f-a57a-cfa8e31bff8b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:40:51.894061 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:51.894035    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb8f3308-393a-4d9f-a57a-cfa8e31bff8b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "eb8f3308-393a-4d9f-a57a-cfa8e31bff8b" (UID: "eb8f3308-393a-4d9f-a57a-cfa8e31bff8b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:40:51.895836 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:51.895814    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb8f3308-393a-4d9f-a57a-cfa8e31bff8b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "eb8f3308-393a-4d9f-a57a-cfa8e31bff8b" (UID: "eb8f3308-393a-4d9f-a57a-cfa8e31bff8b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:40:51.895910 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:51.895874    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb8f3308-393a-4d9f-a57a-cfa8e31bff8b-kube-api-access-df674" (OuterVolumeSpecName: "kube-api-access-df674") pod "eb8f3308-393a-4d9f-a57a-cfa8e31bff8b" (UID: "eb8f3308-393a-4d9f-a57a-cfa8e31bff8b"). InnerVolumeSpecName "kube-api-access-df674". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:40:51.970950 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:51.970915    2881 generic.go:356] "Generic (PLEG): container finished" podID="eb8f3308-393a-4d9f-a57a-cfa8e31bff8b" containerID="d3baa7b815a220dc9de41db8912106b56f3873b26329ab839bd0f138d6245f8c" exitCode=0
Sep 18 17:40:51.971153 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:51.970974    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pznx8"
Sep 18 17:40:51.971153 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:51.970998    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pznx8" event={"ID":"eb8f3308-393a-4d9f-a57a-cfa8e31bff8b","Type":"ContainerDied","Data":"d3baa7b815a220dc9de41db8912106b56f3873b26329ab839bd0f138d6245f8c"}
Sep 18 17:40:51.971153 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:51.971075    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pznx8" event={"ID":"eb8f3308-393a-4d9f-a57a-cfa8e31bff8b","Type":"ContainerDied","Data":"4f9e7296efd2e083102b754deba3bc251c4dc00b18487ad2758addc6add29b6c"}
Sep 18 17:40:51.971153 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:51.971110    2881 scope.go:117] "RemoveContainer" containerID="c65290186624e8c687b261a5bad528acddc1f92a3f43a3a63b6baa280fd81167"
Sep 18 17:40:51.980196 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:51.980172    2881 scope.go:117] "RemoveContainer" containerID="d3baa7b815a220dc9de41db8912106b56f3873b26329ab839bd0f138d6245f8c"
Sep 18 17:40:51.988189 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:51.988169    2881 scope.go:117] "RemoveContainer" containerID="dd13034843bb65e0736f1c21176aeb914c88c0cd4eecda01743787290dda7d54"
Sep 18 17:40:51.994812 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:51.994784    2881 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb8f3308-393a-4d9f-a57a-cfa8e31bff8b-cabundle-cert\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:40:51.994812 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:51.994816    2881 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb8f3308-393a-4d9f-a57a-cfa8e31bff8b-kserve-provision-location\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:40:51.995019 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:51.994830    2881 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-df674\" (UniqueName: \"kubernetes.io/projected/eb8f3308-393a-4d9f-a57a-cfa8e31bff8b-kube-api-access-df674\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:40:51.995019 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:51.994845    2881 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb8f3308-393a-4d9f-a57a-cfa8e31bff8b-proxy-tls\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:40:51.995758 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:51.995735    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pznx8"]
Sep 18 17:40:51.997099 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:51.997082    2881 scope.go:117] "RemoveContainer" containerID="c65290186624e8c687b261a5bad528acddc1f92a3f43a3a63b6baa280fd81167"
Sep 18 17:40:51.997486 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:40:51.997463    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c65290186624e8c687b261a5bad528acddc1f92a3f43a3a63b6baa280fd81167\": container with ID starting with c65290186624e8c687b261a5bad528acddc1f92a3f43a3a63b6baa280fd81167 not found: ID does not exist" containerID="c65290186624e8c687b261a5bad528acddc1f92a3f43a3a63b6baa280fd81167"
Sep 18 17:40:51.997582 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:51.997491    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c65290186624e8c687b261a5bad528acddc1f92a3f43a3a63b6baa280fd81167"} err="failed to get container status \"c65290186624e8c687b261a5bad528acddc1f92a3f43a3a63b6baa280fd81167\": rpc error: code = NotFound desc = could not find container \"c65290186624e8c687b261a5bad528acddc1f92a3f43a3a63b6baa280fd81167\": container with ID starting with c65290186624e8c687b261a5bad528acddc1f92a3f43a3a63b6baa280fd81167 not found: ID does not exist"
Sep 18 17:40:51.997582 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:51.997511    2881 scope.go:117] "RemoveContainer" containerID="d3baa7b815a220dc9de41db8912106b56f3873b26329ab839bd0f138d6245f8c"
Sep 18 17:40:51.997783 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:40:51.997768    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3baa7b815a220dc9de41db8912106b56f3873b26329ab839bd0f138d6245f8c\": container with ID starting with d3baa7b815a220dc9de41db8912106b56f3873b26329ab839bd0f138d6245f8c not found: ID does not exist" containerID="d3baa7b815a220dc9de41db8912106b56f3873b26329ab839bd0f138d6245f8c"
Sep 18 17:40:51.997827 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:51.997787    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3baa7b815a220dc9de41db8912106b56f3873b26329ab839bd0f138d6245f8c"} err="failed to get container status \"d3baa7b815a220dc9de41db8912106b56f3873b26329ab839bd0f138d6245f8c\": rpc error: code = NotFound desc = could not find container \"d3baa7b815a220dc9de41db8912106b56f3873b26329ab839bd0f138d6245f8c\": container with ID starting with d3baa7b815a220dc9de41db8912106b56f3873b26329ab839bd0f138d6245f8c not found: ID does not exist"
Sep 18 17:40:51.997827 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:51.997802    2881 scope.go:117] "RemoveContainer" containerID="dd13034843bb65e0736f1c21176aeb914c88c0cd4eecda01743787290dda7d54"
Sep 18 17:40:51.998054 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:40:51.998034    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd13034843bb65e0736f1c21176aeb914c88c0cd4eecda01743787290dda7d54\": container with ID starting with dd13034843bb65e0736f1c21176aeb914c88c0cd4eecda01743787290dda7d54 not found: ID does not exist" containerID="dd13034843bb65e0736f1c21176aeb914c88c0cd4eecda01743787290dda7d54"
Sep 18 17:40:51.998097 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:51.998059    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd13034843bb65e0736f1c21176aeb914c88c0cd4eecda01743787290dda7d54"} err="failed to get container status \"dd13034843bb65e0736f1c21176aeb914c88c0cd4eecda01743787290dda7d54\": rpc error: code = NotFound desc = could not find container \"dd13034843bb65e0736f1c21176aeb914c88c0cd4eecda01743787290dda7d54\": container with ID starting with dd13034843bb65e0736f1c21176aeb914c88c0cd4eecda01743787290dda7d54 not found: ID does not exist"
Sep 18 17:40:52.002345 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:52.002303    2881 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-pznx8"]
Sep 18 17:40:53.675452 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:53.675357    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nskpc" podUID="f1bf9a13-4bdf-46f6-a80a-ea425450c0b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 18 17:40:53.892902 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:53.892866    2881 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb8f3308-393a-4d9f-a57a-cfa8e31bff8b" path="/var/lib/kubelet/pods/eb8f3308-393a-4d9f-a57a-cfa8e31bff8b/volumes"
Sep 18 17:40:53.983656 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:53.983568    2881 generic.go:356] "Generic (PLEG): container finished" podID="75e97599-fc29-4789-8a43-c26c78a3eaa0" containerID="82da5af8442bd36baf7ba3961edf02dae0401a0d5c8aeeeeac247b7f478d0cae" exitCode=0
Sep 18 17:40:53.983656 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:53.983642    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-7rjg7" event={"ID":"75e97599-fc29-4789-8a43-c26c78a3eaa0","Type":"ContainerDied","Data":"82da5af8442bd36baf7ba3961edf02dae0401a0d5c8aeeeeac247b7f478d0cae"}
Sep 18 17:40:54.991504 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:54.991471    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-7rjg7" event={"ID":"75e97599-fc29-4789-8a43-c26c78a3eaa0","Type":"ContainerStarted","Data":"1f63461a714fdc197949060a3bca41154211f43a9d6395402099c6fb6074adfd"}
Sep 18 17:40:54.991979 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:54.991537    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-7rjg7" event={"ID":"75e97599-fc29-4789-8a43-c26c78a3eaa0","Type":"ContainerStarted","Data":"8aab4e87cb97c36db8d0c30e0f7000e81813829c2ec792c75420b88422575154"}
Sep 18 17:40:54.991979 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:54.991809    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-7rjg7"
Sep 18 17:40:54.991979 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:54.991957    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-7rjg7"
Sep 18 17:40:54.993204 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:54.993178    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-7rjg7" podUID="75e97599-fc29-4789-8a43-c26c78a3eaa0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 18 17:40:55.014215 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:55.014167    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-7rjg7" podStartSLOduration=3.026225569 podStartE2EDuration="7.014151173s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.987925604s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:40:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:40:55.013102599 +0000 UTC m=+2845.212957886" watchObservedRunningTime="2026-09-18 17:40:55.014151173 +0000 UTC m=+2845.214006458"
Sep 18 17:40:56.000623 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:40:56.000571    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-7rjg7" podUID="75e97599-fc29-4789-8a43-c26c78a3eaa0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 18 17:41:01.006580 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:01.006552    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-7rjg7"
Sep 18 17:41:01.007074 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:01.007049    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-7rjg7" podUID="75e97599-fc29-4789-8a43-c26c78a3eaa0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 18 17:41:03.675239 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:03.675194    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nskpc" podUID="f1bf9a13-4bdf-46f6-a80a-ea425450c0b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 18 17:41:09.890392 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:09.890353    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nskpc" podUID="f1bf9a13-4bdf-46f6-a80a-ea425450c0b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 18 17:41:11.007177 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:11.007138    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-7rjg7" podUID="75e97599-fc29-4789-8a43-c26c78a3eaa0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 18 17:41:19.893339 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:19.893308    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nskpc"
Sep 18 17:41:21.007369 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:21.007331    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-7rjg7" podUID="75e97599-fc29-4789-8a43-c26c78a3eaa0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 18 17:41:31.007964 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:31.007923    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-7rjg7" podUID="75e97599-fc29-4789-8a43-c26c78a3eaa0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 18 17:41:31.985379 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:31.985347    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nskpc"]
Sep 18 17:41:31.985602 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:31.985574    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nskpc" podUID="f1bf9a13-4bdf-46f6-a80a-ea425450c0b3" containerName="kserve-container" containerID="cri-o://ac66f71a29f79e2ea01f9d75294e0c406d9f55677f25515f285e9c23064723d5" gracePeriod=30
Sep 18 17:41:31.985731 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:31.985647    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nskpc" podUID="f1bf9a13-4bdf-46f6-a80a-ea425450c0b3" containerName="kube-rbac-proxy" containerID="cri-o://fe2ef96e29a2fdedf10149e95251a9fa6ec18ddc2819038a408bdac8a95a4635" gracePeriod=30
Sep 18 17:41:32.173695 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:32.173658    2881 generic.go:356] "Generic (PLEG): container finished" podID="f1bf9a13-4bdf-46f6-a80a-ea425450c0b3" containerID="fe2ef96e29a2fdedf10149e95251a9fa6ec18ddc2819038a408bdac8a95a4635" exitCode=2
Sep 18 17:41:32.174067 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:32.173709    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nskpc" event={"ID":"f1bf9a13-4bdf-46f6-a80a-ea425450c0b3","Type":"ContainerDied","Data":"fe2ef96e29a2fdedf10149e95251a9fa6ec18ddc2819038a408bdac8a95a4635"}
Sep 18 17:41:33.670060 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:33.670017    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nskpc" podUID="f1bf9a13-4bdf-46f6-a80a-ea425450c0b3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.55:8643/healthz\": dial tcp 10.134.0.55:8643: connect: connection refused"
Sep 18 17:41:37.146717 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:37.146681    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rzbtn"]
Sep 18 17:41:37.147111 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:37.147102    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb8f3308-393a-4d9f-a57a-cfa8e31bff8b" containerName="storage-initializer"
Sep 18 17:41:37.147159 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:37.147114    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb8f3308-393a-4d9f-a57a-cfa8e31bff8b" containerName="storage-initializer"
Sep 18 17:41:37.147159 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:37.147130    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb8f3308-393a-4d9f-a57a-cfa8e31bff8b" containerName="kserve-container"
Sep 18 17:41:37.147159 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:37.147136    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb8f3308-393a-4d9f-a57a-cfa8e31bff8b" containerName="kserve-container"
Sep 18 17:41:37.147159 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:37.147148    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb8f3308-393a-4d9f-a57a-cfa8e31bff8b" containerName="kube-rbac-proxy"
Sep 18 17:41:37.147159 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:37.147154    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb8f3308-393a-4d9f-a57a-cfa8e31bff8b" containerName="kube-rbac-proxy"
Sep 18 17:41:37.147336 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:37.147218    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb8f3308-393a-4d9f-a57a-cfa8e31bff8b" containerName="kube-rbac-proxy"
Sep 18 17:41:37.147336 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:37.147239    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb8f3308-393a-4d9f-a57a-cfa8e31bff8b" containerName="kserve-container"
Sep 18 17:41:37.150498 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:37.150482    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rzbtn"
Sep 18 17:41:37.153024 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:37.152993    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-predictor-serving-cert\""
Sep 18 17:41:37.153155 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:37.153094    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 18 17:41:37.161672 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:37.161649    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rzbtn"]
Sep 18 17:41:37.205677 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:37.205647    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c48397cf-2334-4a81-a26b-6bd818a742f7-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rzbtn\" (UID: \"c48397cf-2334-4a81-a26b-6bd818a742f7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rzbtn"
Sep 18 17:41:37.205852 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:37.205697    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c48397cf-2334-4a81-a26b-6bd818a742f7-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rzbtn\" (UID: \"c48397cf-2334-4a81-a26b-6bd818a742f7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rzbtn"
Sep 18 17:41:37.205852 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:37.205724    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c48397cf-2334-4a81-a26b-6bd818a742f7-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rzbtn\" (UID: \"c48397cf-2334-4a81-a26b-6bd818a742f7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rzbtn"
Sep 18 17:41:37.205852 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:37.205772    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8fw9\" (UniqueName: \"kubernetes.io/projected/c48397cf-2334-4a81-a26b-6bd818a742f7-kube-api-access-b8fw9\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rzbtn\" (UID: \"c48397cf-2334-4a81-a26b-6bd818a742f7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rzbtn"
Sep 18 17:41:37.205852 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:37.205836    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c48397cf-2334-4a81-a26b-6bd818a742f7-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rzbtn\" (UID: \"c48397cf-2334-4a81-a26b-6bd818a742f7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rzbtn"
Sep 18 17:41:37.307057 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:37.307019    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c48397cf-2334-4a81-a26b-6bd818a742f7-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rzbtn\" (UID: \"c48397cf-2334-4a81-a26b-6bd818a742f7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rzbtn"
Sep 18 17:41:37.307057 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:37.307060    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c48397cf-2334-4a81-a26b-6bd818a742f7-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rzbtn\" (UID: \"c48397cf-2334-4a81-a26b-6bd818a742f7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rzbtn"
Sep 18 17:41:37.307356 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:37.307077    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8fw9\" (UniqueName: \"kubernetes.io/projected/c48397cf-2334-4a81-a26b-6bd818a742f7-kube-api-access-b8fw9\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rzbtn\" (UID: \"c48397cf-2334-4a81-a26b-6bd818a742f7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rzbtn"
Sep 18 17:41:37.307356 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:37.307105    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c48397cf-2334-4a81-a26b-6bd818a742f7-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rzbtn\" (UID: \"c48397cf-2334-4a81-a26b-6bd818a742f7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rzbtn"
Sep 18 17:41:37.307356 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:37.307150    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c48397cf-2334-4a81-a26b-6bd818a742f7-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rzbtn\" (UID: \"c48397cf-2334-4a81-a26b-6bd818a742f7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rzbtn"
Sep 18 17:41:37.307538 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:37.307508    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c48397cf-2334-4a81-a26b-6bd818a742f7-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rzbtn\" (UID: \"c48397cf-2334-4a81-a26b-6bd818a742f7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rzbtn"
Sep 18 17:41:37.307752 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:37.307728    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c48397cf-2334-4a81-a26b-6bd818a742f7-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rzbtn\" (UID: \"c48397cf-2334-4a81-a26b-6bd818a742f7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rzbtn"
Sep 18 17:41:37.307857 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:37.307828    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c48397cf-2334-4a81-a26b-6bd818a742f7-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rzbtn\" (UID: \"c48397cf-2334-4a81-a26b-6bd818a742f7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rzbtn"
Sep 18 17:41:37.309590 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:37.309573    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c48397cf-2334-4a81-a26b-6bd818a742f7-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rzbtn\" (UID: \"c48397cf-2334-4a81-a26b-6bd818a742f7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rzbtn"
Sep 18 17:41:37.318022 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:37.317994    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8fw9\" (UniqueName: \"kubernetes.io/projected/c48397cf-2334-4a81-a26b-6bd818a742f7-kube-api-access-b8fw9\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rzbtn\" (UID: \"c48397cf-2334-4a81-a26b-6bd818a742f7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rzbtn"
Sep 18 17:41:37.461134 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:37.461040    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rzbtn"
Sep 18 17:41:37.736547 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:37.736455    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rzbtn"]
Sep 18 17:41:37.740993 ip-10-0-143-5 kubenswrapper[2881]: W0918 17:41:37.740959    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc48397cf_2334_4a81_a26b_6bd818a742f7.slice/crio-ead98e6005406b4124b662ac4f630551efd9cebd3d4e13378dc44e1e7a5bf875 WatchSource:0}: Error finding container ead98e6005406b4124b662ac4f630551efd9cebd3d4e13378dc44e1e7a5bf875: Status 404 returned error can't find the container with id ead98e6005406b4124b662ac4f630551efd9cebd3d4e13378dc44e1e7a5bf875
Sep 18 17:41:38.128858 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:38.128832    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nskpc"
Sep 18 17:41:38.206487 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:38.206456    2881 generic.go:356] "Generic (PLEG): container finished" podID="f1bf9a13-4bdf-46f6-a80a-ea425450c0b3" containerID="ac66f71a29f79e2ea01f9d75294e0c406d9f55677f25515f285e9c23064723d5" exitCode=0
Sep 18 17:41:38.206952 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:38.206510    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nskpc"
Sep 18 17:41:38.206952 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:38.206544    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nskpc" event={"ID":"f1bf9a13-4bdf-46f6-a80a-ea425450c0b3","Type":"ContainerDied","Data":"ac66f71a29f79e2ea01f9d75294e0c406d9f55677f25515f285e9c23064723d5"}
Sep 18 17:41:38.206952 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:38.206608    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nskpc" event={"ID":"f1bf9a13-4bdf-46f6-a80a-ea425450c0b3","Type":"ContainerDied","Data":"19318f0cc4ee749231e4c2ac0696b8ba2408f40ffb7ab32d4dbfdad269e58954"}
Sep 18 17:41:38.206952 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:38.206644    2881 scope.go:117] "RemoveContainer" containerID="fe2ef96e29a2fdedf10149e95251a9fa6ec18ddc2819038a408bdac8a95a4635"
Sep 18 17:41:38.208552 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:38.208512    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rzbtn" event={"ID":"c48397cf-2334-4a81-a26b-6bd818a742f7","Type":"ContainerStarted","Data":"201c4e8d75382cd355f931f8592fda62373d4369c3450a6a5e1a3e32aa164652"}
Sep 18 17:41:38.208689 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:38.208578    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rzbtn" event={"ID":"c48397cf-2334-4a81-a26b-6bd818a742f7","Type":"ContainerStarted","Data":"ead98e6005406b4124b662ac4f630551efd9cebd3d4e13378dc44e1e7a5bf875"}
Sep 18 17:41:38.216939 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:38.216913    2881 scope.go:117] "RemoveContainer" containerID="ac66f71a29f79e2ea01f9d75294e0c406d9f55677f25515f285e9c23064723d5"
Sep 18 17:41:38.217307 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:38.217085    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1bf9a13-4bdf-46f6-a80a-ea425450c0b3-kserve-provision-location\") pod \"f1bf9a13-4bdf-46f6-a80a-ea425450c0b3\" (UID: \"f1bf9a13-4bdf-46f6-a80a-ea425450c0b3\") "
Sep 18 17:41:38.217307 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:38.217165    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f1bf9a13-4bdf-46f6-a80a-ea425450c0b3-proxy-tls\") pod \"f1bf9a13-4bdf-46f6-a80a-ea425450c0b3\" (UID: \"f1bf9a13-4bdf-46f6-a80a-ea425450c0b3\") "
Sep 18 17:41:38.217307 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:38.217218    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-69xdg\" (UniqueName: \"kubernetes.io/projected/f1bf9a13-4bdf-46f6-a80a-ea425450c0b3-kube-api-access-69xdg\") pod \"f1bf9a13-4bdf-46f6-a80a-ea425450c0b3\" (UID: \"f1bf9a13-4bdf-46f6-a80a-ea425450c0b3\") "
Sep 18 17:41:38.217541 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:38.217507    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1bf9a13-4bdf-46f6-a80a-ea425450c0b3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f1bf9a13-4bdf-46f6-a80a-ea425450c0b3" (UID: "f1bf9a13-4bdf-46f6-a80a-ea425450c0b3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:41:38.217718 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:38.217696    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f1bf9a13-4bdf-46f6-a80a-ea425450c0b3-kube-rbac-proxy-sar-config\") pod \"f1bf9a13-4bdf-46f6-a80a-ea425450c0b3\" (UID: \"f1bf9a13-4bdf-46f6-a80a-ea425450c0b3\") "
Sep 18 17:41:38.217833 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:38.217729    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f1bf9a13-4bdf-46f6-a80a-ea425450c0b3-cabundle-cert\") pod \"f1bf9a13-4bdf-46f6-a80a-ea425450c0b3\" (UID: \"f1bf9a13-4bdf-46f6-a80a-ea425450c0b3\") "
Sep 18 17:41:38.218024 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:38.218003    2881 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1bf9a13-4bdf-46f6-a80a-ea425450c0b3-kserve-provision-location\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:41:38.218158 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:38.218103    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1bf9a13-4bdf-46f6-a80a-ea425450c0b3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f1bf9a13-4bdf-46f6-a80a-ea425450c0b3" (UID: "f1bf9a13-4bdf-46f6-a80a-ea425450c0b3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:41:38.218478 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:38.218396    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1bf9a13-4bdf-46f6-a80a-ea425450c0b3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f1bf9a13-4bdf-46f6-a80a-ea425450c0b3" (UID: "f1bf9a13-4bdf-46f6-a80a-ea425450c0b3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:41:38.219411 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:38.219391    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1bf9a13-4bdf-46f6-a80a-ea425450c0b3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f1bf9a13-4bdf-46f6-a80a-ea425450c0b3" (UID: "f1bf9a13-4bdf-46f6-a80a-ea425450c0b3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:41:38.219510 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:38.219478    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1bf9a13-4bdf-46f6-a80a-ea425450c0b3-kube-api-access-69xdg" (OuterVolumeSpecName: "kube-api-access-69xdg") pod "f1bf9a13-4bdf-46f6-a80a-ea425450c0b3" (UID: "f1bf9a13-4bdf-46f6-a80a-ea425450c0b3"). InnerVolumeSpecName "kube-api-access-69xdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:41:38.226552 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:38.226529    2881 scope.go:117] "RemoveContainer" containerID="0e83ce0ada936975aeff43e9c0eea879e5a6a44c315698812e951b305e72e514"
Sep 18 17:41:38.245642 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:38.244546    2881 scope.go:117] "RemoveContainer" containerID="fe2ef96e29a2fdedf10149e95251a9fa6ec18ddc2819038a408bdac8a95a4635"
Sep 18 17:41:38.246204 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:41:38.246171    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe2ef96e29a2fdedf10149e95251a9fa6ec18ddc2819038a408bdac8a95a4635\": container with ID starting with fe2ef96e29a2fdedf10149e95251a9fa6ec18ddc2819038a408bdac8a95a4635 not found: ID does not exist" containerID="fe2ef96e29a2fdedf10149e95251a9fa6ec18ddc2819038a408bdac8a95a4635"
Sep 18 17:41:38.246327 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:38.246216    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe2ef96e29a2fdedf10149e95251a9fa6ec18ddc2819038a408bdac8a95a4635"} err="failed to get container status \"fe2ef96e29a2fdedf10149e95251a9fa6ec18ddc2819038a408bdac8a95a4635\": rpc error: code = NotFound desc = could not find container \"fe2ef96e29a2fdedf10149e95251a9fa6ec18ddc2819038a408bdac8a95a4635\": container with ID starting with fe2ef96e29a2fdedf10149e95251a9fa6ec18ddc2819038a408bdac8a95a4635 not found: ID does not exist"
Sep 18 17:41:38.246327 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:38.246273    2881 scope.go:117] "RemoveContainer" containerID="ac66f71a29f79e2ea01f9d75294e0c406d9f55677f25515f285e9c23064723d5"
Sep 18 17:41:38.246648 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:41:38.246622    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac66f71a29f79e2ea01f9d75294e0c406d9f55677f25515f285e9c23064723d5\": container with ID starting with ac66f71a29f79e2ea01f9d75294e0c406d9f55677f25515f285e9c23064723d5 not found: ID does not exist" containerID="ac66f71a29f79e2ea01f9d75294e0c406d9f55677f25515f285e9c23064723d5"
Sep 18 17:41:38.246707 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:38.246660    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac66f71a29f79e2ea01f9d75294e0c406d9f55677f25515f285e9c23064723d5"} err="failed to get container status \"ac66f71a29f79e2ea01f9d75294e0c406d9f55677f25515f285e9c23064723d5\": rpc error: code = NotFound desc = could not find container \"ac66f71a29f79e2ea01f9d75294e0c406d9f55677f25515f285e9c23064723d5\": container with ID starting with ac66f71a29f79e2ea01f9d75294e0c406d9f55677f25515f285e9c23064723d5 not found: ID does not exist"
Sep 18 17:41:38.246707 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:38.246681    2881 scope.go:117] "RemoveContainer" containerID="0e83ce0ada936975aeff43e9c0eea879e5a6a44c315698812e951b305e72e514"
Sep 18 17:41:38.247027 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:41:38.246999    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e83ce0ada936975aeff43e9c0eea879e5a6a44c315698812e951b305e72e514\": container with ID starting with 0e83ce0ada936975aeff43e9c0eea879e5a6a44c315698812e951b305e72e514 not found: ID does not exist" containerID="0e83ce0ada936975aeff43e9c0eea879e5a6a44c315698812e951b305e72e514"
Sep 18 17:41:38.247128 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:38.247033    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e83ce0ada936975aeff43e9c0eea879e5a6a44c315698812e951b305e72e514"} err="failed to get container status \"0e83ce0ada936975aeff43e9c0eea879e5a6a44c315698812e951b305e72e514\": rpc error: code = NotFound desc = could not find container \"0e83ce0ada936975aeff43e9c0eea879e5a6a44c315698812e951b305e72e514\": container with ID starting with 0e83ce0ada936975aeff43e9c0eea879e5a6a44c315698812e951b305e72e514 not found: ID does not exist"
Sep 18 17:41:38.319725 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:38.319690    2881 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f1bf9a13-4bdf-46f6-a80a-ea425450c0b3-proxy-tls\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:41:38.319725 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:38.319721    2881 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69xdg\" (UniqueName: \"kubernetes.io/projected/f1bf9a13-4bdf-46f6-a80a-ea425450c0b3-kube-api-access-69xdg\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:41:38.319979 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:38.319735    2881 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f1bf9a13-4bdf-46f6-a80a-ea425450c0b3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:41:38.319979 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:38.319750    2881 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f1bf9a13-4bdf-46f6-a80a-ea425450c0b3-cabundle-cert\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:41:38.535413 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:38.535380    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nskpc"]
Sep 18 17:41:38.539800 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:38.539770    2881 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nskpc"]
Sep 18 17:41:39.894403 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:39.894369    2881 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1bf9a13-4bdf-46f6-a80a-ea425450c0b3" path="/var/lib/kubelet/pods/f1bf9a13-4bdf-46f6-a80a-ea425450c0b3/volumes"
Sep 18 17:41:41.007490 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:41.007453    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-7rjg7" podUID="75e97599-fc29-4789-8a43-c26c78a3eaa0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 18 17:41:42.230969 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:42.230887    2881 generic.go:356] "Generic (PLEG): container finished" podID="c48397cf-2334-4a81-a26b-6bd818a742f7" containerID="201c4e8d75382cd355f931f8592fda62373d4369c3450a6a5e1a3e32aa164652" exitCode=0
Sep 18 17:41:42.230969 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:42.230955    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rzbtn" event={"ID":"c48397cf-2334-4a81-a26b-6bd818a742f7","Type":"ContainerDied","Data":"201c4e8d75382cd355f931f8592fda62373d4369c3450a6a5e1a3e32aa164652"}
Sep 18 17:41:43.240764 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:43.240726    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rzbtn" event={"ID":"c48397cf-2334-4a81-a26b-6bd818a742f7","Type":"ContainerStarted","Data":"9b055f1b2598457cfa0591f5054be4c84bc39c929af7a52c91cb4f3cc691e090"}
Sep 18 17:41:43.241127 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:43.240812    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rzbtn" event={"ID":"c48397cf-2334-4a81-a26b-6bd818a742f7","Type":"ContainerStarted","Data":"8c3b63a2f390095d19b05d4ed1bf212c4cf2c34ed6c440b89aaf086880b0bd4d"}
Sep 18 17:41:43.241127 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:43.241033    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rzbtn"
Sep 18 17:41:43.265682 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:43.265620    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rzbtn" podStartSLOduration=2.222475993 podStartE2EDuration="6.265597034s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.043121041s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:41:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:41:43.26517848 +0000 UTC m=+2893.465033771" watchObservedRunningTime="2026-09-18 17:41:43.265597034 +0000 UTC m=+2893.465452321"
Sep 18 17:41:44.246722 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:44.246685    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rzbtn"
Sep 18 17:41:50.257730 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:50.257698    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rzbtn"
Sep 18 17:41:51.007310 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:41:51.007264    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-7rjg7" podUID="75e97599-fc29-4789-8a43-c26c78a3eaa0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 18 17:42:01.008076 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:01.007983    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-7rjg7" podUID="75e97599-fc29-4789-8a43-c26c78a3eaa0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 18 17:42:11.008403 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:11.008367    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-7rjg7"
Sep 18 17:42:19.356709 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:19.356671    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-7rjg7"]
Sep 18 17:42:19.357111 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:19.356997    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-7rjg7" podUID="75e97599-fc29-4789-8a43-c26c78a3eaa0" containerName="kserve-container" containerID="cri-o://8aab4e87cb97c36db8d0c30e0f7000e81813829c2ec792c75420b88422575154" gracePeriod=30
Sep 18 17:42:19.357111 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:19.357089    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-7rjg7" podUID="75e97599-fc29-4789-8a43-c26c78a3eaa0" containerName="kube-rbac-proxy" containerID="cri-o://1f63461a714fdc197949060a3bca41154211f43a9d6395402099c6fb6074adfd" gracePeriod=30
Sep 18 17:42:20.261538 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:20.261509    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rzbtn"
Sep 18 17:42:20.427325 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:20.427289    2881 generic.go:356] "Generic (PLEG): container finished" podID="75e97599-fc29-4789-8a43-c26c78a3eaa0" containerID="1f63461a714fdc197949060a3bca41154211f43a9d6395402099c6fb6074adfd" exitCode=2
Sep 18 17:42:20.427704 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:20.427353    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-7rjg7" event={"ID":"75e97599-fc29-4789-8a43-c26c78a3eaa0","Type":"ContainerDied","Data":"1f63461a714fdc197949060a3bca41154211f43a9d6395402099c6fb6074adfd"}
Sep 18 17:42:21.000875 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:21.000824    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-7rjg7" podUID="75e97599-fc29-4789-8a43-c26c78a3eaa0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.57:8643/healthz\": dial tcp 10.134.0.57:8643: connect: connection refused"
Sep 18 17:42:21.007398 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:21.007366    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-7rjg7" podUID="75e97599-fc29-4789-8a43-c26c78a3eaa0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 18 17:42:23.798927 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:23.798901    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-7rjg7"
Sep 18 17:42:23.836418 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:23.836387    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75e97599-fc29-4789-8a43-c26c78a3eaa0-cabundle-cert\") pod \"75e97599-fc29-4789-8a43-c26c78a3eaa0\" (UID: \"75e97599-fc29-4789-8a43-c26c78a3eaa0\") "
Sep 18 17:42:23.836418 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:23.836421    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75e97599-fc29-4789-8a43-c26c78a3eaa0-kserve-provision-location\") pod \"75e97599-fc29-4789-8a43-c26c78a3eaa0\" (UID: \"75e97599-fc29-4789-8a43-c26c78a3eaa0\") "
Sep 18 17:42:23.836671 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:23.836470    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75e97599-fc29-4789-8a43-c26c78a3eaa0-kube-rbac-proxy-sar-config\") pod \"75e97599-fc29-4789-8a43-c26c78a3eaa0\" (UID: \"75e97599-fc29-4789-8a43-c26c78a3eaa0\") "
Sep 18 17:42:23.836671 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:23.836517    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-55v9n\" (UniqueName: \"kubernetes.io/projected/75e97599-fc29-4789-8a43-c26c78a3eaa0-kube-api-access-55v9n\") pod \"75e97599-fc29-4789-8a43-c26c78a3eaa0\" (UID: \"75e97599-fc29-4789-8a43-c26c78a3eaa0\") "
Sep 18 17:42:23.836671 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:23.836544    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75e97599-fc29-4789-8a43-c26c78a3eaa0-proxy-tls\") pod \"75e97599-fc29-4789-8a43-c26c78a3eaa0\" (UID: \"75e97599-fc29-4789-8a43-c26c78a3eaa0\") "
Sep 18 17:42:23.836854 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:23.836775    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75e97599-fc29-4789-8a43-c26c78a3eaa0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "75e97599-fc29-4789-8a43-c26c78a3eaa0" (UID: "75e97599-fc29-4789-8a43-c26c78a3eaa0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:42:23.836854 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:23.836828    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75e97599-fc29-4789-8a43-c26c78a3eaa0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "75e97599-fc29-4789-8a43-c26c78a3eaa0" (UID: "75e97599-fc29-4789-8a43-c26c78a3eaa0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:42:23.836968 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:23.836863    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75e97599-fc29-4789-8a43-c26c78a3eaa0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "75e97599-fc29-4789-8a43-c26c78a3eaa0" (UID: "75e97599-fc29-4789-8a43-c26c78a3eaa0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:42:23.838717 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:23.838683    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75e97599-fc29-4789-8a43-c26c78a3eaa0-kube-api-access-55v9n" (OuterVolumeSpecName: "kube-api-access-55v9n") pod "75e97599-fc29-4789-8a43-c26c78a3eaa0" (UID: "75e97599-fc29-4789-8a43-c26c78a3eaa0"). InnerVolumeSpecName "kube-api-access-55v9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:42:23.838717 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:23.838712    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75e97599-fc29-4789-8a43-c26c78a3eaa0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "75e97599-fc29-4789-8a43-c26c78a3eaa0" (UID: "75e97599-fc29-4789-8a43-c26c78a3eaa0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:42:23.937442 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:23.937408    2881 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75e97599-fc29-4789-8a43-c26c78a3eaa0-proxy-tls\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:42:23.937442 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:23.937434    2881 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75e97599-fc29-4789-8a43-c26c78a3eaa0-cabundle-cert\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:42:23.937442 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:23.937445    2881 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75e97599-fc29-4789-8a43-c26c78a3eaa0-kserve-provision-location\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:42:23.937680 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:23.937454    2881 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75e97599-fc29-4789-8a43-c26c78a3eaa0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:42:23.937680 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:23.937468    2881 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55v9n\" (UniqueName: \"kubernetes.io/projected/75e97599-fc29-4789-8a43-c26c78a3eaa0-kube-api-access-55v9n\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:42:24.447929 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:24.447892    2881 generic.go:356] "Generic (PLEG): container finished" podID="75e97599-fc29-4789-8a43-c26c78a3eaa0" containerID="8aab4e87cb97c36db8d0c30e0f7000e81813829c2ec792c75420b88422575154" exitCode=0
Sep 18 17:42:24.448101 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:24.447957    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-7rjg7"
Sep 18 17:42:24.448101 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:24.447974    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-7rjg7" event={"ID":"75e97599-fc29-4789-8a43-c26c78a3eaa0","Type":"ContainerDied","Data":"8aab4e87cb97c36db8d0c30e0f7000e81813829c2ec792c75420b88422575154"}
Sep 18 17:42:24.448101 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:24.448035    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-7rjg7" event={"ID":"75e97599-fc29-4789-8a43-c26c78a3eaa0","Type":"ContainerDied","Data":"cc7f42a96699b7001bd0bbcb6528e10c2f084c59b69787441c7e4fdbf774902f"}
Sep 18 17:42:24.448101 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:24.448063    2881 scope.go:117] "RemoveContainer" containerID="1f63461a714fdc197949060a3bca41154211f43a9d6395402099c6fb6074adfd"
Sep 18 17:42:24.456784 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:24.456767    2881 scope.go:117] "RemoveContainer" containerID="8aab4e87cb97c36db8d0c30e0f7000e81813829c2ec792c75420b88422575154"
Sep 18 17:42:24.465161 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:24.465140    2881 scope.go:117] "RemoveContainer" containerID="82da5af8442bd36baf7ba3961edf02dae0401a0d5c8aeeeeac247b7f478d0cae"
Sep 18 17:42:24.470790 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:24.470758    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-7rjg7"]
Sep 18 17:42:24.475586 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:24.475565    2881 scope.go:117] "RemoveContainer" containerID="1f63461a714fdc197949060a3bca41154211f43a9d6395402099c6fb6074adfd"
Sep 18 17:42:24.476529 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:42:24.476408    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f63461a714fdc197949060a3bca41154211f43a9d6395402099c6fb6074adfd\": container with ID starting with 1f63461a714fdc197949060a3bca41154211f43a9d6395402099c6fb6074adfd not found: ID does not exist" containerID="1f63461a714fdc197949060a3bca41154211f43a9d6395402099c6fb6074adfd"
Sep 18 17:42:24.476529 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:24.476445    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f63461a714fdc197949060a3bca41154211f43a9d6395402099c6fb6074adfd"} err="failed to get container status \"1f63461a714fdc197949060a3bca41154211f43a9d6395402099c6fb6074adfd\": rpc error: code = NotFound desc = could not find container \"1f63461a714fdc197949060a3bca41154211f43a9d6395402099c6fb6074adfd\": container with ID starting with 1f63461a714fdc197949060a3bca41154211f43a9d6395402099c6fb6074adfd not found: ID does not exist"
Sep 18 17:42:24.476529 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:24.476470    2881 scope.go:117] "RemoveContainer" containerID="8aab4e87cb97c36db8d0c30e0f7000e81813829c2ec792c75420b88422575154"
Sep 18 17:42:24.476853 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:42:24.476831    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8aab4e87cb97c36db8d0c30e0f7000e81813829c2ec792c75420b88422575154\": container with ID starting with 8aab4e87cb97c36db8d0c30e0f7000e81813829c2ec792c75420b88422575154 not found: ID does not exist" containerID="8aab4e87cb97c36db8d0c30e0f7000e81813829c2ec792c75420b88422575154"
Sep 18 17:42:24.476928 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:24.476862    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8aab4e87cb97c36db8d0c30e0f7000e81813829c2ec792c75420b88422575154"} err="failed to get container status \"8aab4e87cb97c36db8d0c30e0f7000e81813829c2ec792c75420b88422575154\": rpc error: code = NotFound desc = could not find container \"8aab4e87cb97c36db8d0c30e0f7000e81813829c2ec792c75420b88422575154\": container with ID starting with 8aab4e87cb97c36db8d0c30e0f7000e81813829c2ec792c75420b88422575154 not found: ID does not exist"
Sep 18 17:42:24.476928 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:24.476887    2881 scope.go:117] "RemoveContainer" containerID="82da5af8442bd36baf7ba3961edf02dae0401a0d5c8aeeeeac247b7f478d0cae"
Sep 18 17:42:24.477195 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:42:24.477177    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82da5af8442bd36baf7ba3961edf02dae0401a0d5c8aeeeeac247b7f478d0cae\": container with ID starting with 82da5af8442bd36baf7ba3961edf02dae0401a0d5c8aeeeeac247b7f478d0cae not found: ID does not exist" containerID="82da5af8442bd36baf7ba3961edf02dae0401a0d5c8aeeeeac247b7f478d0cae"
Sep 18 17:42:24.477284 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:24.477199    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82da5af8442bd36baf7ba3961edf02dae0401a0d5c8aeeeeac247b7f478d0cae"} err="failed to get container status \"82da5af8442bd36baf7ba3961edf02dae0401a0d5c8aeeeeac247b7f478d0cae\": rpc error: code = NotFound desc = could not find container \"82da5af8442bd36baf7ba3961edf02dae0401a0d5c8aeeeeac247b7f478d0cae\": container with ID starting with 82da5af8442bd36baf7ba3961edf02dae0401a0d5c8aeeeeac247b7f478d0cae not found: ID does not exist"
Sep 18 17:42:24.478460 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:24.478436    2881 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-7rjg7"]
Sep 18 17:42:24.590021 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:24.589987    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-865f87676-4gxjk"]
Sep 18 17:42:24.590431 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:24.590413    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1bf9a13-4bdf-46f6-a80a-ea425450c0b3" containerName="kube-rbac-proxy"
Sep 18 17:42:24.590550 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:24.590434    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1bf9a13-4bdf-46f6-a80a-ea425450c0b3" containerName="kube-rbac-proxy"
Sep 18 17:42:24.590550 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:24.590454    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75e97599-fc29-4789-8a43-c26c78a3eaa0" containerName="storage-initializer"
Sep 18 17:42:24.590550 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:24.590464    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="75e97599-fc29-4789-8a43-c26c78a3eaa0" containerName="storage-initializer"
Sep 18 17:42:24.590550 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:24.590477    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1bf9a13-4bdf-46f6-a80a-ea425450c0b3" containerName="storage-initializer"
Sep 18 17:42:24.590550 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:24.590487    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1bf9a13-4bdf-46f6-a80a-ea425450c0b3" containerName="storage-initializer"
Sep 18 17:42:24.590550 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:24.590500    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1bf9a13-4bdf-46f6-a80a-ea425450c0b3" containerName="kserve-container"
Sep 18 17:42:24.590550 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:24.590507    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1bf9a13-4bdf-46f6-a80a-ea425450c0b3" containerName="kserve-container"
Sep 18 17:42:24.590550 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:24.590515    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75e97599-fc29-4789-8a43-c26c78a3eaa0" containerName="kube-rbac-proxy"
Sep 18 17:42:24.590550 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:24.590523    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="75e97599-fc29-4789-8a43-c26c78a3eaa0" containerName="kube-rbac-proxy"
Sep 18 17:42:24.590550 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:24.590553    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75e97599-fc29-4789-8a43-c26c78a3eaa0" containerName="kserve-container"
Sep 18 17:42:24.591040 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:24.590561    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="75e97599-fc29-4789-8a43-c26c78a3eaa0" containerName="kserve-container"
Sep 18 17:42:24.591040 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:24.590643    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1bf9a13-4bdf-46f6-a80a-ea425450c0b3" containerName="kserve-container"
Sep 18 17:42:24.591040 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:24.590656    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1bf9a13-4bdf-46f6-a80a-ea425450c0b3" containerName="kube-rbac-proxy"
Sep 18 17:42:24.591040 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:24.590666    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="75e97599-fc29-4789-8a43-c26c78a3eaa0" containerName="kube-rbac-proxy"
Sep 18 17:42:24.591040 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:24.590676    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="75e97599-fc29-4789-8a43-c26c78a3eaa0" containerName="kserve-container"
Sep 18 17:42:24.595548 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:24.595523    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-865f87676-4gxjk"
Sep 18 17:42:24.598071 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:24.598051    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 18 17:42:24.598309 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:24.598293    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-predictor-serving-cert\""
Sep 18 17:42:24.598418 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:24.598403    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-kube-rbac-proxy-sar-config\""
Sep 18 17:42:24.605655 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:24.605631    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-865f87676-4gxjk"]
Sep 18 17:42:24.643141 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:24.643102    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db6c5dde-0465-4839-ae5e-0a5293b68cec-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-865f87676-4gxjk\" (UID: \"db6c5dde-0465-4839-ae5e-0a5293b68cec\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-865f87676-4gxjk"
Sep 18 17:42:24.643332 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:24.643156    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db6c5dde-0465-4839-ae5e-0a5293b68cec-proxy-tls\") pod \"isvc-sklearn-s3-predictor-865f87676-4gxjk\" (UID: \"db6c5dde-0465-4839-ae5e-0a5293b68cec\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-865f87676-4gxjk"
Sep 18 17:42:24.643332 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:24.643253    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db6c5dde-0465-4839-ae5e-0a5293b68cec-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-865f87676-4gxjk\" (UID: \"db6c5dde-0465-4839-ae5e-0a5293b68cec\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-865f87676-4gxjk"
Sep 18 17:42:24.643332 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:24.643291    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk76q\" (UniqueName: \"kubernetes.io/projected/db6c5dde-0465-4839-ae5e-0a5293b68cec-kube-api-access-xk76q\") pod \"isvc-sklearn-s3-predictor-865f87676-4gxjk\" (UID: \"db6c5dde-0465-4839-ae5e-0a5293b68cec\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-865f87676-4gxjk"
Sep 18 17:42:24.744165 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:24.744062    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db6c5dde-0465-4839-ae5e-0a5293b68cec-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-865f87676-4gxjk\" (UID: \"db6c5dde-0465-4839-ae5e-0a5293b68cec\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-865f87676-4gxjk"
Sep 18 17:42:24.744165 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:24.744117    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db6c5dde-0465-4839-ae5e-0a5293b68cec-proxy-tls\") pod \"isvc-sklearn-s3-predictor-865f87676-4gxjk\" (UID: \"db6c5dde-0465-4839-ae5e-0a5293b68cec\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-865f87676-4gxjk"
Sep 18 17:42:24.744165 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:24.744155    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db6c5dde-0465-4839-ae5e-0a5293b68cec-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-865f87676-4gxjk\" (UID: \"db6c5dde-0465-4839-ae5e-0a5293b68cec\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-865f87676-4gxjk"
Sep 18 17:42:24.744512 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:24.744176    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk76q\" (UniqueName: \"kubernetes.io/projected/db6c5dde-0465-4839-ae5e-0a5293b68cec-kube-api-access-xk76q\") pod \"isvc-sklearn-s3-predictor-865f87676-4gxjk\" (UID: \"db6c5dde-0465-4839-ae5e-0a5293b68cec\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-865f87676-4gxjk"
Sep 18 17:42:24.744512 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:24.744506    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db6c5dde-0465-4839-ae5e-0a5293b68cec-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-865f87676-4gxjk\" (UID: \"db6c5dde-0465-4839-ae5e-0a5293b68cec\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-865f87676-4gxjk"
Sep 18 17:42:24.744870 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:24.744847    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db6c5dde-0465-4839-ae5e-0a5293b68cec-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-865f87676-4gxjk\" (UID: \"db6c5dde-0465-4839-ae5e-0a5293b68cec\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-865f87676-4gxjk"
Sep 18 17:42:24.746648 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:24.746620    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db6c5dde-0465-4839-ae5e-0a5293b68cec-proxy-tls\") pod \"isvc-sklearn-s3-predictor-865f87676-4gxjk\" (UID: \"db6c5dde-0465-4839-ae5e-0a5293b68cec\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-865f87676-4gxjk"
Sep 18 17:42:24.755108 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:24.755075    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk76q\" (UniqueName: \"kubernetes.io/projected/db6c5dde-0465-4839-ae5e-0a5293b68cec-kube-api-access-xk76q\") pod \"isvc-sklearn-s3-predictor-865f87676-4gxjk\" (UID: \"db6c5dde-0465-4839-ae5e-0a5293b68cec\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-865f87676-4gxjk"
Sep 18 17:42:24.906321 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:24.906281    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-865f87676-4gxjk"
Sep 18 17:42:25.182053 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:25.182018    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-865f87676-4gxjk"]
Sep 18 17:42:25.185543 ip-10-0-143-5 kubenswrapper[2881]: W0918 17:42:25.185516    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb6c5dde_0465_4839_ae5e_0a5293b68cec.slice/crio-6dcb959b68a25980ff5e52f8ae880d5bbb8ccf3cee234286726e6818c1a7fb46 WatchSource:0}: Error finding container 6dcb959b68a25980ff5e52f8ae880d5bbb8ccf3cee234286726e6818c1a7fb46: Status 404 returned error can't find the container with id 6dcb959b68a25980ff5e52f8ae880d5bbb8ccf3cee234286726e6818c1a7fb46
Sep 18 17:42:25.455973 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:25.455889    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-865f87676-4gxjk" event={"ID":"db6c5dde-0465-4839-ae5e-0a5293b68cec","Type":"ContainerStarted","Data":"3496f256b78ff058924055c269e3d2bdee927db5381c478d7eab026afd21595a"}
Sep 18 17:42:25.455973 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:25.455944    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-865f87676-4gxjk" event={"ID":"db6c5dde-0465-4839-ae5e-0a5293b68cec","Type":"ContainerStarted","Data":"6dcb959b68a25980ff5e52f8ae880d5bbb8ccf3cee234286726e6818c1a7fb46"}
Sep 18 17:42:25.893339 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:25.893293    2881 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75e97599-fc29-4789-8a43-c26c78a3eaa0" path="/var/lib/kubelet/pods/75e97599-fc29-4789-8a43-c26c78a3eaa0/volumes"
Sep 18 17:42:27.196575 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:27.196538    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rzbtn"]
Sep 18 17:42:27.197051 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:27.196783    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rzbtn" podUID="c48397cf-2334-4a81-a26b-6bd818a742f7" containerName="kserve-container" containerID="cri-o://8c3b63a2f390095d19b05d4ed1bf212c4cf2c34ed6c440b89aaf086880b0bd4d" gracePeriod=30
Sep 18 17:42:27.197051 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:27.196826    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rzbtn" podUID="c48397cf-2334-4a81-a26b-6bd818a742f7" containerName="kube-rbac-proxy" containerID="cri-o://9b055f1b2598457cfa0591f5054be4c84bc39c929af7a52c91cb4f3cc691e090" gracePeriod=30
Sep 18 17:42:27.469005 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:27.468910    2881 generic.go:356] "Generic (PLEG): container finished" podID="db6c5dde-0465-4839-ae5e-0a5293b68cec" containerID="3496f256b78ff058924055c269e3d2bdee927db5381c478d7eab026afd21595a" exitCode=0
Sep 18 17:42:27.469179 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:27.468990    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-865f87676-4gxjk" event={"ID":"db6c5dde-0465-4839-ae5e-0a5293b68cec","Type":"ContainerDied","Data":"3496f256b78ff058924055c269e3d2bdee927db5381c478d7eab026afd21595a"}
Sep 18 17:42:27.471303 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:27.471280    2881 generic.go:356] "Generic (PLEG): container finished" podID="c48397cf-2334-4a81-a26b-6bd818a742f7" containerID="9b055f1b2598457cfa0591f5054be4c84bc39c929af7a52c91cb4f3cc691e090" exitCode=2
Sep 18 17:42:27.471430 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:27.471314    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rzbtn" event={"ID":"c48397cf-2334-4a81-a26b-6bd818a742f7","Type":"ContainerDied","Data":"9b055f1b2598457cfa0591f5054be4c84bc39c929af7a52c91cb4f3cc691e090"}
Sep 18 17:42:28.355796 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:28.355768    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rzbtn"
Sep 18 17:42:28.478541 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:28.478446    2881 generic.go:356] "Generic (PLEG): container finished" podID="c48397cf-2334-4a81-a26b-6bd818a742f7" containerID="8c3b63a2f390095d19b05d4ed1bf212c4cf2c34ed6c440b89aaf086880b0bd4d" exitCode=0
Sep 18 17:42:28.478541 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:28.478513    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rzbtn"
Sep 18 17:42:28.478541 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:28.478527    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rzbtn" event={"ID":"c48397cf-2334-4a81-a26b-6bd818a742f7","Type":"ContainerDied","Data":"8c3b63a2f390095d19b05d4ed1bf212c4cf2c34ed6c440b89aaf086880b0bd4d"}
Sep 18 17:42:28.478787 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:28.478590    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rzbtn" event={"ID":"c48397cf-2334-4a81-a26b-6bd818a742f7","Type":"ContainerDied","Data":"ead98e6005406b4124b662ac4f630551efd9cebd3d4e13378dc44e1e7a5bf875"}
Sep 18 17:42:28.478787 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:28.478621    2881 scope.go:117] "RemoveContainer" containerID="9b055f1b2598457cfa0591f5054be4c84bc39c929af7a52c91cb4f3cc691e090"
Sep 18 17:42:28.479819 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:28.479802    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c48397cf-2334-4a81-a26b-6bd818a742f7-kserve-provision-location\") pod \"c48397cf-2334-4a81-a26b-6bd818a742f7\" (UID: \"c48397cf-2334-4a81-a26b-6bd818a742f7\") "
Sep 18 17:42:28.479929 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:28.479904    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b8fw9\" (UniqueName: \"kubernetes.io/projected/c48397cf-2334-4a81-a26b-6bd818a742f7-kube-api-access-b8fw9\") pod \"c48397cf-2334-4a81-a26b-6bd818a742f7\" (UID: \"c48397cf-2334-4a81-a26b-6bd818a742f7\") "
Sep 18 17:42:28.479996 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:28.479949    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c48397cf-2334-4a81-a26b-6bd818a742f7-proxy-tls\") pod \"c48397cf-2334-4a81-a26b-6bd818a742f7\" (UID: \"c48397cf-2334-4a81-a26b-6bd818a742f7\") "
Sep 18 17:42:28.480054 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:28.480017    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c48397cf-2334-4a81-a26b-6bd818a742f7-cabundle-cert\") pod \"c48397cf-2334-4a81-a26b-6bd818a742f7\" (UID: \"c48397cf-2334-4a81-a26b-6bd818a742f7\") "
Sep 18 17:42:28.480465 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:28.480106    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c48397cf-2334-4a81-a26b-6bd818a742f7-kube-rbac-proxy-sar-config\") pod \"c48397cf-2334-4a81-a26b-6bd818a742f7\" (UID: \"c48397cf-2334-4a81-a26b-6bd818a742f7\") "
Sep 18 17:42:28.480465 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:28.480393    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c48397cf-2334-4a81-a26b-6bd818a742f7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c48397cf-2334-4a81-a26b-6bd818a742f7" (UID: "c48397cf-2334-4a81-a26b-6bd818a742f7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:42:28.480465 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:28.480451    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c48397cf-2334-4a81-a26b-6bd818a742f7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c48397cf-2334-4a81-a26b-6bd818a742f7" (UID: "c48397cf-2334-4a81-a26b-6bd818a742f7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:42:28.480707 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:28.480539    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c48397cf-2334-4a81-a26b-6bd818a742f7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c48397cf-2334-4a81-a26b-6bd818a742f7" (UID: "c48397cf-2334-4a81-a26b-6bd818a742f7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:42:28.481669 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:28.481644    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-865f87676-4gxjk" event={"ID":"db6c5dde-0465-4839-ae5e-0a5293b68cec","Type":"ContainerStarted","Data":"3af5885b8ebd89cf7db9de38765ddf5d1fc43765015b2e27014f11deafbc6fba"}
Sep 18 17:42:28.481805 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:28.481702    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-865f87676-4gxjk" event={"ID":"db6c5dde-0465-4839-ae5e-0a5293b68cec","Type":"ContainerStarted","Data":"185a40d013d91647cbe6c37a718f811bd5990f30dad004d4c2aa514dc87e54f8"}
Sep 18 17:42:28.482070 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:28.482047    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-865f87676-4gxjk"
Sep 18 17:42:28.482185 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:28.482103    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-865f87676-4gxjk"
Sep 18 17:42:28.482323 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:28.482297    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c48397cf-2334-4a81-a26b-6bd818a742f7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c48397cf-2334-4a81-a26b-6bd818a742f7" (UID: "c48397cf-2334-4a81-a26b-6bd818a742f7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:42:28.482323 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:28.482319    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c48397cf-2334-4a81-a26b-6bd818a742f7-kube-api-access-b8fw9" (OuterVolumeSpecName: "kube-api-access-b8fw9") pod "c48397cf-2334-4a81-a26b-6bd818a742f7" (UID: "c48397cf-2334-4a81-a26b-6bd818a742f7"). InnerVolumeSpecName "kube-api-access-b8fw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:42:28.483734 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:28.483704    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-865f87676-4gxjk" podUID="db6c5dde-0465-4839-ae5e-0a5293b68cec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 18 17:42:28.498167 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:28.498145    2881 scope.go:117] "RemoveContainer" containerID="8c3b63a2f390095d19b05d4ed1bf212c4cf2c34ed6c440b89aaf086880b0bd4d"
Sep 18 17:42:28.504698 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:28.504652    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-865f87676-4gxjk" podStartSLOduration=3.288218534 podStartE2EDuration="4.50463705s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.216418516s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:42:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:42:28.502609748 +0000 UTC m=+2938.702465036" watchObservedRunningTime="2026-09-18 17:42:28.50463705 +0000 UTC m=+2938.704492336"
Sep 18 17:42:28.507396 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:28.507376    2881 scope.go:117] "RemoveContainer" containerID="201c4e8d75382cd355f931f8592fda62373d4369c3450a6a5e1a3e32aa164652"
Sep 18 17:42:28.515582 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:28.515564    2881 scope.go:117] "RemoveContainer" containerID="9b055f1b2598457cfa0591f5054be4c84bc39c929af7a52c91cb4f3cc691e090"
Sep 18 17:42:28.515920 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:42:28.515894    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b055f1b2598457cfa0591f5054be4c84bc39c929af7a52c91cb4f3cc691e090\": container with ID starting with 9b055f1b2598457cfa0591f5054be4c84bc39c929af7a52c91cb4f3cc691e090 not found: ID does not exist" containerID="9b055f1b2598457cfa0591f5054be4c84bc39c929af7a52c91cb4f3cc691e090"
Sep 18 17:42:28.516009 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:28.515926    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b055f1b2598457cfa0591f5054be4c84bc39c929af7a52c91cb4f3cc691e090"} err="failed to get container status \"9b055f1b2598457cfa0591f5054be4c84bc39c929af7a52c91cb4f3cc691e090\": rpc error: code = NotFound desc = could not find container \"9b055f1b2598457cfa0591f5054be4c84bc39c929af7a52c91cb4f3cc691e090\": container with ID starting with 9b055f1b2598457cfa0591f5054be4c84bc39c929af7a52c91cb4f3cc691e090 not found: ID does not exist"
Sep 18 17:42:28.516009 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:28.515944    2881 scope.go:117] "RemoveContainer" containerID="8c3b63a2f390095d19b05d4ed1bf212c4cf2c34ed6c440b89aaf086880b0bd4d"
Sep 18 17:42:28.516264 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:42:28.516242    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c3b63a2f390095d19b05d4ed1bf212c4cf2c34ed6c440b89aaf086880b0bd4d\": container with ID starting with 8c3b63a2f390095d19b05d4ed1bf212c4cf2c34ed6c440b89aaf086880b0bd4d not found: ID does not exist" containerID="8c3b63a2f390095d19b05d4ed1bf212c4cf2c34ed6c440b89aaf086880b0bd4d"
Sep 18 17:42:28.516317 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:28.516271    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c3b63a2f390095d19b05d4ed1bf212c4cf2c34ed6c440b89aaf086880b0bd4d"} err="failed to get container status \"8c3b63a2f390095d19b05d4ed1bf212c4cf2c34ed6c440b89aaf086880b0bd4d\": rpc error: code = NotFound desc = could not find container \"8c3b63a2f390095d19b05d4ed1bf212c4cf2c34ed6c440b89aaf086880b0bd4d\": container with ID starting with 8c3b63a2f390095d19b05d4ed1bf212c4cf2c34ed6c440b89aaf086880b0bd4d not found: ID does not exist"
Sep 18 17:42:28.516317 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:28.516285    2881 scope.go:117] "RemoveContainer" containerID="201c4e8d75382cd355f931f8592fda62373d4369c3450a6a5e1a3e32aa164652"
Sep 18 17:42:28.516524 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:42:28.516506    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"201c4e8d75382cd355f931f8592fda62373d4369c3450a6a5e1a3e32aa164652\": container with ID starting with 201c4e8d75382cd355f931f8592fda62373d4369c3450a6a5e1a3e32aa164652 not found: ID does not exist" containerID="201c4e8d75382cd355f931f8592fda62373d4369c3450a6a5e1a3e32aa164652"
Sep 18 17:42:28.516567 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:28.516533    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"201c4e8d75382cd355f931f8592fda62373d4369c3450a6a5e1a3e32aa164652"} err="failed to get container status \"201c4e8d75382cd355f931f8592fda62373d4369c3450a6a5e1a3e32aa164652\": rpc error: code = NotFound desc = could not find container \"201c4e8d75382cd355f931f8592fda62373d4369c3450a6a5e1a3e32aa164652\": container with ID starting with 201c4e8d75382cd355f931f8592fda62373d4369c3450a6a5e1a3e32aa164652 not found: ID does not exist"
Sep 18 17:42:28.581672 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:28.581636    2881 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c48397cf-2334-4a81-a26b-6bd818a742f7-kserve-provision-location\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:42:28.581672 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:28.581665    2881 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8fw9\" (UniqueName: \"kubernetes.io/projected/c48397cf-2334-4a81-a26b-6bd818a742f7-kube-api-access-b8fw9\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:42:28.581672 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:28.581676    2881 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c48397cf-2334-4a81-a26b-6bd818a742f7-proxy-tls\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:42:28.581672 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:28.581685    2881 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c48397cf-2334-4a81-a26b-6bd818a742f7-cabundle-cert\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:42:28.581997 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:28.581694    2881 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c48397cf-2334-4a81-a26b-6bd818a742f7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:42:28.816156 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:28.816124    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rzbtn"]
Sep 18 17:42:28.820230 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:28.820196    2881 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rzbtn"]
Sep 18 17:42:29.487456 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:29.487406    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-865f87676-4gxjk" podUID="db6c5dde-0465-4839-ae5e-0a5293b68cec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 18 17:42:29.896067 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:29.896027    2881 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c48397cf-2334-4a81-a26b-6bd818a742f7" path="/var/lib/kubelet/pods/c48397cf-2334-4a81-a26b-6bd818a742f7/volumes"
Sep 18 17:42:32.369281 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:32.369241    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n6fd9"]
Sep 18 17:42:32.369658 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:32.369624    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c48397cf-2334-4a81-a26b-6bd818a742f7" containerName="kube-rbac-proxy"
Sep 18 17:42:32.369658 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:32.369637    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="c48397cf-2334-4a81-a26b-6bd818a742f7" containerName="kube-rbac-proxy"
Sep 18 17:42:32.369658 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:32.369650    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c48397cf-2334-4a81-a26b-6bd818a742f7" containerName="storage-initializer"
Sep 18 17:42:32.369658 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:32.369655    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="c48397cf-2334-4a81-a26b-6bd818a742f7" containerName="storage-initializer"
Sep 18 17:42:32.369787 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:32.369674    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c48397cf-2334-4a81-a26b-6bd818a742f7" containerName="kserve-container"
Sep 18 17:42:32.369787 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:32.369684    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="c48397cf-2334-4a81-a26b-6bd818a742f7" containerName="kserve-container"
Sep 18 17:42:32.369787 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:32.369740    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="c48397cf-2334-4a81-a26b-6bd818a742f7" containerName="kserve-container"
Sep 18 17:42:32.369787 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:32.369751    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="c48397cf-2334-4a81-a26b-6bd818a742f7" containerName="kube-rbac-proxy"
Sep 18 17:42:32.374703 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:32.374683    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n6fd9"
Sep 18 17:42:32.376957 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:32.376934    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-predictor-serving-cert\""
Sep 18 17:42:32.377264 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:32.377247    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 18 17:42:32.377342 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:32.377280    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 17:42:32.377342 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:32.377280    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 17:42:32.384511 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:32.384485    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n6fd9"]
Sep 18 17:42:32.413875 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:32.413841    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6vm5\" (UniqueName: \"kubernetes.io/projected/3eee5273-1f28-448e-b4dd-2d7451c4aa35-kube-api-access-x6vm5\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n6fd9\" (UID: \"3eee5273-1f28-448e-b4dd-2d7451c4aa35\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n6fd9"
Sep 18 17:42:32.414047 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:32.413885    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3eee5273-1f28-448e-b4dd-2d7451c4aa35-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n6fd9\" (UID: \"3eee5273-1f28-448e-b4dd-2d7451c4aa35\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n6fd9"
Sep 18 17:42:32.414047 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:32.413920    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3eee5273-1f28-448e-b4dd-2d7451c4aa35-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n6fd9\" (UID: \"3eee5273-1f28-448e-b4dd-2d7451c4aa35\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n6fd9"
Sep 18 17:42:32.414047 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:32.413970    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3eee5273-1f28-448e-b4dd-2d7451c4aa35-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n6fd9\" (UID: \"3eee5273-1f28-448e-b4dd-2d7451c4aa35\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n6fd9"
Sep 18 17:42:32.414047 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:32.413991    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3eee5273-1f28-448e-b4dd-2d7451c4aa35-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n6fd9\" (UID: \"3eee5273-1f28-448e-b4dd-2d7451c4aa35\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n6fd9"
Sep 18 17:42:32.514993 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:32.514958    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6vm5\" (UniqueName: \"kubernetes.io/projected/3eee5273-1f28-448e-b4dd-2d7451c4aa35-kube-api-access-x6vm5\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n6fd9\" (UID: \"3eee5273-1f28-448e-b4dd-2d7451c4aa35\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n6fd9"
Sep 18 17:42:32.515163 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:32.515000    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3eee5273-1f28-448e-b4dd-2d7451c4aa35-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n6fd9\" (UID: \"3eee5273-1f28-448e-b4dd-2d7451c4aa35\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n6fd9"
Sep 18 17:42:32.515163 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:32.515038    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3eee5273-1f28-448e-b4dd-2d7451c4aa35-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n6fd9\" (UID: \"3eee5273-1f28-448e-b4dd-2d7451c4aa35\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n6fd9"
Sep 18 17:42:32.515163 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:32.515067    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3eee5273-1f28-448e-b4dd-2d7451c4aa35-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n6fd9\" (UID: \"3eee5273-1f28-448e-b4dd-2d7451c4aa35\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n6fd9"
Sep 18 17:42:32.515163 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:32.515090    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3eee5273-1f28-448e-b4dd-2d7451c4aa35-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n6fd9\" (UID: \"3eee5273-1f28-448e-b4dd-2d7451c4aa35\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n6fd9"
Sep 18 17:42:32.515544 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:32.515519    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3eee5273-1f28-448e-b4dd-2d7451c4aa35-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n6fd9\" (UID: \"3eee5273-1f28-448e-b4dd-2d7451c4aa35\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n6fd9"
Sep 18 17:42:32.515826 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:32.515806    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3eee5273-1f28-448e-b4dd-2d7451c4aa35-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n6fd9\" (UID: \"3eee5273-1f28-448e-b4dd-2d7451c4aa35\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n6fd9"
Sep 18 17:42:32.515918 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:32.515831    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3eee5273-1f28-448e-b4dd-2d7451c4aa35-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n6fd9\" (UID: \"3eee5273-1f28-448e-b4dd-2d7451c4aa35\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n6fd9"
Sep 18 17:42:32.517448 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:32.517430    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3eee5273-1f28-448e-b4dd-2d7451c4aa35-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n6fd9\" (UID: \"3eee5273-1f28-448e-b4dd-2d7451c4aa35\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n6fd9"
Sep 18 17:42:32.524814 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:32.524794    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6vm5\" (UniqueName: \"kubernetes.io/projected/3eee5273-1f28-448e-b4dd-2d7451c4aa35-kube-api-access-x6vm5\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n6fd9\" (UID: \"3eee5273-1f28-448e-b4dd-2d7451c4aa35\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n6fd9"
Sep 18 17:42:32.685554 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:32.685460    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n6fd9"
Sep 18 17:42:32.965188 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:32.965161    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n6fd9"]
Sep 18 17:42:32.967635 ip-10-0-143-5 kubenswrapper[2881]: W0918 17:42:32.967599    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eee5273_1f28_448e_b4dd_2d7451c4aa35.slice/crio-58d7658169da596a425ec0a80b5241aed2f048d9c986521ba6605e04b86c79a8 WatchSource:0}: Error finding container 58d7658169da596a425ec0a80b5241aed2f048d9c986521ba6605e04b86c79a8: Status 404 returned error can't find the container with id 58d7658169da596a425ec0a80b5241aed2f048d9c986521ba6605e04b86c79a8
Sep 18 17:42:33.508015 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:33.507970    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n6fd9" event={"ID":"3eee5273-1f28-448e-b4dd-2d7451c4aa35","Type":"ContainerStarted","Data":"46f3c2306f8c0c7761eaa9956f89c6405f841e6f1957d7c45899e84be9636a76"}
Sep 18 17:42:33.508537 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:33.508048    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n6fd9" event={"ID":"3eee5273-1f28-448e-b4dd-2d7451c4aa35","Type":"ContainerStarted","Data":"58d7658169da596a425ec0a80b5241aed2f048d9c986521ba6605e04b86c79a8"}
Sep 18 17:42:34.492260 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:34.492217    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-865f87676-4gxjk"
Sep 18 17:42:34.492907 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:34.492880    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-865f87676-4gxjk" podUID="db6c5dde-0465-4839-ae5e-0a5293b68cec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 18 17:42:37.528133 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:37.528099    2881 generic.go:356] "Generic (PLEG): container finished" podID="3eee5273-1f28-448e-b4dd-2d7451c4aa35" containerID="46f3c2306f8c0c7761eaa9956f89c6405f841e6f1957d7c45899e84be9636a76" exitCode=0
Sep 18 17:42:37.528522 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:37.528136    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n6fd9" event={"ID":"3eee5273-1f28-448e-b4dd-2d7451c4aa35","Type":"ContainerDied","Data":"46f3c2306f8c0c7761eaa9956f89c6405f841e6f1957d7c45899e84be9636a76"}
Sep 18 17:42:38.536617 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:38.536578    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n6fd9" event={"ID":"3eee5273-1f28-448e-b4dd-2d7451c4aa35","Type":"ContainerStarted","Data":"6f17659354ec62608bb9cdb409c482358f154e5b1ba2fa595b1c9ef133d91bdc"}
Sep 18 17:42:38.537115 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:38.536648    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n6fd9" event={"ID":"3eee5273-1f28-448e-b4dd-2d7451c4aa35","Type":"ContainerStarted","Data":"f8e9db5e1967eda6cab2072c27e401568bdb8946d814f176403cba5f49ef35fd"}
Sep 18 17:42:38.537115 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:38.536872    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n6fd9"
Sep 18 17:42:38.561710 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:38.561650    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n6fd9" podStartSLOduration=2.954429279 podStartE2EDuration="6.56163265s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.607203371s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:42:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:42:38.559257699 +0000 UTC m=+2948.759112985" watchObservedRunningTime="2026-09-18 17:42:38.56163265 +0000 UTC m=+2948.761487936"
Sep 18 17:42:39.542711 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:39.542678    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n6fd9"
Sep 18 17:42:39.543941 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:39.543913    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n6fd9" podUID="3eee5273-1f28-448e-b4dd-2d7451c4aa35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 18 17:42:40.547419 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:40.547377    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n6fd9" podUID="3eee5273-1f28-448e-b4dd-2d7451c4aa35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 18 17:42:44.493487 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:44.493440    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-865f87676-4gxjk" podUID="db6c5dde-0465-4839-ae5e-0a5293b68cec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 18 17:42:45.552079 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:45.552048    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n6fd9"
Sep 18 17:42:45.552719 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:45.552690    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n6fd9" podUID="3eee5273-1f28-448e-b4dd-2d7451c4aa35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 18 17:42:54.492917 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:54.492876    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-865f87676-4gxjk" podUID="db6c5dde-0465-4839-ae5e-0a5293b68cec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 18 17:42:55.554252 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:42:55.554199    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n6fd9"
Sep 18 17:43:02.504066 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:02.504033    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n6fd9"]
Sep 18 17:43:02.504513 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:02.504312    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n6fd9" podUID="3eee5273-1f28-448e-b4dd-2d7451c4aa35" containerName="kserve-container" containerID="cri-o://f8e9db5e1967eda6cab2072c27e401568bdb8946d814f176403cba5f49ef35fd" gracePeriod=30
Sep 18 17:43:02.504513 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:02.504349    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n6fd9" podUID="3eee5273-1f28-448e-b4dd-2d7451c4aa35" containerName="kube-rbac-proxy" containerID="cri-o://6f17659354ec62608bb9cdb409c482358f154e5b1ba2fa595b1c9ef133d91bdc" gracePeriod=30
Sep 18 17:43:02.649467 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:02.649433    2881 generic.go:356] "Generic (PLEG): container finished" podID="3eee5273-1f28-448e-b4dd-2d7451c4aa35" containerID="6f17659354ec62608bb9cdb409c482358f154e5b1ba2fa595b1c9ef133d91bdc" exitCode=2
Sep 18 17:43:02.649630 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:02.649508    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n6fd9" event={"ID":"3eee5273-1f28-448e-b4dd-2d7451c4aa35","Type":"ContainerDied","Data":"6f17659354ec62608bb9cdb409c482358f154e5b1ba2fa595b1c9ef133d91bdc"}
Sep 18 17:43:03.355506 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:03.355481    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n6fd9"
Sep 18 17:43:03.397391 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:03.397311    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x6vm5\" (UniqueName: \"kubernetes.io/projected/3eee5273-1f28-448e-b4dd-2d7451c4aa35-kube-api-access-x6vm5\") pod \"3eee5273-1f28-448e-b4dd-2d7451c4aa35\" (UID: \"3eee5273-1f28-448e-b4dd-2d7451c4aa35\") "
Sep 18 17:43:03.397391 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:03.397357    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3eee5273-1f28-448e-b4dd-2d7451c4aa35-cabundle-cert\") pod \"3eee5273-1f28-448e-b4dd-2d7451c4aa35\" (UID: \"3eee5273-1f28-448e-b4dd-2d7451c4aa35\") "
Sep 18 17:43:03.397597 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:03.397403    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3eee5273-1f28-448e-b4dd-2d7451c4aa35-kserve-provision-location\") pod \"3eee5273-1f28-448e-b4dd-2d7451c4aa35\" (UID: \"3eee5273-1f28-448e-b4dd-2d7451c4aa35\") "
Sep 18 17:43:03.397597 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:03.397452    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3eee5273-1f28-448e-b4dd-2d7451c4aa35-kube-rbac-proxy-sar-config\") pod \"3eee5273-1f28-448e-b4dd-2d7451c4aa35\" (UID: \"3eee5273-1f28-448e-b4dd-2d7451c4aa35\") "
Sep 18 17:43:03.397597 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:03.397512    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3eee5273-1f28-448e-b4dd-2d7451c4aa35-proxy-tls\") pod \"3eee5273-1f28-448e-b4dd-2d7451c4aa35\" (UID: \"3eee5273-1f28-448e-b4dd-2d7451c4aa35\") "
Sep 18 17:43:03.397823 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:03.397799    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eee5273-1f28-448e-b4dd-2d7451c4aa35-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3eee5273-1f28-448e-b4dd-2d7451c4aa35" (UID: "3eee5273-1f28-448e-b4dd-2d7451c4aa35"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:43:03.397898 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:03.397798    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eee5273-1f28-448e-b4dd-2d7451c4aa35-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3eee5273-1f28-448e-b4dd-2d7451c4aa35" (UID: "3eee5273-1f28-448e-b4dd-2d7451c4aa35"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:43:03.397898 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:03.397798    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eee5273-1f28-448e-b4dd-2d7451c4aa35-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3eee5273-1f28-448e-b4dd-2d7451c4aa35" (UID: "3eee5273-1f28-448e-b4dd-2d7451c4aa35"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:43:03.399607 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:03.399579    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eee5273-1f28-448e-b4dd-2d7451c4aa35-kube-api-access-x6vm5" (OuterVolumeSpecName: "kube-api-access-x6vm5") pod "3eee5273-1f28-448e-b4dd-2d7451c4aa35" (UID: "3eee5273-1f28-448e-b4dd-2d7451c4aa35"). InnerVolumeSpecName "kube-api-access-x6vm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:43:03.399686 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:03.399587    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3eee5273-1f28-448e-b4dd-2d7451c4aa35-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3eee5273-1f28-448e-b4dd-2d7451c4aa35" (UID: "3eee5273-1f28-448e-b4dd-2d7451c4aa35"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:43:03.498238 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:03.498191    2881 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3eee5273-1f28-448e-b4dd-2d7451c4aa35-kserve-provision-location\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:43:03.498238 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:03.498244    2881 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3eee5273-1f28-448e-b4dd-2d7451c4aa35-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:43:03.498448 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:03.498254    2881 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3eee5273-1f28-448e-b4dd-2d7451c4aa35-proxy-tls\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:43:03.498448 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:03.498263    2881 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6vm5\" (UniqueName: \"kubernetes.io/projected/3eee5273-1f28-448e-b4dd-2d7451c4aa35-kube-api-access-x6vm5\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:43:03.498448 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:03.498276    2881 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3eee5273-1f28-448e-b4dd-2d7451c4aa35-cabundle-cert\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:43:03.656297 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:03.656187    2881 generic.go:356] "Generic (PLEG): container finished" podID="3eee5273-1f28-448e-b4dd-2d7451c4aa35" containerID="f8e9db5e1967eda6cab2072c27e401568bdb8946d814f176403cba5f49ef35fd" exitCode=0
Sep 18 17:43:03.656297 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:03.656268    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n6fd9"
Sep 18 17:43:03.656815 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:03.656263    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n6fd9" event={"ID":"3eee5273-1f28-448e-b4dd-2d7451c4aa35","Type":"ContainerDied","Data":"f8e9db5e1967eda6cab2072c27e401568bdb8946d814f176403cba5f49ef35fd"}
Sep 18 17:43:03.656815 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:03.656408    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n6fd9" event={"ID":"3eee5273-1f28-448e-b4dd-2d7451c4aa35","Type":"ContainerDied","Data":"58d7658169da596a425ec0a80b5241aed2f048d9c986521ba6605e04b86c79a8"}
Sep 18 17:43:03.656815 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:03.656435    2881 scope.go:117] "RemoveContainer" containerID="6f17659354ec62608bb9cdb409c482358f154e5b1ba2fa595b1c9ef133d91bdc"
Sep 18 17:43:03.665105 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:03.665081    2881 scope.go:117] "RemoveContainer" containerID="f8e9db5e1967eda6cab2072c27e401568bdb8946d814f176403cba5f49ef35fd"
Sep 18 17:43:03.673369 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:03.673350    2881 scope.go:117] "RemoveContainer" containerID="46f3c2306f8c0c7761eaa9956f89c6405f841e6f1957d7c45899e84be9636a76"
Sep 18 17:43:03.681976 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:03.681956    2881 scope.go:117] "RemoveContainer" containerID="6f17659354ec62608bb9cdb409c482358f154e5b1ba2fa595b1c9ef133d91bdc"
Sep 18 17:43:03.682311 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:43:03.682290    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f17659354ec62608bb9cdb409c482358f154e5b1ba2fa595b1c9ef133d91bdc\": container with ID starting with 6f17659354ec62608bb9cdb409c482358f154e5b1ba2fa595b1c9ef133d91bdc not found: ID does not exist" containerID="6f17659354ec62608bb9cdb409c482358f154e5b1ba2fa595b1c9ef133d91bdc"
Sep 18 17:43:03.682373 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:03.682324    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f17659354ec62608bb9cdb409c482358f154e5b1ba2fa595b1c9ef133d91bdc"} err="failed to get container status \"6f17659354ec62608bb9cdb409c482358f154e5b1ba2fa595b1c9ef133d91bdc\": rpc error: code = NotFound desc = could not find container \"6f17659354ec62608bb9cdb409c482358f154e5b1ba2fa595b1c9ef133d91bdc\": container with ID starting with 6f17659354ec62608bb9cdb409c482358f154e5b1ba2fa595b1c9ef133d91bdc not found: ID does not exist"
Sep 18 17:43:03.682373 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:03.682345    2881 scope.go:117] "RemoveContainer" containerID="f8e9db5e1967eda6cab2072c27e401568bdb8946d814f176403cba5f49ef35fd"
Sep 18 17:43:03.682653 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:43:03.682634    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8e9db5e1967eda6cab2072c27e401568bdb8946d814f176403cba5f49ef35fd\": container with ID starting with f8e9db5e1967eda6cab2072c27e401568bdb8946d814f176403cba5f49ef35fd not found: ID does not exist" containerID="f8e9db5e1967eda6cab2072c27e401568bdb8946d814f176403cba5f49ef35fd"
Sep 18 17:43:03.682695 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:03.682666    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8e9db5e1967eda6cab2072c27e401568bdb8946d814f176403cba5f49ef35fd"} err="failed to get container status \"f8e9db5e1967eda6cab2072c27e401568bdb8946d814f176403cba5f49ef35fd\": rpc error: code = NotFound desc = could not find container \"f8e9db5e1967eda6cab2072c27e401568bdb8946d814f176403cba5f49ef35fd\": container with ID starting with f8e9db5e1967eda6cab2072c27e401568bdb8946d814f176403cba5f49ef35fd not found: ID does not exist"
Sep 18 17:43:03.682695 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:03.682686    2881 scope.go:117] "RemoveContainer" containerID="46f3c2306f8c0c7761eaa9956f89c6405f841e6f1957d7c45899e84be9636a76"
Sep 18 17:43:03.682954 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:43:03.682937    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46f3c2306f8c0c7761eaa9956f89c6405f841e6f1957d7c45899e84be9636a76\": container with ID starting with 46f3c2306f8c0c7761eaa9956f89c6405f841e6f1957d7c45899e84be9636a76 not found: ID does not exist" containerID="46f3c2306f8c0c7761eaa9956f89c6405f841e6f1957d7c45899e84be9636a76"
Sep 18 17:43:03.683015 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:03.682957    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46f3c2306f8c0c7761eaa9956f89c6405f841e6f1957d7c45899e84be9636a76"} err="failed to get container status \"46f3c2306f8c0c7761eaa9956f89c6405f841e6f1957d7c45899e84be9636a76\": rpc error: code = NotFound desc = could not find container \"46f3c2306f8c0c7761eaa9956f89c6405f841e6f1957d7c45899e84be9636a76\": container with ID starting with 46f3c2306f8c0c7761eaa9956f89c6405f841e6f1957d7c45899e84be9636a76 not found: ID does not exist"
Sep 18 17:43:03.683099 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:03.683081    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n6fd9"]
Sep 18 17:43:03.687623 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:03.687598    2881 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-n6fd9"]
Sep 18 17:43:03.893005 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:03.892970    2881 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3eee5273-1f28-448e-b4dd-2d7451c4aa35" path="/var/lib/kubelet/pods/3eee5273-1f28-448e-b4dd-2d7451c4aa35/volumes"
Sep 18 17:43:04.492873 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:04.492838    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-865f87676-4gxjk" podUID="db6c5dde-0465-4839-ae5e-0a5293b68cec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 18 17:43:07.719481 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:07.719439    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jzrln"]
Sep 18 17:43:07.719880 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:07.719836    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3eee5273-1f28-448e-b4dd-2d7451c4aa35" containerName="storage-initializer"
Sep 18 17:43:07.719880 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:07.719849    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="3eee5273-1f28-448e-b4dd-2d7451c4aa35" containerName="storage-initializer"
Sep 18 17:43:07.719880 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:07.719856    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3eee5273-1f28-448e-b4dd-2d7451c4aa35" containerName="kserve-container"
Sep 18 17:43:07.719880 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:07.719869    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="3eee5273-1f28-448e-b4dd-2d7451c4aa35" containerName="kserve-container"
Sep 18 17:43:07.720017 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:07.719904    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3eee5273-1f28-448e-b4dd-2d7451c4aa35" containerName="kube-rbac-proxy"
Sep 18 17:43:07.720017 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:07.719910    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="3eee5273-1f28-448e-b4dd-2d7451c4aa35" containerName="kube-rbac-proxy"
Sep 18 17:43:07.720017 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:07.719967    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="3eee5273-1f28-448e-b4dd-2d7451c4aa35" containerName="kserve-container"
Sep 18 17:43:07.720017 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:07.719976    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="3eee5273-1f28-448e-b4dd-2d7451c4aa35" containerName="kube-rbac-proxy"
Sep 18 17:43:07.724741 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:07.724725    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jzrln"
Sep 18 17:43:07.728058 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:07.728035    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 17:43:07.728273 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:07.728256    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 18 17:43:07.728329 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:07.728293    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-predictor-serving-cert\""
Sep 18 17:43:07.728377 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:07.728301    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 17:43:07.736272 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:07.736239    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae35c019-31ea-40b2-ad88-fd9778afd203-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-jzrln\" (UID: \"ae35c019-31ea-40b2-ad88-fd9778afd203\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jzrln"
Sep 18 17:43:07.736411 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:07.736323    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae35c019-31ea-40b2-ad88-fd9778afd203-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-jzrln\" (UID: \"ae35c019-31ea-40b2-ad88-fd9778afd203\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jzrln"
Sep 18 17:43:07.736411 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:07.736343    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jzrln"]
Sep 18 17:43:07.736411 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:07.736355    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae35c019-31ea-40b2-ad88-fd9778afd203-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-jzrln\" (UID: \"ae35c019-31ea-40b2-ad88-fd9778afd203\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jzrln"
Sep 18 17:43:07.736594 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:07.736413    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzq7c\" (UniqueName: \"kubernetes.io/projected/ae35c019-31ea-40b2-ad88-fd9778afd203-kube-api-access-wzq7c\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-jzrln\" (UID: \"ae35c019-31ea-40b2-ad88-fd9778afd203\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jzrln"
Sep 18 17:43:07.736594 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:07.736483    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae35c019-31ea-40b2-ad88-fd9778afd203-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-jzrln\" (UID: \"ae35c019-31ea-40b2-ad88-fd9778afd203\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jzrln"
Sep 18 17:43:07.837332 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:07.837294    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae35c019-31ea-40b2-ad88-fd9778afd203-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-jzrln\" (UID: \"ae35c019-31ea-40b2-ad88-fd9778afd203\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jzrln"
Sep 18 17:43:07.837332 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:07.837341    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae35c019-31ea-40b2-ad88-fd9778afd203-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-jzrln\" (UID: \"ae35c019-31ea-40b2-ad88-fd9778afd203\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jzrln"
Sep 18 17:43:07.837604 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:07.837360    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae35c019-31ea-40b2-ad88-fd9778afd203-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-jzrln\" (UID: \"ae35c019-31ea-40b2-ad88-fd9778afd203\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jzrln"
Sep 18 17:43:07.837604 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:07.837393    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzq7c\" (UniqueName: \"kubernetes.io/projected/ae35c019-31ea-40b2-ad88-fd9778afd203-kube-api-access-wzq7c\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-jzrln\" (UID: \"ae35c019-31ea-40b2-ad88-fd9778afd203\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jzrln"
Sep 18 17:43:07.837604 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:07.837443    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae35c019-31ea-40b2-ad88-fd9778afd203-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-jzrln\" (UID: \"ae35c019-31ea-40b2-ad88-fd9778afd203\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jzrln"
Sep 18 17:43:07.837901 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:07.837875    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae35c019-31ea-40b2-ad88-fd9778afd203-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-jzrln\" (UID: \"ae35c019-31ea-40b2-ad88-fd9778afd203\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jzrln"
Sep 18 17:43:07.838098 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:07.838081    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae35c019-31ea-40b2-ad88-fd9778afd203-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-jzrln\" (UID: \"ae35c019-31ea-40b2-ad88-fd9778afd203\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jzrln"
Sep 18 17:43:07.838165 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:07.838082    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae35c019-31ea-40b2-ad88-fd9778afd203-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-jzrln\" (UID: \"ae35c019-31ea-40b2-ad88-fd9778afd203\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jzrln"
Sep 18 17:43:07.839959 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:07.839933    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae35c019-31ea-40b2-ad88-fd9778afd203-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-jzrln\" (UID: \"ae35c019-31ea-40b2-ad88-fd9778afd203\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jzrln"
Sep 18 17:43:07.847771 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:07.847748    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzq7c\" (UniqueName: \"kubernetes.io/projected/ae35c019-31ea-40b2-ad88-fd9778afd203-kube-api-access-wzq7c\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-jzrln\" (UID: \"ae35c019-31ea-40b2-ad88-fd9778afd203\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jzrln"
Sep 18 17:43:08.036978 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:08.036937    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jzrln"
Sep 18 17:43:08.317737 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:08.317707    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jzrln"]
Sep 18 17:43:08.320303 ip-10-0-143-5 kubenswrapper[2881]: W0918 17:43:08.320261    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae35c019_31ea_40b2_ad88_fd9778afd203.slice/crio-d2a9a3d8e47ef379ad8a0be9e169df6c67b90cff601863e164dd5da11d4626e5 WatchSource:0}: Error finding container d2a9a3d8e47ef379ad8a0be9e169df6c67b90cff601863e164dd5da11d4626e5: Status 404 returned error can't find the container with id d2a9a3d8e47ef379ad8a0be9e169df6c67b90cff601863e164dd5da11d4626e5
Sep 18 17:43:08.680816 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:08.680725    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jzrln" event={"ID":"ae35c019-31ea-40b2-ad88-fd9778afd203","Type":"ContainerStarted","Data":"a6695bdb97e4f3abd7d2945eafec91afa0e5f265cb09803a3b5278a3c7f587fa"}
Sep 18 17:43:08.680816 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:08.680786    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jzrln" event={"ID":"ae35c019-31ea-40b2-ad88-fd9778afd203","Type":"ContainerStarted","Data":"d2a9a3d8e47ef379ad8a0be9e169df6c67b90cff601863e164dd5da11d4626e5"}
Sep 18 17:43:12.700275 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:12.700162    2881 generic.go:356] "Generic (PLEG): container finished" podID="ae35c019-31ea-40b2-ad88-fd9778afd203" containerID="a6695bdb97e4f3abd7d2945eafec91afa0e5f265cb09803a3b5278a3c7f587fa" exitCode=0
Sep 18 17:43:12.700275 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:12.700259    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jzrln" event={"ID":"ae35c019-31ea-40b2-ad88-fd9778afd203","Type":"ContainerDied","Data":"a6695bdb97e4f3abd7d2945eafec91afa0e5f265cb09803a3b5278a3c7f587fa"}
Sep 18 17:43:13.707716 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:13.707682    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jzrln" event={"ID":"ae35c019-31ea-40b2-ad88-fd9778afd203","Type":"ContainerStarted","Data":"fef16eace596a3f179761f059820486e8ee72f8c1a8836449a07c8c40ce33643"}
Sep 18 17:43:13.708129 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:13.707746    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jzrln" event={"ID":"ae35c019-31ea-40b2-ad88-fd9778afd203","Type":"ContainerStarted","Data":"3d6fcf8ba756ac48f11cb0160de1de497a19347b86db8d133fe6f7ae2051b474"}
Sep 18 17:43:13.708129 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:13.707958    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jzrln"
Sep 18 17:43:13.708129 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:13.708115    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jzrln"
Sep 18 17:43:14.493324 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:14.493284    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-865f87676-4gxjk" podUID="db6c5dde-0465-4839-ae5e-0a5293b68cec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 18 17:43:19.718336 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:19.718307    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jzrln"
Sep 18 17:43:19.741786 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:19.741732    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jzrln" podStartSLOduration=8.804109177 podStartE2EDuration="12.741717761s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.937608584s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:43:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:43:13.745848937 +0000 UTC m=+2983.945704239" watchObservedRunningTime="2026-09-18 17:43:19.741717761 +0000 UTC m=+2989.941573047"
Sep 18 17:43:24.493720 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:24.493676    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-865f87676-4gxjk" podUID="db6c5dde-0465-4839-ae5e-0a5293b68cec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 18 17:43:34.493630 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:34.493589    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-865f87676-4gxjk" podUID="db6c5dde-0465-4839-ae5e-0a5293b68cec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 18 17:43:44.493349 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:44.493305    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-865f87676-4gxjk" podUID="db6c5dde-0465-4839-ae5e-0a5293b68cec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 18 17:43:49.722351 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:49.722313    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jzrln"
Sep 18 17:43:51.893521 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:51.893489    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-865f87676-4gxjk"
Sep 18 17:43:57.811702 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:57.811667    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jzrln"]
Sep 18 17:43:57.812180 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:57.811907    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jzrln" podUID="ae35c019-31ea-40b2-ad88-fd9778afd203" containerName="kserve-container" containerID="cri-o://3d6fcf8ba756ac48f11cb0160de1de497a19347b86db8d133fe6f7ae2051b474" gracePeriod=30
Sep 18 17:43:57.812180 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:57.811982    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jzrln" podUID="ae35c019-31ea-40b2-ad88-fd9778afd203" containerName="kube-rbac-proxy" containerID="cri-o://fef16eace596a3f179761f059820486e8ee72f8c1a8836449a07c8c40ce33643" gracePeriod=30
Sep 18 17:43:58.909172 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:58.909138    2881 generic.go:356] "Generic (PLEG): container finished" podID="ae35c019-31ea-40b2-ad88-fd9778afd203" containerID="fef16eace596a3f179761f059820486e8ee72f8c1a8836449a07c8c40ce33643" exitCode=2
Sep 18 17:43:58.909668 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:58.909215    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jzrln" event={"ID":"ae35c019-31ea-40b2-ad88-fd9778afd203","Type":"ContainerDied","Data":"fef16eace596a3f179761f059820486e8ee72f8c1a8836449a07c8c40ce33643"}
Sep 18 17:43:59.161602 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:59.161532    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jzrln"
Sep 18 17:43:59.197509 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:59.197479    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wzq7c\" (UniqueName: \"kubernetes.io/projected/ae35c019-31ea-40b2-ad88-fd9778afd203-kube-api-access-wzq7c\") pod \"ae35c019-31ea-40b2-ad88-fd9778afd203\" (UID: \"ae35c019-31ea-40b2-ad88-fd9778afd203\") "
Sep 18 17:43:59.197697 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:59.197545    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae35c019-31ea-40b2-ad88-fd9778afd203-cabundle-cert\") pod \"ae35c019-31ea-40b2-ad88-fd9778afd203\" (UID: \"ae35c019-31ea-40b2-ad88-fd9778afd203\") "
Sep 18 17:43:59.197697 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:59.197596    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae35c019-31ea-40b2-ad88-fd9778afd203-kserve-provision-location\") pod \"ae35c019-31ea-40b2-ad88-fd9778afd203\" (UID: \"ae35c019-31ea-40b2-ad88-fd9778afd203\") "
Sep 18 17:43:59.197697 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:59.197669    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae35c019-31ea-40b2-ad88-fd9778afd203-proxy-tls\") pod \"ae35c019-31ea-40b2-ad88-fd9778afd203\" (UID: \"ae35c019-31ea-40b2-ad88-fd9778afd203\") "
Sep 18 17:43:59.197860 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:59.197711    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae35c019-31ea-40b2-ad88-fd9778afd203-kube-rbac-proxy-sar-config\") pod \"ae35c019-31ea-40b2-ad88-fd9778afd203\" (UID: \"ae35c019-31ea-40b2-ad88-fd9778afd203\") "
Sep 18 17:43:59.197949 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:59.197920    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae35c019-31ea-40b2-ad88-fd9778afd203-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ae35c019-31ea-40b2-ad88-fd9778afd203" (UID: "ae35c019-31ea-40b2-ad88-fd9778afd203"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:43:59.198100 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:59.197960    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae35c019-31ea-40b2-ad88-fd9778afd203-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ae35c019-31ea-40b2-ad88-fd9778afd203" (UID: "ae35c019-31ea-40b2-ad88-fd9778afd203"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:43:59.198341 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:59.198316    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae35c019-31ea-40b2-ad88-fd9778afd203-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ae35c019-31ea-40b2-ad88-fd9778afd203" (UID: "ae35c019-31ea-40b2-ad88-fd9778afd203"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:43:59.199731 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:59.199704    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae35c019-31ea-40b2-ad88-fd9778afd203-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ae35c019-31ea-40b2-ad88-fd9778afd203" (UID: "ae35c019-31ea-40b2-ad88-fd9778afd203"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:43:59.200040 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:59.199789    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae35c019-31ea-40b2-ad88-fd9778afd203-kube-api-access-wzq7c" (OuterVolumeSpecName: "kube-api-access-wzq7c") pod "ae35c019-31ea-40b2-ad88-fd9778afd203" (UID: "ae35c019-31ea-40b2-ad88-fd9778afd203"). InnerVolumeSpecName "kube-api-access-wzq7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:43:59.298628 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:59.298592    2881 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae35c019-31ea-40b2-ad88-fd9778afd203-proxy-tls\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:43:59.298628 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:59.298621    2881 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae35c019-31ea-40b2-ad88-fd9778afd203-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:43:59.298842 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:59.298641    2881 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzq7c\" (UniqueName: \"kubernetes.io/projected/ae35c019-31ea-40b2-ad88-fd9778afd203-kube-api-access-wzq7c\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:43:59.298842 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:59.298650    2881 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae35c019-31ea-40b2-ad88-fd9778afd203-cabundle-cert\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:43:59.298842 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:59.298659    2881 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae35c019-31ea-40b2-ad88-fd9778afd203-kserve-provision-location\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:43:59.416378 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:59.416285    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-865f87676-4gxjk"]
Sep 18 17:43:59.416591 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:59.416560    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-865f87676-4gxjk" podUID="db6c5dde-0465-4839-ae5e-0a5293b68cec" containerName="kserve-container" containerID="cri-o://185a40d013d91647cbe6c37a718f811bd5990f30dad004d4c2aa514dc87e54f8" gracePeriod=30
Sep 18 17:43:59.416591 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:59.416584    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-865f87676-4gxjk" podUID="db6c5dde-0465-4839-ae5e-0a5293b68cec" containerName="kube-rbac-proxy" containerID="cri-o://3af5885b8ebd89cf7db9de38765ddf5d1fc43765015b2e27014f11deafbc6fba" gracePeriod=30
Sep 18 17:43:59.488576 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:59.488535    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-865f87676-4gxjk" podUID="db6c5dde-0465-4839-ae5e-0a5293b68cec" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.59:8643/healthz\": dial tcp 10.134.0.59:8643: connect: connection refused"
Sep 18 17:43:59.570989 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:59.570945    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8"]
Sep 18 17:43:59.571410 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:59.571392    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae35c019-31ea-40b2-ad88-fd9778afd203" containerName="kserve-container"
Sep 18 17:43:59.571513 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:59.571413    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae35c019-31ea-40b2-ad88-fd9778afd203" containerName="kserve-container"
Sep 18 17:43:59.571513 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:59.571439    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae35c019-31ea-40b2-ad88-fd9778afd203" containerName="kube-rbac-proxy"
Sep 18 17:43:59.571513 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:59.571448    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae35c019-31ea-40b2-ad88-fd9778afd203" containerName="kube-rbac-proxy"
Sep 18 17:43:59.571513 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:59.571473    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae35c019-31ea-40b2-ad88-fd9778afd203" containerName="storage-initializer"
Sep 18 17:43:59.571513 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:59.571482    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae35c019-31ea-40b2-ad88-fd9778afd203" containerName="storage-initializer"
Sep 18 17:43:59.571774 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:59.571582    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae35c019-31ea-40b2-ad88-fd9778afd203" containerName="kserve-container"
Sep 18 17:43:59.571774 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:59.571598    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae35c019-31ea-40b2-ad88-fd9778afd203" containerName="kube-rbac-proxy"
Sep 18 17:43:59.574758 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:59.574735    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8"
Sep 18 17:43:59.581003 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:59.580978    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\""
Sep 18 17:43:59.581134 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:59.580977    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-predictor-serving-cert\""
Sep 18 17:43:59.590647 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:59.590620    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8"]
Sep 18 17:43:59.602012 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:59.601979    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4fd1fe5e-7044-4283-aa43-495bce3a18dc-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8\" (UID: \"4fd1fe5e-7044-4283-aa43-495bce3a18dc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8"
Sep 18 17:43:59.602189 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:59.602042    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4fd1fe5e-7044-4283-aa43-495bce3a18dc-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8\" (UID: \"4fd1fe5e-7044-4283-aa43-495bce3a18dc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8"
Sep 18 17:43:59.602189 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:59.602104    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6pkr\" (UniqueName: \"kubernetes.io/projected/4fd1fe5e-7044-4283-aa43-495bce3a18dc-kube-api-access-c6pkr\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8\" (UID: \"4fd1fe5e-7044-4283-aa43-495bce3a18dc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8"
Sep 18 17:43:59.602189 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:59.602134    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4fd1fe5e-7044-4283-aa43-495bce3a18dc-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8\" (UID: \"4fd1fe5e-7044-4283-aa43-495bce3a18dc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8"
Sep 18 17:43:59.602338 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:59.602211    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4fd1fe5e-7044-4283-aa43-495bce3a18dc-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8\" (UID: \"4fd1fe5e-7044-4283-aa43-495bce3a18dc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8"
Sep 18 17:43:59.703262 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:59.703132    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6pkr\" (UniqueName: \"kubernetes.io/projected/4fd1fe5e-7044-4283-aa43-495bce3a18dc-kube-api-access-c6pkr\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8\" (UID: \"4fd1fe5e-7044-4283-aa43-495bce3a18dc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8"
Sep 18 17:43:59.703262 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:59.703186    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4fd1fe5e-7044-4283-aa43-495bce3a18dc-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8\" (UID: \"4fd1fe5e-7044-4283-aa43-495bce3a18dc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8"
Sep 18 17:43:59.703262 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:59.703250    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4fd1fe5e-7044-4283-aa43-495bce3a18dc-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8\" (UID: \"4fd1fe5e-7044-4283-aa43-495bce3a18dc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8"
Sep 18 17:43:59.703536 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:59.703299    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4fd1fe5e-7044-4283-aa43-495bce3a18dc-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8\" (UID: \"4fd1fe5e-7044-4283-aa43-495bce3a18dc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8"
Sep 18 17:43:59.703536 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:59.703320    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4fd1fe5e-7044-4283-aa43-495bce3a18dc-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8\" (UID: \"4fd1fe5e-7044-4283-aa43-495bce3a18dc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8"
Sep 18 17:43:59.703536 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:43:59.703335    2881 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-global-pass-predictor-serving-cert" not found
Sep 18 17:43:59.703536 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:43:59.703407    2881 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fd1fe5e-7044-4283-aa43-495bce3a18dc-proxy-tls podName:4fd1fe5e-7044-4283-aa43-495bce3a18dc nodeName:}" failed. No retries permitted until 2026-09-18 17:44:00.203386382 +0000 UTC m=+3030.403241660 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4fd1fe5e-7044-4283-aa43-495bce3a18dc-proxy-tls") pod "isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8" (UID: "4fd1fe5e-7044-4283-aa43-495bce3a18dc") : secret "isvc-sklearn-s3-tls-global-pass-predictor-serving-cert" not found
Sep 18 17:43:59.703799 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:59.703778    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4fd1fe5e-7044-4283-aa43-495bce3a18dc-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8\" (UID: \"4fd1fe5e-7044-4283-aa43-495bce3a18dc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8"
Sep 18 17:43:59.704048 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:59.704032    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4fd1fe5e-7044-4283-aa43-495bce3a18dc-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8\" (UID: \"4fd1fe5e-7044-4283-aa43-495bce3a18dc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8"
Sep 18 17:43:59.704048 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:59.704039    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4fd1fe5e-7044-4283-aa43-495bce3a18dc-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8\" (UID: \"4fd1fe5e-7044-4283-aa43-495bce3a18dc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8"
Sep 18 17:43:59.714672 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:59.714635    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6pkr\" (UniqueName: \"kubernetes.io/projected/4fd1fe5e-7044-4283-aa43-495bce3a18dc-kube-api-access-c6pkr\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8\" (UID: \"4fd1fe5e-7044-4283-aa43-495bce3a18dc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8"
Sep 18 17:43:59.915388 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:59.915352    2881 generic.go:356] "Generic (PLEG): container finished" podID="ae35c019-31ea-40b2-ad88-fd9778afd203" containerID="3d6fcf8ba756ac48f11cb0160de1de497a19347b86db8d133fe6f7ae2051b474" exitCode=0
Sep 18 17:43:59.915787 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:59.915427    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jzrln"
Sep 18 17:43:59.915787 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:59.915437    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jzrln" event={"ID":"ae35c019-31ea-40b2-ad88-fd9778afd203","Type":"ContainerDied","Data":"3d6fcf8ba756ac48f11cb0160de1de497a19347b86db8d133fe6f7ae2051b474"}
Sep 18 17:43:59.915787 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:59.915497    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jzrln" event={"ID":"ae35c019-31ea-40b2-ad88-fd9778afd203","Type":"ContainerDied","Data":"d2a9a3d8e47ef379ad8a0be9e169df6c67b90cff601863e164dd5da11d4626e5"}
Sep 18 17:43:59.915787 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:59.915528    2881 scope.go:117] "RemoveContainer" containerID="fef16eace596a3f179761f059820486e8ee72f8c1a8836449a07c8c40ce33643"
Sep 18 17:43:59.921127 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:59.921102    2881 generic.go:356] "Generic (PLEG): container finished" podID="db6c5dde-0465-4839-ae5e-0a5293b68cec" containerID="3af5885b8ebd89cf7db9de38765ddf5d1fc43765015b2e27014f11deafbc6fba" exitCode=2
Sep 18 17:43:59.921288 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:59.921150    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-865f87676-4gxjk" event={"ID":"db6c5dde-0465-4839-ae5e-0a5293b68cec","Type":"ContainerDied","Data":"3af5885b8ebd89cf7db9de38765ddf5d1fc43765015b2e27014f11deafbc6fba"}
Sep 18 17:43:59.926959 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:59.926930    2881 scope.go:117] "RemoveContainer" containerID="3d6fcf8ba756ac48f11cb0160de1de497a19347b86db8d133fe6f7ae2051b474"
Sep 18 17:43:59.935315 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:59.935293    2881 scope.go:117] "RemoveContainer" containerID="a6695bdb97e4f3abd7d2945eafec91afa0e5f265cb09803a3b5278a3c7f587fa"
Sep 18 17:43:59.936888 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:59.936866    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jzrln"]
Sep 18 17:43:59.943192 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:59.943169    2881 scope.go:117] "RemoveContainer" containerID="fef16eace596a3f179761f059820486e8ee72f8c1a8836449a07c8c40ce33643"
Sep 18 17:43:59.943509 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:43:59.943489    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fef16eace596a3f179761f059820486e8ee72f8c1a8836449a07c8c40ce33643\": container with ID starting with fef16eace596a3f179761f059820486e8ee72f8c1a8836449a07c8c40ce33643 not found: ID does not exist" containerID="fef16eace596a3f179761f059820486e8ee72f8c1a8836449a07c8c40ce33643"
Sep 18 17:43:59.943577 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:59.943517    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fef16eace596a3f179761f059820486e8ee72f8c1a8836449a07c8c40ce33643"} err="failed to get container status \"fef16eace596a3f179761f059820486e8ee72f8c1a8836449a07c8c40ce33643\": rpc error: code = NotFound desc = could not find container \"fef16eace596a3f179761f059820486e8ee72f8c1a8836449a07c8c40ce33643\": container with ID starting with fef16eace596a3f179761f059820486e8ee72f8c1a8836449a07c8c40ce33643 not found: ID does not exist"
Sep 18 17:43:59.943577 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:59.943535    2881 scope.go:117] "RemoveContainer" containerID="3d6fcf8ba756ac48f11cb0160de1de497a19347b86db8d133fe6f7ae2051b474"
Sep 18 17:43:59.943817 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:43:59.943800    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d6fcf8ba756ac48f11cb0160de1de497a19347b86db8d133fe6f7ae2051b474\": container with ID starting with 3d6fcf8ba756ac48f11cb0160de1de497a19347b86db8d133fe6f7ae2051b474 not found: ID does not exist" containerID="3d6fcf8ba756ac48f11cb0160de1de497a19347b86db8d133fe6f7ae2051b474"
Sep 18 17:43:59.943874 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:59.943830    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d6fcf8ba756ac48f11cb0160de1de497a19347b86db8d133fe6f7ae2051b474"} err="failed to get container status \"3d6fcf8ba756ac48f11cb0160de1de497a19347b86db8d133fe6f7ae2051b474\": rpc error: code = NotFound desc = could not find container \"3d6fcf8ba756ac48f11cb0160de1de497a19347b86db8d133fe6f7ae2051b474\": container with ID starting with 3d6fcf8ba756ac48f11cb0160de1de497a19347b86db8d133fe6f7ae2051b474 not found: ID does not exist"
Sep 18 17:43:59.943874 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:59.943847    2881 scope.go:117] "RemoveContainer" containerID="a6695bdb97e4f3abd7d2945eafec91afa0e5f265cb09803a3b5278a3c7f587fa"
Sep 18 17:43:59.944107 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:43:59.944090    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6695bdb97e4f3abd7d2945eafec91afa0e5f265cb09803a3b5278a3c7f587fa\": container with ID starting with a6695bdb97e4f3abd7d2945eafec91afa0e5f265cb09803a3b5278a3c7f587fa not found: ID does not exist" containerID="a6695bdb97e4f3abd7d2945eafec91afa0e5f265cb09803a3b5278a3c7f587fa"
Sep 18 17:43:59.944165 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:59.944110    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6695bdb97e4f3abd7d2945eafec91afa0e5f265cb09803a3b5278a3c7f587fa"} err="failed to get container status \"a6695bdb97e4f3abd7d2945eafec91afa0e5f265cb09803a3b5278a3c7f587fa\": rpc error: code = NotFound desc = could not find container \"a6695bdb97e4f3abd7d2945eafec91afa0e5f265cb09803a3b5278a3c7f587fa\": container with ID starting with a6695bdb97e4f3abd7d2945eafec91afa0e5f265cb09803a3b5278a3c7f587fa not found: ID does not exist"
Sep 18 17:43:59.949812 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:43:59.949790    2881 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jzrln"]
Sep 18 17:44:00.209071 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:00.209032    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4fd1fe5e-7044-4283-aa43-495bce3a18dc-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8\" (UID: \"4fd1fe5e-7044-4283-aa43-495bce3a18dc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8"
Sep 18 17:44:00.211564 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:00.211536    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4fd1fe5e-7044-4283-aa43-495bce3a18dc-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8\" (UID: \"4fd1fe5e-7044-4283-aa43-495bce3a18dc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8"
Sep 18 17:44:00.485206 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:00.485174    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8"
Sep 18 17:44:00.763835 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:00.763799    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8"]
Sep 18 17:44:00.767680 ip-10-0-143-5 kubenswrapper[2881]: W0918 17:44:00.767645    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fd1fe5e_7044_4283_aa43_495bce3a18dc.slice/crio-93555e2a23eebecdc92681bad18fab671ac2587e334efeaf46784824ecb66bab WatchSource:0}: Error finding container 93555e2a23eebecdc92681bad18fab671ac2587e334efeaf46784824ecb66bab: Status 404 returned error can't find the container with id 93555e2a23eebecdc92681bad18fab671ac2587e334efeaf46784824ecb66bab
Sep 18 17:44:00.928785 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:00.928750    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8" event={"ID":"4fd1fe5e-7044-4283-aa43-495bce3a18dc","Type":"ContainerStarted","Data":"e919135e405aba0c009eda9265a8391725ad2beec85b7eeb67bfeb1963319179"}
Sep 18 17:44:00.929145 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:00.928806    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8" event={"ID":"4fd1fe5e-7044-4283-aa43-495bce3a18dc","Type":"ContainerStarted","Data":"93555e2a23eebecdc92681bad18fab671ac2587e334efeaf46784824ecb66bab"}
Sep 18 17:44:01.888598 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:01.888552    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-865f87676-4gxjk" podUID="db6c5dde-0465-4839-ae5e-0a5293b68cec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 18 17:44:01.893849 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:01.893822    2881 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae35c019-31ea-40b2-ad88-fd9778afd203" path="/var/lib/kubelet/pods/ae35c019-31ea-40b2-ad88-fd9778afd203/volumes"
Sep 18 17:44:02.940477 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:02.940446    2881 generic.go:356] "Generic (PLEG): container finished" podID="4fd1fe5e-7044-4283-aa43-495bce3a18dc" containerID="e919135e405aba0c009eda9265a8391725ad2beec85b7eeb67bfeb1963319179" exitCode=0
Sep 18 17:44:02.940873 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:02.940516    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8" event={"ID":"4fd1fe5e-7044-4283-aa43-495bce3a18dc","Type":"ContainerDied","Data":"e919135e405aba0c009eda9265a8391725ad2beec85b7eeb67bfeb1963319179"}
Sep 18 17:44:02.974868 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:02.974839    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2"]
Sep 18 17:44:02.979879 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:02.979852    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2"
Sep 18 17:44:02.982653 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:02.982632    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 17:44:02.982805 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:02.982652    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\""
Sep 18 17:44:02.982906 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:02.982892    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-predictor-serving-cert\""
Sep 18 17:44:02.991394 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:02.991366    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2"]
Sep 18 17:44:03.037978 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:03.037924    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bc7c7586-87e3-4ce0-9f38-909e21c2dded-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2\" (UID: \"bc7c7586-87e3-4ce0-9f38-909e21c2dded\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2"
Sep 18 17:44:03.038175 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:03.038005    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6wgw\" (UniqueName: \"kubernetes.io/projected/bc7c7586-87e3-4ce0-9f38-909e21c2dded-kube-api-access-s6wgw\") pod \"isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2\" (UID: \"bc7c7586-87e3-4ce0-9f38-909e21c2dded\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2"
Sep 18 17:44:03.038175 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:03.038033    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc7c7586-87e3-4ce0-9f38-909e21c2dded-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2\" (UID: \"bc7c7586-87e3-4ce0-9f38-909e21c2dded\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2"
Sep 18 17:44:03.038331 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:03.038163    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bc7c7586-87e3-4ce0-9f38-909e21c2dded-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2\" (UID: \"bc7c7586-87e3-4ce0-9f38-909e21c2dded\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2"
Sep 18 17:44:03.038331 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:03.038218    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc7c7586-87e3-4ce0-9f38-909e21c2dded-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2\" (UID: \"bc7c7586-87e3-4ce0-9f38-909e21c2dded\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2"
Sep 18 17:44:03.139552 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:03.139514    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bc7c7586-87e3-4ce0-9f38-909e21c2dded-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2\" (UID: \"bc7c7586-87e3-4ce0-9f38-909e21c2dded\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2"
Sep 18 17:44:03.139552 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:03.139565    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc7c7586-87e3-4ce0-9f38-909e21c2dded-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2\" (UID: \"bc7c7586-87e3-4ce0-9f38-909e21c2dded\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2"
Sep 18 17:44:03.139827 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:03.139608    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bc7c7586-87e3-4ce0-9f38-909e21c2dded-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2\" (UID: \"bc7c7586-87e3-4ce0-9f38-909e21c2dded\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2"
Sep 18 17:44:03.139827 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:03.139630    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6wgw\" (UniqueName: \"kubernetes.io/projected/bc7c7586-87e3-4ce0-9f38-909e21c2dded-kube-api-access-s6wgw\") pod \"isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2\" (UID: \"bc7c7586-87e3-4ce0-9f38-909e21c2dded\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2"
Sep 18 17:44:03.139827 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:03.139648    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc7c7586-87e3-4ce0-9f38-909e21c2dded-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2\" (UID: \"bc7c7586-87e3-4ce0-9f38-909e21c2dded\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2"
Sep 18 17:44:03.140366 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:03.140338    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bc7c7586-87e3-4ce0-9f38-909e21c2dded-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2\" (UID: \"bc7c7586-87e3-4ce0-9f38-909e21c2dded\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2"
Sep 18 17:44:03.140486 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:03.140467    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc7c7586-87e3-4ce0-9f38-909e21c2dded-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2\" (UID: \"bc7c7586-87e3-4ce0-9f38-909e21c2dded\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2"
Sep 18 17:44:03.140567 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:03.140516    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bc7c7586-87e3-4ce0-9f38-909e21c2dded-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2\" (UID: \"bc7c7586-87e3-4ce0-9f38-909e21c2dded\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2"
Sep 18 17:44:03.142362 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:03.142340    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc7c7586-87e3-4ce0-9f38-909e21c2dded-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2\" (UID: \"bc7c7586-87e3-4ce0-9f38-909e21c2dded\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2"
Sep 18 17:44:03.150438 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:03.150412    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6wgw\" (UniqueName: \"kubernetes.io/projected/bc7c7586-87e3-4ce0-9f38-909e21c2dded-kube-api-access-s6wgw\") pod \"isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2\" (UID: \"bc7c7586-87e3-4ce0-9f38-909e21c2dded\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2"
Sep 18 17:44:03.291714 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:03.291672    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2"
Sep 18 17:44:03.570390 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:03.570366    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2"]
Sep 18 17:44:03.572213 ip-10-0-143-5 kubenswrapper[2881]: W0918 17:44:03.572171    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc7c7586_87e3_4ce0_9f38_909e21c2dded.slice/crio-c1676495d36f039f9ea618b79b7940ea212d46190f1d7f9e9b32a14013680632 WatchSource:0}: Error finding container c1676495d36f039f9ea618b79b7940ea212d46190f1d7f9e9b32a14013680632: Status 404 returned error can't find the container with id c1676495d36f039f9ea618b79b7940ea212d46190f1d7f9e9b32a14013680632
Sep 18 17:44:03.947756 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:03.947654    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8" event={"ID":"4fd1fe5e-7044-4283-aa43-495bce3a18dc","Type":"ContainerStarted","Data":"e5bb9c44e8bf2d621dff4506d3603b26536cb6b5f3c77b1dd199d8bac0ec4215"}
Sep 18 17:44:03.947756 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:03.947727    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8" event={"ID":"4fd1fe5e-7044-4283-aa43-495bce3a18dc","Type":"ContainerStarted","Data":"96f8b785be11dcc5376d0328cd67e1ce6fd7d708112d324c5dabe04c2c49c8e3"}
Sep 18 17:44:03.948295 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:03.948038    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8"
Sep 18 17:44:03.948295 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:03.948183    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8"
Sep 18 17:44:03.949631 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:03.949602    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8" podUID="4fd1fe5e-7044-4283-aa43-495bce3a18dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Sep 18 17:44:03.949809 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:03.949786    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2" event={"ID":"bc7c7586-87e3-4ce0-9f38-909e21c2dded","Type":"ContainerStarted","Data":"19c33e1da02d01122da718abaa80db48833206bbdb0830417091be6eaa245b90"}
Sep 18 17:44:03.949874 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:03.949862    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2" event={"ID":"bc7c7586-87e3-4ce0-9f38-909e21c2dded","Type":"ContainerStarted","Data":"c1676495d36f039f9ea618b79b7940ea212d46190f1d7f9e9b32a14013680632"}
Sep 18 17:44:03.969438 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:03.969384    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8" podStartSLOduration=3.826856456 podStartE2EDuration="4.969370628s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.142514172s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:43:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:44:03.968279539 +0000 UTC m=+3034.168134825" watchObservedRunningTime="2026-09-18 17:44:03.969370628 +0000 UTC m=+3034.169225914"
Sep 18 17:44:04.487605 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:04.487562    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-865f87676-4gxjk" podUID="db6c5dde-0465-4839-ae5e-0a5293b68cec" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.59:8643/healthz\": dial tcp 10.134.0.59:8643: connect: connection refused"
Sep 18 17:44:04.867538 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:04.867510    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-865f87676-4gxjk"
Sep 18 17:44:04.954354 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:04.954324    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xk76q\" (UniqueName: \"kubernetes.io/projected/db6c5dde-0465-4839-ae5e-0a5293b68cec-kube-api-access-xk76q\") pod \"db6c5dde-0465-4839-ae5e-0a5293b68cec\" (UID: \"db6c5dde-0465-4839-ae5e-0a5293b68cec\") "
Sep 18 17:44:04.954790 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:04.954383    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db6c5dde-0465-4839-ae5e-0a5293b68cec-kube-rbac-proxy-sar-config\") pod \"db6c5dde-0465-4839-ae5e-0a5293b68cec\" (UID: \"db6c5dde-0465-4839-ae5e-0a5293b68cec\") "
Sep 18 17:44:04.954790 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:04.954459    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db6c5dde-0465-4839-ae5e-0a5293b68cec-kserve-provision-location\") pod \"db6c5dde-0465-4839-ae5e-0a5293b68cec\" (UID: \"db6c5dde-0465-4839-ae5e-0a5293b68cec\") "
Sep 18 17:44:04.954790 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:04.954491    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db6c5dde-0465-4839-ae5e-0a5293b68cec-proxy-tls\") pod \"db6c5dde-0465-4839-ae5e-0a5293b68cec\" (UID: \"db6c5dde-0465-4839-ae5e-0a5293b68cec\") "
Sep 18 17:44:04.954979 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:04.954821    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db6c5dde-0465-4839-ae5e-0a5293b68cec-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "db6c5dde-0465-4839-ae5e-0a5293b68cec" (UID: "db6c5dde-0465-4839-ae5e-0a5293b68cec"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:44:04.954979 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:04.954831    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db6c5dde-0465-4839-ae5e-0a5293b68cec-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "db6c5dde-0465-4839-ae5e-0a5293b68cec" (UID: "db6c5dde-0465-4839-ae5e-0a5293b68cec"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:44:04.956643 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:04.956615    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db6c5dde-0465-4839-ae5e-0a5293b68cec-kube-api-access-xk76q" (OuterVolumeSpecName: "kube-api-access-xk76q") pod "db6c5dde-0465-4839-ae5e-0a5293b68cec" (UID: "db6c5dde-0465-4839-ae5e-0a5293b68cec"). InnerVolumeSpecName "kube-api-access-xk76q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:44:04.956767 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:04.956696    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db6c5dde-0465-4839-ae5e-0a5293b68cec-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "db6c5dde-0465-4839-ae5e-0a5293b68cec" (UID: "db6c5dde-0465-4839-ae5e-0a5293b68cec"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:44:04.958309 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:04.958279    2881 generic.go:356] "Generic (PLEG): container finished" podID="db6c5dde-0465-4839-ae5e-0a5293b68cec" containerID="185a40d013d91647cbe6c37a718f811bd5990f30dad004d4c2aa514dc87e54f8" exitCode=0
Sep 18 17:44:04.958415 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:04.958347    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-865f87676-4gxjk"
Sep 18 17:44:04.958415 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:04.958355    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-865f87676-4gxjk" event={"ID":"db6c5dde-0465-4839-ae5e-0a5293b68cec","Type":"ContainerDied","Data":"185a40d013d91647cbe6c37a718f811bd5990f30dad004d4c2aa514dc87e54f8"}
Sep 18 17:44:04.958522 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:04.958422    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-865f87676-4gxjk" event={"ID":"db6c5dde-0465-4839-ae5e-0a5293b68cec","Type":"ContainerDied","Data":"6dcb959b68a25980ff5e52f8ae880d5bbb8ccf3cee234286726e6818c1a7fb46"}
Sep 18 17:44:04.958522 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:04.958449    2881 scope.go:117] "RemoveContainer" containerID="3af5885b8ebd89cf7db9de38765ddf5d1fc43765015b2e27014f11deafbc6fba"
Sep 18 17:44:04.959403 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:04.959369    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8" podUID="4fd1fe5e-7044-4283-aa43-495bce3a18dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Sep 18 17:44:04.969264 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:04.969248    2881 scope.go:117] "RemoveContainer" containerID="185a40d013d91647cbe6c37a718f811bd5990f30dad004d4c2aa514dc87e54f8"
Sep 18 17:44:04.977479 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:04.977461    2881 scope.go:117] "RemoveContainer" containerID="3496f256b78ff058924055c269e3d2bdee927db5381c478d7eab026afd21595a"
Sep 18 17:44:04.984001 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:04.983975    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-865f87676-4gxjk"]
Sep 18 17:44:04.986289 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:04.986269    2881 scope.go:117] "RemoveContainer" containerID="3af5885b8ebd89cf7db9de38765ddf5d1fc43765015b2e27014f11deafbc6fba"
Sep 18 17:44:04.986647 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:44:04.986615    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3af5885b8ebd89cf7db9de38765ddf5d1fc43765015b2e27014f11deafbc6fba\": container with ID starting with 3af5885b8ebd89cf7db9de38765ddf5d1fc43765015b2e27014f11deafbc6fba not found: ID does not exist" containerID="3af5885b8ebd89cf7db9de38765ddf5d1fc43765015b2e27014f11deafbc6fba"
Sep 18 17:44:04.986748 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:04.986655    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3af5885b8ebd89cf7db9de38765ddf5d1fc43765015b2e27014f11deafbc6fba"} err="failed to get container status \"3af5885b8ebd89cf7db9de38765ddf5d1fc43765015b2e27014f11deafbc6fba\": rpc error: code = NotFound desc = could not find container \"3af5885b8ebd89cf7db9de38765ddf5d1fc43765015b2e27014f11deafbc6fba\": container with ID starting with 3af5885b8ebd89cf7db9de38765ddf5d1fc43765015b2e27014f11deafbc6fba not found: ID does not exist"
Sep 18 17:44:04.986748 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:04.986678    2881 scope.go:117] "RemoveContainer" containerID="185a40d013d91647cbe6c37a718f811bd5990f30dad004d4c2aa514dc87e54f8"
Sep 18 17:44:04.986978 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:44:04.986956    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"185a40d013d91647cbe6c37a718f811bd5990f30dad004d4c2aa514dc87e54f8\": container with ID starting with 185a40d013d91647cbe6c37a718f811bd5990f30dad004d4c2aa514dc87e54f8 not found: ID does not exist" containerID="185a40d013d91647cbe6c37a718f811bd5990f30dad004d4c2aa514dc87e54f8"
Sep 18 17:44:04.987042 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:04.986982    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"185a40d013d91647cbe6c37a718f811bd5990f30dad004d4c2aa514dc87e54f8"} err="failed to get container status \"185a40d013d91647cbe6c37a718f811bd5990f30dad004d4c2aa514dc87e54f8\": rpc error: code = NotFound desc = could not find container \"185a40d013d91647cbe6c37a718f811bd5990f30dad004d4c2aa514dc87e54f8\": container with ID starting with 185a40d013d91647cbe6c37a718f811bd5990f30dad004d4c2aa514dc87e54f8 not found: ID does not exist"
Sep 18 17:44:04.987042 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:04.986998    2881 scope.go:117] "RemoveContainer" containerID="3496f256b78ff058924055c269e3d2bdee927db5381c478d7eab026afd21595a"
Sep 18 17:44:04.987276 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:44:04.987257    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3496f256b78ff058924055c269e3d2bdee927db5381c478d7eab026afd21595a\": container with ID starting with 3496f256b78ff058924055c269e3d2bdee927db5381c478d7eab026afd21595a not found: ID does not exist" containerID="3496f256b78ff058924055c269e3d2bdee927db5381c478d7eab026afd21595a"
Sep 18 17:44:04.987372 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:04.987279    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3496f256b78ff058924055c269e3d2bdee927db5381c478d7eab026afd21595a"} err="failed to get container status \"3496f256b78ff058924055c269e3d2bdee927db5381c478d7eab026afd21595a\": rpc error: code = NotFound desc = could not find container \"3496f256b78ff058924055c269e3d2bdee927db5381c478d7eab026afd21595a\": container with ID starting with 3496f256b78ff058924055c269e3d2bdee927db5381c478d7eab026afd21595a not found: ID does not exist"
Sep 18 17:44:04.988748 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:04.988729    2881 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-865f87676-4gxjk"]
Sep 18 17:44:05.056217 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:05.056079    2881 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db6c5dde-0465-4839-ae5e-0a5293b68cec-kserve-provision-location\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:44:05.056217 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:05.056112    2881 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db6c5dde-0465-4839-ae5e-0a5293b68cec-proxy-tls\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:44:05.056217 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:05.056131    2881 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk76q\" (UniqueName: \"kubernetes.io/projected/db6c5dde-0465-4839-ae5e-0a5293b68cec-kube-api-access-xk76q\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:44:05.056217 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:05.056147    2881 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db6c5dde-0465-4839-ae5e-0a5293b68cec-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:44:05.892941 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:05.892910    2881 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db6c5dde-0465-4839-ae5e-0a5293b68cec" path="/var/lib/kubelet/pods/db6c5dde-0465-4839-ae5e-0a5293b68cec/volumes"
Sep 18 17:44:07.975092 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:07.975064    2881 generic.go:356] "Generic (PLEG): container finished" podID="bc7c7586-87e3-4ce0-9f38-909e21c2dded" containerID="19c33e1da02d01122da718abaa80db48833206bbdb0830417091be6eaa245b90" exitCode=0
Sep 18 17:44:07.975589 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:07.975138    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2" event={"ID":"bc7c7586-87e3-4ce0-9f38-909e21c2dded","Type":"ContainerDied","Data":"19c33e1da02d01122da718abaa80db48833206bbdb0830417091be6eaa245b90"}
Sep 18 17:44:08.037835 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:08.037807    2881 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 17:44:08.985432 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:08.985285    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2" event={"ID":"bc7c7586-87e3-4ce0-9f38-909e21c2dded","Type":"ContainerStarted","Data":"bfef22f15bff262b6a32403517d62d470aafa99e30d3a119f52721d693ea619c"}
Sep 18 17:44:09.964001 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:09.963972    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8"
Sep 18 17:44:09.964460 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:09.964432    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8" podUID="4fd1fe5e-7044-4283-aa43-495bce3a18dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Sep 18 17:44:10.998638 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:10.998605    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2" event={"ID":"bc7c7586-87e3-4ce0-9f38-909e21c2dded","Type":"ContainerStarted","Data":"ae4046750275192a07d411306dff1182d7ff07ee1c5455f635136280250c7a18"}
Sep 18 17:44:10.999049 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:10.998672    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2" event={"ID":"bc7c7586-87e3-4ce0-9f38-909e21c2dded","Type":"ContainerStarted","Data":"a1f170862549a6c625d8c57e0efc30e50ef30d714ef6f3b7a8219db40a6fc0b7"}
Sep 18 17:44:10.999049 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:10.998707    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2"
Sep 18 17:44:10.999049 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:10.998735    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2"
Sep 18 17:44:11.036111 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:11.036043    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2" podStartSLOduration=2.568536645 podStartE2EDuration="9.036016857s" totalImagesPullingTime="2.752020201s" totalInitContainerRuntime="3.715460011s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:44:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:44:11.034533584 +0000 UTC m=+3041.234388872" watchObservedRunningTime="2026-09-18 17:44:11.036016857 +0000 UTC m=+3041.235872145"
Sep 18 17:44:12.002985 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:12.002955    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2"
Sep 18 17:44:18.012470 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:18.012440    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2"
Sep 18 17:44:19.964489 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:19.964451    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8" podUID="4fd1fe5e-7044-4283-aa43-495bce3a18dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Sep 18 17:44:29.965378 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:29.965334    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8" podUID="4fd1fe5e-7044-4283-aa43-495bce3a18dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Sep 18 17:44:38.013877 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:38.013833    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2" podUID="bc7c7586-87e3-4ce0-9f38-909e21c2dded" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.63:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 18 17:44:39.965153 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:39.965115    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8" podUID="4fd1fe5e-7044-4283-aa43-495bce3a18dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Sep 18 17:44:43.292812 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:43.292760    2881 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2" podUID="bc7c7586-87e3-4ce0-9f38-909e21c2dded" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.63:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 18 17:44:48.233714 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:48.233685    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2"
Sep 18 17:44:49.964462 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:49.964420    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8" podUID="4fd1fe5e-7044-4283-aa43-495bce3a18dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Sep 18 17:44:59.964416 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:44:59.964376    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8" podUID="4fd1fe5e-7044-4283-aa43-495bce3a18dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Sep 18 17:45:09.964967 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:09.964927    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8" podUID="4fd1fe5e-7044-4283-aa43-495bce3a18dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Sep 18 17:45:18.235395 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:18.235366    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2"
Sep 18 17:45:19.890313 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:19.890273    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8" podUID="4fd1fe5e-7044-4283-aa43-495bce3a18dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Sep 18 17:45:22.999708 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:22.999675    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2"]
Sep 18 17:45:23.000133 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:22.999954    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2" podUID="bc7c7586-87e3-4ce0-9f38-909e21c2dded" containerName="kserve-container" containerID="cri-o://bfef22f15bff262b6a32403517d62d470aafa99e30d3a119f52721d693ea619c" gracePeriod=30
Sep 18 17:45:23.000133 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:23.000008    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2" podUID="bc7c7586-87e3-4ce0-9f38-909e21c2dded" containerName="kube-rbac-proxy" containerID="cri-o://ae4046750275192a07d411306dff1182d7ff07ee1c5455f635136280250c7a18" gracePeriod=30
Sep 18 17:45:23.000133 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:23.000037    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2" podUID="bc7c7586-87e3-4ce0-9f38-909e21c2dded" containerName="kserve-agent" containerID="cri-o://a1f170862549a6c625d8c57e0efc30e50ef30d714ef6f3b7a8219db40a6fc0b7" gracePeriod=30
Sep 18 17:45:23.007858 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:23.007815    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2" podUID="bc7c7586-87e3-4ce0-9f38-909e21c2dded" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.63:8643/healthz\": dial tcp 10.134.0.63:8643: connect: connection refused"
Sep 18 17:45:23.336119 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:23.336087    2881 generic.go:356] "Generic (PLEG): container finished" podID="bc7c7586-87e3-4ce0-9f38-909e21c2dded" containerID="ae4046750275192a07d411306dff1182d7ff07ee1c5455f635136280250c7a18" exitCode=2
Sep 18 17:45:23.336341 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:23.336169    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2" event={"ID":"bc7c7586-87e3-4ce0-9f38-909e21c2dded","Type":"ContainerDied","Data":"ae4046750275192a07d411306dff1182d7ff07ee1c5455f635136280250c7a18"}
Sep 18 17:45:26.352086 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:26.352049    2881 generic.go:356] "Generic (PLEG): container finished" podID="bc7c7586-87e3-4ce0-9f38-909e21c2dded" containerID="bfef22f15bff262b6a32403517d62d470aafa99e30d3a119f52721d693ea619c" exitCode=0
Sep 18 17:45:26.352492 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:26.352126    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2" event={"ID":"bc7c7586-87e3-4ce0-9f38-909e21c2dded","Type":"ContainerDied","Data":"bfef22f15bff262b6a32403517d62d470aafa99e30d3a119f52721d693ea619c"}
Sep 18 17:45:28.007465 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:28.007430    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2" podUID="bc7c7586-87e3-4ce0-9f38-909e21c2dded" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.63:8643/healthz\": dial tcp 10.134.0.63:8643: connect: connection refused"
Sep 18 17:45:28.013898 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:28.013867    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2" podUID="bc7c7586-87e3-4ce0-9f38-909e21c2dded" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.63:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 18 17:45:28.194763 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:28.194727    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rv4g4"]
Sep 18 17:45:28.195126 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:28.195114    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db6c5dde-0465-4839-ae5e-0a5293b68cec" containerName="storage-initializer"
Sep 18 17:45:28.195179 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:28.195128    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="db6c5dde-0465-4839-ae5e-0a5293b68cec" containerName="storage-initializer"
Sep 18 17:45:28.195179 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:28.195137    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db6c5dde-0465-4839-ae5e-0a5293b68cec" containerName="kube-rbac-proxy"
Sep 18 17:45:28.195179 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:28.195142    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="db6c5dde-0465-4839-ae5e-0a5293b68cec" containerName="kube-rbac-proxy"
Sep 18 17:45:28.195179 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:28.195156    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db6c5dde-0465-4839-ae5e-0a5293b68cec" containerName="kserve-container"
Sep 18 17:45:28.195179 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:28.195162    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="db6c5dde-0465-4839-ae5e-0a5293b68cec" containerName="kserve-container"
Sep 18 17:45:28.195361 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:28.195214    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="db6c5dde-0465-4839-ae5e-0a5293b68cec" containerName="kserve-container"
Sep 18 17:45:28.195361 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:28.195238    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="db6c5dde-0465-4839-ae5e-0a5293b68cec" containerName="kube-rbac-proxy"
Sep 18 17:45:28.198460 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:28.198435    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rv4g4"
Sep 18 17:45:28.204435 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:28.204410    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-predictor-serving-cert\""
Sep 18 17:45:28.204567 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:28.204454    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-kube-rbac-proxy-sar-config\""
Sep 18 17:45:28.216970 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:28.216938    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rv4g4"]
Sep 18 17:45:28.305751 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:28.305716    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e18f677-9b83-4138-9a41-36d19bf881fb-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-rv4g4\" (UID: \"1e18f677-9b83-4138-9a41-36d19bf881fb\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rv4g4"
Sep 18 17:45:28.305751 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:28.305752    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e18f677-9b83-4138-9a41-36d19bf881fb-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-rv4g4\" (UID: \"1e18f677-9b83-4138-9a41-36d19bf881fb\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rv4g4"
Sep 18 17:45:28.305991 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:28.305777    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e18f677-9b83-4138-9a41-36d19bf881fb-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-rv4g4\" (UID: \"1e18f677-9b83-4138-9a41-36d19bf881fb\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rv4g4"
Sep 18 17:45:28.305991 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:28.305880    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e18f677-9b83-4138-9a41-36d19bf881fb-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-rv4g4\" (UID: \"1e18f677-9b83-4138-9a41-36d19bf881fb\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rv4g4"
Sep 18 17:45:28.305991 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:28.305920    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwdj5\" (UniqueName: \"kubernetes.io/projected/1e18f677-9b83-4138-9a41-36d19bf881fb-kube-api-access-kwdj5\") pod \"isvc-paddle-predictor-6d5c9fd4f6-rv4g4\" (UID: \"1e18f677-9b83-4138-9a41-36d19bf881fb\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rv4g4"
Sep 18 17:45:28.406645 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:28.406609    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e18f677-9b83-4138-9a41-36d19bf881fb-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-rv4g4\" (UID: \"1e18f677-9b83-4138-9a41-36d19bf881fb\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rv4g4"
Sep 18 17:45:28.406827 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:28.406654    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwdj5\" (UniqueName: \"kubernetes.io/projected/1e18f677-9b83-4138-9a41-36d19bf881fb-kube-api-access-kwdj5\") pod \"isvc-paddle-predictor-6d5c9fd4f6-rv4g4\" (UID: \"1e18f677-9b83-4138-9a41-36d19bf881fb\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rv4g4"
Sep 18 17:45:28.406827 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:28.406734    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e18f677-9b83-4138-9a41-36d19bf881fb-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-rv4g4\" (UID: \"1e18f677-9b83-4138-9a41-36d19bf881fb\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rv4g4"
Sep 18 17:45:28.406827 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:28.406759    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e18f677-9b83-4138-9a41-36d19bf881fb-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-rv4g4\" (UID: \"1e18f677-9b83-4138-9a41-36d19bf881fb\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rv4g4"
Sep 18 17:45:28.406981 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:28.406788    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e18f677-9b83-4138-9a41-36d19bf881fb-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-rv4g4\" (UID: \"1e18f677-9b83-4138-9a41-36d19bf881fb\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rv4g4"
Sep 18 17:45:28.406981 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:45:28.406952    2881 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-predictor-serving-cert: secret "isvc-paddle-predictor-serving-cert" not found
Sep 18 17:45:28.407047 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:45:28.407029    2881 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e18f677-9b83-4138-9a41-36d19bf881fb-proxy-tls podName:1e18f677-9b83-4138-9a41-36d19bf881fb nodeName:}" failed. No retries permitted until 2026-09-18 17:45:28.907007032 +0000 UTC m=+3119.106862303 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1e18f677-9b83-4138-9a41-36d19bf881fb-proxy-tls") pod "isvc-paddle-predictor-6d5c9fd4f6-rv4g4" (UID: "1e18f677-9b83-4138-9a41-36d19bf881fb") : secret "isvc-paddle-predictor-serving-cert" not found
Sep 18 17:45:28.407160 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:28.407142    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e18f677-9b83-4138-9a41-36d19bf881fb-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-rv4g4\" (UID: \"1e18f677-9b83-4138-9a41-36d19bf881fb\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rv4g4"
Sep 18 17:45:28.407509 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:28.407490    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e18f677-9b83-4138-9a41-36d19bf881fb-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-rv4g4\" (UID: \"1e18f677-9b83-4138-9a41-36d19bf881fb\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rv4g4"
Sep 18 17:45:28.407569 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:28.407552    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e18f677-9b83-4138-9a41-36d19bf881fb-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-rv4g4\" (UID: \"1e18f677-9b83-4138-9a41-36d19bf881fb\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rv4g4"
Sep 18 17:45:28.416076 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:28.416051    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwdj5\" (UniqueName: \"kubernetes.io/projected/1e18f677-9b83-4138-9a41-36d19bf881fb-kube-api-access-kwdj5\") pod \"isvc-paddle-predictor-6d5c9fd4f6-rv4g4\" (UID: \"1e18f677-9b83-4138-9a41-36d19bf881fb\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rv4g4"
Sep 18 17:45:28.910721 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:28.910682    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e18f677-9b83-4138-9a41-36d19bf881fb-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-rv4g4\" (UID: \"1e18f677-9b83-4138-9a41-36d19bf881fb\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rv4g4"
Sep 18 17:45:28.913420 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:28.913393    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e18f677-9b83-4138-9a41-36d19bf881fb-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-rv4g4\" (UID: \"1e18f677-9b83-4138-9a41-36d19bf881fb\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rv4g4"
Sep 18 17:45:29.108377 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:29.108341    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rv4g4"
Sep 18 17:45:29.382472 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:29.382447    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rv4g4"]
Sep 18 17:45:29.384609 ip-10-0-143-5 kubenswrapper[2881]: W0918 17:45:29.384584    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e18f677_9b83_4138_9a41_36d19bf881fb.slice/crio-97d1a35b2d5e3c116f194cbf33d8d7bb842edc7d50a0508a7233316331ae774f WatchSource:0}: Error finding container 97d1a35b2d5e3c116f194cbf33d8d7bb842edc7d50a0508a7233316331ae774f: Status 404 returned error can't find the container with id 97d1a35b2d5e3c116f194cbf33d8d7bb842edc7d50a0508a7233316331ae774f
Sep 18 17:45:29.894109 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:29.894079    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8"
Sep 18 17:45:30.371796 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:30.371754    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rv4g4" event={"ID":"1e18f677-9b83-4138-9a41-36d19bf881fb","Type":"ContainerStarted","Data":"8f552cb4eb6975bd61313870a84e44cc8c45aee043afea1fee55a295ad0ccbcd"}
Sep 18 17:45:30.372294 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:30.371842    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rv4g4" event={"ID":"1e18f677-9b83-4138-9a41-36d19bf881fb","Type":"ContainerStarted","Data":"97d1a35b2d5e3c116f194cbf33d8d7bb842edc7d50a0508a7233316331ae774f"}
Sep 18 17:45:33.007889 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:33.007845    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2" podUID="bc7c7586-87e3-4ce0-9f38-909e21c2dded" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.63:8643/healthz\": dial tcp 10.134.0.63:8643: connect: connection refused"
Sep 18 17:45:33.008285 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:33.007977    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2"
Sep 18 17:45:34.392159 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:34.392123    2881 generic.go:356] "Generic (PLEG): container finished" podID="1e18f677-9b83-4138-9a41-36d19bf881fb" containerID="8f552cb4eb6975bd61313870a84e44cc8c45aee043afea1fee55a295ad0ccbcd" exitCode=0
Sep 18 17:45:34.392577 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:34.392196    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rv4g4" event={"ID":"1e18f677-9b83-4138-9a41-36d19bf881fb","Type":"ContainerDied","Data":"8f552cb4eb6975bd61313870a84e44cc8c45aee043afea1fee55a295ad0ccbcd"}
Sep 18 17:45:38.007590 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:38.007516    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2" podUID="bc7c7586-87e3-4ce0-9f38-909e21c2dded" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.63:8643/healthz\": dial tcp 10.134.0.63:8643: connect: connection refused"
Sep 18 17:45:38.013288 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:38.013219    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2" podUID="bc7c7586-87e3-4ce0-9f38-909e21c2dded" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.63:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 18 17:45:39.653141 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:39.653104    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8"]
Sep 18 17:45:39.653703 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:39.653404    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8" podUID="4fd1fe5e-7044-4283-aa43-495bce3a18dc" containerName="kserve-container" containerID="cri-o://96f8b785be11dcc5376d0328cd67e1ce6fd7d708112d324c5dabe04c2c49c8e3" gracePeriod=30
Sep 18 17:45:39.653703 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:39.653546    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8" podUID="4fd1fe5e-7044-4283-aa43-495bce3a18dc" containerName="kube-rbac-proxy" containerID="cri-o://e5bb9c44e8bf2d621dff4506d3603b26536cb6b5f3c77b1dd199d8bac0ec4215" gracePeriod=30
Sep 18 17:45:39.890803 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:39.890750    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8" podUID="4fd1fe5e-7044-4283-aa43-495bce3a18dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Sep 18 17:45:39.960504 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:39.960407    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8" podUID="4fd1fe5e-7044-4283-aa43-495bce3a18dc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.62:8643/healthz\": dial tcp 10.134.0.62:8643: connect: connection refused"
Sep 18 17:45:40.427640 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:40.427609    2881 generic.go:356] "Generic (PLEG): container finished" podID="4fd1fe5e-7044-4283-aa43-495bce3a18dc" containerID="e5bb9c44e8bf2d621dff4506d3603b26536cb6b5f3c77b1dd199d8bac0ec4215" exitCode=2
Sep 18 17:45:40.427839 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:40.427688    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8" event={"ID":"4fd1fe5e-7044-4283-aa43-495bce3a18dc","Type":"ContainerDied","Data":"e5bb9c44e8bf2d621dff4506d3603b26536cb6b5f3c77b1dd199d8bac0ec4215"}
Sep 18 17:45:40.735388 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:40.735295    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-97d9f4864-t4z8v"]
Sep 18 17:45:40.757869 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:40.757838    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-97d9f4864-t4z8v"]
Sep 18 17:45:40.758037 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:40.757966    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-97d9f4864-t4z8v"
Sep 18 17:45:40.762532 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:40.762497    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-predictor-serving-cert\""
Sep 18 17:45:40.762711 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:40.762545    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\""
Sep 18 17:45:40.831161 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:40.831123    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/395493bc-8b7c-4ebc-85e0-0c76a6be6898-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-97d9f4864-t4z8v\" (UID: \"395493bc-8b7c-4ebc-85e0-0c76a6be6898\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-97d9f4864-t4z8v"
Sep 18 17:45:40.831161 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:40.831174    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb9c4\" (UniqueName: \"kubernetes.io/projected/395493bc-8b7c-4ebc-85e0-0c76a6be6898-kube-api-access-bb9c4\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-97d9f4864-t4z8v\" (UID: \"395493bc-8b7c-4ebc-85e0-0c76a6be6898\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-97d9f4864-t4z8v"
Sep 18 17:45:40.831434 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:40.831208    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/395493bc-8b7c-4ebc-85e0-0c76a6be6898-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-97d9f4864-t4z8v\" (UID: \"395493bc-8b7c-4ebc-85e0-0c76a6be6898\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-97d9f4864-t4z8v"
Sep 18 17:45:40.831434 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:40.831323    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/395493bc-8b7c-4ebc-85e0-0c76a6be6898-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-97d9f4864-t4z8v\" (UID: \"395493bc-8b7c-4ebc-85e0-0c76a6be6898\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-97d9f4864-t4z8v"
Sep 18 17:45:40.932846 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:40.932799    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/395493bc-8b7c-4ebc-85e0-0c76a6be6898-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-97d9f4864-t4z8v\" (UID: \"395493bc-8b7c-4ebc-85e0-0c76a6be6898\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-97d9f4864-t4z8v"
Sep 18 17:45:40.933043 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:40.932912    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/395493bc-8b7c-4ebc-85e0-0c76a6be6898-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-97d9f4864-t4z8v\" (UID: \"395493bc-8b7c-4ebc-85e0-0c76a6be6898\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-97d9f4864-t4z8v"
Sep 18 17:45:40.933043 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:40.932949    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb9c4\" (UniqueName: \"kubernetes.io/projected/395493bc-8b7c-4ebc-85e0-0c76a6be6898-kube-api-access-bb9c4\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-97d9f4864-t4z8v\" (UID: \"395493bc-8b7c-4ebc-85e0-0c76a6be6898\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-97d9f4864-t4z8v"
Sep 18 17:45:40.933043 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:45:40.932955    2881 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-global-fail-predictor-serving-cert" not found
Sep 18 17:45:40.933043 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:40.932989    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/395493bc-8b7c-4ebc-85e0-0c76a6be6898-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-97d9f4864-t4z8v\" (UID: \"395493bc-8b7c-4ebc-85e0-0c76a6be6898\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-97d9f4864-t4z8v"
Sep 18 17:45:40.933295 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:45:40.933053    2881 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/395493bc-8b7c-4ebc-85e0-0c76a6be6898-proxy-tls podName:395493bc-8b7c-4ebc-85e0-0c76a6be6898 nodeName:}" failed. No retries permitted until 2026-09-18 17:45:41.433027807 +0000 UTC m=+3131.632883098 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/395493bc-8b7c-4ebc-85e0-0c76a6be6898-proxy-tls") pod "isvc-sklearn-s3-tls-global-fail-predictor-97d9f4864-t4z8v" (UID: "395493bc-8b7c-4ebc-85e0-0c76a6be6898") : secret "isvc-sklearn-s3-tls-global-fail-predictor-serving-cert" not found
Sep 18 17:45:40.934376 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:40.933803    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/395493bc-8b7c-4ebc-85e0-0c76a6be6898-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-97d9f4864-t4z8v\" (UID: \"395493bc-8b7c-4ebc-85e0-0c76a6be6898\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-97d9f4864-t4z8v"
Sep 18 17:45:40.934376 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:40.933869    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/395493bc-8b7c-4ebc-85e0-0c76a6be6898-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-97d9f4864-t4z8v\" (UID: \"395493bc-8b7c-4ebc-85e0-0c76a6be6898\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-97d9f4864-t4z8v"
Sep 18 17:45:40.943707 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:40.943679    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb9c4\" (UniqueName: \"kubernetes.io/projected/395493bc-8b7c-4ebc-85e0-0c76a6be6898-kube-api-access-bb9c4\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-97d9f4864-t4z8v\" (UID: \"395493bc-8b7c-4ebc-85e0-0c76a6be6898\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-97d9f4864-t4z8v"
Sep 18 17:45:41.439349 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:41.439313    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/395493bc-8b7c-4ebc-85e0-0c76a6be6898-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-97d9f4864-t4z8v\" (UID: \"395493bc-8b7c-4ebc-85e0-0c76a6be6898\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-97d9f4864-t4z8v"
Sep 18 17:45:41.442103 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:41.442068    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/395493bc-8b7c-4ebc-85e0-0c76a6be6898-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-97d9f4864-t4z8v\" (UID: \"395493bc-8b7c-4ebc-85e0-0c76a6be6898\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-97d9f4864-t4z8v"
Sep 18 17:45:41.669976 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:41.669937    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-97d9f4864-t4z8v"
Sep 18 17:45:43.008244 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:43.008178    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2" podUID="bc7c7586-87e3-4ce0-9f38-909e21c2dded" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.63:8643/healthz\": dial tcp 10.134.0.63:8643: connect: connection refused"
Sep 18 17:45:44.960021 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:44.959973    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8" podUID="4fd1fe5e-7044-4283-aa43-495bce3a18dc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.62:8643/healthz\": dial tcp 10.134.0.62:8643: connect: connection refused"
Sep 18 17:45:45.459572 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:45.459530    2881 generic.go:356] "Generic (PLEG): container finished" podID="4fd1fe5e-7044-4283-aa43-495bce3a18dc" containerID="96f8b785be11dcc5376d0328cd67e1ce6fd7d708112d324c5dabe04c2c49c8e3" exitCode=0
Sep 18 17:45:45.459721 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:45.459601    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8" event={"ID":"4fd1fe5e-7044-4283-aa43-495bce3a18dc","Type":"ContainerDied","Data":"96f8b785be11dcc5376d0328cd67e1ce6fd7d708112d324c5dabe04c2c49c8e3"}
Sep 18 17:45:45.543514 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:45.543479    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8"
Sep 18 17:45:45.682066 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:45.682039    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4fd1fe5e-7044-4283-aa43-495bce3a18dc-proxy-tls\") pod \"4fd1fe5e-7044-4283-aa43-495bce3a18dc\" (UID: \"4fd1fe5e-7044-4283-aa43-495bce3a18dc\") "
Sep 18 17:45:45.682193 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:45.682117    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4fd1fe5e-7044-4283-aa43-495bce3a18dc-kserve-provision-location\") pod \"4fd1fe5e-7044-4283-aa43-495bce3a18dc\" (UID: \"4fd1fe5e-7044-4283-aa43-495bce3a18dc\") "
Sep 18 17:45:45.682193 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:45.682162    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c6pkr\" (UniqueName: \"kubernetes.io/projected/4fd1fe5e-7044-4283-aa43-495bce3a18dc-kube-api-access-c6pkr\") pod \"4fd1fe5e-7044-4283-aa43-495bce3a18dc\" (UID: \"4fd1fe5e-7044-4283-aa43-495bce3a18dc\") "
Sep 18 17:45:45.682337 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:45.682289    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4fd1fe5e-7044-4283-aa43-495bce3a18dc-kube-rbac-proxy-sar-config\") pod \"4fd1fe5e-7044-4283-aa43-495bce3a18dc\" (UID: \"4fd1fe5e-7044-4283-aa43-495bce3a18dc\") "
Sep 18 17:45:45.682337 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:45.682327    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4fd1fe5e-7044-4283-aa43-495bce3a18dc-cabundle-cert\") pod \"4fd1fe5e-7044-4283-aa43-495bce3a18dc\" (UID: \"4fd1fe5e-7044-4283-aa43-495bce3a18dc\") "
Sep 18 17:45:45.682442 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:45.682421    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fd1fe5e-7044-4283-aa43-495bce3a18dc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4fd1fe5e-7044-4283-aa43-495bce3a18dc" (UID: "4fd1fe5e-7044-4283-aa43-495bce3a18dc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:45:45.682680 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:45.682655    2881 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4fd1fe5e-7044-4283-aa43-495bce3a18dc-kserve-provision-location\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:45:45.682760 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:45.682669    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fd1fe5e-7044-4283-aa43-495bce3a18dc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4fd1fe5e-7044-4283-aa43-495bce3a18dc" (UID: "4fd1fe5e-7044-4283-aa43-495bce3a18dc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:45:45.682760 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:45.682711    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fd1fe5e-7044-4283-aa43-495bce3a18dc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4fd1fe5e-7044-4283-aa43-495bce3a18dc" (UID: "4fd1fe5e-7044-4283-aa43-495bce3a18dc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:45:45.684371 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:45.684349    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fd1fe5e-7044-4283-aa43-495bce3a18dc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4fd1fe5e-7044-4283-aa43-495bce3a18dc" (UID: "4fd1fe5e-7044-4283-aa43-495bce3a18dc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:45:45.684459 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:45.684346    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fd1fe5e-7044-4283-aa43-495bce3a18dc-kube-api-access-c6pkr" (OuterVolumeSpecName: "kube-api-access-c6pkr") pod "4fd1fe5e-7044-4283-aa43-495bce3a18dc" (UID: "4fd1fe5e-7044-4283-aa43-495bce3a18dc"). InnerVolumeSpecName "kube-api-access-c6pkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:45:45.784008 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:45.783968    2881 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4fd1fe5e-7044-4283-aa43-495bce3a18dc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:45:45.784008 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:45.783993    2881 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4fd1fe5e-7044-4283-aa43-495bce3a18dc-cabundle-cert\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:45:45.784008 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:45.784004    2881 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4fd1fe5e-7044-4283-aa43-495bce3a18dc-proxy-tls\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:45:45.784126 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:45.784012    2881 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6pkr\" (UniqueName: \"kubernetes.io/projected/4fd1fe5e-7044-4283-aa43-495bce3a18dc-kube-api-access-c6pkr\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:45:45.875052 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:45.875002    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-97d9f4864-t4z8v"]
Sep 18 17:45:45.877271 ip-10-0-143-5 kubenswrapper[2881]: W0918 17:45:45.877214    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod395493bc_8b7c_4ebc_85e0_0c76a6be6898.slice/crio-1f14c77ab5a8e1e6e446b2fd410454b59f62c8d75826cf80c89b5c192a20667d WatchSource:0}: Error finding container 1f14c77ab5a8e1e6e446b2fd410454b59f62c8d75826cf80c89b5c192a20667d: Status 404 returned error can't find the container with id 1f14c77ab5a8e1e6e446b2fd410454b59f62c8d75826cf80c89b5c192a20667d
Sep 18 17:45:46.466429 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:46.466385    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-97d9f4864-t4z8v" event={"ID":"395493bc-8b7c-4ebc-85e0-0c76a6be6898","Type":"ContainerStarted","Data":"a3dc7680f4c753810b55dc7ad995b16ea1fccc7a8cc2e9814a8e6781558b0184"}
Sep 18 17:45:46.466897 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:46.466465    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-97d9f4864-t4z8v" event={"ID":"395493bc-8b7c-4ebc-85e0-0c76a6be6898","Type":"ContainerStarted","Data":"1f14c77ab5a8e1e6e446b2fd410454b59f62c8d75826cf80c89b5c192a20667d"}
Sep 18 17:45:46.468721 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:46.468694    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rv4g4" event={"ID":"1e18f677-9b83-4138-9a41-36d19bf881fb","Type":"ContainerStarted","Data":"acf93b439856898d14f8303c78f01cd6f4bf7a64d9b1c4b58a589d757e72e133"}
Sep 18 17:45:46.468853 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:46.468746    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rv4g4" event={"ID":"1e18f677-9b83-4138-9a41-36d19bf881fb","Type":"ContainerStarted","Data":"08ea7bbca0d56fea748a3bbfde00715f54046e1619baa8058a6339fdc3896743"}
Sep 18 17:45:46.469036 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:46.469016    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rv4g4"
Sep 18 17:45:46.469155 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:46.469065    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rv4g4"
Sep 18 17:45:46.470774 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:46.470745    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rv4g4" podUID="1e18f677-9b83-4138-9a41-36d19bf881fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 18 17:45:46.471056 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:46.471035    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8" event={"ID":"4fd1fe5e-7044-4283-aa43-495bce3a18dc","Type":"ContainerDied","Data":"93555e2a23eebecdc92681bad18fab671ac2587e334efeaf46784824ecb66bab"}
Sep 18 17:45:46.471137 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:46.471084    2881 scope.go:117] "RemoveContainer" containerID="e5bb9c44e8bf2d621dff4506d3603b26536cb6b5f3c77b1dd199d8bac0ec4215"
Sep 18 17:45:46.471137 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:46.471041    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8"
Sep 18 17:45:46.479514 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:46.479496    2881 scope.go:117] "RemoveContainer" containerID="96f8b785be11dcc5376d0328cd67e1ce6fd7d708112d324c5dabe04c2c49c8e3"
Sep 18 17:45:46.488270 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:46.488248    2881 scope.go:117] "RemoveContainer" containerID="e919135e405aba0c009eda9265a8391725ad2beec85b7eeb67bfeb1963319179"
Sep 18 17:45:46.505393 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:46.505343    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rv4g4" podStartSLOduration=2.769851201 podStartE2EDuration="18.505324918s" totalImagesPullingTime="11.284109948s" totalInitContainerRuntime="4.451363769s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:45:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:45:46.504301609 +0000 UTC m=+3136.704156913" watchObservedRunningTime="2026-09-18 17:45:46.505324918 +0000 UTC m=+3136.705180201"
Sep 18 17:45:46.518709 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:46.518673    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8"]
Sep 18 17:45:46.520951 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:46.520925    2881 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66874dd6f6-d7nl8"]
Sep 18 17:45:47.477530 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:47.477491    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rv4g4" podUID="1e18f677-9b83-4138-9a41-36d19bf881fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 18 17:45:47.893345 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:47.893313    2881 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fd1fe5e-7044-4283-aa43-495bce3a18dc" path="/var/lib/kubelet/pods/4fd1fe5e-7044-4283-aa43-495bce3a18dc/volumes"
Sep 18 17:45:48.008297 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:48.008255    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2" podUID="bc7c7586-87e3-4ce0-9f38-909e21c2dded" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.63:8643/healthz\": dial tcp 10.134.0.63:8643: connect: connection refused"
Sep 18 17:45:48.013085 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:48.013053    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2" podUID="bc7c7586-87e3-4ce0-9f38-909e21c2dded" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.63:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 18 17:45:48.013241 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:48.013169    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2"
Sep 18 17:45:51.498897 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:51.498818    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-97d9f4864-t4z8v_395493bc-8b7c-4ebc-85e0-0c76a6be6898/storage-initializer/0.log"
Sep 18 17:45:51.498897 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:51.498858    2881 generic.go:356] "Generic (PLEG): container finished" podID="395493bc-8b7c-4ebc-85e0-0c76a6be6898" containerID="a3dc7680f4c753810b55dc7ad995b16ea1fccc7a8cc2e9814a8e6781558b0184" exitCode=1
Sep 18 17:45:51.499388 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:51.498937    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-97d9f4864-t4z8v" event={"ID":"395493bc-8b7c-4ebc-85e0-0c76a6be6898","Type":"ContainerDied","Data":"a3dc7680f4c753810b55dc7ad995b16ea1fccc7a8cc2e9814a8e6781558b0184"}
Sep 18 17:45:52.481877 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:52.481851    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rv4g4"
Sep 18 17:45:52.482399 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:52.482377    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rv4g4" podUID="1e18f677-9b83-4138-9a41-36d19bf881fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 18 17:45:52.505853 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:52.505829    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-97d9f4864-t4z8v_395493bc-8b7c-4ebc-85e0-0c76a6be6898/storage-initializer/0.log"
Sep 18 17:45:52.506266 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:52.505903    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-97d9f4864-t4z8v" event={"ID":"395493bc-8b7c-4ebc-85e0-0c76a6be6898","Type":"ContainerStarted","Data":"6caf3921985669a1b072bf4566fa2184d54dfb506b8a1e959e0907be9538c24f"}
Sep 18 17:45:53.007335 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:53.007288    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2" podUID="bc7c7586-87e3-4ce0-9f38-909e21c2dded" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.63:8643/healthz\": dial tcp 10.134.0.63:8643: connect: connection refused"
Sep 18 17:45:53.174013 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:53.173988    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2"
Sep 18 17:45:53.355959 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:53.355921    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc7c7586-87e3-4ce0-9f38-909e21c2dded-kube-rbac-proxy-sar-config\") pod \"bc7c7586-87e3-4ce0-9f38-909e21c2dded\" (UID: \"bc7c7586-87e3-4ce0-9f38-909e21c2dded\") "
Sep 18 17:45:53.356149 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:53.356005    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bc7c7586-87e3-4ce0-9f38-909e21c2dded-kserve-provision-location\") pod \"bc7c7586-87e3-4ce0-9f38-909e21c2dded\" (UID: \"bc7c7586-87e3-4ce0-9f38-909e21c2dded\") "
Sep 18 17:45:53.356149 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:53.356066    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6wgw\" (UniqueName: \"kubernetes.io/projected/bc7c7586-87e3-4ce0-9f38-909e21c2dded-kube-api-access-s6wgw\") pod \"bc7c7586-87e3-4ce0-9f38-909e21c2dded\" (UID: \"bc7c7586-87e3-4ce0-9f38-909e21c2dded\") "
Sep 18 17:45:53.356274 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:53.356153    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc7c7586-87e3-4ce0-9f38-909e21c2dded-proxy-tls\") pod \"bc7c7586-87e3-4ce0-9f38-909e21c2dded\" (UID: \"bc7c7586-87e3-4ce0-9f38-909e21c2dded\") "
Sep 18 17:45:53.356274 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:53.356181    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bc7c7586-87e3-4ce0-9f38-909e21c2dded-cabundle-cert\") pod \"bc7c7586-87e3-4ce0-9f38-909e21c2dded\" (UID: \"bc7c7586-87e3-4ce0-9f38-909e21c2dded\") "
Sep 18 17:45:53.356401 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:53.356364    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc7c7586-87e3-4ce0-9f38-909e21c2dded-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bc7c7586-87e3-4ce0-9f38-909e21c2dded" (UID: "bc7c7586-87e3-4ce0-9f38-909e21c2dded"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:45:53.356401 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:53.356394    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc7c7586-87e3-4ce0-9f38-909e21c2dded-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bc7c7586-87e3-4ce0-9f38-909e21c2dded" (UID: "bc7c7586-87e3-4ce0-9f38-909e21c2dded"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:45:53.356516 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:53.356498    2881 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc7c7586-87e3-4ce0-9f38-909e21c2dded-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:45:53.356572 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:53.356520    2881 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bc7c7586-87e3-4ce0-9f38-909e21c2dded-kserve-provision-location\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:45:53.356626 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:53.356608    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc7c7586-87e3-4ce0-9f38-909e21c2dded-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bc7c7586-87e3-4ce0-9f38-909e21c2dded" (UID: "bc7c7586-87e3-4ce0-9f38-909e21c2dded"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:45:53.358259 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:53.358218    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc7c7586-87e3-4ce0-9f38-909e21c2dded-kube-api-access-s6wgw" (OuterVolumeSpecName: "kube-api-access-s6wgw") pod "bc7c7586-87e3-4ce0-9f38-909e21c2dded" (UID: "bc7c7586-87e3-4ce0-9f38-909e21c2dded"). InnerVolumeSpecName "kube-api-access-s6wgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:45:53.358369 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:53.358272    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc7c7586-87e3-4ce0-9f38-909e21c2dded-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bc7c7586-87e3-4ce0-9f38-909e21c2dded" (UID: "bc7c7586-87e3-4ce0-9f38-909e21c2dded"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:45:53.457525 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:53.457484    2881 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc7c7586-87e3-4ce0-9f38-909e21c2dded-proxy-tls\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:45:53.457525 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:53.457517    2881 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bc7c7586-87e3-4ce0-9f38-909e21c2dded-cabundle-cert\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:45:53.457525 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:53.457531    2881 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6wgw\" (UniqueName: \"kubernetes.io/projected/bc7c7586-87e3-4ce0-9f38-909e21c2dded-kube-api-access-s6wgw\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:45:53.513952 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:53.513913    2881 generic.go:356] "Generic (PLEG): container finished" podID="bc7c7586-87e3-4ce0-9f38-909e21c2dded" containerID="a1f170862549a6c625d8c57e0efc30e50ef30d714ef6f3b7a8219db40a6fc0b7" exitCode=137
Sep 18 17:45:53.514391 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:53.513991    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2"
Sep 18 17:45:53.514391 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:53.513995    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2" event={"ID":"bc7c7586-87e3-4ce0-9f38-909e21c2dded","Type":"ContainerDied","Data":"a1f170862549a6c625d8c57e0efc30e50ef30d714ef6f3b7a8219db40a6fc0b7"}
Sep 18 17:45:53.514391 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:53.514061    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2" event={"ID":"bc7c7586-87e3-4ce0-9f38-909e21c2dded","Type":"ContainerDied","Data":"c1676495d36f039f9ea618b79b7940ea212d46190f1d7f9e9b32a14013680632"}
Sep 18 17:45:53.514391 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:53.514094    2881 scope.go:117] "RemoveContainer" containerID="ae4046750275192a07d411306dff1182d7ff07ee1c5455f635136280250c7a18"
Sep 18 17:45:53.523360 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:53.523339    2881 scope.go:117] "RemoveContainer" containerID="a1f170862549a6c625d8c57e0efc30e50ef30d714ef6f3b7a8219db40a6fc0b7"
Sep 18 17:45:53.531010 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:53.530989    2881 scope.go:117] "RemoveContainer" containerID="bfef22f15bff262b6a32403517d62d470aafa99e30d3a119f52721d693ea619c"
Sep 18 17:45:53.539717 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:53.539694    2881 scope.go:117] "RemoveContainer" containerID="19c33e1da02d01122da718abaa80db48833206bbdb0830417091be6eaa245b90"
Sep 18 17:45:53.542959 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:53.542934    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2"]
Sep 18 17:45:53.547036 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:53.547008    2881 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7477cd8d95-jl6t2"]
Sep 18 17:45:53.549483 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:53.549466    2881 scope.go:117] "RemoveContainer" containerID="ae4046750275192a07d411306dff1182d7ff07ee1c5455f635136280250c7a18"
Sep 18 17:45:53.549831 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:45:53.549813    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae4046750275192a07d411306dff1182d7ff07ee1c5455f635136280250c7a18\": container with ID starting with ae4046750275192a07d411306dff1182d7ff07ee1c5455f635136280250c7a18 not found: ID does not exist" containerID="ae4046750275192a07d411306dff1182d7ff07ee1c5455f635136280250c7a18"
Sep 18 17:45:53.549918 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:53.549844    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae4046750275192a07d411306dff1182d7ff07ee1c5455f635136280250c7a18"} err="failed to get container status \"ae4046750275192a07d411306dff1182d7ff07ee1c5455f635136280250c7a18\": rpc error: code = NotFound desc = could not find container \"ae4046750275192a07d411306dff1182d7ff07ee1c5455f635136280250c7a18\": container with ID starting with ae4046750275192a07d411306dff1182d7ff07ee1c5455f635136280250c7a18 not found: ID does not exist"
Sep 18 17:45:53.549918 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:53.549868    2881 scope.go:117] "RemoveContainer" containerID="a1f170862549a6c625d8c57e0efc30e50ef30d714ef6f3b7a8219db40a6fc0b7"
Sep 18 17:45:53.550175 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:45:53.550156    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1f170862549a6c625d8c57e0efc30e50ef30d714ef6f3b7a8219db40a6fc0b7\": container with ID starting with a1f170862549a6c625d8c57e0efc30e50ef30d714ef6f3b7a8219db40a6fc0b7 not found: ID does not exist" containerID="a1f170862549a6c625d8c57e0efc30e50ef30d714ef6f3b7a8219db40a6fc0b7"
Sep 18 17:45:53.550215 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:53.550182    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1f170862549a6c625d8c57e0efc30e50ef30d714ef6f3b7a8219db40a6fc0b7"} err="failed to get container status \"a1f170862549a6c625d8c57e0efc30e50ef30d714ef6f3b7a8219db40a6fc0b7\": rpc error: code = NotFound desc = could not find container \"a1f170862549a6c625d8c57e0efc30e50ef30d714ef6f3b7a8219db40a6fc0b7\": container with ID starting with a1f170862549a6c625d8c57e0efc30e50ef30d714ef6f3b7a8219db40a6fc0b7 not found: ID does not exist"
Sep 18 17:45:53.550215 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:53.550198    2881 scope.go:117] "RemoveContainer" containerID="bfef22f15bff262b6a32403517d62d470aafa99e30d3a119f52721d693ea619c"
Sep 18 17:45:53.550550 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:45:53.550529    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfef22f15bff262b6a32403517d62d470aafa99e30d3a119f52721d693ea619c\": container with ID starting with bfef22f15bff262b6a32403517d62d470aafa99e30d3a119f52721d693ea619c not found: ID does not exist" containerID="bfef22f15bff262b6a32403517d62d470aafa99e30d3a119f52721d693ea619c"
Sep 18 17:45:53.550590 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:53.550560    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfef22f15bff262b6a32403517d62d470aafa99e30d3a119f52721d693ea619c"} err="failed to get container status \"bfef22f15bff262b6a32403517d62d470aafa99e30d3a119f52721d693ea619c\": rpc error: code = NotFound desc = could not find container \"bfef22f15bff262b6a32403517d62d470aafa99e30d3a119f52721d693ea619c\": container with ID starting with bfef22f15bff262b6a32403517d62d470aafa99e30d3a119f52721d693ea619c not found: ID does not exist"
Sep 18 17:45:53.550590 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:53.550578    2881 scope.go:117] "RemoveContainer" containerID="19c33e1da02d01122da718abaa80db48833206bbdb0830417091be6eaa245b90"
Sep 18 17:45:53.550838 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:45:53.550822    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19c33e1da02d01122da718abaa80db48833206bbdb0830417091be6eaa245b90\": container with ID starting with 19c33e1da02d01122da718abaa80db48833206bbdb0830417091be6eaa245b90 not found: ID does not exist" containerID="19c33e1da02d01122da718abaa80db48833206bbdb0830417091be6eaa245b90"
Sep 18 17:45:53.550895 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:53.550842    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19c33e1da02d01122da718abaa80db48833206bbdb0830417091be6eaa245b90"} err="failed to get container status \"19c33e1da02d01122da718abaa80db48833206bbdb0830417091be6eaa245b90\": rpc error: code = NotFound desc = could not find container \"19c33e1da02d01122da718abaa80db48833206bbdb0830417091be6eaa245b90\": container with ID starting with 19c33e1da02d01122da718abaa80db48833206bbdb0830417091be6eaa245b90 not found: ID does not exist"
Sep 18 17:45:53.897791 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:53.895301    2881 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc7c7586-87e3-4ce0-9f38-909e21c2dded" path="/var/lib/kubelet/pods/bc7c7586-87e3-4ce0-9f38-909e21c2dded/volumes"
Sep 18 17:45:54.522105 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:54.522078    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-97d9f4864-t4z8v_395493bc-8b7c-4ebc-85e0-0c76a6be6898/storage-initializer/1.log"
Sep 18 17:45:54.522542 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:54.522525    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-97d9f4864-t4z8v_395493bc-8b7c-4ebc-85e0-0c76a6be6898/storage-initializer/0.log"
Sep 18 17:45:54.522584 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:54.522561    2881 generic.go:356] "Generic (PLEG): container finished" podID="395493bc-8b7c-4ebc-85e0-0c76a6be6898" containerID="6caf3921985669a1b072bf4566fa2184d54dfb506b8a1e959e0907be9538c24f" exitCode=1
Sep 18 17:45:54.522628 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:54.522608    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-97d9f4864-t4z8v" event={"ID":"395493bc-8b7c-4ebc-85e0-0c76a6be6898","Type":"ContainerDied","Data":"6caf3921985669a1b072bf4566fa2184d54dfb506b8a1e959e0907be9538c24f"}
Sep 18 17:45:54.522677 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:54.522665    2881 scope.go:117] "RemoveContainer" containerID="a3dc7680f4c753810b55dc7ad995b16ea1fccc7a8cc2e9814a8e6781558b0184"
Sep 18 17:45:54.523042 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:54.523020    2881 scope.go:117] "RemoveContainer" containerID="a3dc7680f4c753810b55dc7ad995b16ea1fccc7a8cc2e9814a8e6781558b0184"
Sep 18 17:45:54.533576 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:45:54.533555    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-global-fail-predictor-97d9f4864-t4z8v_kserve-ci-e2e-test(395493bc-8b7c-4ebc-85e0-0c76a6be6898)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-97d9f4864-t4z8v" podUID="395493bc-8b7c-4ebc-85e0-0c76a6be6898"
Sep 18 17:45:54.533667 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:45:54.533553    2881 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-global-fail-predictor-97d9f4864-t4z8v_kserve-ci-e2e-test_395493bc-8b7c-4ebc-85e0-0c76a6be6898_0 in pod sandbox 1f14c77ab5a8e1e6e446b2fd410454b59f62c8d75826cf80c89b5c192a20667d from index: no such id: 'a3dc7680f4c753810b55dc7ad995b16ea1fccc7a8cc2e9814a8e6781558b0184'" containerID="a3dc7680f4c753810b55dc7ad995b16ea1fccc7a8cc2e9814a8e6781558b0184"
Sep 18 17:45:54.533667 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:54.533591    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3dc7680f4c753810b55dc7ad995b16ea1fccc7a8cc2e9814a8e6781558b0184"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-global-fail-predictor-97d9f4864-t4z8v_kserve-ci-e2e-test_395493bc-8b7c-4ebc-85e0-0c76a6be6898_0 in pod sandbox 1f14c77ab5a8e1e6e446b2fd410454b59f62c8d75826cf80c89b5c192a20667d from index: no such id: 'a3dc7680f4c753810b55dc7ad995b16ea1fccc7a8cc2e9814a8e6781558b0184'"
Sep 18 17:45:55.529766 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:45:55.529743    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-97d9f4864-t4z8v_395493bc-8b7c-4ebc-85e0-0c76a6be6898/storage-initializer/1.log"
Sep 18 17:46:00.700517 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:00.700485    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-97d9f4864-t4z8v"]
Sep 18 17:46:00.836538 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:00.836507    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-97d9f4864-t4z8v_395493bc-8b7c-4ebc-85e0-0c76a6be6898/storage-initializer/1.log"
Sep 18 17:46:00.840015 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:00.839990    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-97d9f4864-t4z8v"
Sep 18 17:46:00.916812 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:00.916775    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/395493bc-8b7c-4ebc-85e0-0c76a6be6898-proxy-tls\") pod \"395493bc-8b7c-4ebc-85e0-0c76a6be6898\" (UID: \"395493bc-8b7c-4ebc-85e0-0c76a6be6898\") "
Sep 18 17:46:00.916995 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:00.916842    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bb9c4\" (UniqueName: \"kubernetes.io/projected/395493bc-8b7c-4ebc-85e0-0c76a6be6898-kube-api-access-bb9c4\") pod \"395493bc-8b7c-4ebc-85e0-0c76a6be6898\" (UID: \"395493bc-8b7c-4ebc-85e0-0c76a6be6898\") "
Sep 18 17:46:00.916995 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:00.916914    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/395493bc-8b7c-4ebc-85e0-0c76a6be6898-kube-rbac-proxy-sar-config\") pod \"395493bc-8b7c-4ebc-85e0-0c76a6be6898\" (UID: \"395493bc-8b7c-4ebc-85e0-0c76a6be6898\") "
Sep 18 17:46:00.917109 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:00.917017    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/395493bc-8b7c-4ebc-85e0-0c76a6be6898-kserve-provision-location\") pod \"395493bc-8b7c-4ebc-85e0-0c76a6be6898\" (UID: \"395493bc-8b7c-4ebc-85e0-0c76a6be6898\") "
Sep 18 17:46:00.917314 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:00.917292    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/395493bc-8b7c-4ebc-85e0-0c76a6be6898-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "395493bc-8b7c-4ebc-85e0-0c76a6be6898" (UID: "395493bc-8b7c-4ebc-85e0-0c76a6be6898"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:46:00.917417 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:00.917320    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/395493bc-8b7c-4ebc-85e0-0c76a6be6898-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "395493bc-8b7c-4ebc-85e0-0c76a6be6898" (UID: "395493bc-8b7c-4ebc-85e0-0c76a6be6898"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:46:00.918991 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:00.918970    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/395493bc-8b7c-4ebc-85e0-0c76a6be6898-kube-api-access-bb9c4" (OuterVolumeSpecName: "kube-api-access-bb9c4") pod "395493bc-8b7c-4ebc-85e0-0c76a6be6898" (UID: "395493bc-8b7c-4ebc-85e0-0c76a6be6898"). InnerVolumeSpecName "kube-api-access-bb9c4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:46:00.919051 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:00.918975    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/395493bc-8b7c-4ebc-85e0-0c76a6be6898-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "395493bc-8b7c-4ebc-85e0-0c76a6be6898" (UID: "395493bc-8b7c-4ebc-85e0-0c76a6be6898"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:46:01.017919 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:01.017887    2881 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/395493bc-8b7c-4ebc-85e0-0c76a6be6898-proxy-tls\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:46:01.017919 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:01.017917    2881 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bb9c4\" (UniqueName: \"kubernetes.io/projected/395493bc-8b7c-4ebc-85e0-0c76a6be6898-kube-api-access-bb9c4\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:46:01.018126 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:01.017928    2881 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/395493bc-8b7c-4ebc-85e0-0c76a6be6898-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:46:01.018126 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:01.017940    2881 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/395493bc-8b7c-4ebc-85e0-0c76a6be6898-kserve-provision-location\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:46:01.560768 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:01.560739    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-97d9f4864-t4z8v_395493bc-8b7c-4ebc-85e0-0c76a6be6898/storage-initializer/1.log"
Sep 18 17:46:01.560925 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:01.560789    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-97d9f4864-t4z8v" event={"ID":"395493bc-8b7c-4ebc-85e0-0c76a6be6898","Type":"ContainerDied","Data":"1f14c77ab5a8e1e6e446b2fd410454b59f62c8d75826cf80c89b5c192a20667d"}
Sep 18 17:46:01.560925 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:01.560836    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-97d9f4864-t4z8v"
Sep 18 17:46:01.560925 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:01.560854    2881 scope.go:117] "RemoveContainer" containerID="6caf3921985669a1b072bf4566fa2184d54dfb506b8a1e959e0907be9538c24f"
Sep 18 17:46:01.599784 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:01.599753    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-97d9f4864-t4z8v"]
Sep 18 17:46:01.605925 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:01.605894    2881 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-97d9f4864-t4z8v"]
Sep 18 17:46:01.862240 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:01.862138    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5d89dfcbf6-bqmmj"]
Sep 18 17:46:01.862618 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:01.862586    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="395493bc-8b7c-4ebc-85e0-0c76a6be6898" containerName="storage-initializer"
Sep 18 17:46:01.862618 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:01.862602    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="395493bc-8b7c-4ebc-85e0-0c76a6be6898" containerName="storage-initializer"
Sep 18 17:46:01.862710 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:01.862623    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fd1fe5e-7044-4283-aa43-495bce3a18dc" containerName="kserve-container"
Sep 18 17:46:01.862710 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:01.862629    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fd1fe5e-7044-4283-aa43-495bce3a18dc" containerName="kserve-container"
Sep 18 17:46:01.862710 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:01.862638    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc7c7586-87e3-4ce0-9f38-909e21c2dded" containerName="kserve-container"
Sep 18 17:46:01.862710 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:01.862643    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc7c7586-87e3-4ce0-9f38-909e21c2dded" containerName="kserve-container"
Sep 18 17:46:01.862710 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:01.862656    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fd1fe5e-7044-4283-aa43-495bce3a18dc" containerName="kube-rbac-proxy"
Sep 18 17:46:01.862710 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:01.862661    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fd1fe5e-7044-4283-aa43-495bce3a18dc" containerName="kube-rbac-proxy"
Sep 18 17:46:01.862710 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:01.862667    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc7c7586-87e3-4ce0-9f38-909e21c2dded" containerName="storage-initializer"
Sep 18 17:46:01.862710 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:01.862672    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc7c7586-87e3-4ce0-9f38-909e21c2dded" containerName="storage-initializer"
Sep 18 17:46:01.862710 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:01.862677    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc7c7586-87e3-4ce0-9f38-909e21c2dded" containerName="kserve-agent"
Sep 18 17:46:01.862710 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:01.862682    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc7c7586-87e3-4ce0-9f38-909e21c2dded" containerName="kserve-agent"
Sep 18 17:46:01.862710 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:01.862690    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fd1fe5e-7044-4283-aa43-495bce3a18dc" containerName="storage-initializer"
Sep 18 17:46:01.862710 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:01.862696    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fd1fe5e-7044-4283-aa43-495bce3a18dc" containerName="storage-initializer"
Sep 18 17:46:01.862710 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:01.862702    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc7c7586-87e3-4ce0-9f38-909e21c2dded" containerName="kube-rbac-proxy"
Sep 18 17:46:01.862710 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:01.862707    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc7c7586-87e3-4ce0-9f38-909e21c2dded" containerName="kube-rbac-proxy"
Sep 18 17:46:01.863111 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:01.862773    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="395493bc-8b7c-4ebc-85e0-0c76a6be6898" containerName="storage-initializer"
Sep 18 17:46:01.863111 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:01.862781    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc7c7586-87e3-4ce0-9f38-909e21c2dded" containerName="kserve-agent"
Sep 18 17:46:01.863111 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:01.862789    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fd1fe5e-7044-4283-aa43-495bce3a18dc" containerName="kserve-container"
Sep 18 17:46:01.863111 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:01.862795    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fd1fe5e-7044-4283-aa43-495bce3a18dc" containerName="kube-rbac-proxy"
Sep 18 17:46:01.863111 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:01.862800    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="395493bc-8b7c-4ebc-85e0-0c76a6be6898" containerName="storage-initializer"
Sep 18 17:46:01.863111 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:01.862807    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc7c7586-87e3-4ce0-9f38-909e21c2dded" containerName="kserve-container"
Sep 18 17:46:01.863111 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:01.862813    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc7c7586-87e3-4ce0-9f38-909e21c2dded" containerName="kube-rbac-proxy"
Sep 18 17:46:01.863111 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:01.862878    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="395493bc-8b7c-4ebc-85e0-0c76a6be6898" containerName="storage-initializer"
Sep 18 17:46:01.863111 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:01.862884    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="395493bc-8b7c-4ebc-85e0-0c76a6be6898" containerName="storage-initializer"
Sep 18 17:46:01.866839 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:01.866823    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5d89dfcbf6-bqmmj"
Sep 18 17:46:01.869403 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:01.869375    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 18 17:46:01.869557 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:01.869408    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert\""
Sep 18 17:46:01.869686 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:01.869670    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\""
Sep 18 17:46:01.876170 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:01.876142    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5d89dfcbf6-bqmmj"]
Sep 18 17:46:01.894436 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:01.894410    2881 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="395493bc-8b7c-4ebc-85e0-0c76a6be6898" path="/var/lib/kubelet/pods/395493bc-8b7c-4ebc-85e0-0c76a6be6898/volumes"
Sep 18 17:46:01.926300 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:01.926261    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3090f174-c2e3-417f-bbbf-e7bec2757b8f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5d89dfcbf6-bqmmj\" (UID: \"3090f174-c2e3-417f-bbbf-e7bec2757b8f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5d89dfcbf6-bqmmj"
Sep 18 17:46:01.926500 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:01.926322    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3090f174-c2e3-417f-bbbf-e7bec2757b8f-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5d89dfcbf6-bqmmj\" (UID: \"3090f174-c2e3-417f-bbbf-e7bec2757b8f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5d89dfcbf6-bqmmj"
Sep 18 17:46:01.926500 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:01.926359    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3090f174-c2e3-417f-bbbf-e7bec2757b8f-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5d89dfcbf6-bqmmj\" (UID: \"3090f174-c2e3-417f-bbbf-e7bec2757b8f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5d89dfcbf6-bqmmj"
Sep 18 17:46:01.926617 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:01.926502    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgglq\" (UniqueName: \"kubernetes.io/projected/3090f174-c2e3-417f-bbbf-e7bec2757b8f-kube-api-access-bgglq\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5d89dfcbf6-bqmmj\" (UID: \"3090f174-c2e3-417f-bbbf-e7bec2757b8f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5d89dfcbf6-bqmmj"
Sep 18 17:46:01.926617 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:01.926533    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3090f174-c2e3-417f-bbbf-e7bec2757b8f-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5d89dfcbf6-bqmmj\" (UID: \"3090f174-c2e3-417f-bbbf-e7bec2757b8f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5d89dfcbf6-bqmmj"
Sep 18 17:46:02.027028 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:02.026984    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3090f174-c2e3-417f-bbbf-e7bec2757b8f-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5d89dfcbf6-bqmmj\" (UID: \"3090f174-c2e3-417f-bbbf-e7bec2757b8f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5d89dfcbf6-bqmmj"
Sep 18 17:46:02.027028 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:02.027037    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3090f174-c2e3-417f-bbbf-e7bec2757b8f-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5d89dfcbf6-bqmmj\" (UID: \"3090f174-c2e3-417f-bbbf-e7bec2757b8f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5d89dfcbf6-bqmmj"
Sep 18 17:46:02.027341 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:02.027090    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgglq\" (UniqueName: \"kubernetes.io/projected/3090f174-c2e3-417f-bbbf-e7bec2757b8f-kube-api-access-bgglq\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5d89dfcbf6-bqmmj\" (UID: \"3090f174-c2e3-417f-bbbf-e7bec2757b8f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5d89dfcbf6-bqmmj"
Sep 18 17:46:02.027341 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:02.027255    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3090f174-c2e3-417f-bbbf-e7bec2757b8f-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5d89dfcbf6-bqmmj\" (UID: \"3090f174-c2e3-417f-bbbf-e7bec2757b8f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5d89dfcbf6-bqmmj"
Sep 18 17:46:02.027462 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:02.027347    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3090f174-c2e3-417f-bbbf-e7bec2757b8f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5d89dfcbf6-bqmmj\" (UID: \"3090f174-c2e3-417f-bbbf-e7bec2757b8f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5d89dfcbf6-bqmmj"
Sep 18 17:46:02.027680 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:02.027655    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3090f174-c2e3-417f-bbbf-e7bec2757b8f-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5d89dfcbf6-bqmmj\" (UID: \"3090f174-c2e3-417f-bbbf-e7bec2757b8f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5d89dfcbf6-bqmmj"
Sep 18 17:46:02.027836 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:02.027818    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3090f174-c2e3-417f-bbbf-e7bec2757b8f-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5d89dfcbf6-bqmmj\" (UID: \"3090f174-c2e3-417f-bbbf-e7bec2757b8f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5d89dfcbf6-bqmmj"
Sep 18 17:46:02.028054 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:02.028035    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3090f174-c2e3-417f-bbbf-e7bec2757b8f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5d89dfcbf6-bqmmj\" (UID: \"3090f174-c2e3-417f-bbbf-e7bec2757b8f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5d89dfcbf6-bqmmj"
Sep 18 17:46:02.029564 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:02.029533    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3090f174-c2e3-417f-bbbf-e7bec2757b8f-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5d89dfcbf6-bqmmj\" (UID: \"3090f174-c2e3-417f-bbbf-e7bec2757b8f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5d89dfcbf6-bqmmj"
Sep 18 17:46:02.036388 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:02.036365    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgglq\" (UniqueName: \"kubernetes.io/projected/3090f174-c2e3-417f-bbbf-e7bec2757b8f-kube-api-access-bgglq\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5d89dfcbf6-bqmmj\" (UID: \"3090f174-c2e3-417f-bbbf-e7bec2757b8f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5d89dfcbf6-bqmmj"
Sep 18 17:46:02.177579 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:02.177465    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5d89dfcbf6-bqmmj"
Sep 18 17:46:02.454372 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:02.454343    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5d89dfcbf6-bqmmj"]
Sep 18 17:46:02.456581 ip-10-0-143-5 kubenswrapper[2881]: W0918 17:46:02.456552    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3090f174_c2e3_417f_bbbf_e7bec2757b8f.slice/crio-253e895573b91394288f457a91b9d9f8907c096a6ed76a2c8013ccd7f2782c26 WatchSource:0}: Error finding container 253e895573b91394288f457a91b9d9f8907c096a6ed76a2c8013ccd7f2782c26: Status 404 returned error can't find the container with id 253e895573b91394288f457a91b9d9f8907c096a6ed76a2c8013ccd7f2782c26
Sep 18 17:46:02.482939 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:02.482903    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rv4g4" podUID="1e18f677-9b83-4138-9a41-36d19bf881fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 18 17:46:02.571032 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:02.570987    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5d89dfcbf6-bqmmj" event={"ID":"3090f174-c2e3-417f-bbbf-e7bec2757b8f","Type":"ContainerStarted","Data":"54f0b2c492c7aac41e34455425515def23ae5a3ee5e7a8e6d4f91de2b87d3c59"}
Sep 18 17:46:02.571207 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:02.571067    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5d89dfcbf6-bqmmj" event={"ID":"3090f174-c2e3-417f-bbbf-e7bec2757b8f","Type":"ContainerStarted","Data":"253e895573b91394288f457a91b9d9f8907c096a6ed76a2c8013ccd7f2782c26"}
Sep 18 17:46:04.584619 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:04.584586    2881 generic.go:356] "Generic (PLEG): container finished" podID="3090f174-c2e3-417f-bbbf-e7bec2757b8f" containerID="54f0b2c492c7aac41e34455425515def23ae5a3ee5e7a8e6d4f91de2b87d3c59" exitCode=0
Sep 18 17:46:04.584982 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:04.584659    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5d89dfcbf6-bqmmj" event={"ID":"3090f174-c2e3-417f-bbbf-e7bec2757b8f","Type":"ContainerDied","Data":"54f0b2c492c7aac41e34455425515def23ae5a3ee5e7a8e6d4f91de2b87d3c59"}
Sep 18 17:46:05.592645 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:05.592612    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5d89dfcbf6-bqmmj" event={"ID":"3090f174-c2e3-417f-bbbf-e7bec2757b8f","Type":"ContainerStarted","Data":"338b048539a5ae72408976b226ec24516c42e2f07b78e2d62c2f7ef4680b4b62"}
Sep 18 17:46:05.593036 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:05.592679    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5d89dfcbf6-bqmmj" event={"ID":"3090f174-c2e3-417f-bbbf-e7bec2757b8f","Type":"ContainerStarted","Data":"b8234a1c67e91ddcacbadd1edcd7d38fdcc8a36a0d582e92bc6f7b757c4312aa"}
Sep 18 17:46:05.593036 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:05.592875    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5d89dfcbf6-bqmmj"
Sep 18 17:46:05.617909 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:05.617853    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5d89dfcbf6-bqmmj" podStartSLOduration=3.489318923 podStartE2EDuration="4.617837613s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.12851869s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:46:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:46:05.616507547 +0000 UTC m=+3155.816362842" watchObservedRunningTime="2026-09-18 17:46:05.617837613 +0000 UTC m=+3155.817692899"
Sep 18 17:46:06.597082 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:06.597046    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5d89dfcbf6-bqmmj"
Sep 18 17:46:06.598216 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:06.598190    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5d89dfcbf6-bqmmj" podUID="3090f174-c2e3-417f-bbbf-e7bec2757b8f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Sep 18 17:46:07.601899 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:07.601852    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5d89dfcbf6-bqmmj" podUID="3090f174-c2e3-417f-bbbf-e7bec2757b8f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Sep 18 17:46:12.482667 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:12.482631    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rv4g4" podUID="1e18f677-9b83-4138-9a41-36d19bf881fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 18 17:46:12.606258 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:12.606206    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5d89dfcbf6-bqmmj"
Sep 18 17:46:12.606750 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:12.606725    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5d89dfcbf6-bqmmj" podUID="3090f174-c2e3-417f-bbbf-e7bec2757b8f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Sep 18 17:46:22.482419 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:22.482370    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rv4g4" podUID="1e18f677-9b83-4138-9a41-36d19bf881fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 18 17:46:22.606998 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:22.606962    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5d89dfcbf6-bqmmj" podUID="3090f174-c2e3-417f-bbbf-e7bec2757b8f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Sep 18 17:46:32.482365 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:32.482325    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rv4g4" podUID="1e18f677-9b83-4138-9a41-36d19bf881fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 18 17:46:32.607384 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:32.607350    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5d89dfcbf6-bqmmj" podUID="3090f174-c2e3-417f-bbbf-e7bec2757b8f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Sep 18 17:46:42.483965 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:42.483936    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rv4g4"
Sep 18 17:46:42.607300 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:42.607255    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5d89dfcbf6-bqmmj" podUID="3090f174-c2e3-417f-bbbf-e7bec2757b8f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Sep 18 17:46:49.669870 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:49.669830    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rv4g4"]
Sep 18 17:46:49.670435 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:49.670128    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rv4g4" podUID="1e18f677-9b83-4138-9a41-36d19bf881fb" containerName="kserve-container" containerID="cri-o://08ea7bbca0d56fea748a3bbfde00715f54046e1619baa8058a6339fdc3896743" gracePeriod=30
Sep 18 17:46:49.670435 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:49.670175    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rv4g4" podUID="1e18f677-9b83-4138-9a41-36d19bf881fb" containerName="kube-rbac-proxy" containerID="cri-o://acf93b439856898d14f8303c78f01cd6f4bf7a64d9b1c4b58a589d757e72e133" gracePeriod=30
Sep 18 17:46:49.798511 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:49.798481    2881 generic.go:356] "Generic (PLEG): container finished" podID="1e18f677-9b83-4138-9a41-36d19bf881fb" containerID="acf93b439856898d14f8303c78f01cd6f4bf7a64d9b1c4b58a589d757e72e133" exitCode=2
Sep 18 17:46:49.798676 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:49.798552    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rv4g4" event={"ID":"1e18f677-9b83-4138-9a41-36d19bf881fb","Type":"ContainerDied","Data":"acf93b439856898d14f8303c78f01cd6f4bf7a64d9b1c4b58a589d757e72e133"}
Sep 18 17:46:52.477671 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:52.477631    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rv4g4" podUID="1e18f677-9b83-4138-9a41-36d19bf881fb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.64:8643/healthz\": dial tcp 10.134.0.64:8643: connect: connection refused"
Sep 18 17:46:52.482724 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:52.482695    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rv4g4" podUID="1e18f677-9b83-4138-9a41-36d19bf881fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 18 17:46:52.606964 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:52.606924    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5d89dfcbf6-bqmmj" podUID="3090f174-c2e3-417f-bbbf-e7bec2757b8f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Sep 18 17:46:52.813324 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:52.813293    2881 generic.go:356] "Generic (PLEG): container finished" podID="1e18f677-9b83-4138-9a41-36d19bf881fb" containerID="08ea7bbca0d56fea748a3bbfde00715f54046e1619baa8058a6339fdc3896743" exitCode=0
Sep 18 17:46:52.813469 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:52.813364    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rv4g4" event={"ID":"1e18f677-9b83-4138-9a41-36d19bf881fb","Type":"ContainerDied","Data":"08ea7bbca0d56fea748a3bbfde00715f54046e1619baa8058a6339fdc3896743"}
Sep 18 17:46:52.813469 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:52.813420    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rv4g4" event={"ID":"1e18f677-9b83-4138-9a41-36d19bf881fb","Type":"ContainerDied","Data":"97d1a35b2d5e3c116f194cbf33d8d7bb842edc7d50a0508a7233316331ae774f"}
Sep 18 17:46:52.813469 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:52.813439    2881 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97d1a35b2d5e3c116f194cbf33d8d7bb842edc7d50a0508a7233316331ae774f"
Sep 18 17:46:52.814496 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:52.814481    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rv4g4"
Sep 18 17:46:52.886285 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:52.886250    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kwdj5\" (UniqueName: \"kubernetes.io/projected/1e18f677-9b83-4138-9a41-36d19bf881fb-kube-api-access-kwdj5\") pod \"1e18f677-9b83-4138-9a41-36d19bf881fb\" (UID: \"1e18f677-9b83-4138-9a41-36d19bf881fb\") "
Sep 18 17:46:52.886466 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:52.886307    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e18f677-9b83-4138-9a41-36d19bf881fb-kserve-provision-location\") pod \"1e18f677-9b83-4138-9a41-36d19bf881fb\" (UID: \"1e18f677-9b83-4138-9a41-36d19bf881fb\") "
Sep 18 17:46:52.886466 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:52.886335    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e18f677-9b83-4138-9a41-36d19bf881fb-kube-rbac-proxy-sar-config\") pod \"1e18f677-9b83-4138-9a41-36d19bf881fb\" (UID: \"1e18f677-9b83-4138-9a41-36d19bf881fb\") "
Sep 18 17:46:52.886466 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:52.886358    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e18f677-9b83-4138-9a41-36d19bf881fb-proxy-tls\") pod \"1e18f677-9b83-4138-9a41-36d19bf881fb\" (UID: \"1e18f677-9b83-4138-9a41-36d19bf881fb\") "
Sep 18 17:46:52.886466 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:52.886388    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e18f677-9b83-4138-9a41-36d19bf881fb-cabundle-cert\") pod \"1e18f677-9b83-4138-9a41-36d19bf881fb\" (UID: \"1e18f677-9b83-4138-9a41-36d19bf881fb\") "
Sep 18 17:46:52.886704 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:52.886676    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e18f677-9b83-4138-9a41-36d19bf881fb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1e18f677-9b83-4138-9a41-36d19bf881fb" (UID: "1e18f677-9b83-4138-9a41-36d19bf881fb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:46:52.886894 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:52.886869    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e18f677-9b83-4138-9a41-36d19bf881fb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1e18f677-9b83-4138-9a41-36d19bf881fb" (UID: "1e18f677-9b83-4138-9a41-36d19bf881fb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:46:52.888549 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:52.888521    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e18f677-9b83-4138-9a41-36d19bf881fb-kube-api-access-kwdj5" (OuterVolumeSpecName: "kube-api-access-kwdj5") pod "1e18f677-9b83-4138-9a41-36d19bf881fb" (UID: "1e18f677-9b83-4138-9a41-36d19bf881fb"). InnerVolumeSpecName "kube-api-access-kwdj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:46:52.888653 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:52.888560    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e18f677-9b83-4138-9a41-36d19bf881fb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1e18f677-9b83-4138-9a41-36d19bf881fb" (UID: "1e18f677-9b83-4138-9a41-36d19bf881fb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:46:52.895203 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:52.895172    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e18f677-9b83-4138-9a41-36d19bf881fb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1e18f677-9b83-4138-9a41-36d19bf881fb" (UID: "1e18f677-9b83-4138-9a41-36d19bf881fb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:46:52.987289 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:52.987196    2881 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwdj5\" (UniqueName: \"kubernetes.io/projected/1e18f677-9b83-4138-9a41-36d19bf881fb-kube-api-access-kwdj5\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:46:52.987289 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:52.987235    2881 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e18f677-9b83-4138-9a41-36d19bf881fb-kserve-provision-location\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:46:52.987289 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:52.987246    2881 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e18f677-9b83-4138-9a41-36d19bf881fb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:46:52.987289 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:52.987256    2881 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e18f677-9b83-4138-9a41-36d19bf881fb-proxy-tls\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:46:52.987289 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:52.987265    2881 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e18f677-9b83-4138-9a41-36d19bf881fb-cabundle-cert\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:46:53.817243 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:53.817197    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rv4g4"
Sep 18 17:46:53.842344 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:53.842312    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rv4g4"]
Sep 18 17:46:53.848951 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:53.848921    2881 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rv4g4"]
Sep 18 17:46:53.893075 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:53.893042    2881 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e18f677-9b83-4138-9a41-36d19bf881fb" path="/var/lib/kubelet/pods/1e18f677-9b83-4138-9a41-36d19bf881fb/volumes"
Sep 18 17:46:54.887366 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:54.887334    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-xpbh6"]
Sep 18 17:46:54.887778 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:54.887685    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e18f677-9b83-4138-9a41-36d19bf881fb" containerName="storage-initializer"
Sep 18 17:46:54.887778 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:54.887695    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e18f677-9b83-4138-9a41-36d19bf881fb" containerName="storage-initializer"
Sep 18 17:46:54.887778 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:54.887707    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e18f677-9b83-4138-9a41-36d19bf881fb" containerName="kube-rbac-proxy"
Sep 18 17:46:54.887778 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:54.887714    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e18f677-9b83-4138-9a41-36d19bf881fb" containerName="kube-rbac-proxy"
Sep 18 17:46:54.887778 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:54.887725    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e18f677-9b83-4138-9a41-36d19bf881fb" containerName="kserve-container"
Sep 18 17:46:54.887778 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:54.887731    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e18f677-9b83-4138-9a41-36d19bf881fb" containerName="kserve-container"
Sep 18 17:46:54.888029 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:54.887786    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e18f677-9b83-4138-9a41-36d19bf881fb" containerName="kserve-container"
Sep 18 17:46:54.888029 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:54.887795    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e18f677-9b83-4138-9a41-36d19bf881fb" containerName="kube-rbac-proxy"
Sep 18 17:46:54.891147 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:54.891129    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-xpbh6"
Sep 18 17:46:54.895202 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:54.895181    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 17:46:54.895366 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:54.895239    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-predictor-serving-cert\""
Sep 18 17:46:54.895509 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:54.895490    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-kube-rbac-proxy-sar-config\""
Sep 18 17:46:54.903160 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:54.903132    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-xpbh6"]
Sep 18 17:46:55.001948 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:55.001907    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/244d1afa-7185-49c2-8cb3-4e48ba7b5aa7-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-xpbh6\" (UID: \"244d1afa-7185-49c2-8cb3-4e48ba7b5aa7\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-xpbh6"
Sep 18 17:46:55.002245 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:55.001974    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2w5k\" (UniqueName: \"kubernetes.io/projected/244d1afa-7185-49c2-8cb3-4e48ba7b5aa7-kube-api-access-h2w5k\") pod \"isvc-paddle-runtime-predictor-58b99fff76-xpbh6\" (UID: \"244d1afa-7185-49c2-8cb3-4e48ba7b5aa7\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-xpbh6"
Sep 18 17:46:55.002245 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:55.002005    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/244d1afa-7185-49c2-8cb3-4e48ba7b5aa7-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-xpbh6\" (UID: \"244d1afa-7185-49c2-8cb3-4e48ba7b5aa7\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-xpbh6"
Sep 18 17:46:55.002245 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:55.002034    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/244d1afa-7185-49c2-8cb3-4e48ba7b5aa7-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-xpbh6\" (UID: \"244d1afa-7185-49c2-8cb3-4e48ba7b5aa7\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-xpbh6"
Sep 18 17:46:55.002245 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:55.002095    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/244d1afa-7185-49c2-8cb3-4e48ba7b5aa7-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-xpbh6\" (UID: \"244d1afa-7185-49c2-8cb3-4e48ba7b5aa7\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-xpbh6"
Sep 18 17:46:55.103300 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:55.103257    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/244d1afa-7185-49c2-8cb3-4e48ba7b5aa7-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-xpbh6\" (UID: \"244d1afa-7185-49c2-8cb3-4e48ba7b5aa7\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-xpbh6"
Sep 18 17:46:55.103515 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:55.103332    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2w5k\" (UniqueName: \"kubernetes.io/projected/244d1afa-7185-49c2-8cb3-4e48ba7b5aa7-kube-api-access-h2w5k\") pod \"isvc-paddle-runtime-predictor-58b99fff76-xpbh6\" (UID: \"244d1afa-7185-49c2-8cb3-4e48ba7b5aa7\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-xpbh6"
Sep 18 17:46:55.103515 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:55.103360    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/244d1afa-7185-49c2-8cb3-4e48ba7b5aa7-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-xpbh6\" (UID: \"244d1afa-7185-49c2-8cb3-4e48ba7b5aa7\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-xpbh6"
Sep 18 17:46:55.103515 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:55.103388    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/244d1afa-7185-49c2-8cb3-4e48ba7b5aa7-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-xpbh6\" (UID: \"244d1afa-7185-49c2-8cb3-4e48ba7b5aa7\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-xpbh6"
Sep 18 17:46:55.103515 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:55.103420    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/244d1afa-7185-49c2-8cb3-4e48ba7b5aa7-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-xpbh6\" (UID: \"244d1afa-7185-49c2-8cb3-4e48ba7b5aa7\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-xpbh6"
Sep 18 17:46:55.103788 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:55.103763    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/244d1afa-7185-49c2-8cb3-4e48ba7b5aa7-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-xpbh6\" (UID: \"244d1afa-7185-49c2-8cb3-4e48ba7b5aa7\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-xpbh6"
Sep 18 17:46:55.104088 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:55.104066    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/244d1afa-7185-49c2-8cb3-4e48ba7b5aa7-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-xpbh6\" (UID: \"244d1afa-7185-49c2-8cb3-4e48ba7b5aa7\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-xpbh6"
Sep 18 17:46:55.104175 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:55.104156    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/244d1afa-7185-49c2-8cb3-4e48ba7b5aa7-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-xpbh6\" (UID: \"244d1afa-7185-49c2-8cb3-4e48ba7b5aa7\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-xpbh6"
Sep 18 17:46:55.105930 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:55.105906    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/244d1afa-7185-49c2-8cb3-4e48ba7b5aa7-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-xpbh6\" (UID: \"244d1afa-7185-49c2-8cb3-4e48ba7b5aa7\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-xpbh6"
Sep 18 17:46:55.113255 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:55.113215    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2w5k\" (UniqueName: \"kubernetes.io/projected/244d1afa-7185-49c2-8cb3-4e48ba7b5aa7-kube-api-access-h2w5k\") pod \"isvc-paddle-runtime-predictor-58b99fff76-xpbh6\" (UID: \"244d1afa-7185-49c2-8cb3-4e48ba7b5aa7\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-xpbh6"
Sep 18 17:46:55.201748 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:55.201650    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-xpbh6"
Sep 18 17:46:55.478258 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:55.478217    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-xpbh6"]
Sep 18 17:46:55.480316 ip-10-0-143-5 kubenswrapper[2881]: W0918 17:46:55.480270    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod244d1afa_7185_49c2_8cb3_4e48ba7b5aa7.slice/crio-8db80d8f8d6aa62f33e0f9374e5529469a2587edfd606f9659f6f87ff37cc322 WatchSource:0}: Error finding container 8db80d8f8d6aa62f33e0f9374e5529469a2587edfd606f9659f6f87ff37cc322: Status 404 returned error can't find the container with id 8db80d8f8d6aa62f33e0f9374e5529469a2587edfd606f9659f6f87ff37cc322
Sep 18 17:46:55.828891 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:55.828856    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-xpbh6" event={"ID":"244d1afa-7185-49c2-8cb3-4e48ba7b5aa7","Type":"ContainerStarted","Data":"7f9e69d027eecc2babe3a0b337ad9a50d7b9c88da3d3f27e04c91172673ac05b"}
Sep 18 17:46:55.829075 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:46:55.828915    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-xpbh6" event={"ID":"244d1afa-7185-49c2-8cb3-4e48ba7b5aa7","Type":"ContainerStarted","Data":"8db80d8f8d6aa62f33e0f9374e5529469a2587edfd606f9659f6f87ff37cc322"}
Sep 18 17:47:00.853491 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:00.853457    2881 generic.go:356] "Generic (PLEG): container finished" podID="244d1afa-7185-49c2-8cb3-4e48ba7b5aa7" containerID="7f9e69d027eecc2babe3a0b337ad9a50d7b9c88da3d3f27e04c91172673ac05b" exitCode=0
Sep 18 17:47:00.853877 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:00.853524    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-xpbh6" event={"ID":"244d1afa-7185-49c2-8cb3-4e48ba7b5aa7","Type":"ContainerDied","Data":"7f9e69d027eecc2babe3a0b337ad9a50d7b9c88da3d3f27e04c91172673ac05b"}
Sep 18 17:47:01.860478 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:01.860439    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-xpbh6" event={"ID":"244d1afa-7185-49c2-8cb3-4e48ba7b5aa7","Type":"ContainerStarted","Data":"846d5cafdccfe893858008cf948485728b08cd63ec1948288e8541ac7330c806"}
Sep 18 17:47:01.860888 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:01.860502    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-xpbh6" event={"ID":"244d1afa-7185-49c2-8cb3-4e48ba7b5aa7","Type":"ContainerStarted","Data":"0d4c7ba16dacbb937e7e7f3bf2e72655b19795cb14c76334e0a9a155f6179f60"}
Sep 18 17:47:01.860888 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:01.860693    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-xpbh6"
Sep 18 17:47:01.899938 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:01.899883    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-xpbh6" podStartSLOduration=3.103999885 podStartE2EDuration="7.899864894s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.795865009s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:46:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:47:01.896081107 +0000 UTC m=+3212.095936393" watchObservedRunningTime="2026-09-18 17:47:01.899864894 +0000 UTC m=+3212.099720179"
Sep 18 17:47:02.607613 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:02.607564    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5d89dfcbf6-bqmmj" podUID="3090f174-c2e3-417f-bbbf-e7bec2757b8f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Sep 18 17:47:02.865512 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:02.865427    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-xpbh6"
Sep 18 17:47:02.866588 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:02.866562    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-xpbh6" podUID="244d1afa-7185-49c2-8cb3-4e48ba7b5aa7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Sep 18 17:47:03.870053 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:03.870012    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-xpbh6" podUID="244d1afa-7185-49c2-8cb3-4e48ba7b5aa7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Sep 18 17:47:08.874145 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:08.874112    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-xpbh6"
Sep 18 17:47:08.874708 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:08.874607    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-xpbh6" podUID="244d1afa-7185-49c2-8cb3-4e48ba7b5aa7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Sep 18 17:47:12.607695 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:12.607655    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5d89dfcbf6-bqmmj" podUID="3090f174-c2e3-417f-bbbf-e7bec2757b8f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Sep 18 17:47:18.875023 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:18.874983    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-xpbh6" podUID="244d1afa-7185-49c2-8cb3-4e48ba7b5aa7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Sep 18 17:47:22.607517 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:22.607472    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5d89dfcbf6-bqmmj" podUID="3090f174-c2e3-417f-bbbf-e7bec2757b8f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Sep 18 17:47:28.875198 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:28.875156    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-xpbh6" podUID="244d1afa-7185-49c2-8cb3-4e48ba7b5aa7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Sep 18 17:47:32.608143 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:32.608110    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5d89dfcbf6-bqmmj"
Sep 18 17:47:38.875099 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:38.875059    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-xpbh6" podUID="244d1afa-7185-49c2-8cb3-4e48ba7b5aa7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Sep 18 17:47:41.893662 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:41.893627    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5d89dfcbf6-bqmmj"]
Sep 18 17:47:41.894054 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:41.893906    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5d89dfcbf6-bqmmj" podUID="3090f174-c2e3-417f-bbbf-e7bec2757b8f" containerName="kserve-container" containerID="cri-o://b8234a1c67e91ddcacbadd1edcd7d38fdcc8a36a0d582e92bc6f7b757c4312aa" gracePeriod=30
Sep 18 17:47:41.894054 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:41.893904    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5d89dfcbf6-bqmmj" podUID="3090f174-c2e3-417f-bbbf-e7bec2757b8f" containerName="kube-rbac-proxy" containerID="cri-o://338b048539a5ae72408976b226ec24516c42e2f07b78e2d62c2f7ef4680b4b62" gracePeriod=30
Sep 18 17:47:42.057200 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:42.057164    2881 generic.go:356] "Generic (PLEG): container finished" podID="3090f174-c2e3-417f-bbbf-e7bec2757b8f" containerID="338b048539a5ae72408976b226ec24516c42e2f07b78e2d62c2f7ef4680b4b62" exitCode=2
Sep 18 17:47:42.057393 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:42.057263    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5d89dfcbf6-bqmmj" event={"ID":"3090f174-c2e3-417f-bbbf-e7bec2757b8f","Type":"ContainerDied","Data":"338b048539a5ae72408976b226ec24516c42e2f07b78e2d62c2f7ef4680b4b62"}
Sep 18 17:47:42.602551 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:42.602509    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5d89dfcbf6-bqmmj" podUID="3090f174-c2e3-417f-bbbf-e7bec2757b8f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.66:8643/healthz\": dial tcp 10.134.0.66:8643: connect: connection refused"
Sep 18 17:47:42.606897 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:42.606868    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5d89dfcbf6-bqmmj" podUID="3090f174-c2e3-417f-bbbf-e7bec2757b8f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Sep 18 17:47:43.049219 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:43.049183    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-c8cc6df88-nh9mz"]
Sep 18 17:47:43.053194 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:43.053170    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-c8cc6df88-nh9mz"
Sep 18 17:47:43.055709 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:43.055689    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert\""
Sep 18 17:47:43.055912 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:43.055896    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\""
Sep 18 17:47:43.063421 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:43.063398    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-c8cc6df88-nh9mz"]
Sep 18 17:47:43.141257 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:43.141212    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55d77ce8-c8c8-4093-88a6-cd08da6f4870-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-c8cc6df88-nh9mz\" (UID: \"55d77ce8-c8c8-4093-88a6-cd08da6f4870\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-c8cc6df88-nh9mz"
Sep 18 17:47:43.141456 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:43.141271    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55d77ce8-c8c8-4093-88a6-cd08da6f4870-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-c8cc6df88-nh9mz\" (UID: \"55d77ce8-c8c8-4093-88a6-cd08da6f4870\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-c8cc6df88-nh9mz"
Sep 18 17:47:43.141456 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:43.141343    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6546n\" (UniqueName: \"kubernetes.io/projected/55d77ce8-c8c8-4093-88a6-cd08da6f4870-kube-api-access-6546n\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-c8cc6df88-nh9mz\" (UID: \"55d77ce8-c8c8-4093-88a6-cd08da6f4870\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-c8cc6df88-nh9mz"
Sep 18 17:47:43.141456 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:43.141391    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55d77ce8-c8c8-4093-88a6-cd08da6f4870-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-c8cc6df88-nh9mz\" (UID: \"55d77ce8-c8c8-4093-88a6-cd08da6f4870\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-c8cc6df88-nh9mz"
Sep 18 17:47:43.242249 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:43.242198    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6546n\" (UniqueName: \"kubernetes.io/projected/55d77ce8-c8c8-4093-88a6-cd08da6f4870-kube-api-access-6546n\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-c8cc6df88-nh9mz\" (UID: \"55d77ce8-c8c8-4093-88a6-cd08da6f4870\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-c8cc6df88-nh9mz"
Sep 18 17:47:43.242453 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:43.242360    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55d77ce8-c8c8-4093-88a6-cd08da6f4870-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-c8cc6df88-nh9mz\" (UID: \"55d77ce8-c8c8-4093-88a6-cd08da6f4870\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-c8cc6df88-nh9mz"
Sep 18 17:47:43.242453 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:43.242441    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55d77ce8-c8c8-4093-88a6-cd08da6f4870-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-c8cc6df88-nh9mz\" (UID: \"55d77ce8-c8c8-4093-88a6-cd08da6f4870\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-c8cc6df88-nh9mz"
Sep 18 17:47:43.242578 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:43.242466    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55d77ce8-c8c8-4093-88a6-cd08da6f4870-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-c8cc6df88-nh9mz\" (UID: \"55d77ce8-c8c8-4093-88a6-cd08da6f4870\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-c8cc6df88-nh9mz"
Sep 18 17:47:43.242941 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:43.242921    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55d77ce8-c8c8-4093-88a6-cd08da6f4870-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-c8cc6df88-nh9mz\" (UID: \"55d77ce8-c8c8-4093-88a6-cd08da6f4870\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-c8cc6df88-nh9mz"
Sep 18 17:47:43.243395 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:43.243368    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55d77ce8-c8c8-4093-88a6-cd08da6f4870-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-c8cc6df88-nh9mz\" (UID: \"55d77ce8-c8c8-4093-88a6-cd08da6f4870\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-c8cc6df88-nh9mz"
Sep 18 17:47:43.244816 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:43.244793    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55d77ce8-c8c8-4093-88a6-cd08da6f4870-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-c8cc6df88-nh9mz\" (UID: \"55d77ce8-c8c8-4093-88a6-cd08da6f4870\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-c8cc6df88-nh9mz"
Sep 18 17:47:43.251947 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:43.251926    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6546n\" (UniqueName: \"kubernetes.io/projected/55d77ce8-c8c8-4093-88a6-cd08da6f4870-kube-api-access-6546n\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-c8cc6df88-nh9mz\" (UID: \"55d77ce8-c8c8-4093-88a6-cd08da6f4870\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-c8cc6df88-nh9mz"
Sep 18 17:47:43.364089 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:43.363993    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-c8cc6df88-nh9mz"
Sep 18 17:47:43.643979 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:43.643934    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-c8cc6df88-nh9mz"]
Sep 18 17:47:43.646578 ip-10-0-143-5 kubenswrapper[2881]: W0918 17:47:43.646545    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55d77ce8_c8c8_4093_88a6_cd08da6f4870.slice/crio-cb27a23aa4d22e9ddc1b13d47bce66637785ddaaf060f5306b20035a67a71d34 WatchSource:0}: Error finding container cb27a23aa4d22e9ddc1b13d47bce66637785ddaaf060f5306b20035a67a71d34: Status 404 returned error can't find the container with id cb27a23aa4d22e9ddc1b13d47bce66637785ddaaf060f5306b20035a67a71d34
Sep 18 17:47:44.070922 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:44.070880    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-c8cc6df88-nh9mz" event={"ID":"55d77ce8-c8c8-4093-88a6-cd08da6f4870","Type":"ContainerStarted","Data":"083d1313acb540b5e1cf304b1d6d66a650e3b68a0745cd98f60e06bba5cfe3aa"}
Sep 18 17:47:44.071347 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:44.070963    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-c8cc6df88-nh9mz" event={"ID":"55d77ce8-c8c8-4093-88a6-cd08da6f4870","Type":"ContainerStarted","Data":"cb27a23aa4d22e9ddc1b13d47bce66637785ddaaf060f5306b20035a67a71d34"}
Sep 18 17:47:47.247189 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:47.247162    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5d89dfcbf6-bqmmj"
Sep 18 17:47:47.380731 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:47.380628    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bgglq\" (UniqueName: \"kubernetes.io/projected/3090f174-c2e3-417f-bbbf-e7bec2757b8f-kube-api-access-bgglq\") pod \"3090f174-c2e3-417f-bbbf-e7bec2757b8f\" (UID: \"3090f174-c2e3-417f-bbbf-e7bec2757b8f\") "
Sep 18 17:47:47.380731 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:47.380711    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3090f174-c2e3-417f-bbbf-e7bec2757b8f-cabundle-cert\") pod \"3090f174-c2e3-417f-bbbf-e7bec2757b8f\" (UID: \"3090f174-c2e3-417f-bbbf-e7bec2757b8f\") "
Sep 18 17:47:47.380975 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:47.380740    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3090f174-c2e3-417f-bbbf-e7bec2757b8f-kube-rbac-proxy-sar-config\") pod \"3090f174-c2e3-417f-bbbf-e7bec2757b8f\" (UID: \"3090f174-c2e3-417f-bbbf-e7bec2757b8f\") "
Sep 18 17:47:47.380975 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:47.380876    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3090f174-c2e3-417f-bbbf-e7bec2757b8f-kserve-provision-location\") pod \"3090f174-c2e3-417f-bbbf-e7bec2757b8f\" (UID: \"3090f174-c2e3-417f-bbbf-e7bec2757b8f\") "
Sep 18 17:47:47.381087 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:47.380974    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3090f174-c2e3-417f-bbbf-e7bec2757b8f-proxy-tls\") pod \"3090f174-c2e3-417f-bbbf-e7bec2757b8f\" (UID: \"3090f174-c2e3-417f-bbbf-e7bec2757b8f\") "
Sep 18 17:47:47.381158 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:47.381119    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3090f174-c2e3-417f-bbbf-e7bec2757b8f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3090f174-c2e3-417f-bbbf-e7bec2757b8f" (UID: "3090f174-c2e3-417f-bbbf-e7bec2757b8f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:47:47.381203 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:47.381160    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3090f174-c2e3-417f-bbbf-e7bec2757b8f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3090f174-c2e3-417f-bbbf-e7bec2757b8f" (UID: "3090f174-c2e3-417f-bbbf-e7bec2757b8f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:47:47.381203 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:47.381171    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3090f174-c2e3-417f-bbbf-e7bec2757b8f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3090f174-c2e3-417f-bbbf-e7bec2757b8f" (UID: "3090f174-c2e3-417f-bbbf-e7bec2757b8f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:47:47.381374 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:47.381354    2881 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3090f174-c2e3-417f-bbbf-e7bec2757b8f-cabundle-cert\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:47:47.381374 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:47.381373    2881 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3090f174-c2e3-417f-bbbf-e7bec2757b8f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:47:47.381534 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:47.381382    2881 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3090f174-c2e3-417f-bbbf-e7bec2757b8f-kserve-provision-location\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:47:47.382951 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:47.382923    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3090f174-c2e3-417f-bbbf-e7bec2757b8f-kube-api-access-bgglq" (OuterVolumeSpecName: "kube-api-access-bgglq") pod "3090f174-c2e3-417f-bbbf-e7bec2757b8f" (UID: "3090f174-c2e3-417f-bbbf-e7bec2757b8f"). InnerVolumeSpecName "kube-api-access-bgglq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:47:47.383064 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:47.382956    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3090f174-c2e3-417f-bbbf-e7bec2757b8f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3090f174-c2e3-417f-bbbf-e7bec2757b8f" (UID: "3090f174-c2e3-417f-bbbf-e7bec2757b8f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:47:47.482368 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:47.482319    2881 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3090f174-c2e3-417f-bbbf-e7bec2757b8f-proxy-tls\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:47:47.482368 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:47.482364    2881 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgglq\" (UniqueName: \"kubernetes.io/projected/3090f174-c2e3-417f-bbbf-e7bec2757b8f-kube-api-access-bgglq\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:47:48.092349 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:48.092316    2881 generic.go:356] "Generic (PLEG): container finished" podID="3090f174-c2e3-417f-bbbf-e7bec2757b8f" containerID="b8234a1c67e91ddcacbadd1edcd7d38fdcc8a36a0d582e92bc6f7b757c4312aa" exitCode=0
Sep 18 17:47:48.092506 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:48.092377    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5d89dfcbf6-bqmmj"
Sep 18 17:47:48.092506 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:48.092407    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5d89dfcbf6-bqmmj" event={"ID":"3090f174-c2e3-417f-bbbf-e7bec2757b8f","Type":"ContainerDied","Data":"b8234a1c67e91ddcacbadd1edcd7d38fdcc8a36a0d582e92bc6f7b757c4312aa"}
Sep 18 17:47:48.092506 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:48.092483    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5d89dfcbf6-bqmmj" event={"ID":"3090f174-c2e3-417f-bbbf-e7bec2757b8f","Type":"ContainerDied","Data":"253e895573b91394288f457a91b9d9f8907c096a6ed76a2c8013ccd7f2782c26"}
Sep 18 17:47:48.092639 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:48.092521    2881 scope.go:117] "RemoveContainer" containerID="338b048539a5ae72408976b226ec24516c42e2f07b78e2d62c2f7ef4680b4b62"
Sep 18 17:47:48.101776 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:48.101752    2881 scope.go:117] "RemoveContainer" containerID="b8234a1c67e91ddcacbadd1edcd7d38fdcc8a36a0d582e92bc6f7b757c4312aa"
Sep 18 17:47:48.109971 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:48.109948    2881 scope.go:117] "RemoveContainer" containerID="54f0b2c492c7aac41e34455425515def23ae5a3ee5e7a8e6d4f91de2b87d3c59"
Sep 18 17:47:48.113910 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:48.113881    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5d89dfcbf6-bqmmj"]
Sep 18 17:47:48.118109 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:48.118084    2881 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5d89dfcbf6-bqmmj"]
Sep 18 17:47:48.119328 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:48.119304    2881 scope.go:117] "RemoveContainer" containerID="338b048539a5ae72408976b226ec24516c42e2f07b78e2d62c2f7ef4680b4b62"
Sep 18 17:47:48.119636 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:47:48.119615    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"338b048539a5ae72408976b226ec24516c42e2f07b78e2d62c2f7ef4680b4b62\": container with ID starting with 338b048539a5ae72408976b226ec24516c42e2f07b78e2d62c2f7ef4680b4b62 not found: ID does not exist" containerID="338b048539a5ae72408976b226ec24516c42e2f07b78e2d62c2f7ef4680b4b62"
Sep 18 17:47:48.119704 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:48.119652    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"338b048539a5ae72408976b226ec24516c42e2f07b78e2d62c2f7ef4680b4b62"} err="failed to get container status \"338b048539a5ae72408976b226ec24516c42e2f07b78e2d62c2f7ef4680b4b62\": rpc error: code = NotFound desc = could not find container \"338b048539a5ae72408976b226ec24516c42e2f07b78e2d62c2f7ef4680b4b62\": container with ID starting with 338b048539a5ae72408976b226ec24516c42e2f07b78e2d62c2f7ef4680b4b62 not found: ID does not exist"
Sep 18 17:47:48.119704 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:48.119679    2881 scope.go:117] "RemoveContainer" containerID="b8234a1c67e91ddcacbadd1edcd7d38fdcc8a36a0d582e92bc6f7b757c4312aa"
Sep 18 17:47:48.119998 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:47:48.119980    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8234a1c67e91ddcacbadd1edcd7d38fdcc8a36a0d582e92bc6f7b757c4312aa\": container with ID starting with b8234a1c67e91ddcacbadd1edcd7d38fdcc8a36a0d582e92bc6f7b757c4312aa not found: ID does not exist" containerID="b8234a1c67e91ddcacbadd1edcd7d38fdcc8a36a0d582e92bc6f7b757c4312aa"
Sep 18 17:47:48.120045 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:48.120004    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8234a1c67e91ddcacbadd1edcd7d38fdcc8a36a0d582e92bc6f7b757c4312aa"} err="failed to get container status \"b8234a1c67e91ddcacbadd1edcd7d38fdcc8a36a0d582e92bc6f7b757c4312aa\": rpc error: code = NotFound desc = could not find container \"b8234a1c67e91ddcacbadd1edcd7d38fdcc8a36a0d582e92bc6f7b757c4312aa\": container with ID starting with b8234a1c67e91ddcacbadd1edcd7d38fdcc8a36a0d582e92bc6f7b757c4312aa not found: ID does not exist"
Sep 18 17:47:48.120045 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:48.120019    2881 scope.go:117] "RemoveContainer" containerID="54f0b2c492c7aac41e34455425515def23ae5a3ee5e7a8e6d4f91de2b87d3c59"
Sep 18 17:47:48.120270 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:47:48.120251    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54f0b2c492c7aac41e34455425515def23ae5a3ee5e7a8e6d4f91de2b87d3c59\": container with ID starting with 54f0b2c492c7aac41e34455425515def23ae5a3ee5e7a8e6d4f91de2b87d3c59 not found: ID does not exist" containerID="54f0b2c492c7aac41e34455425515def23ae5a3ee5e7a8e6d4f91de2b87d3c59"
Sep 18 17:47:48.120327 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:48.120276    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54f0b2c492c7aac41e34455425515def23ae5a3ee5e7a8e6d4f91de2b87d3c59"} err="failed to get container status \"54f0b2c492c7aac41e34455425515def23ae5a3ee5e7a8e6d4f91de2b87d3c59\": rpc error: code = NotFound desc = could not find container \"54f0b2c492c7aac41e34455425515def23ae5a3ee5e7a8e6d4f91de2b87d3c59\": container with ID starting with 54f0b2c492c7aac41e34455425515def23ae5a3ee5e7a8e6d4f91de2b87d3c59 not found: ID does not exist"
Sep 18 17:47:48.875611 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:48.875573    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-xpbh6" podUID="244d1afa-7185-49c2-8cb3-4e48ba7b5aa7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Sep 18 17:47:49.894216 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:49.894133    2881 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3090f174-c2e3-417f-bbbf-e7bec2757b8f" path="/var/lib/kubelet/pods/3090f174-c2e3-417f-bbbf-e7bec2757b8f/volumes"
Sep 18 17:47:50.105182 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:50.105151    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-c8cc6df88-nh9mz_55d77ce8-c8c8-4093-88a6-cd08da6f4870/storage-initializer/0.log"
Sep 18 17:47:50.105370 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:50.105196    2881 generic.go:356] "Generic (PLEG): container finished" podID="55d77ce8-c8c8-4093-88a6-cd08da6f4870" containerID="083d1313acb540b5e1cf304b1d6d66a650e3b68a0745cd98f60e06bba5cfe3aa" exitCode=1
Sep 18 17:47:50.105370 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:50.105282    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-c8cc6df88-nh9mz" event={"ID":"55d77ce8-c8c8-4093-88a6-cd08da6f4870","Type":"ContainerDied","Data":"083d1313acb540b5e1cf304b1d6d66a650e3b68a0745cd98f60e06bba5cfe3aa"}
Sep 18 17:47:51.111684 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:51.111648    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-c8cc6df88-nh9mz_55d77ce8-c8c8-4093-88a6-cd08da6f4870/storage-initializer/0.log"
Sep 18 17:47:51.112195 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:51.111712    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-c8cc6df88-nh9mz" event={"ID":"55d77ce8-c8c8-4093-88a6-cd08da6f4870","Type":"ContainerStarted","Data":"f9454d2e47bf587a006f62fda5e9d03ec01031abc6dadc03f12d497ac3ee7564"}
Sep 18 17:47:52.986582 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:52.986544    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-c8cc6df88-nh9mz"]
Sep 18 17:47:52.986982 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:52.986787    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-c8cc6df88-nh9mz" podUID="55d77ce8-c8c8-4093-88a6-cd08da6f4870" containerName="storage-initializer" containerID="cri-o://f9454d2e47bf587a006f62fda5e9d03ec01031abc6dadc03f12d497ac3ee7564" gracePeriod=30
Sep 18 17:47:54.186682 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:54.186644    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-f9f48c85f-x4cmz"]
Sep 18 17:47:54.187179 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:54.186999    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3090f174-c2e3-417f-bbbf-e7bec2757b8f" containerName="kube-rbac-proxy"
Sep 18 17:47:54.187179 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:54.187010    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="3090f174-c2e3-417f-bbbf-e7bec2757b8f" containerName="kube-rbac-proxy"
Sep 18 17:47:54.187179 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:54.187021    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3090f174-c2e3-417f-bbbf-e7bec2757b8f" containerName="kserve-container"
Sep 18 17:47:54.187179 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:54.187026    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="3090f174-c2e3-417f-bbbf-e7bec2757b8f" containerName="kserve-container"
Sep 18 17:47:54.187179 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:54.187044    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3090f174-c2e3-417f-bbbf-e7bec2757b8f" containerName="storage-initializer"
Sep 18 17:47:54.187179 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:54.187051    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="3090f174-c2e3-417f-bbbf-e7bec2757b8f" containerName="storage-initializer"
Sep 18 17:47:54.187179 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:54.187130    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="3090f174-c2e3-417f-bbbf-e7bec2757b8f" containerName="kserve-container"
Sep 18 17:47:54.187179 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:54.187137    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="3090f174-c2e3-417f-bbbf-e7bec2757b8f" containerName="kube-rbac-proxy"
Sep 18 17:47:54.190448 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:54.190424    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-f9f48c85f-x4cmz"
Sep 18 17:47:54.194118 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:54.194097    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert\""
Sep 18 17:47:54.194261 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:54.194197    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\""
Sep 18 17:47:54.206507 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:54.206473    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-f9f48c85f-x4cmz"]
Sep 18 17:47:54.344100 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:54.344040    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57666c50-c51f-4142-8007-0e2a4e3ea84f-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-f9f48c85f-x4cmz\" (UID: \"57666c50-c51f-4142-8007-0e2a4e3ea84f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-f9f48c85f-x4cmz"
Sep 18 17:47:54.344341 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:54.344128    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57666c50-c51f-4142-8007-0e2a4e3ea84f-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-f9f48c85f-x4cmz\" (UID: \"57666c50-c51f-4142-8007-0e2a4e3ea84f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-f9f48c85f-x4cmz"
Sep 18 17:47:54.344341 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:54.344153    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57666c50-c51f-4142-8007-0e2a4e3ea84f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-f9f48c85f-x4cmz\" (UID: \"57666c50-c51f-4142-8007-0e2a4e3ea84f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-f9f48c85f-x4cmz"
Sep 18 17:47:54.344341 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:54.344180    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8brvh\" (UniqueName: \"kubernetes.io/projected/57666c50-c51f-4142-8007-0e2a4e3ea84f-kube-api-access-8brvh\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-f9f48c85f-x4cmz\" (UID: \"57666c50-c51f-4142-8007-0e2a4e3ea84f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-f9f48c85f-x4cmz"
Sep 18 17:47:54.344341 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:54.344244    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57666c50-c51f-4142-8007-0e2a4e3ea84f-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-f9f48c85f-x4cmz\" (UID: \"57666c50-c51f-4142-8007-0e2a4e3ea84f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-f9f48c85f-x4cmz"
Sep 18 17:47:54.445411 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:54.445301    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57666c50-c51f-4142-8007-0e2a4e3ea84f-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-f9f48c85f-x4cmz\" (UID: \"57666c50-c51f-4142-8007-0e2a4e3ea84f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-f9f48c85f-x4cmz"
Sep 18 17:47:54.445411 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:54.445370    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57666c50-c51f-4142-8007-0e2a4e3ea84f-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-f9f48c85f-x4cmz\" (UID: \"57666c50-c51f-4142-8007-0e2a4e3ea84f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-f9f48c85f-x4cmz"
Sep 18 17:47:54.445411 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:54.445394    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57666c50-c51f-4142-8007-0e2a4e3ea84f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-f9f48c85f-x4cmz\" (UID: \"57666c50-c51f-4142-8007-0e2a4e3ea84f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-f9f48c85f-x4cmz"
Sep 18 17:47:54.445411 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:54.445410    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8brvh\" (UniqueName: \"kubernetes.io/projected/57666c50-c51f-4142-8007-0e2a4e3ea84f-kube-api-access-8brvh\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-f9f48c85f-x4cmz\" (UID: \"57666c50-c51f-4142-8007-0e2a4e3ea84f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-f9f48c85f-x4cmz"
Sep 18 17:47:54.445789 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:54.445429    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57666c50-c51f-4142-8007-0e2a4e3ea84f-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-f9f48c85f-x4cmz\" (UID: \"57666c50-c51f-4142-8007-0e2a4e3ea84f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-f9f48c85f-x4cmz"
Sep 18 17:47:54.445789 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:47:54.445500    2881 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert" not found
Sep 18 17:47:54.445789 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:47:54.445596    2881 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57666c50-c51f-4142-8007-0e2a4e3ea84f-proxy-tls podName:57666c50-c51f-4142-8007-0e2a4e3ea84f nodeName:}" failed. No retries permitted until 2026-09-18 17:47:54.945573637 +0000 UTC m=+3265.145428917 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/57666c50-c51f-4142-8007-0e2a4e3ea84f-proxy-tls") pod "isvc-sklearn-s3-tls-serving-pass-predictor-f9f48c85f-x4cmz" (UID: "57666c50-c51f-4142-8007-0e2a4e3ea84f") : secret "isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert" not found
Sep 18 17:47:54.445936 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:54.445854    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57666c50-c51f-4142-8007-0e2a4e3ea84f-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-f9f48c85f-x4cmz\" (UID: \"57666c50-c51f-4142-8007-0e2a4e3ea84f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-f9f48c85f-x4cmz"
Sep 18 17:47:54.446109 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:54.446088    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57666c50-c51f-4142-8007-0e2a4e3ea84f-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-f9f48c85f-x4cmz\" (UID: \"57666c50-c51f-4142-8007-0e2a4e3ea84f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-f9f48c85f-x4cmz"
Sep 18 17:47:54.446149 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:54.446126    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57666c50-c51f-4142-8007-0e2a4e3ea84f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-f9f48c85f-x4cmz\" (UID: \"57666c50-c51f-4142-8007-0e2a4e3ea84f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-f9f48c85f-x4cmz"
Sep 18 17:47:54.456690 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:54.456641    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8brvh\" (UniqueName: \"kubernetes.io/projected/57666c50-c51f-4142-8007-0e2a4e3ea84f-kube-api-access-8brvh\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-f9f48c85f-x4cmz\" (UID: \"57666c50-c51f-4142-8007-0e2a4e3ea84f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-f9f48c85f-x4cmz"
Sep 18 17:47:54.929647 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:54.929622    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-c8cc6df88-nh9mz_55d77ce8-c8c8-4093-88a6-cd08da6f4870/storage-initializer/1.log"
Sep 18 17:47:54.930103 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:54.930085    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-c8cc6df88-nh9mz_55d77ce8-c8c8-4093-88a6-cd08da6f4870/storage-initializer/0.log"
Sep 18 17:47:54.932560 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:54.932543    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-c8cc6df88-nh9mz"
Sep 18 17:47:54.950158 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:54.950089    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57666c50-c51f-4142-8007-0e2a4e3ea84f-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-f9f48c85f-x4cmz\" (UID: \"57666c50-c51f-4142-8007-0e2a4e3ea84f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-f9f48c85f-x4cmz"
Sep 18 17:47:54.952639 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:54.952608    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57666c50-c51f-4142-8007-0e2a4e3ea84f-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-f9f48c85f-x4cmz\" (UID: \"57666c50-c51f-4142-8007-0e2a4e3ea84f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-f9f48c85f-x4cmz"
Sep 18 17:47:55.050571 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:55.050532    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55d77ce8-c8c8-4093-88a6-cd08da6f4870-proxy-tls\") pod \"55d77ce8-c8c8-4093-88a6-cd08da6f4870\" (UID: \"55d77ce8-c8c8-4093-88a6-cd08da6f4870\") "
Sep 18 17:47:55.050571 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:55.050575    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6546n\" (UniqueName: \"kubernetes.io/projected/55d77ce8-c8c8-4093-88a6-cd08da6f4870-kube-api-access-6546n\") pod \"55d77ce8-c8c8-4093-88a6-cd08da6f4870\" (UID: \"55d77ce8-c8c8-4093-88a6-cd08da6f4870\") "
Sep 18 17:47:55.050843 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:55.050642    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55d77ce8-c8c8-4093-88a6-cd08da6f4870-kserve-provision-location\") pod \"55d77ce8-c8c8-4093-88a6-cd08da6f4870\" (UID: \"55d77ce8-c8c8-4093-88a6-cd08da6f4870\") "
Sep 18 17:47:55.050843 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:55.050663    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55d77ce8-c8c8-4093-88a6-cd08da6f4870-kube-rbac-proxy-sar-config\") pod \"55d77ce8-c8c8-4093-88a6-cd08da6f4870\" (UID: \"55d77ce8-c8c8-4093-88a6-cd08da6f4870\") "
Sep 18 17:47:55.050976 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:55.050948    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55d77ce8-c8c8-4093-88a6-cd08da6f4870-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "55d77ce8-c8c8-4093-88a6-cd08da6f4870" (UID: "55d77ce8-c8c8-4093-88a6-cd08da6f4870"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:47:55.051123 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:55.051093    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55d77ce8-c8c8-4093-88a6-cd08da6f4870-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "55d77ce8-c8c8-4093-88a6-cd08da6f4870" (UID: "55d77ce8-c8c8-4093-88a6-cd08da6f4870"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:47:55.052814 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:55.052790    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55d77ce8-c8c8-4093-88a6-cd08da6f4870-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "55d77ce8-c8c8-4093-88a6-cd08da6f4870" (UID: "55d77ce8-c8c8-4093-88a6-cd08da6f4870"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:47:55.052814 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:55.052803    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55d77ce8-c8c8-4093-88a6-cd08da6f4870-kube-api-access-6546n" (OuterVolumeSpecName: "kube-api-access-6546n") pod "55d77ce8-c8c8-4093-88a6-cd08da6f4870" (UID: "55d77ce8-c8c8-4093-88a6-cd08da6f4870"). InnerVolumeSpecName "kube-api-access-6546n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:47:55.101174 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:55.101135    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-f9f48c85f-x4cmz"
Sep 18 17:47:55.133666 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:55.133642    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-c8cc6df88-nh9mz_55d77ce8-c8c8-4093-88a6-cd08da6f4870/storage-initializer/1.log"
Sep 18 17:47:55.134102 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:55.134086    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-c8cc6df88-nh9mz_55d77ce8-c8c8-4093-88a6-cd08da6f4870/storage-initializer/0.log"
Sep 18 17:47:55.134176 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:55.134121    2881 generic.go:356] "Generic (PLEG): container finished" podID="55d77ce8-c8c8-4093-88a6-cd08da6f4870" containerID="f9454d2e47bf587a006f62fda5e9d03ec01031abc6dadc03f12d497ac3ee7564" exitCode=1
Sep 18 17:47:55.134176 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:55.134156    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-c8cc6df88-nh9mz" event={"ID":"55d77ce8-c8c8-4093-88a6-cd08da6f4870","Type":"ContainerDied","Data":"f9454d2e47bf587a006f62fda5e9d03ec01031abc6dadc03f12d497ac3ee7564"}
Sep 18 17:47:55.134306 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:55.134205    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-c8cc6df88-nh9mz" event={"ID":"55d77ce8-c8c8-4093-88a6-cd08da6f4870","Type":"ContainerDied","Data":"cb27a23aa4d22e9ddc1b13d47bce66637785ddaaf060f5306b20035a67a71d34"}
Sep 18 17:47:55.134306 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:55.134206    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-c8cc6df88-nh9mz"
Sep 18 17:47:55.134306 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:55.134254    2881 scope.go:117] "RemoveContainer" containerID="f9454d2e47bf587a006f62fda5e9d03ec01031abc6dadc03f12d497ac3ee7564"
Sep 18 17:47:55.144000 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:55.143950    2881 scope.go:117] "RemoveContainer" containerID="083d1313acb540b5e1cf304b1d6d66a650e3b68a0745cd98f60e06bba5cfe3aa"
Sep 18 17:47:55.151935 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:55.151906    2881 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55d77ce8-c8c8-4093-88a6-cd08da6f4870-kserve-provision-location\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:47:55.151935 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:55.151932    2881 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55d77ce8-c8c8-4093-88a6-cd08da6f4870-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:47:55.152163 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:55.151944    2881 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55d77ce8-c8c8-4093-88a6-cd08da6f4870-proxy-tls\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:47:55.152163 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:55.151955    2881 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6546n\" (UniqueName: \"kubernetes.io/projected/55d77ce8-c8c8-4093-88a6-cd08da6f4870-kube-api-access-6546n\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:47:55.153015 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:55.152902    2881 scope.go:117] "RemoveContainer" containerID="f9454d2e47bf587a006f62fda5e9d03ec01031abc6dadc03f12d497ac3ee7564"
Sep 18 17:47:55.153398 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:47:55.153374    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9454d2e47bf587a006f62fda5e9d03ec01031abc6dadc03f12d497ac3ee7564\": container with ID starting with f9454d2e47bf587a006f62fda5e9d03ec01031abc6dadc03f12d497ac3ee7564 not found: ID does not exist" containerID="f9454d2e47bf587a006f62fda5e9d03ec01031abc6dadc03f12d497ac3ee7564"
Sep 18 17:47:55.153627 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:55.153411    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9454d2e47bf587a006f62fda5e9d03ec01031abc6dadc03f12d497ac3ee7564"} err="failed to get container status \"f9454d2e47bf587a006f62fda5e9d03ec01031abc6dadc03f12d497ac3ee7564\": rpc error: code = NotFound desc = could not find container \"f9454d2e47bf587a006f62fda5e9d03ec01031abc6dadc03f12d497ac3ee7564\": container with ID starting with f9454d2e47bf587a006f62fda5e9d03ec01031abc6dadc03f12d497ac3ee7564 not found: ID does not exist"
Sep 18 17:47:55.153627 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:55.153440    2881 scope.go:117] "RemoveContainer" containerID="083d1313acb540b5e1cf304b1d6d66a650e3b68a0745cd98f60e06bba5cfe3aa"
Sep 18 17:47:55.154260 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:47:55.154215    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"083d1313acb540b5e1cf304b1d6d66a650e3b68a0745cd98f60e06bba5cfe3aa\": container with ID starting with 083d1313acb540b5e1cf304b1d6d66a650e3b68a0745cd98f60e06bba5cfe3aa not found: ID does not exist" containerID="083d1313acb540b5e1cf304b1d6d66a650e3b68a0745cd98f60e06bba5cfe3aa"
Sep 18 17:47:55.154390 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:55.154269    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"083d1313acb540b5e1cf304b1d6d66a650e3b68a0745cd98f60e06bba5cfe3aa"} err="failed to get container status \"083d1313acb540b5e1cf304b1d6d66a650e3b68a0745cd98f60e06bba5cfe3aa\": rpc error: code = NotFound desc = could not find container \"083d1313acb540b5e1cf304b1d6d66a650e3b68a0745cd98f60e06bba5cfe3aa\": container with ID starting with 083d1313acb540b5e1cf304b1d6d66a650e3b68a0745cd98f60e06bba5cfe3aa not found: ID does not exist"
Sep 18 17:47:55.175330 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:55.175297    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-c8cc6df88-nh9mz"]
Sep 18 17:47:55.177973 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:55.177946    2881 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-c8cc6df88-nh9mz"]
Sep 18 17:47:55.383148 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:55.382975    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-f9f48c85f-x4cmz"]
Sep 18 17:47:55.385700 ip-10-0-143-5 kubenswrapper[2881]: W0918 17:47:55.385670    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57666c50_c51f_4142_8007_0e2a4e3ea84f.slice/crio-b9cfcc5fd8d35cf19425e3c8655d60ef500007124db374dfcb85ea4c5c68f90a WatchSource:0}: Error finding container b9cfcc5fd8d35cf19425e3c8655d60ef500007124db374dfcb85ea4c5c68f90a: Status 404 returned error can't find the container with id b9cfcc5fd8d35cf19425e3c8655d60ef500007124db374dfcb85ea4c5c68f90a
Sep 18 17:47:55.893554 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:55.893514    2881 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55d77ce8-c8c8-4093-88a6-cd08da6f4870" path="/var/lib/kubelet/pods/55d77ce8-c8c8-4093-88a6-cd08da6f4870/volumes"
Sep 18 17:47:56.140679 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:56.140638    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-f9f48c85f-x4cmz" event={"ID":"57666c50-c51f-4142-8007-0e2a4e3ea84f","Type":"ContainerStarted","Data":"a89803133fa56dc43e9e274a377ff625e86274afb64099af29f63e3cf759486b"}
Sep 18 17:47:56.140894 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:56.140714    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-f9f48c85f-x4cmz" event={"ID":"57666c50-c51f-4142-8007-0e2a4e3ea84f","Type":"ContainerStarted","Data":"b9cfcc5fd8d35cf19425e3c8655d60ef500007124db374dfcb85ea4c5c68f90a"}
Sep 18 17:47:57.148073 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:57.148040    2881 generic.go:356] "Generic (PLEG): container finished" podID="57666c50-c51f-4142-8007-0e2a4e3ea84f" containerID="a89803133fa56dc43e9e274a377ff625e86274afb64099af29f63e3cf759486b" exitCode=0
Sep 18 17:47:57.148584 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:57.148086    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-f9f48c85f-x4cmz" event={"ID":"57666c50-c51f-4142-8007-0e2a4e3ea84f","Type":"ContainerDied","Data":"a89803133fa56dc43e9e274a377ff625e86274afb64099af29f63e3cf759486b"}
Sep 18 17:47:58.155691 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:58.155655    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-f9f48c85f-x4cmz" event={"ID":"57666c50-c51f-4142-8007-0e2a4e3ea84f","Type":"ContainerStarted","Data":"3aee5c01b113da88a7baf99dadd9666b16b15f05a56787bd5d55cb427564a368"}
Sep 18 17:47:58.156091 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:58.155720    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-f9f48c85f-x4cmz" event={"ID":"57666c50-c51f-4142-8007-0e2a4e3ea84f","Type":"ContainerStarted","Data":"f6cc63abaf6a57e08ea7be84d07ad95c169a9d42d448cad17897f74f9ffcfd0e"}
Sep 18 17:47:58.156091 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:58.155838    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-f9f48c85f-x4cmz"
Sep 18 17:47:58.156091 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:58.155894    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-f9f48c85f-x4cmz"
Sep 18 17:47:58.157294 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:58.157270    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-f9f48c85f-x4cmz" podUID="57666c50-c51f-4142-8007-0e2a4e3ea84f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.69:8080: connect: connection refused"
Sep 18 17:47:58.176994 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:58.176929    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-f9f48c85f-x4cmz" podStartSLOduration=2.9842227340000003 podStartE2EDuration="4.176910851s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.192688117s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:47:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:47:58.175781982 +0000 UTC m=+3268.375637280" watchObservedRunningTime="2026-09-18 17:47:58.176910851 +0000 UTC m=+3268.376766136"
Sep 18 17:47:58.876191 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:58.876161    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-xpbh6"
Sep 18 17:47:59.160708 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:47:59.160609    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-f9f48c85f-x4cmz" podUID="57666c50-c51f-4142-8007-0e2a4e3ea84f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.69:8080: connect: connection refused"
Sep 18 17:48:04.165740 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:04.165658    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-f9f48c85f-x4cmz"
Sep 18 17:48:04.166194 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:04.166167    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-f9f48c85f-x4cmz" podUID="57666c50-c51f-4142-8007-0e2a4e3ea84f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.69:8080: connect: connection refused"
Sep 18 17:48:06.361737 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:06.361702    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-xpbh6"]
Sep 18 17:48:06.362197 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:06.361968    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-xpbh6" podUID="244d1afa-7185-49c2-8cb3-4e48ba7b5aa7" containerName="kserve-container" containerID="cri-o://0d4c7ba16dacbb937e7e7f3bf2e72655b19795cb14c76334e0a9a155f6179f60" gracePeriod=30
Sep 18 17:48:06.362197 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:06.361996    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-xpbh6" podUID="244d1afa-7185-49c2-8cb3-4e48ba7b5aa7" containerName="kube-rbac-proxy" containerID="cri-o://846d5cafdccfe893858008cf948485728b08cd63ec1948288e8541ac7330c806" gracePeriod=30
Sep 18 17:48:07.206946 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:07.206910    2881 generic.go:356] "Generic (PLEG): container finished" podID="244d1afa-7185-49c2-8cb3-4e48ba7b5aa7" containerID="846d5cafdccfe893858008cf948485728b08cd63ec1948288e8541ac7330c806" exitCode=2
Sep 18 17:48:07.207128 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:07.206964    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-xpbh6" event={"ID":"244d1afa-7185-49c2-8cb3-4e48ba7b5aa7","Type":"ContainerDied","Data":"846d5cafdccfe893858008cf948485728b08cd63ec1948288e8541ac7330c806"}
Sep 18 17:48:08.870415 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:08.870371    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-xpbh6" podUID="244d1afa-7185-49c2-8cb3-4e48ba7b5aa7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.67:8643/healthz\": dial tcp 10.134.0.67:8643: connect: connection refused"
Sep 18 17:48:08.874633 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:08.874599    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-xpbh6" podUID="244d1afa-7185-49c2-8cb3-4e48ba7b5aa7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Sep 18 17:48:09.907549 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:09.907515    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-xpbh6"
Sep 18 17:48:09.989315 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:09.989191    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/244d1afa-7185-49c2-8cb3-4e48ba7b5aa7-cabundle-cert\") pod \"244d1afa-7185-49c2-8cb3-4e48ba7b5aa7\" (UID: \"244d1afa-7185-49c2-8cb3-4e48ba7b5aa7\") "
Sep 18 17:48:09.989315 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:09.989271    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/244d1afa-7185-49c2-8cb3-4e48ba7b5aa7-proxy-tls\") pod \"244d1afa-7185-49c2-8cb3-4e48ba7b5aa7\" (UID: \"244d1afa-7185-49c2-8cb3-4e48ba7b5aa7\") "
Sep 18 17:48:09.989526 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:09.989318    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/244d1afa-7185-49c2-8cb3-4e48ba7b5aa7-kube-rbac-proxy-sar-config\") pod \"244d1afa-7185-49c2-8cb3-4e48ba7b5aa7\" (UID: \"244d1afa-7185-49c2-8cb3-4e48ba7b5aa7\") "
Sep 18 17:48:09.989526 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:09.989374    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/244d1afa-7185-49c2-8cb3-4e48ba7b5aa7-kserve-provision-location\") pod \"244d1afa-7185-49c2-8cb3-4e48ba7b5aa7\" (UID: \"244d1afa-7185-49c2-8cb3-4e48ba7b5aa7\") "
Sep 18 17:48:09.989526 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:09.989429    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2w5k\" (UniqueName: \"kubernetes.io/projected/244d1afa-7185-49c2-8cb3-4e48ba7b5aa7-kube-api-access-h2w5k\") pod \"244d1afa-7185-49c2-8cb3-4e48ba7b5aa7\" (UID: \"244d1afa-7185-49c2-8cb3-4e48ba7b5aa7\") "
Sep 18 17:48:09.989697 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:09.989653    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/244d1afa-7185-49c2-8cb3-4e48ba7b5aa7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "244d1afa-7185-49c2-8cb3-4e48ba7b5aa7" (UID: "244d1afa-7185-49c2-8cb3-4e48ba7b5aa7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:48:09.989761 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:09.989723    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/244d1afa-7185-49c2-8cb3-4e48ba7b5aa7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "244d1afa-7185-49c2-8cb3-4e48ba7b5aa7" (UID: "244d1afa-7185-49c2-8cb3-4e48ba7b5aa7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:48:09.991544 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:09.991507    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/244d1afa-7185-49c2-8cb3-4e48ba7b5aa7-kube-api-access-h2w5k" (OuterVolumeSpecName: "kube-api-access-h2w5k") pod "244d1afa-7185-49c2-8cb3-4e48ba7b5aa7" (UID: "244d1afa-7185-49c2-8cb3-4e48ba7b5aa7"). InnerVolumeSpecName "kube-api-access-h2w5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:48:09.991671 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:09.991620    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/244d1afa-7185-49c2-8cb3-4e48ba7b5aa7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "244d1afa-7185-49c2-8cb3-4e48ba7b5aa7" (UID: "244d1afa-7185-49c2-8cb3-4e48ba7b5aa7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:48:09.999408 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:09.999371    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/244d1afa-7185-49c2-8cb3-4e48ba7b5aa7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "244d1afa-7185-49c2-8cb3-4e48ba7b5aa7" (UID: "244d1afa-7185-49c2-8cb3-4e48ba7b5aa7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:48:10.090640 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:10.090598    2881 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/244d1afa-7185-49c2-8cb3-4e48ba7b5aa7-cabundle-cert\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:48:10.090640 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:10.090631    2881 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/244d1afa-7185-49c2-8cb3-4e48ba7b5aa7-proxy-tls\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:48:10.090640 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:10.090641    2881 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/244d1afa-7185-49c2-8cb3-4e48ba7b5aa7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:48:10.090640 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:10.090652    2881 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/244d1afa-7185-49c2-8cb3-4e48ba7b5aa7-kserve-provision-location\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:48:10.090969 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:10.090662    2881 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2w5k\" (UniqueName: \"kubernetes.io/projected/244d1afa-7185-49c2-8cb3-4e48ba7b5aa7-kube-api-access-h2w5k\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:48:10.224355 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:10.224312    2881 generic.go:356] "Generic (PLEG): container finished" podID="244d1afa-7185-49c2-8cb3-4e48ba7b5aa7" containerID="0d4c7ba16dacbb937e7e7f3bf2e72655b19795cb14c76334e0a9a155f6179f60" exitCode=0
Sep 18 17:48:10.224563 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:10.224372    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-xpbh6"
Sep 18 17:48:10.224563 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:10.224395    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-xpbh6" event={"ID":"244d1afa-7185-49c2-8cb3-4e48ba7b5aa7","Type":"ContainerDied","Data":"0d4c7ba16dacbb937e7e7f3bf2e72655b19795cb14c76334e0a9a155f6179f60"}
Sep 18 17:48:10.224563 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:10.224460    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-xpbh6" event={"ID":"244d1afa-7185-49c2-8cb3-4e48ba7b5aa7","Type":"ContainerDied","Data":"8db80d8f8d6aa62f33e0f9374e5529469a2587edfd606f9659f6f87ff37cc322"}
Sep 18 17:48:10.224563 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:10.224492    2881 scope.go:117] "RemoveContainer" containerID="846d5cafdccfe893858008cf948485728b08cd63ec1948288e8541ac7330c806"
Sep 18 17:48:10.234018 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:10.233998    2881 scope.go:117] "RemoveContainer" containerID="0d4c7ba16dacbb937e7e7f3bf2e72655b19795cb14c76334e0a9a155f6179f60"
Sep 18 17:48:10.242737 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:10.242718    2881 scope.go:117] "RemoveContainer" containerID="7f9e69d027eecc2babe3a0b337ad9a50d7b9c88da3d3f27e04c91172673ac05b"
Sep 18 17:48:10.248715 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:10.248688    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-xpbh6"]
Sep 18 17:48:10.251666 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:10.251647    2881 scope.go:117] "RemoveContainer" containerID="846d5cafdccfe893858008cf948485728b08cd63ec1948288e8541ac7330c806"
Sep 18 17:48:10.252037 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:48:10.252015    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"846d5cafdccfe893858008cf948485728b08cd63ec1948288e8541ac7330c806\": container with ID starting with 846d5cafdccfe893858008cf948485728b08cd63ec1948288e8541ac7330c806 not found: ID does not exist" containerID="846d5cafdccfe893858008cf948485728b08cd63ec1948288e8541ac7330c806"
Sep 18 17:48:10.252089 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:10.252048    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"846d5cafdccfe893858008cf948485728b08cd63ec1948288e8541ac7330c806"} err="failed to get container status \"846d5cafdccfe893858008cf948485728b08cd63ec1948288e8541ac7330c806\": rpc error: code = NotFound desc = could not find container \"846d5cafdccfe893858008cf948485728b08cd63ec1948288e8541ac7330c806\": container with ID starting with 846d5cafdccfe893858008cf948485728b08cd63ec1948288e8541ac7330c806 not found: ID does not exist"
Sep 18 17:48:10.252089 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:10.252067    2881 scope.go:117] "RemoveContainer" containerID="0d4c7ba16dacbb937e7e7f3bf2e72655b19795cb14c76334e0a9a155f6179f60"
Sep 18 17:48:10.252383 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:48:10.252365    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d4c7ba16dacbb937e7e7f3bf2e72655b19795cb14c76334e0a9a155f6179f60\": container with ID starting with 0d4c7ba16dacbb937e7e7f3bf2e72655b19795cb14c76334e0a9a155f6179f60 not found: ID does not exist" containerID="0d4c7ba16dacbb937e7e7f3bf2e72655b19795cb14c76334e0a9a155f6179f60"
Sep 18 17:48:10.252435 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:10.252390    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d4c7ba16dacbb937e7e7f3bf2e72655b19795cb14c76334e0a9a155f6179f60"} err="failed to get container status \"0d4c7ba16dacbb937e7e7f3bf2e72655b19795cb14c76334e0a9a155f6179f60\": rpc error: code = NotFound desc = could not find container \"0d4c7ba16dacbb937e7e7f3bf2e72655b19795cb14c76334e0a9a155f6179f60\": container with ID starting with 0d4c7ba16dacbb937e7e7f3bf2e72655b19795cb14c76334e0a9a155f6179f60 not found: ID does not exist"
Sep 18 17:48:10.252435 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:10.252410    2881 scope.go:117] "RemoveContainer" containerID="7f9e69d027eecc2babe3a0b337ad9a50d7b9c88da3d3f27e04c91172673ac05b"
Sep 18 17:48:10.252724 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:48:10.252695    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f9e69d027eecc2babe3a0b337ad9a50d7b9c88da3d3f27e04c91172673ac05b\": container with ID starting with 7f9e69d027eecc2babe3a0b337ad9a50d7b9c88da3d3f27e04c91172673ac05b not found: ID does not exist" containerID="7f9e69d027eecc2babe3a0b337ad9a50d7b9c88da3d3f27e04c91172673ac05b"
Sep 18 17:48:10.252837 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:10.252728    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f9e69d027eecc2babe3a0b337ad9a50d7b9c88da3d3f27e04c91172673ac05b"} err="failed to get container status \"7f9e69d027eecc2babe3a0b337ad9a50d7b9c88da3d3f27e04c91172673ac05b\": rpc error: code = NotFound desc = could not find container \"7f9e69d027eecc2babe3a0b337ad9a50d7b9c88da3d3f27e04c91172673ac05b\": container with ID starting with 7f9e69d027eecc2babe3a0b337ad9a50d7b9c88da3d3f27e04c91172673ac05b not found: ID does not exist"
Sep 18 17:48:10.255298 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:10.255273    2881 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-xpbh6"]
Sep 18 17:48:11.656422 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:11.656386    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-s679h"]
Sep 18 17:48:11.656820 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:11.656751    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="244d1afa-7185-49c2-8cb3-4e48ba7b5aa7" containerName="kube-rbac-proxy"
Sep 18 17:48:11.656820 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:11.656762    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="244d1afa-7185-49c2-8cb3-4e48ba7b5aa7" containerName="kube-rbac-proxy"
Sep 18 17:48:11.656820 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:11.656776    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="244d1afa-7185-49c2-8cb3-4e48ba7b5aa7" containerName="storage-initializer"
Sep 18 17:48:11.656820 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:11.656781    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="244d1afa-7185-49c2-8cb3-4e48ba7b5aa7" containerName="storage-initializer"
Sep 18 17:48:11.656820 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:11.656790    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="244d1afa-7185-49c2-8cb3-4e48ba7b5aa7" containerName="kserve-container"
Sep 18 17:48:11.656820 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:11.656797    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="244d1afa-7185-49c2-8cb3-4e48ba7b5aa7" containerName="kserve-container"
Sep 18 17:48:11.656820 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:11.656802    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55d77ce8-c8c8-4093-88a6-cd08da6f4870" containerName="storage-initializer"
Sep 18 17:48:11.656820 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:11.656808    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="55d77ce8-c8c8-4093-88a6-cd08da6f4870" containerName="storage-initializer"
Sep 18 17:48:11.656820 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:11.656816    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55d77ce8-c8c8-4093-88a6-cd08da6f4870" containerName="storage-initializer"
Sep 18 17:48:11.656820 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:11.656821    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="55d77ce8-c8c8-4093-88a6-cd08da6f4870" containerName="storage-initializer"
Sep 18 17:48:11.657121 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:11.656877    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="55d77ce8-c8c8-4093-88a6-cd08da6f4870" containerName="storage-initializer"
Sep 18 17:48:11.657121 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:11.656886    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="244d1afa-7185-49c2-8cb3-4e48ba7b5aa7" containerName="kserve-container"
Sep 18 17:48:11.657121 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:11.656898    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="244d1afa-7185-49c2-8cb3-4e48ba7b5aa7" containerName="kube-rbac-proxy"
Sep 18 17:48:11.657121 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:11.657001    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="55d77ce8-c8c8-4093-88a6-cd08da6f4870" containerName="storage-initializer"
Sep 18 17:48:11.660670 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:11.660650    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-s679h"
Sep 18 17:48:11.665599 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:11.665571    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 17:48:11.665599 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:11.665591    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-predictor-serving-cert\""
Sep 18 17:48:11.665806 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:11.665605    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 18 17:48:11.675029 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:11.675002    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-s679h"]
Sep 18 17:48:11.703958 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:11.703906    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75b46e80-39a1-43ac-90e7-cbf0e7816043-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-s679h\" (UID: \"75b46e80-39a1-43ac-90e7-cbf0e7816043\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-s679h"
Sep 18 17:48:11.703958 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:11.703956    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75b46e80-39a1-43ac-90e7-cbf0e7816043-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-s679h\" (UID: \"75b46e80-39a1-43ac-90e7-cbf0e7816043\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-s679h"
Sep 18 17:48:11.704219 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:11.704010    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6hbb\" (UniqueName: \"kubernetes.io/projected/75b46e80-39a1-43ac-90e7-cbf0e7816043-kube-api-access-p6hbb\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-s679h\" (UID: \"75b46e80-39a1-43ac-90e7-cbf0e7816043\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-s679h"
Sep 18 17:48:11.704219 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:11.704048    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75b46e80-39a1-43ac-90e7-cbf0e7816043-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-s679h\" (UID: \"75b46e80-39a1-43ac-90e7-cbf0e7816043\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-s679h"
Sep 18 17:48:11.704219 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:11.704092    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75b46e80-39a1-43ac-90e7-cbf0e7816043-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-s679h\" (UID: \"75b46e80-39a1-43ac-90e7-cbf0e7816043\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-s679h"
Sep 18 17:48:11.805261 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:11.805199    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75b46e80-39a1-43ac-90e7-cbf0e7816043-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-s679h\" (UID: \"75b46e80-39a1-43ac-90e7-cbf0e7816043\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-s679h"
Sep 18 17:48:11.805511 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:11.805374    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75b46e80-39a1-43ac-90e7-cbf0e7816043-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-s679h\" (UID: \"75b46e80-39a1-43ac-90e7-cbf0e7816043\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-s679h"
Sep 18 17:48:11.805511 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:11.805413    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75b46e80-39a1-43ac-90e7-cbf0e7816043-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-s679h\" (UID: \"75b46e80-39a1-43ac-90e7-cbf0e7816043\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-s679h"
Sep 18 17:48:11.805511 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:11.805494    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6hbb\" (UniqueName: \"kubernetes.io/projected/75b46e80-39a1-43ac-90e7-cbf0e7816043-kube-api-access-p6hbb\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-s679h\" (UID: \"75b46e80-39a1-43ac-90e7-cbf0e7816043\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-s679h"
Sep 18 17:48:11.805700 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:11.805553    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75b46e80-39a1-43ac-90e7-cbf0e7816043-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-s679h\" (UID: \"75b46e80-39a1-43ac-90e7-cbf0e7816043\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-s679h"
Sep 18 17:48:11.805700 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:48:11.805497    2881 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-serving-cert: secret "isvc-paddle-v2-kserve-predictor-serving-cert" not found
Sep 18 17:48:11.805700 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:48:11.805624    2881 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75b46e80-39a1-43ac-90e7-cbf0e7816043-proxy-tls podName:75b46e80-39a1-43ac-90e7-cbf0e7816043 nodeName:}" failed. No retries permitted until 2026-09-18 17:48:12.305607245 +0000 UTC m=+3282.505462508 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/75b46e80-39a1-43ac-90e7-cbf0e7816043-proxy-tls") pod "isvc-paddle-v2-kserve-predictor-64d4c74b4d-s679h" (UID: "75b46e80-39a1-43ac-90e7-cbf0e7816043") : secret "isvc-paddle-v2-kserve-predictor-serving-cert" not found
Sep 18 17:48:11.805869 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:11.805811    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75b46e80-39a1-43ac-90e7-cbf0e7816043-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-s679h\" (UID: \"75b46e80-39a1-43ac-90e7-cbf0e7816043\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-s679h"
Sep 18 17:48:11.806029 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:11.806007    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75b46e80-39a1-43ac-90e7-cbf0e7816043-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-s679h\" (UID: \"75b46e80-39a1-43ac-90e7-cbf0e7816043\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-s679h"
Sep 18 17:48:11.806250 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:11.806205    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75b46e80-39a1-43ac-90e7-cbf0e7816043-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-s679h\" (UID: \"75b46e80-39a1-43ac-90e7-cbf0e7816043\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-s679h"
Sep 18 17:48:11.816710 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:11.816682    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6hbb\" (UniqueName: \"kubernetes.io/projected/75b46e80-39a1-43ac-90e7-cbf0e7816043-kube-api-access-p6hbb\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-s679h\" (UID: \"75b46e80-39a1-43ac-90e7-cbf0e7816043\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-s679h"
Sep 18 17:48:11.893021 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:11.892988    2881 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="244d1afa-7185-49c2-8cb3-4e48ba7b5aa7" path="/var/lib/kubelet/pods/244d1afa-7185-49c2-8cb3-4e48ba7b5aa7/volumes"
Sep 18 17:48:12.309980 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:12.309940    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75b46e80-39a1-43ac-90e7-cbf0e7816043-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-s679h\" (UID: \"75b46e80-39a1-43ac-90e7-cbf0e7816043\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-s679h"
Sep 18 17:48:12.312449 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:12.312423    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75b46e80-39a1-43ac-90e7-cbf0e7816043-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-s679h\" (UID: \"75b46e80-39a1-43ac-90e7-cbf0e7816043\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-s679h"
Sep 18 17:48:12.571472 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:12.571369    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-s679h"
Sep 18 17:48:12.856766 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:12.856740    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-s679h"]
Sep 18 17:48:12.858899 ip-10-0-143-5 kubenswrapper[2881]: W0918 17:48:12.858865    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75b46e80_39a1_43ac_90e7_cbf0e7816043.slice/crio-d60f4e5c443ba780977d005844b1fe77d47d616df5a8b4c9577da2ffbd9d089a WatchSource:0}: Error finding container d60f4e5c443ba780977d005844b1fe77d47d616df5a8b4c9577da2ffbd9d089a: Status 404 returned error can't find the container with id d60f4e5c443ba780977d005844b1fe77d47d616df5a8b4c9577da2ffbd9d089a
Sep 18 17:48:13.243840 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:13.243724    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-s679h" event={"ID":"75b46e80-39a1-43ac-90e7-cbf0e7816043","Type":"ContainerStarted","Data":"0ac39dd227346e36c1b95f131ac0a9321bc4fe8861bfcfb6c924fa55c274b9bc"}
Sep 18 17:48:13.243840 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:13.243793    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-s679h" event={"ID":"75b46e80-39a1-43ac-90e7-cbf0e7816043","Type":"ContainerStarted","Data":"d60f4e5c443ba780977d005844b1fe77d47d616df5a8b4c9577da2ffbd9d089a"}
Sep 18 17:48:14.166290 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:14.166248    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-f9f48c85f-x4cmz" podUID="57666c50-c51f-4142-8007-0e2a4e3ea84f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.69:8080: connect: connection refused"
Sep 18 17:48:18.269004 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:18.268968    2881 generic.go:356] "Generic (PLEG): container finished" podID="75b46e80-39a1-43ac-90e7-cbf0e7816043" containerID="0ac39dd227346e36c1b95f131ac0a9321bc4fe8861bfcfb6c924fa55c274b9bc" exitCode=0
Sep 18 17:48:18.269454 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:18.269039    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-s679h" event={"ID":"75b46e80-39a1-43ac-90e7-cbf0e7816043","Type":"ContainerDied","Data":"0ac39dd227346e36c1b95f131ac0a9321bc4fe8861bfcfb6c924fa55c274b9bc"}
Sep 18 17:48:19.277710 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:19.277673    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-s679h" event={"ID":"75b46e80-39a1-43ac-90e7-cbf0e7816043","Type":"ContainerStarted","Data":"f76c0cd53738634740094553e510ccfe6d93a753281271da033d0d00d4944abd"}
Sep 18 17:48:19.278119 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:19.277742    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-s679h" event={"ID":"75b46e80-39a1-43ac-90e7-cbf0e7816043","Type":"ContainerStarted","Data":"20a5b1f5b50b62a30d71f618eac98393ccc15040012be85c379815ebe405c7b5"}
Sep 18 17:48:19.278119 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:19.278014    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-s679h"
Sep 18 17:48:19.278259 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:19.278167    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-s679h"
Sep 18 17:48:19.279400 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:19.279376    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-s679h" podUID="75b46e80-39a1-43ac-90e7-cbf0e7816043" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.70:8080: connect: connection refused"
Sep 18 17:48:19.298416 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:19.298361    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-s679h" podStartSLOduration=3.8046659910000002 podStartE2EDuration="8.298342354s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.493676363s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:48:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:48:19.2970269 +0000 UTC m=+3289.496882187" watchObservedRunningTime="2026-09-18 17:48:19.298342354 +0000 UTC m=+3289.498197639"
Sep 18 17:48:20.283450 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:20.283404    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-s679h" podUID="75b46e80-39a1-43ac-90e7-cbf0e7816043" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.70:8080: connect: connection refused"
Sep 18 17:48:24.166911 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:24.166871    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-f9f48c85f-x4cmz" podUID="57666c50-c51f-4142-8007-0e2a4e3ea84f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.69:8080: connect: connection refused"
Sep 18 17:48:25.288098 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:25.288065    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-s679h"
Sep 18 17:48:25.288612 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:25.288546    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-s679h" podUID="75b46e80-39a1-43ac-90e7-cbf0e7816043" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.70:8080: connect: connection refused"
Sep 18 17:48:34.167078 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:34.167037    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-f9f48c85f-x4cmz" podUID="57666c50-c51f-4142-8007-0e2a4e3ea84f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.69:8080: connect: connection refused"
Sep 18 17:48:35.289387 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:35.289345    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-s679h" podUID="75b46e80-39a1-43ac-90e7-cbf0e7816043" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.70:8080: connect: connection refused"
Sep 18 17:48:44.166245 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:44.166188    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-f9f48c85f-x4cmz" podUID="57666c50-c51f-4142-8007-0e2a4e3ea84f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.69:8080: connect: connection refused"
Sep 18 17:48:45.289070 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:45.289023    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-s679h" podUID="75b46e80-39a1-43ac-90e7-cbf0e7816043" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.70:8080: connect: connection refused"
Sep 18 17:48:54.167067 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:54.167019    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-f9f48c85f-x4cmz" podUID="57666c50-c51f-4142-8007-0e2a4e3ea84f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.69:8080: connect: connection refused"
Sep 18 17:48:55.288804 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:48:55.288763    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-s679h" podUID="75b46e80-39a1-43ac-90e7-cbf0e7816043" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.70:8080: connect: connection refused"
Sep 18 17:49:04.166544 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:04.166499    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-f9f48c85f-x4cmz" podUID="57666c50-c51f-4142-8007-0e2a4e3ea84f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.69:8080: connect: connection refused"
Sep 18 17:49:05.288864 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:05.288824    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-s679h" podUID="75b46e80-39a1-43ac-90e7-cbf0e7816043" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.70:8080: connect: connection refused"
Sep 18 17:49:14.167164 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:14.167116    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-f9f48c85f-x4cmz" podUID="57666c50-c51f-4142-8007-0e2a4e3ea84f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.69:8080: connect: connection refused"
Sep 18 17:49:15.290018 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:15.289983    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-s679h"
Sep 18 17:49:23.373157 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:23.373118    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-s679h"]
Sep 18 17:49:23.373641 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:23.373411    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-s679h" podUID="75b46e80-39a1-43ac-90e7-cbf0e7816043" containerName="kserve-container" containerID="cri-o://20a5b1f5b50b62a30d71f618eac98393ccc15040012be85c379815ebe405c7b5" gracePeriod=30
Sep 18 17:49:23.373641 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:23.373460    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-s679h" podUID="75b46e80-39a1-43ac-90e7-cbf0e7816043" containerName="kube-rbac-proxy" containerID="cri-o://f76c0cd53738634740094553e510ccfe6d93a753281271da033d0d00d4944abd" gracePeriod=30
Sep 18 17:49:23.595515 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:23.595477    2881 generic.go:356] "Generic (PLEG): container finished" podID="75b46e80-39a1-43ac-90e7-cbf0e7816043" containerID="f76c0cd53738634740094553e510ccfe6d93a753281271da033d0d00d4944abd" exitCode=2
Sep 18 17:49:23.595707 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:23.595532    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-s679h" event={"ID":"75b46e80-39a1-43ac-90e7-cbf0e7816043","Type":"ContainerDied","Data":"f76c0cd53738634740094553e510ccfe6d93a753281271da033d0d00d4944abd"}
Sep 18 17:49:24.167382 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:24.167353    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-f9f48c85f-x4cmz"
Sep 18 17:49:25.283868 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:25.283817    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-s679h" podUID="75b46e80-39a1-43ac-90e7-cbf0e7816043" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.70:8643/healthz\": dial tcp 10.134.0.70:8643: connect: connection refused"
Sep 18 17:49:25.289293 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:25.289259    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-s679h" podUID="75b46e80-39a1-43ac-90e7-cbf0e7816043" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.70:8080: connect: connection refused"
Sep 18 17:49:26.914522 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:26.914493    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-s679h"
Sep 18 17:49:27.003873 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:27.003751    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75b46e80-39a1-43ac-90e7-cbf0e7816043-cabundle-cert\") pod \"75b46e80-39a1-43ac-90e7-cbf0e7816043\" (UID: \"75b46e80-39a1-43ac-90e7-cbf0e7816043\") "
Sep 18 17:49:27.003873 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:27.003879    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p6hbb\" (UniqueName: \"kubernetes.io/projected/75b46e80-39a1-43ac-90e7-cbf0e7816043-kube-api-access-p6hbb\") pod \"75b46e80-39a1-43ac-90e7-cbf0e7816043\" (UID: \"75b46e80-39a1-43ac-90e7-cbf0e7816043\") "
Sep 18 17:49:27.004110 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:27.003903    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75b46e80-39a1-43ac-90e7-cbf0e7816043-kube-rbac-proxy-sar-config\") pod \"75b46e80-39a1-43ac-90e7-cbf0e7816043\" (UID: \"75b46e80-39a1-43ac-90e7-cbf0e7816043\") "
Sep 18 17:49:27.004110 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:27.003955    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75b46e80-39a1-43ac-90e7-cbf0e7816043-kserve-provision-location\") pod \"75b46e80-39a1-43ac-90e7-cbf0e7816043\" (UID: \"75b46e80-39a1-43ac-90e7-cbf0e7816043\") "
Sep 18 17:49:27.004110 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:27.003975    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75b46e80-39a1-43ac-90e7-cbf0e7816043-proxy-tls\") pod \"75b46e80-39a1-43ac-90e7-cbf0e7816043\" (UID: \"75b46e80-39a1-43ac-90e7-cbf0e7816043\") "
Sep 18 17:49:27.004325 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:27.004197    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75b46e80-39a1-43ac-90e7-cbf0e7816043-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "75b46e80-39a1-43ac-90e7-cbf0e7816043" (UID: "75b46e80-39a1-43ac-90e7-cbf0e7816043"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:49:27.004443 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:27.004418    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75b46e80-39a1-43ac-90e7-cbf0e7816043-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "75b46e80-39a1-43ac-90e7-cbf0e7816043" (UID: "75b46e80-39a1-43ac-90e7-cbf0e7816043"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:49:27.006185 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:27.006149    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75b46e80-39a1-43ac-90e7-cbf0e7816043-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "75b46e80-39a1-43ac-90e7-cbf0e7816043" (UID: "75b46e80-39a1-43ac-90e7-cbf0e7816043"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:49:27.006336 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:27.006233    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75b46e80-39a1-43ac-90e7-cbf0e7816043-kube-api-access-p6hbb" (OuterVolumeSpecName: "kube-api-access-p6hbb") pod "75b46e80-39a1-43ac-90e7-cbf0e7816043" (UID: "75b46e80-39a1-43ac-90e7-cbf0e7816043"). InnerVolumeSpecName "kube-api-access-p6hbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:49:27.011931 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:27.011895    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75b46e80-39a1-43ac-90e7-cbf0e7816043-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "75b46e80-39a1-43ac-90e7-cbf0e7816043" (UID: "75b46e80-39a1-43ac-90e7-cbf0e7816043"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:49:27.105395 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:27.105340    2881 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6hbb\" (UniqueName: \"kubernetes.io/projected/75b46e80-39a1-43ac-90e7-cbf0e7816043-kube-api-access-p6hbb\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:49:27.105395 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:27.105388    2881 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75b46e80-39a1-43ac-90e7-cbf0e7816043-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:49:27.105395 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:27.105397    2881 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75b46e80-39a1-43ac-90e7-cbf0e7816043-kserve-provision-location\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:49:27.105395 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:27.105409    2881 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75b46e80-39a1-43ac-90e7-cbf0e7816043-proxy-tls\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:49:27.105700 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:27.105419    2881 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75b46e80-39a1-43ac-90e7-cbf0e7816043-cabundle-cert\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:49:27.616563 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:27.616518    2881 generic.go:356] "Generic (PLEG): container finished" podID="75b46e80-39a1-43ac-90e7-cbf0e7816043" containerID="20a5b1f5b50b62a30d71f618eac98393ccc15040012be85c379815ebe405c7b5" exitCode=0
Sep 18 17:49:27.616743 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:27.616574    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-s679h" event={"ID":"75b46e80-39a1-43ac-90e7-cbf0e7816043","Type":"ContainerDied","Data":"20a5b1f5b50b62a30d71f618eac98393ccc15040012be85c379815ebe405c7b5"}
Sep 18 17:49:27.616743 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:27.616593    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-s679h"
Sep 18 17:49:27.616743 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:27.616659    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-s679h" event={"ID":"75b46e80-39a1-43ac-90e7-cbf0e7816043","Type":"ContainerDied","Data":"d60f4e5c443ba780977d005844b1fe77d47d616df5a8b4c9577da2ffbd9d089a"}
Sep 18 17:49:27.616743 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:27.616687    2881 scope.go:117] "RemoveContainer" containerID="f76c0cd53738634740094553e510ccfe6d93a753281271da033d0d00d4944abd"
Sep 18 17:49:27.626339 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:27.626318    2881 scope.go:117] "RemoveContainer" containerID="20a5b1f5b50b62a30d71f618eac98393ccc15040012be85c379815ebe405c7b5"
Sep 18 17:49:27.637414 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:27.637393    2881 scope.go:117] "RemoveContainer" containerID="0ac39dd227346e36c1b95f131ac0a9321bc4fe8861bfcfb6c924fa55c274b9bc"
Sep 18 17:49:27.642659 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:27.642629    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-s679h"]
Sep 18 17:49:27.646208 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:27.646184    2881 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-s679h"]
Sep 18 17:49:27.646695 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:27.646679    2881 scope.go:117] "RemoveContainer" containerID="f76c0cd53738634740094553e510ccfe6d93a753281271da033d0d00d4944abd"
Sep 18 17:49:27.647009 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:49:27.646991    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f76c0cd53738634740094553e510ccfe6d93a753281271da033d0d00d4944abd\": container with ID starting with f76c0cd53738634740094553e510ccfe6d93a753281271da033d0d00d4944abd not found: ID does not exist" containerID="f76c0cd53738634740094553e510ccfe6d93a753281271da033d0d00d4944abd"
Sep 18 17:49:27.647062 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:27.647019    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f76c0cd53738634740094553e510ccfe6d93a753281271da033d0d00d4944abd"} err="failed to get container status \"f76c0cd53738634740094553e510ccfe6d93a753281271da033d0d00d4944abd\": rpc error: code = NotFound desc = could not find container \"f76c0cd53738634740094553e510ccfe6d93a753281271da033d0d00d4944abd\": container with ID starting with f76c0cd53738634740094553e510ccfe6d93a753281271da033d0d00d4944abd not found: ID does not exist"
Sep 18 17:49:27.647062 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:27.647039    2881 scope.go:117] "RemoveContainer" containerID="20a5b1f5b50b62a30d71f618eac98393ccc15040012be85c379815ebe405c7b5"
Sep 18 17:49:27.647306 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:49:27.647290    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20a5b1f5b50b62a30d71f618eac98393ccc15040012be85c379815ebe405c7b5\": container with ID starting with 20a5b1f5b50b62a30d71f618eac98393ccc15040012be85c379815ebe405c7b5 not found: ID does not exist" containerID="20a5b1f5b50b62a30d71f618eac98393ccc15040012be85c379815ebe405c7b5"
Sep 18 17:49:27.647352 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:27.647311    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20a5b1f5b50b62a30d71f618eac98393ccc15040012be85c379815ebe405c7b5"} err="failed to get container status \"20a5b1f5b50b62a30d71f618eac98393ccc15040012be85c379815ebe405c7b5\": rpc error: code = NotFound desc = could not find container \"20a5b1f5b50b62a30d71f618eac98393ccc15040012be85c379815ebe405c7b5\": container with ID starting with 20a5b1f5b50b62a30d71f618eac98393ccc15040012be85c379815ebe405c7b5 not found: ID does not exist"
Sep 18 17:49:27.647352 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:27.647325    2881 scope.go:117] "RemoveContainer" containerID="0ac39dd227346e36c1b95f131ac0a9321bc4fe8861bfcfb6c924fa55c274b9bc"
Sep 18 17:49:27.647551 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:49:27.647533    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ac39dd227346e36c1b95f131ac0a9321bc4fe8861bfcfb6c924fa55c274b9bc\": container with ID starting with 0ac39dd227346e36c1b95f131ac0a9321bc4fe8861bfcfb6c924fa55c274b9bc not found: ID does not exist" containerID="0ac39dd227346e36c1b95f131ac0a9321bc4fe8861bfcfb6c924fa55c274b9bc"
Sep 18 17:49:27.647593 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:27.647556    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ac39dd227346e36c1b95f131ac0a9321bc4fe8861bfcfb6c924fa55c274b9bc"} err="failed to get container status \"0ac39dd227346e36c1b95f131ac0a9321bc4fe8861bfcfb6c924fa55c274b9bc\": rpc error: code = NotFound desc = could not find container \"0ac39dd227346e36c1b95f131ac0a9321bc4fe8861bfcfb6c924fa55c274b9bc\": container with ID starting with 0ac39dd227346e36c1b95f131ac0a9321bc4fe8861bfcfb6c924fa55c274b9bc not found: ID does not exist"
Sep 18 17:49:27.893573 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:27.893486    2881 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75b46e80-39a1-43ac-90e7-cbf0e7816043" path="/var/lib/kubelet/pods/75b46e80-39a1-43ac-90e7-cbf0e7816043/volumes"
Sep 18 17:49:28.550787 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:28.550749    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-74lfl"]
Sep 18 17:49:28.551192 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:28.551120    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75b46e80-39a1-43ac-90e7-cbf0e7816043" containerName="storage-initializer"
Sep 18 17:49:28.551192 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:28.551136    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="75b46e80-39a1-43ac-90e7-cbf0e7816043" containerName="storage-initializer"
Sep 18 17:49:28.551192 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:28.551155    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75b46e80-39a1-43ac-90e7-cbf0e7816043" containerName="kserve-container"
Sep 18 17:49:28.551192 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:28.551160    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="75b46e80-39a1-43ac-90e7-cbf0e7816043" containerName="kserve-container"
Sep 18 17:49:28.551192 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:28.551180    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75b46e80-39a1-43ac-90e7-cbf0e7816043" containerName="kube-rbac-proxy"
Sep 18 17:49:28.551192 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:28.551186    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="75b46e80-39a1-43ac-90e7-cbf0e7816043" containerName="kube-rbac-proxy"
Sep 18 17:49:28.551444 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:28.551261    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="75b46e80-39a1-43ac-90e7-cbf0e7816043" containerName="kube-rbac-proxy"
Sep 18 17:49:28.551444 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:28.551275    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="75b46e80-39a1-43ac-90e7-cbf0e7816043" containerName="kserve-container"
Sep 18 17:49:28.556073 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:28.556050    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-74lfl"
Sep 18 17:49:28.558904 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:28.558879    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-predictor-serving-cert\""
Sep 18 17:49:28.559069 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:28.558879    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 17:49:28.559069 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:28.558951    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-kube-rbac-proxy-sar-config\""
Sep 18 17:49:28.568827 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:28.568801    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-74lfl"]
Sep 18 17:49:28.722114 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:28.722068    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/edf79def-78af-4176-8a84-e4743b413dd6-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-74lfl\" (UID: \"edf79def-78af-4176-8a84-e4743b413dd6\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-74lfl"
Sep 18 17:49:28.722359 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:28.722124    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7q8m\" (UniqueName: \"kubernetes.io/projected/edf79def-78af-4176-8a84-e4743b413dd6-kube-api-access-p7q8m\") pod \"isvc-pmml-predictor-8674795f59-74lfl\" (UID: \"edf79def-78af-4176-8a84-e4743b413dd6\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-74lfl"
Sep 18 17:49:28.722359 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:28.722196    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/edf79def-78af-4176-8a84-e4743b413dd6-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-74lfl\" (UID: \"edf79def-78af-4176-8a84-e4743b413dd6\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-74lfl"
Sep 18 17:49:28.722359 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:28.722282    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/edf79def-78af-4176-8a84-e4743b413dd6-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-74lfl\" (UID: \"edf79def-78af-4176-8a84-e4743b413dd6\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-74lfl"
Sep 18 17:49:28.722359 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:28.722342    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/edf79def-78af-4176-8a84-e4743b413dd6-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-74lfl\" (UID: \"edf79def-78af-4176-8a84-e4743b413dd6\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-74lfl"
Sep 18 17:49:28.823208 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:28.823108    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/edf79def-78af-4176-8a84-e4743b413dd6-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-74lfl\" (UID: \"edf79def-78af-4176-8a84-e4743b413dd6\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-74lfl"
Sep 18 17:49:28.823208 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:28.823152    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7q8m\" (UniqueName: \"kubernetes.io/projected/edf79def-78af-4176-8a84-e4743b413dd6-kube-api-access-p7q8m\") pod \"isvc-pmml-predictor-8674795f59-74lfl\" (UID: \"edf79def-78af-4176-8a84-e4743b413dd6\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-74lfl"
Sep 18 17:49:28.823208 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:28.823193    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/edf79def-78af-4176-8a84-e4743b413dd6-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-74lfl\" (UID: \"edf79def-78af-4176-8a84-e4743b413dd6\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-74lfl"
Sep 18 17:49:28.823553 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:28.823218    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/edf79def-78af-4176-8a84-e4743b413dd6-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-74lfl\" (UID: \"edf79def-78af-4176-8a84-e4743b413dd6\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-74lfl"
Sep 18 17:49:28.823553 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:28.823299    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/edf79def-78af-4176-8a84-e4743b413dd6-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-74lfl\" (UID: \"edf79def-78af-4176-8a84-e4743b413dd6\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-74lfl"
Sep 18 17:49:28.823803 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:28.823780    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/edf79def-78af-4176-8a84-e4743b413dd6-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-74lfl\" (UID: \"edf79def-78af-4176-8a84-e4743b413dd6\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-74lfl"
Sep 18 17:49:28.824005 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:28.823981    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/edf79def-78af-4176-8a84-e4743b413dd6-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-74lfl\" (UID: \"edf79def-78af-4176-8a84-e4743b413dd6\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-74lfl"
Sep 18 17:49:28.824077 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:28.824043    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/edf79def-78af-4176-8a84-e4743b413dd6-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-74lfl\" (UID: \"edf79def-78af-4176-8a84-e4743b413dd6\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-74lfl"
Sep 18 17:49:28.825731 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:28.825712    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/edf79def-78af-4176-8a84-e4743b413dd6-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-74lfl\" (UID: \"edf79def-78af-4176-8a84-e4743b413dd6\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-74lfl"
Sep 18 17:49:28.833178 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:28.833151    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7q8m\" (UniqueName: \"kubernetes.io/projected/edf79def-78af-4176-8a84-e4743b413dd6-kube-api-access-p7q8m\") pod \"isvc-pmml-predictor-8674795f59-74lfl\" (UID: \"edf79def-78af-4176-8a84-e4743b413dd6\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-74lfl"
Sep 18 17:49:28.867589 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:28.867529    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-74lfl"
Sep 18 17:49:29.147465 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:29.147438    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-74lfl"]
Sep 18 17:49:29.149361 ip-10-0-143-5 kubenswrapper[2881]: W0918 17:49:29.149333    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedf79def_78af_4176_8a84_e4743b413dd6.slice/crio-f2edf86e8e5bf542029d0bdaf896a73ce691dd68e34dcf219444d4569def61bc WatchSource:0}: Error finding container f2edf86e8e5bf542029d0bdaf896a73ce691dd68e34dcf219444d4569def61bc: Status 404 returned error can't find the container with id f2edf86e8e5bf542029d0bdaf896a73ce691dd68e34dcf219444d4569def61bc
Sep 18 17:49:29.630593 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:29.630557    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-74lfl" event={"ID":"edf79def-78af-4176-8a84-e4743b413dd6","Type":"ContainerStarted","Data":"e6cc8133b7bfbee3fdd902a06a0236f6fda6d0dd15f25033c1e4b4b6b1f06553"}
Sep 18 17:49:29.631024 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:29.630621    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-74lfl" event={"ID":"edf79def-78af-4176-8a84-e4743b413dd6","Type":"ContainerStarted","Data":"f2edf86e8e5bf542029d0bdaf896a73ce691dd68e34dcf219444d4569def61bc"}
Sep 18 17:49:33.651789 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:33.651755    2881 generic.go:356] "Generic (PLEG): container finished" podID="edf79def-78af-4176-8a84-e4743b413dd6" containerID="e6cc8133b7bfbee3fdd902a06a0236f6fda6d0dd15f25033c1e4b4b6b1f06553" exitCode=0
Sep 18 17:49:33.652198 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:33.651820    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-74lfl" event={"ID":"edf79def-78af-4176-8a84-e4743b413dd6","Type":"ContainerDied","Data":"e6cc8133b7bfbee3fdd902a06a0236f6fda6d0dd15f25033c1e4b4b6b1f06553"}
Sep 18 17:49:33.653101 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:33.653084    2881 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 17:49:34.233798 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:34.233753    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-f9f48c85f-x4cmz"]
Sep 18 17:49:34.234128 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:34.234090    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-f9f48c85f-x4cmz" podUID="57666c50-c51f-4142-8007-0e2a4e3ea84f" containerName="kserve-container" containerID="cri-o://f6cc63abaf6a57e08ea7be84d07ad95c169a9d42d448cad17897f74f9ffcfd0e" gracePeriod=30
Sep 18 17:49:34.234515 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:34.234485    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-f9f48c85f-x4cmz" podUID="57666c50-c51f-4142-8007-0e2a4e3ea84f" containerName="kube-rbac-proxy" containerID="cri-o://3aee5c01b113da88a7baf99dadd9666b16b15f05a56787bd5d55cb427564a368" gracePeriod=30
Sep 18 17:49:34.663748 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:34.663619    2881 generic.go:356] "Generic (PLEG): container finished" podID="57666c50-c51f-4142-8007-0e2a4e3ea84f" containerID="3aee5c01b113da88a7baf99dadd9666b16b15f05a56787bd5d55cb427564a368" exitCode=2
Sep 18 17:49:34.663748 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:34.663691    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-f9f48c85f-x4cmz" event={"ID":"57666c50-c51f-4142-8007-0e2a4e3ea84f","Type":"ContainerDied","Data":"3aee5c01b113da88a7baf99dadd9666b16b15f05a56787bd5d55cb427564a368"}
Sep 18 17:49:35.319532 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:35.318620    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5748bb89bf-fk78d"]
Sep 18 17:49:35.327561 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:35.327518    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5748bb89bf-fk78d"
Sep 18 17:49:35.333491 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:35.332072    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\""
Sep 18 17:49:35.333491 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:35.332694    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert\""
Sep 18 17:49:35.338673 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:35.335865    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5748bb89bf-fk78d"]
Sep 18 17:49:35.388520 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:35.388243    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c32c7723-96a5-4976-b738-cd266e9857f9-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5748bb89bf-fk78d\" (UID: \"c32c7723-96a5-4976-b738-cd266e9857f9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5748bb89bf-fk78d"
Sep 18 17:49:35.388520 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:35.388313    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c32c7723-96a5-4976-b738-cd266e9857f9-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5748bb89bf-fk78d\" (UID: \"c32c7723-96a5-4976-b738-cd266e9857f9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5748bb89bf-fk78d"
Sep 18 17:49:35.388520 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:35.388349    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c32c7723-96a5-4976-b738-cd266e9857f9-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5748bb89bf-fk78d\" (UID: \"c32c7723-96a5-4976-b738-cd266e9857f9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5748bb89bf-fk78d"
Sep 18 17:49:35.388520 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:35.388424    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88wmt\" (UniqueName: \"kubernetes.io/projected/c32c7723-96a5-4976-b738-cd266e9857f9-kube-api-access-88wmt\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5748bb89bf-fk78d\" (UID: \"c32c7723-96a5-4976-b738-cd266e9857f9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5748bb89bf-fk78d"
Sep 18 17:49:35.489557 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:35.489506    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88wmt\" (UniqueName: \"kubernetes.io/projected/c32c7723-96a5-4976-b738-cd266e9857f9-kube-api-access-88wmt\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5748bb89bf-fk78d\" (UID: \"c32c7723-96a5-4976-b738-cd266e9857f9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5748bb89bf-fk78d"
Sep 18 17:49:35.489763 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:35.489606    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c32c7723-96a5-4976-b738-cd266e9857f9-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5748bb89bf-fk78d\" (UID: \"c32c7723-96a5-4976-b738-cd266e9857f9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5748bb89bf-fk78d"
Sep 18 17:49:35.489763 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:35.489653    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c32c7723-96a5-4976-b738-cd266e9857f9-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5748bb89bf-fk78d\" (UID: \"c32c7723-96a5-4976-b738-cd266e9857f9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5748bb89bf-fk78d"
Sep 18 17:49:35.489763 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:35.489688    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c32c7723-96a5-4976-b738-cd266e9857f9-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5748bb89bf-fk78d\" (UID: \"c32c7723-96a5-4976-b738-cd266e9857f9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5748bb89bf-fk78d"
Sep 18 17:49:35.491435 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:35.490375    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c32c7723-96a5-4976-b738-cd266e9857f9-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5748bb89bf-fk78d\" (UID: \"c32c7723-96a5-4976-b738-cd266e9857f9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5748bb89bf-fk78d"
Sep 18 17:49:35.491435 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:35.491380    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c32c7723-96a5-4976-b738-cd266e9857f9-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5748bb89bf-fk78d\" (UID: \"c32c7723-96a5-4976-b738-cd266e9857f9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5748bb89bf-fk78d"
Sep 18 17:49:35.493125 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:35.493102    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c32c7723-96a5-4976-b738-cd266e9857f9-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5748bb89bf-fk78d\" (UID: \"c32c7723-96a5-4976-b738-cd266e9857f9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5748bb89bf-fk78d"
Sep 18 17:49:35.503131 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:35.503068    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88wmt\" (UniqueName: \"kubernetes.io/projected/c32c7723-96a5-4976-b738-cd266e9857f9-kube-api-access-88wmt\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5748bb89bf-fk78d\" (UID: \"c32c7723-96a5-4976-b738-cd266e9857f9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5748bb89bf-fk78d"
Sep 18 17:49:35.653252 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:35.653143    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5748bb89bf-fk78d"
Sep 18 17:49:36.801980 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:36.801870    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5748bb89bf-fk78d"]
Sep 18 17:49:38.159071 ip-10-0-143-5 kubenswrapper[2881]: W0918 17:49:38.159039    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc32c7723_96a5_4976_b738_cd266e9857f9.slice/crio-a700325189388245d6a615e478894dea0f28e78bfb3d85ab4aed8f2a8129112d WatchSource:0}: Error finding container a700325189388245d6a615e478894dea0f28e78bfb3d85ab4aed8f2a8129112d: Status 404 returned error can't find the container with id a700325189388245d6a615e478894dea0f28e78bfb3d85ab4aed8f2a8129112d
Sep 18 17:49:38.691574 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:38.691475    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5748bb89bf-fk78d" event={"ID":"c32c7723-96a5-4976-b738-cd266e9857f9","Type":"ContainerStarted","Data":"a700325189388245d6a615e478894dea0f28e78bfb3d85ab4aed8f2a8129112d"}
Sep 18 17:49:39.161521 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:39.161470    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-f9f48c85f-x4cmz" podUID="57666c50-c51f-4142-8007-0e2a4e3ea84f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.69:8643/healthz\": dial tcp 10.134.0.69:8643: connect: connection refused"
Sep 18 17:49:41.179320 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:41.179294    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-f9f48c85f-x4cmz"
Sep 18 17:49:41.244657 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:41.244627    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57666c50-c51f-4142-8007-0e2a4e3ea84f-kube-rbac-proxy-sar-config\") pod \"57666c50-c51f-4142-8007-0e2a4e3ea84f\" (UID: \"57666c50-c51f-4142-8007-0e2a4e3ea84f\") "
Sep 18 17:49:41.244787 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:41.244683    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57666c50-c51f-4142-8007-0e2a4e3ea84f-kserve-provision-location\") pod \"57666c50-c51f-4142-8007-0e2a4e3ea84f\" (UID: \"57666c50-c51f-4142-8007-0e2a4e3ea84f\") "
Sep 18 17:49:41.244787 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:41.244712    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57666c50-c51f-4142-8007-0e2a4e3ea84f-cabundle-cert\") pod \"57666c50-c51f-4142-8007-0e2a4e3ea84f\" (UID: \"57666c50-c51f-4142-8007-0e2a4e3ea84f\") "
Sep 18 17:49:41.244924 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:41.244900    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57666c50-c51f-4142-8007-0e2a4e3ea84f-proxy-tls\") pod \"57666c50-c51f-4142-8007-0e2a4e3ea84f\" (UID: \"57666c50-c51f-4142-8007-0e2a4e3ea84f\") "
Sep 18 17:49:41.245020 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:41.244947    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8brvh\" (UniqueName: \"kubernetes.io/projected/57666c50-c51f-4142-8007-0e2a4e3ea84f-kube-api-access-8brvh\") pod \"57666c50-c51f-4142-8007-0e2a4e3ea84f\" (UID: \"57666c50-c51f-4142-8007-0e2a4e3ea84f\") "
Sep 18 17:49:41.245359 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:41.245081    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57666c50-c51f-4142-8007-0e2a4e3ea84f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "57666c50-c51f-4142-8007-0e2a4e3ea84f" (UID: "57666c50-c51f-4142-8007-0e2a4e3ea84f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:49:41.245359 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:41.245123    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57666c50-c51f-4142-8007-0e2a4e3ea84f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "57666c50-c51f-4142-8007-0e2a4e3ea84f" (UID: "57666c50-c51f-4142-8007-0e2a4e3ea84f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:49:41.245359 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:41.245141    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57666c50-c51f-4142-8007-0e2a4e3ea84f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "57666c50-c51f-4142-8007-0e2a4e3ea84f" (UID: "57666c50-c51f-4142-8007-0e2a4e3ea84f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:49:41.245618 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:41.245374    2881 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57666c50-c51f-4142-8007-0e2a4e3ea84f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:49:41.245618 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:41.245458    2881 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57666c50-c51f-4142-8007-0e2a4e3ea84f-kserve-provision-location\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:49:41.245618 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:41.245477    2881 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57666c50-c51f-4142-8007-0e2a4e3ea84f-cabundle-cert\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:49:41.247516 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:41.247488    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57666c50-c51f-4142-8007-0e2a4e3ea84f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "57666c50-c51f-4142-8007-0e2a4e3ea84f" (UID: "57666c50-c51f-4142-8007-0e2a4e3ea84f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:49:41.247609 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:41.247512    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57666c50-c51f-4142-8007-0e2a4e3ea84f-kube-api-access-8brvh" (OuterVolumeSpecName: "kube-api-access-8brvh") pod "57666c50-c51f-4142-8007-0e2a4e3ea84f" (UID: "57666c50-c51f-4142-8007-0e2a4e3ea84f"). InnerVolumeSpecName "kube-api-access-8brvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:49:41.345945 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:41.345914    2881 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57666c50-c51f-4142-8007-0e2a4e3ea84f-proxy-tls\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:49:41.345945 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:41.345944    2881 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8brvh\" (UniqueName: \"kubernetes.io/projected/57666c50-c51f-4142-8007-0e2a4e3ea84f-kube-api-access-8brvh\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:49:41.710994 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:41.710954    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-74lfl" event={"ID":"edf79def-78af-4176-8a84-e4743b413dd6","Type":"ContainerStarted","Data":"e8cd6c41e639b2b456fa8013c0f67661371423bac7044a1f2cbe0a4e3aa1924f"}
Sep 18 17:49:41.711197 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:41.711033    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-74lfl" event={"ID":"edf79def-78af-4176-8a84-e4743b413dd6","Type":"ContainerStarted","Data":"494322e37768d9dc65bf6297379e8cd03ce2c217f32ea10e3199e727cd7308da"}
Sep 18 17:49:41.711365 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:41.711343    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-74lfl"
Sep 18 17:49:41.711536 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:41.711520    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-74lfl"
Sep 18 17:49:41.713120 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:41.713086    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5748bb89bf-fk78d" event={"ID":"c32c7723-96a5-4976-b738-cd266e9857f9","Type":"ContainerStarted","Data":"59352c951be8caa8c98f32260bd4c8af1d235581bc752407b4cb50c30e01a3e1"}
Sep 18 17:49:41.713283 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:41.713153    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-74lfl" podUID="edf79def-78af-4176-8a84-e4743b413dd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.71:8080: connect: connection refused"
Sep 18 17:49:41.715324 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:41.715302    2881 generic.go:356] "Generic (PLEG): container finished" podID="57666c50-c51f-4142-8007-0e2a4e3ea84f" containerID="f6cc63abaf6a57e08ea7be84d07ad95c169a9d42d448cad17897f74f9ffcfd0e" exitCode=0
Sep 18 17:49:41.715528 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:41.715343    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-f9f48c85f-x4cmz" event={"ID":"57666c50-c51f-4142-8007-0e2a4e3ea84f","Type":"ContainerDied","Data":"f6cc63abaf6a57e08ea7be84d07ad95c169a9d42d448cad17897f74f9ffcfd0e"}
Sep 18 17:49:41.715528 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:41.715373    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-f9f48c85f-x4cmz"
Sep 18 17:49:41.715528 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:41.715384    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-f9f48c85f-x4cmz" event={"ID":"57666c50-c51f-4142-8007-0e2a4e3ea84f","Type":"ContainerDied","Data":"b9cfcc5fd8d35cf19425e3c8655d60ef500007124db374dfcb85ea4c5c68f90a"}
Sep 18 17:49:41.715528 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:41.715406    2881 scope.go:117] "RemoveContainer" containerID="3aee5c01b113da88a7baf99dadd9666b16b15f05a56787bd5d55cb427564a368"
Sep 18 17:49:41.724140 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:41.724119    2881 scope.go:117] "RemoveContainer" containerID="f6cc63abaf6a57e08ea7be84d07ad95c169a9d42d448cad17897f74f9ffcfd0e"
Sep 18 17:49:41.733455 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:41.733288    2881 scope.go:117] "RemoveContainer" containerID="a89803133fa56dc43e9e274a377ff625e86274afb64099af29f63e3cf759486b"
Sep 18 17:49:41.734366 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:41.734296    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-74lfl" podStartSLOduration=2.124046055 podStartE2EDuration="13.734279654s" totalImagesPullingTime="7.571878099s" totalInitContainerRuntime="4.0383555s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:49:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:49:41.732458046 +0000 UTC m=+3371.932313356" watchObservedRunningTime="2026-09-18 17:49:41.734279654 +0000 UTC m=+3371.934134940"
Sep 18 17:49:41.742718 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:41.742699    2881 scope.go:117] "RemoveContainer" containerID="3aee5c01b113da88a7baf99dadd9666b16b15f05a56787bd5d55cb427564a368"
Sep 18 17:49:41.743049 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:49:41.743027    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3aee5c01b113da88a7baf99dadd9666b16b15f05a56787bd5d55cb427564a368\": container with ID starting with 3aee5c01b113da88a7baf99dadd9666b16b15f05a56787bd5d55cb427564a368 not found: ID does not exist" containerID="3aee5c01b113da88a7baf99dadd9666b16b15f05a56787bd5d55cb427564a368"
Sep 18 17:49:41.743145 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:41.743055    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3aee5c01b113da88a7baf99dadd9666b16b15f05a56787bd5d55cb427564a368"} err="failed to get container status \"3aee5c01b113da88a7baf99dadd9666b16b15f05a56787bd5d55cb427564a368\": rpc error: code = NotFound desc = could not find container \"3aee5c01b113da88a7baf99dadd9666b16b15f05a56787bd5d55cb427564a368\": container with ID starting with 3aee5c01b113da88a7baf99dadd9666b16b15f05a56787bd5d55cb427564a368 not found: ID does not exist"
Sep 18 17:49:41.743145 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:41.743074    2881 scope.go:117] "RemoveContainer" containerID="f6cc63abaf6a57e08ea7be84d07ad95c169a9d42d448cad17897f74f9ffcfd0e"
Sep 18 17:49:41.743444 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:49:41.743425    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6cc63abaf6a57e08ea7be84d07ad95c169a9d42d448cad17897f74f9ffcfd0e\": container with ID starting with f6cc63abaf6a57e08ea7be84d07ad95c169a9d42d448cad17897f74f9ffcfd0e not found: ID does not exist" containerID="f6cc63abaf6a57e08ea7be84d07ad95c169a9d42d448cad17897f74f9ffcfd0e"
Sep 18 17:49:41.743498 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:41.743449    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6cc63abaf6a57e08ea7be84d07ad95c169a9d42d448cad17897f74f9ffcfd0e"} err="failed to get container status \"f6cc63abaf6a57e08ea7be84d07ad95c169a9d42d448cad17897f74f9ffcfd0e\": rpc error: code = NotFound desc = could not find container \"f6cc63abaf6a57e08ea7be84d07ad95c169a9d42d448cad17897f74f9ffcfd0e\": container with ID starting with f6cc63abaf6a57e08ea7be84d07ad95c169a9d42d448cad17897f74f9ffcfd0e not found: ID does not exist"
Sep 18 17:49:41.743498 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:41.743465    2881 scope.go:117] "RemoveContainer" containerID="a89803133fa56dc43e9e274a377ff625e86274afb64099af29f63e3cf759486b"
Sep 18 17:49:41.743735 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:49:41.743721    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a89803133fa56dc43e9e274a377ff625e86274afb64099af29f63e3cf759486b\": container with ID starting with a89803133fa56dc43e9e274a377ff625e86274afb64099af29f63e3cf759486b not found: ID does not exist" containerID="a89803133fa56dc43e9e274a377ff625e86274afb64099af29f63e3cf759486b"
Sep 18 17:49:41.743778 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:41.743737    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a89803133fa56dc43e9e274a377ff625e86274afb64099af29f63e3cf759486b"} err="failed to get container status \"a89803133fa56dc43e9e274a377ff625e86274afb64099af29f63e3cf759486b\": rpc error: code = NotFound desc = could not find container \"a89803133fa56dc43e9e274a377ff625e86274afb64099af29f63e3cf759486b\": container with ID starting with a89803133fa56dc43e9e274a377ff625e86274afb64099af29f63e3cf759486b not found: ID does not exist"
Sep 18 17:49:41.767311 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:41.767272    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-f9f48c85f-x4cmz"]
Sep 18 17:49:41.771996 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:41.771968    2881 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-f9f48c85f-x4cmz"]
Sep 18 17:49:41.896954 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:41.896918    2881 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57666c50-c51f-4142-8007-0e2a4e3ea84f" path="/var/lib/kubelet/pods/57666c50-c51f-4142-8007-0e2a4e3ea84f/volumes"
Sep 18 17:49:42.722535 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:42.722497    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-74lfl" podUID="edf79def-78af-4176-8a84-e4743b413dd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.71:8080: connect: connection refused"
Sep 18 17:49:46.746193 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:46.746165    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-5748bb89bf-fk78d_c32c7723-96a5-4976-b738-cd266e9857f9/storage-initializer/0.log"
Sep 18 17:49:46.746738 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:46.746206    2881 generic.go:356] "Generic (PLEG): container finished" podID="c32c7723-96a5-4976-b738-cd266e9857f9" containerID="59352c951be8caa8c98f32260bd4c8af1d235581bc752407b4cb50c30e01a3e1" exitCode=1
Sep 18 17:49:46.746738 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:46.746267    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5748bb89bf-fk78d" event={"ID":"c32c7723-96a5-4976-b738-cd266e9857f9","Type":"ContainerDied","Data":"59352c951be8caa8c98f32260bd4c8af1d235581bc752407b4cb50c30e01a3e1"}
Sep 18 17:49:47.727561 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:47.727527    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-74lfl"
Sep 18 17:49:47.728088 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:47.728061    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-74lfl" podUID="edf79def-78af-4176-8a84-e4743b413dd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.71:8080: connect: connection refused"
Sep 18 17:49:47.752818 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:47.752794    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-5748bb89bf-fk78d_c32c7723-96a5-4976-b738-cd266e9857f9/storage-initializer/0.log"
Sep 18 17:49:47.753202 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:47.752852    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5748bb89bf-fk78d" event={"ID":"c32c7723-96a5-4976-b738-cd266e9857f9","Type":"ContainerStarted","Data":"05a30361bd4c6efa6f87a95cde58f46ea76dddf96bcd6284b0f0f7f691539668"}
Sep 18 17:49:52.779819 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:52.779791    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-5748bb89bf-fk78d_c32c7723-96a5-4976-b738-cd266e9857f9/storage-initializer/1.log"
Sep 18 17:49:52.780277 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:52.780247    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-5748bb89bf-fk78d_c32c7723-96a5-4976-b738-cd266e9857f9/storage-initializer/0.log"
Sep 18 17:49:52.780336 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:52.780279    2881 generic.go:356] "Generic (PLEG): container finished" podID="c32c7723-96a5-4976-b738-cd266e9857f9" containerID="05a30361bd4c6efa6f87a95cde58f46ea76dddf96bcd6284b0f0f7f691539668" exitCode=1
Sep 18 17:49:52.780393 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:52.780371    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5748bb89bf-fk78d" event={"ID":"c32c7723-96a5-4976-b738-cd266e9857f9","Type":"ContainerDied","Data":"05a30361bd4c6efa6f87a95cde58f46ea76dddf96bcd6284b0f0f7f691539668"}
Sep 18 17:49:52.780471 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:52.780458    2881 scope.go:117] "RemoveContainer" containerID="59352c951be8caa8c98f32260bd4c8af1d235581bc752407b4cb50c30e01a3e1"
Sep 18 17:49:52.780906 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:52.780888    2881 scope.go:117] "RemoveContainer" containerID="59352c951be8caa8c98f32260bd4c8af1d235581bc752407b4cb50c30e01a3e1"
Sep 18 17:49:52.791685 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:49:52.791627    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-serving-fail-predictor-5748bb89bf-fk78d_kserve-ci-e2e-test(c32c7723-96a5-4976-b738-cd266e9857f9)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5748bb89bf-fk78d" podUID="c32c7723-96a5-4976-b738-cd266e9857f9"
Sep 18 17:49:52.791685 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:49:52.791650    2881 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-serving-fail-predictor-5748bb89bf-fk78d_kserve-ci-e2e-test_c32c7723-96a5-4976-b738-cd266e9857f9_0 in pod sandbox a700325189388245d6a615e478894dea0f28e78bfb3d85ab4aed8f2a8129112d from index: no such id: '59352c951be8caa8c98f32260bd4c8af1d235581bc752407b4cb50c30e01a3e1'" containerID="59352c951be8caa8c98f32260bd4c8af1d235581bc752407b4cb50c30e01a3e1"
Sep 18 17:49:52.791865 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:52.791682    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59352c951be8caa8c98f32260bd4c8af1d235581bc752407b4cb50c30e01a3e1"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-serving-fail-predictor-5748bb89bf-fk78d_kserve-ci-e2e-test_c32c7723-96a5-4976-b738-cd266e9857f9_0 in pod sandbox a700325189388245d6a615e478894dea0f28e78bfb3d85ab4aed8f2a8129112d from index: no such id: '59352c951be8caa8c98f32260bd4c8af1d235581bc752407b4cb50c30e01a3e1'"
Sep 18 17:49:53.786702 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:53.786676    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-5748bb89bf-fk78d_c32c7723-96a5-4976-b738-cd266e9857f9/storage-initializer/1.log"
Sep 18 17:49:55.276415 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:55.276374    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5748bb89bf-fk78d"]
Sep 18 17:49:55.409445 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:55.409419    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-5748bb89bf-fk78d_c32c7723-96a5-4976-b738-cd266e9857f9/storage-initializer/1.log"
Sep 18 17:49:55.411619 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:55.411601    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5748bb89bf-fk78d"
Sep 18 17:49:55.475763 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:55.475725    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-88wmt\" (UniqueName: \"kubernetes.io/projected/c32c7723-96a5-4976-b738-cd266e9857f9-kube-api-access-88wmt\") pod \"c32c7723-96a5-4976-b738-cd266e9857f9\" (UID: \"c32c7723-96a5-4976-b738-cd266e9857f9\") "
Sep 18 17:49:55.475763 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:55.475769    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c32c7723-96a5-4976-b738-cd266e9857f9-kserve-provision-location\") pod \"c32c7723-96a5-4976-b738-cd266e9857f9\" (UID: \"c32c7723-96a5-4976-b738-cd266e9857f9\") "
Sep 18 17:49:55.476060 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:55.475798    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c32c7723-96a5-4976-b738-cd266e9857f9-kube-rbac-proxy-sar-config\") pod \"c32c7723-96a5-4976-b738-cd266e9857f9\" (UID: \"c32c7723-96a5-4976-b738-cd266e9857f9\") "
Sep 18 17:49:55.476060 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:55.475817    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c32c7723-96a5-4976-b738-cd266e9857f9-proxy-tls\") pod \"c32c7723-96a5-4976-b738-cd266e9857f9\" (UID: \"c32c7723-96a5-4976-b738-cd266e9857f9\") "
Sep 18 17:49:55.476172 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:55.476091    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c32c7723-96a5-4976-b738-cd266e9857f9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c32c7723-96a5-4976-b738-cd266e9857f9" (UID: "c32c7723-96a5-4976-b738-cd266e9857f9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:49:55.476172 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:55.476127    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c32c7723-96a5-4976-b738-cd266e9857f9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c32c7723-96a5-4976-b738-cd266e9857f9" (UID: "c32c7723-96a5-4976-b738-cd266e9857f9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:49:55.477897 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:55.477874    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c32c7723-96a5-4976-b738-cd266e9857f9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c32c7723-96a5-4976-b738-cd266e9857f9" (UID: "c32c7723-96a5-4976-b738-cd266e9857f9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:49:55.477978 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:55.477929    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c32c7723-96a5-4976-b738-cd266e9857f9-kube-api-access-88wmt" (OuterVolumeSpecName: "kube-api-access-88wmt") pod "c32c7723-96a5-4976-b738-cd266e9857f9" (UID: "c32c7723-96a5-4976-b738-cd266e9857f9"). InnerVolumeSpecName "kube-api-access-88wmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:49:55.577061 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:55.576969    2881 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88wmt\" (UniqueName: \"kubernetes.io/projected/c32c7723-96a5-4976-b738-cd266e9857f9-kube-api-access-88wmt\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:49:55.577061 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:55.577005    2881 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c32c7723-96a5-4976-b738-cd266e9857f9-kserve-provision-location\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:49:55.577061 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:55.577016    2881 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c32c7723-96a5-4976-b738-cd266e9857f9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:49:55.577061 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:55.577025    2881 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c32c7723-96a5-4976-b738-cd266e9857f9-proxy-tls\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:49:55.798369 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:55.798341    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-5748bb89bf-fk78d_c32c7723-96a5-4976-b738-cd266e9857f9/storage-initializer/1.log"
Sep 18 17:49:55.798552 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:55.798441    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5748bb89bf-fk78d"
Sep 18 17:49:55.798552 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:55.798457    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5748bb89bf-fk78d" event={"ID":"c32c7723-96a5-4976-b738-cd266e9857f9","Type":"ContainerDied","Data":"a700325189388245d6a615e478894dea0f28e78bfb3d85ab4aed8f2a8129112d"}
Sep 18 17:49:55.798552 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:55.798524    2881 scope.go:117] "RemoveContainer" containerID="05a30361bd4c6efa6f87a95cde58f46ea76dddf96bcd6284b0f0f7f691539668"
Sep 18 17:49:55.838853 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:55.838768    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5748bb89bf-fk78d"]
Sep 18 17:49:55.845100 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:55.845070    2881 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5748bb89bf-fk78d"]
Sep 18 17:49:55.893216 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:55.893184    2881 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c32c7723-96a5-4976-b738-cd266e9857f9" path="/var/lib/kubelet/pods/c32c7723-96a5-4976-b738-cd266e9857f9/volumes"
Sep 18 17:49:56.434798 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:56.434760    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-sqp78"]
Sep 18 17:49:56.435323 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:56.435303    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57666c50-c51f-4142-8007-0e2a4e3ea84f" containerName="kserve-container"
Sep 18 17:49:56.435389 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:56.435328    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="57666c50-c51f-4142-8007-0e2a4e3ea84f" containerName="kserve-container"
Sep 18 17:49:56.435389 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:56.435344    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c32c7723-96a5-4976-b738-cd266e9857f9" containerName="storage-initializer"
Sep 18 17:49:56.435389 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:56.435353    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="c32c7723-96a5-4976-b738-cd266e9857f9" containerName="storage-initializer"
Sep 18 17:49:56.435389 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:56.435371    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57666c50-c51f-4142-8007-0e2a4e3ea84f" containerName="storage-initializer"
Sep 18 17:49:56.435389 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:56.435381    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="57666c50-c51f-4142-8007-0e2a4e3ea84f" containerName="storage-initializer"
Sep 18 17:49:56.435560 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:56.435419    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57666c50-c51f-4142-8007-0e2a4e3ea84f" containerName="kube-rbac-proxy"
Sep 18 17:49:56.435560 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:56.435428    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="57666c50-c51f-4142-8007-0e2a4e3ea84f" containerName="kube-rbac-proxy"
Sep 18 17:49:56.435560 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:56.435505    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="c32c7723-96a5-4976-b738-cd266e9857f9" containerName="storage-initializer"
Sep 18 17:49:56.435560 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:56.435517    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="57666c50-c51f-4142-8007-0e2a4e3ea84f" containerName="kserve-container"
Sep 18 17:49:56.435560 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:56.435527    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="c32c7723-96a5-4976-b738-cd266e9857f9" containerName="storage-initializer"
Sep 18 17:49:56.435560 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:56.435537    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="57666c50-c51f-4142-8007-0e2a4e3ea84f" containerName="kube-rbac-proxy"
Sep 18 17:49:56.435742 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:56.435645    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c32c7723-96a5-4976-b738-cd266e9857f9" containerName="storage-initializer"
Sep 18 17:49:56.435742 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:56.435654    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="c32c7723-96a5-4976-b738-cd266e9857f9" containerName="storage-initializer"
Sep 18 17:49:56.440270 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:56.440254    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-sqp78"
Sep 18 17:49:56.443019 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:56.442994    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-kube-rbac-proxy-sar-config\""
Sep 18 17:49:56.443178 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:56.442995    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-predictor-serving-cert\""
Sep 18 17:49:56.448642 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:56.448617    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-sqp78"]
Sep 18 17:49:56.486747 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:56.486704    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55s5n\" (UniqueName: \"kubernetes.io/projected/3e434de9-e182-41b8-9a4a-e6dd17684bf4-kube-api-access-55s5n\") pod \"isvc-pmml-runtime-predictor-885fc8777-sqp78\" (UID: \"3e434de9-e182-41b8-9a4a-e6dd17684bf4\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-sqp78"
Sep 18 17:49:56.486969 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:56.486760    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e434de9-e182-41b8-9a4a-e6dd17684bf4-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-sqp78\" (UID: \"3e434de9-e182-41b8-9a4a-e6dd17684bf4\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-sqp78"
Sep 18 17:49:56.486969 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:56.486885    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e434de9-e182-41b8-9a4a-e6dd17684bf4-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-sqp78\" (UID: \"3e434de9-e182-41b8-9a4a-e6dd17684bf4\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-sqp78"
Sep 18 17:49:56.486969 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:56.486917    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e434de9-e182-41b8-9a4a-e6dd17684bf4-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-sqp78\" (UID: \"3e434de9-e182-41b8-9a4a-e6dd17684bf4\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-sqp78"
Sep 18 17:49:56.486969 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:56.486950    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e434de9-e182-41b8-9a4a-e6dd17684bf4-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-sqp78\" (UID: \"3e434de9-e182-41b8-9a4a-e6dd17684bf4\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-sqp78"
Sep 18 17:49:56.588070 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:56.588024    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e434de9-e182-41b8-9a4a-e6dd17684bf4-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-sqp78\" (UID: \"3e434de9-e182-41b8-9a4a-e6dd17684bf4\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-sqp78"
Sep 18 17:49:56.588070 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:56.588075    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e434de9-e182-41b8-9a4a-e6dd17684bf4-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-sqp78\" (UID: \"3e434de9-e182-41b8-9a4a-e6dd17684bf4\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-sqp78"
Sep 18 17:49:56.588390 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:56.588100    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e434de9-e182-41b8-9a4a-e6dd17684bf4-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-sqp78\" (UID: \"3e434de9-e182-41b8-9a4a-e6dd17684bf4\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-sqp78"
Sep 18 17:49:56.588390 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:56.588130    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55s5n\" (UniqueName: \"kubernetes.io/projected/3e434de9-e182-41b8-9a4a-e6dd17684bf4-kube-api-access-55s5n\") pod \"isvc-pmml-runtime-predictor-885fc8777-sqp78\" (UID: \"3e434de9-e182-41b8-9a4a-e6dd17684bf4\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-sqp78"
Sep 18 17:49:56.588390 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:56.588156    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e434de9-e182-41b8-9a4a-e6dd17684bf4-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-sqp78\" (UID: \"3e434de9-e182-41b8-9a4a-e6dd17684bf4\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-sqp78"
Sep 18 17:49:56.588390 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:49:56.588193    2881 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-pmml-runtime-predictor-serving-cert: secret "isvc-pmml-runtime-predictor-serving-cert" not found
Sep 18 17:49:56.588390 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:49:56.588309    2881 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e434de9-e182-41b8-9a4a-e6dd17684bf4-proxy-tls podName:3e434de9-e182-41b8-9a4a-e6dd17684bf4 nodeName:}" failed. No retries permitted until 2026-09-18 17:49:57.088285766 +0000 UTC m=+3387.288141035 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3e434de9-e182-41b8-9a4a-e6dd17684bf4-proxy-tls") pod "isvc-pmml-runtime-predictor-885fc8777-sqp78" (UID: "3e434de9-e182-41b8-9a4a-e6dd17684bf4") : secret "isvc-pmml-runtime-predictor-serving-cert" not found
Sep 18 17:49:56.588646 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:56.588557    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e434de9-e182-41b8-9a4a-e6dd17684bf4-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-sqp78\" (UID: \"3e434de9-e182-41b8-9a4a-e6dd17684bf4\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-sqp78"
Sep 18 17:49:56.588820 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:56.588803    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e434de9-e182-41b8-9a4a-e6dd17684bf4-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-sqp78\" (UID: \"3e434de9-e182-41b8-9a4a-e6dd17684bf4\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-sqp78"
Sep 18 17:49:56.588861 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:56.588839    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e434de9-e182-41b8-9a4a-e6dd17684bf4-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-sqp78\" (UID: \"3e434de9-e182-41b8-9a4a-e6dd17684bf4\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-sqp78"
Sep 18 17:49:56.603940 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:56.603903    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55s5n\" (UniqueName: \"kubernetes.io/projected/3e434de9-e182-41b8-9a4a-e6dd17684bf4-kube-api-access-55s5n\") pod \"isvc-pmml-runtime-predictor-885fc8777-sqp78\" (UID: \"3e434de9-e182-41b8-9a4a-e6dd17684bf4\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-sqp78"
Sep 18 17:49:57.093726 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:57.093667    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e434de9-e182-41b8-9a4a-e6dd17684bf4-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-sqp78\" (UID: \"3e434de9-e182-41b8-9a4a-e6dd17684bf4\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-sqp78"
Sep 18 17:49:57.093915 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:49:57.093831    2881 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-pmml-runtime-predictor-serving-cert: secret "isvc-pmml-runtime-predictor-serving-cert" not found
Sep 18 17:49:57.093915 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:49:57.093894    2881 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e434de9-e182-41b8-9a4a-e6dd17684bf4-proxy-tls podName:3e434de9-e182-41b8-9a4a-e6dd17684bf4 nodeName:}" failed. No retries permitted until 2026-09-18 17:49:58.09387549 +0000 UTC m=+3388.293730773 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3e434de9-e182-41b8-9a4a-e6dd17684bf4-proxy-tls") pod "isvc-pmml-runtime-predictor-885fc8777-sqp78" (UID: "3e434de9-e182-41b8-9a4a-e6dd17684bf4") : secret "isvc-pmml-runtime-predictor-serving-cert" not found
Sep 18 17:49:57.728474 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:57.728432    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-74lfl" podUID="edf79def-78af-4176-8a84-e4743b413dd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.71:8080: connect: connection refused"
Sep 18 17:49:58.102918 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:58.102859    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e434de9-e182-41b8-9a4a-e6dd17684bf4-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-sqp78\" (UID: \"3e434de9-e182-41b8-9a4a-e6dd17684bf4\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-sqp78"
Sep 18 17:49:58.105490 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:58.105457    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e434de9-e182-41b8-9a4a-e6dd17684bf4-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-sqp78\" (UID: \"3e434de9-e182-41b8-9a4a-e6dd17684bf4\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-sqp78"
Sep 18 17:49:58.251622 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:58.251581    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-sqp78"
Sep 18 17:49:58.528967 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:58.528935    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-sqp78"]
Sep 18 17:49:58.531618 ip-10-0-143-5 kubenswrapper[2881]: W0918 17:49:58.531590    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e434de9_e182_41b8_9a4a_e6dd17684bf4.slice/crio-081a89f972c8d15f27a7292b25cd3359e6259ac2c92f6124532d784c33a09faa WatchSource:0}: Error finding container 081a89f972c8d15f27a7292b25cd3359e6259ac2c92f6124532d784c33a09faa: Status 404 returned error can't find the container with id 081a89f972c8d15f27a7292b25cd3359e6259ac2c92f6124532d784c33a09faa
Sep 18 17:49:58.816496 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:58.816399    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-sqp78" event={"ID":"3e434de9-e182-41b8-9a4a-e6dd17684bf4","Type":"ContainerStarted","Data":"b683f1449c23a7b2b393b0a8686cb6f566a368c5f7fd99bc9099e1543f354bb8"}
Sep 18 17:49:58.816496 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:49:58.816471    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-sqp78" event={"ID":"3e434de9-e182-41b8-9a4a-e6dd17684bf4","Type":"ContainerStarted","Data":"081a89f972c8d15f27a7292b25cd3359e6259ac2c92f6124532d784c33a09faa"}
Sep 18 17:50:02.837747 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:50:02.837648    2881 generic.go:356] "Generic (PLEG): container finished" podID="3e434de9-e182-41b8-9a4a-e6dd17684bf4" containerID="b683f1449c23a7b2b393b0a8686cb6f566a368c5f7fd99bc9099e1543f354bb8" exitCode=0
Sep 18 17:50:02.837747 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:50:02.837725    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-sqp78" event={"ID":"3e434de9-e182-41b8-9a4a-e6dd17684bf4","Type":"ContainerDied","Data":"b683f1449c23a7b2b393b0a8686cb6f566a368c5f7fd99bc9099e1543f354bb8"}
Sep 18 17:50:03.845813 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:50:03.845781    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-sqp78" event={"ID":"3e434de9-e182-41b8-9a4a-e6dd17684bf4","Type":"ContainerStarted","Data":"a52ad0718337c662cce339808d605fdf5b9b196e351e0e3209341f05494e25af"}
Sep 18 17:50:03.846259 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:50:03.845850    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-sqp78" event={"ID":"3e434de9-e182-41b8-9a4a-e6dd17684bf4","Type":"ContainerStarted","Data":"fb98b7ebb399f7611dd7a2af7c6628eed08ed86f214c1f37fe7fb20a7934e8ce"}
Sep 18 17:50:03.846259 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:50:03.846043    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-sqp78"
Sep 18 17:50:03.869543 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:50:03.869491    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-sqp78" podStartSLOduration=3.876276905 podStartE2EDuration="7.869473784s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.993196879s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:49:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:50:03.868456798 +0000 UTC m=+3394.068312084" watchObservedRunningTime="2026-09-18 17:50:03.869473784 +0000 UTC m=+3394.069329070"
Sep 18 17:50:04.850813 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:50:04.850772    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-sqp78"
Sep 18 17:50:04.852093 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:50:04.852063    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-sqp78" podUID="3e434de9-e182-41b8-9a4a-e6dd17684bf4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.73:8080: connect: connection refused"
Sep 18 17:50:05.856236 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:50:05.856194    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-sqp78" podUID="3e434de9-e182-41b8-9a4a-e6dd17684bf4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.73:8080: connect: connection refused"
Sep 18 17:50:07.728838 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:50:07.728801    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-74lfl" podUID="edf79def-78af-4176-8a84-e4743b413dd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.71:8080: connect: connection refused"
Sep 18 17:50:10.860609 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:50:10.860576    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-sqp78"
Sep 18 17:50:10.874900 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:50:10.861128    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-sqp78" podUID="3e434de9-e182-41b8-9a4a-e6dd17684bf4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.73:8080: connect: connection refused"
Sep 18 17:50:17.728326 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:50:17.728275    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-74lfl" podUID="edf79def-78af-4176-8a84-e4743b413dd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.71:8080: connect: connection refused"
Sep 18 17:50:20.861546 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:50:20.861509    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-sqp78" podUID="3e434de9-e182-41b8-9a4a-e6dd17684bf4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.73:8080: connect: connection refused"
Sep 18 17:50:27.728727 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:50:27.728680    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-74lfl" podUID="edf79def-78af-4176-8a84-e4743b413dd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.71:8080: connect: connection refused"
Sep 18 17:50:30.861408 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:50:30.861364    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-sqp78" podUID="3e434de9-e182-41b8-9a4a-e6dd17684bf4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.73:8080: connect: connection refused"
Sep 18 17:50:37.728579 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:50:37.728534    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-74lfl" podUID="edf79def-78af-4176-8a84-e4743b413dd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.71:8080: connect: connection refused"
Sep 18 17:50:40.861179 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:50:40.861137    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-sqp78" podUID="3e434de9-e182-41b8-9a4a-e6dd17684bf4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.73:8080: connect: connection refused"
Sep 18 17:50:47.728311 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:50:47.728267    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-74lfl" podUID="edf79def-78af-4176-8a84-e4743b413dd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.71:8080: connect: connection refused"
Sep 18 17:50:50.861080 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:50:50.861033    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-sqp78" podUID="3e434de9-e182-41b8-9a4a-e6dd17684bf4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.73:8080: connect: connection refused"
Sep 18 17:50:57.728856 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:50:57.728813    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-74lfl" podUID="edf79def-78af-4176-8a84-e4743b413dd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.71:8080: connect: connection refused"
Sep 18 17:51:00.862116 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:00.862025    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-sqp78" podUID="3e434de9-e182-41b8-9a4a-e6dd17684bf4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.73:8080: connect: connection refused"
Sep 18 17:51:06.888678 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:06.888630    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-74lfl" podUID="edf79def-78af-4176-8a84-e4743b413dd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.71:8080: connect: connection refused"
Sep 18 17:51:10.861039 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:10.861002    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-sqp78" podUID="3e434de9-e182-41b8-9a4a-e6dd17684bf4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.73:8080: connect: connection refused"
Sep 18 17:51:16.889296 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:16.889259    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-74lfl"
Sep 18 17:51:20.450417 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:20.450384    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-74lfl"]
Sep 18 17:51:20.450888 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:20.450617    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-74lfl" podUID="edf79def-78af-4176-8a84-e4743b413dd6" containerName="kserve-container" containerID="cri-o://494322e37768d9dc65bf6297379e8cd03ce2c217f32ea10e3199e727cd7308da" gracePeriod=30
Sep 18 17:51:20.450888 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:20.450737    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-74lfl" podUID="edf79def-78af-4176-8a84-e4743b413dd6" containerName="kube-rbac-proxy" containerID="cri-o://e8cd6c41e639b2b456fa8013c0f67661371423bac7044a1f2cbe0a4e3aa1924f" gracePeriod=30
Sep 18 17:51:20.861747 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:20.861707    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-sqp78" podUID="3e434de9-e182-41b8-9a4a-e6dd17684bf4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.73:8080: connect: connection refused"
Sep 18 17:51:21.217196 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:21.217112    2881 generic.go:356] "Generic (PLEG): container finished" podID="edf79def-78af-4176-8a84-e4743b413dd6" containerID="e8cd6c41e639b2b456fa8013c0f67661371423bac7044a1f2cbe0a4e3aa1924f" exitCode=2
Sep 18 17:51:21.217365 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:21.217196    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-74lfl" event={"ID":"edf79def-78af-4176-8a84-e4743b413dd6","Type":"ContainerDied","Data":"e8cd6c41e639b2b456fa8013c0f67661371423bac7044a1f2cbe0a4e3aa1924f"}
Sep 18 17:51:22.723071 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:22.723022    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-74lfl" podUID="edf79def-78af-4176-8a84-e4743b413dd6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.71:8643/healthz\": dial tcp 10.134.0.71:8643: connect: connection refused"
Sep 18 17:51:25.491729 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:25.491702    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-74lfl"
Sep 18 17:51:25.589472 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:25.589375    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/edf79def-78af-4176-8a84-e4743b413dd6-cabundle-cert\") pod \"edf79def-78af-4176-8a84-e4743b413dd6\" (UID: \"edf79def-78af-4176-8a84-e4743b413dd6\") "
Sep 18 17:51:25.589472 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:25.589417    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/edf79def-78af-4176-8a84-e4743b413dd6-kserve-provision-location\") pod \"edf79def-78af-4176-8a84-e4743b413dd6\" (UID: \"edf79def-78af-4176-8a84-e4743b413dd6\") "
Sep 18 17:51:25.589472 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:25.589467    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/edf79def-78af-4176-8a84-e4743b413dd6-proxy-tls\") pod \"edf79def-78af-4176-8a84-e4743b413dd6\" (UID: \"edf79def-78af-4176-8a84-e4743b413dd6\") "
Sep 18 17:51:25.589767 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:25.589642    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/edf79def-78af-4176-8a84-e4743b413dd6-kube-rbac-proxy-sar-config\") pod \"edf79def-78af-4176-8a84-e4743b413dd6\" (UID: \"edf79def-78af-4176-8a84-e4743b413dd6\") "
Sep 18 17:51:25.589767 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:25.589685    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p7q8m\" (UniqueName: \"kubernetes.io/projected/edf79def-78af-4176-8a84-e4743b413dd6-kube-api-access-p7q8m\") pod \"edf79def-78af-4176-8a84-e4743b413dd6\" (UID: \"edf79def-78af-4176-8a84-e4743b413dd6\") "
Sep 18 17:51:25.589889 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:25.589774    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edf79def-78af-4176-8a84-e4743b413dd6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "edf79def-78af-4176-8a84-e4743b413dd6" (UID: "edf79def-78af-4176-8a84-e4743b413dd6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:51:25.589889 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:25.589818    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edf79def-78af-4176-8a84-e4743b413dd6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "edf79def-78af-4176-8a84-e4743b413dd6" (UID: "edf79def-78af-4176-8a84-e4743b413dd6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:51:25.590039 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:25.590012    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edf79def-78af-4176-8a84-e4743b413dd6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "edf79def-78af-4176-8a84-e4743b413dd6" (UID: "edf79def-78af-4176-8a84-e4743b413dd6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:51:25.590128 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:25.590103    2881 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/edf79def-78af-4176-8a84-e4743b413dd6-cabundle-cert\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:51:25.590128 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:25.590125    2881 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/edf79def-78af-4176-8a84-e4743b413dd6-kserve-provision-location\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:51:25.590362 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:25.590137    2881 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/edf79def-78af-4176-8a84-e4743b413dd6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:51:25.591763 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:25.591741    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edf79def-78af-4176-8a84-e4743b413dd6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "edf79def-78af-4176-8a84-e4743b413dd6" (UID: "edf79def-78af-4176-8a84-e4743b413dd6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:51:25.591881 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:25.591862    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edf79def-78af-4176-8a84-e4743b413dd6-kube-api-access-p7q8m" (OuterVolumeSpecName: "kube-api-access-p7q8m") pod "edf79def-78af-4176-8a84-e4743b413dd6" (UID: "edf79def-78af-4176-8a84-e4743b413dd6"). InnerVolumeSpecName "kube-api-access-p7q8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:51:25.691091 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:25.691053    2881 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7q8m\" (UniqueName: \"kubernetes.io/projected/edf79def-78af-4176-8a84-e4743b413dd6-kube-api-access-p7q8m\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:51:25.691091 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:25.691087    2881 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/edf79def-78af-4176-8a84-e4743b413dd6-proxy-tls\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:51:25.740534 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:25.740502    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-56241a-predictor-5449ff77bd-jlptd"]
Sep 18 17:51:25.740854 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:25.740841    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edf79def-78af-4176-8a84-e4743b413dd6" containerName="storage-initializer"
Sep 18 17:51:25.740901 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:25.740856    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="edf79def-78af-4176-8a84-e4743b413dd6" containerName="storage-initializer"
Sep 18 17:51:25.740901 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:25.740871    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edf79def-78af-4176-8a84-e4743b413dd6" containerName="kube-rbac-proxy"
Sep 18 17:51:25.740901 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:25.740878    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="edf79def-78af-4176-8a84-e4743b413dd6" containerName="kube-rbac-proxy"
Sep 18 17:51:25.740901 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:25.740886    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edf79def-78af-4176-8a84-e4743b413dd6" containerName="kserve-container"
Sep 18 17:51:25.740901 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:25.740892    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="edf79def-78af-4176-8a84-e4743b413dd6" containerName="kserve-container"
Sep 18 17:51:25.741056 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:25.740956    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="edf79def-78af-4176-8a84-e4743b413dd6" containerName="kserve-container"
Sep 18 17:51:25.741056 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:25.740968    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="edf79def-78af-4176-8a84-e4743b413dd6" containerName="kube-rbac-proxy"
Sep 18 17:51:25.744300 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:25.744284    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-56241a-predictor-5449ff77bd-jlptd"
Sep 18 17:51:25.747272 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:25.747243    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-56241a-kube-rbac-proxy-sar-config\""
Sep 18 17:51:25.747422 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:25.747375    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-56241a-predictor-serving-cert\""
Sep 18 17:51:25.755586 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:25.755561    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-56241a-predictor-5449ff77bd-jlptd"]
Sep 18 17:51:25.792093 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:25.792055    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d15df05b-a9de-45d5-8142-a0547993ad0f-cabundle-cert\") pod \"isvc-primary-56241a-predictor-5449ff77bd-jlptd\" (UID: \"d15df05b-a9de-45d5-8142-a0547993ad0f\") " pod="kserve-ci-e2e-test/isvc-primary-56241a-predictor-5449ff77bd-jlptd"
Sep 18 17:51:25.792356 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:25.792112    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtksc\" (UniqueName: \"kubernetes.io/projected/d15df05b-a9de-45d5-8142-a0547993ad0f-kube-api-access-qtksc\") pod \"isvc-primary-56241a-predictor-5449ff77bd-jlptd\" (UID: \"d15df05b-a9de-45d5-8142-a0547993ad0f\") " pod="kserve-ci-e2e-test/isvc-primary-56241a-predictor-5449ff77bd-jlptd"
Sep 18 17:51:25.792356 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:25.792178    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d15df05b-a9de-45d5-8142-a0547993ad0f-kserve-provision-location\") pod \"isvc-primary-56241a-predictor-5449ff77bd-jlptd\" (UID: \"d15df05b-a9de-45d5-8142-a0547993ad0f\") " pod="kserve-ci-e2e-test/isvc-primary-56241a-predictor-5449ff77bd-jlptd"
Sep 18 17:51:25.792356 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:25.792265    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d15df05b-a9de-45d5-8142-a0547993ad0f-proxy-tls\") pod \"isvc-primary-56241a-predictor-5449ff77bd-jlptd\" (UID: \"d15df05b-a9de-45d5-8142-a0547993ad0f\") " pod="kserve-ci-e2e-test/isvc-primary-56241a-predictor-5449ff77bd-jlptd"
Sep 18 17:51:25.792356 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:25.792311    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d15df05b-a9de-45d5-8142-a0547993ad0f-kube-rbac-proxy-sar-config\") pod \"isvc-primary-56241a-predictor-5449ff77bd-jlptd\" (UID: \"d15df05b-a9de-45d5-8142-a0547993ad0f\") " pod="kserve-ci-e2e-test/isvc-primary-56241a-predictor-5449ff77bd-jlptd"
Sep 18 17:51:25.893207 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:25.893120    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d15df05b-a9de-45d5-8142-a0547993ad0f-cabundle-cert\") pod \"isvc-primary-56241a-predictor-5449ff77bd-jlptd\" (UID: \"d15df05b-a9de-45d5-8142-a0547993ad0f\") " pod="kserve-ci-e2e-test/isvc-primary-56241a-predictor-5449ff77bd-jlptd"
Sep 18 17:51:25.893207 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:25.893164    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtksc\" (UniqueName: \"kubernetes.io/projected/d15df05b-a9de-45d5-8142-a0547993ad0f-kube-api-access-qtksc\") pod \"isvc-primary-56241a-predictor-5449ff77bd-jlptd\" (UID: \"d15df05b-a9de-45d5-8142-a0547993ad0f\") " pod="kserve-ci-e2e-test/isvc-primary-56241a-predictor-5449ff77bd-jlptd"
Sep 18 17:51:25.893207 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:25.893181    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d15df05b-a9de-45d5-8142-a0547993ad0f-kserve-provision-location\") pod \"isvc-primary-56241a-predictor-5449ff77bd-jlptd\" (UID: \"d15df05b-a9de-45d5-8142-a0547993ad0f\") " pod="kserve-ci-e2e-test/isvc-primary-56241a-predictor-5449ff77bd-jlptd"
Sep 18 17:51:25.893207 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:25.893202    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d15df05b-a9de-45d5-8142-a0547993ad0f-proxy-tls\") pod \"isvc-primary-56241a-predictor-5449ff77bd-jlptd\" (UID: \"d15df05b-a9de-45d5-8142-a0547993ad0f\") " pod="kserve-ci-e2e-test/isvc-primary-56241a-predictor-5449ff77bd-jlptd"
Sep 18 17:51:25.893586 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:25.893237    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d15df05b-a9de-45d5-8142-a0547993ad0f-kube-rbac-proxy-sar-config\") pod \"isvc-primary-56241a-predictor-5449ff77bd-jlptd\" (UID: \"d15df05b-a9de-45d5-8142-a0547993ad0f\") " pod="kserve-ci-e2e-test/isvc-primary-56241a-predictor-5449ff77bd-jlptd"
Sep 18 17:51:25.893586 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:51:25.893320    2881 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-primary-56241a-predictor-serving-cert: secret "isvc-primary-56241a-predictor-serving-cert" not found
Sep 18 17:51:25.893586 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:51:25.893390    2881 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d15df05b-a9de-45d5-8142-a0547993ad0f-proxy-tls podName:d15df05b-a9de-45d5-8142-a0547993ad0f nodeName:}" failed. No retries permitted until 2026-09-18 17:51:26.393369082 +0000 UTC m=+3476.593224355 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d15df05b-a9de-45d5-8142-a0547993ad0f-proxy-tls") pod "isvc-primary-56241a-predictor-5449ff77bd-jlptd" (UID: "d15df05b-a9de-45d5-8142-a0547993ad0f") : secret "isvc-primary-56241a-predictor-serving-cert" not found
Sep 18 17:51:25.893757 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:25.893661    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d15df05b-a9de-45d5-8142-a0547993ad0f-kserve-provision-location\") pod \"isvc-primary-56241a-predictor-5449ff77bd-jlptd\" (UID: \"d15df05b-a9de-45d5-8142-a0547993ad0f\") " pod="kserve-ci-e2e-test/isvc-primary-56241a-predictor-5449ff77bd-jlptd"
Sep 18 17:51:25.893905 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:25.893884    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d15df05b-a9de-45d5-8142-a0547993ad0f-cabundle-cert\") pod \"isvc-primary-56241a-predictor-5449ff77bd-jlptd\" (UID: \"d15df05b-a9de-45d5-8142-a0547993ad0f\") " pod="kserve-ci-e2e-test/isvc-primary-56241a-predictor-5449ff77bd-jlptd"
Sep 18 17:51:25.893966 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:25.893936    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d15df05b-a9de-45d5-8142-a0547993ad0f-kube-rbac-proxy-sar-config\") pod \"isvc-primary-56241a-predictor-5449ff77bd-jlptd\" (UID: \"d15df05b-a9de-45d5-8142-a0547993ad0f\") " pod="kserve-ci-e2e-test/isvc-primary-56241a-predictor-5449ff77bd-jlptd"
Sep 18 17:51:25.903195 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:25.903170    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtksc\" (UniqueName: \"kubernetes.io/projected/d15df05b-a9de-45d5-8142-a0547993ad0f-kube-api-access-qtksc\") pod \"isvc-primary-56241a-predictor-5449ff77bd-jlptd\" (UID: \"d15df05b-a9de-45d5-8142-a0547993ad0f\") " pod="kserve-ci-e2e-test/isvc-primary-56241a-predictor-5449ff77bd-jlptd"
Sep 18 17:51:26.242662 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:26.242570    2881 generic.go:356] "Generic (PLEG): container finished" podID="edf79def-78af-4176-8a84-e4743b413dd6" containerID="494322e37768d9dc65bf6297379e8cd03ce2c217f32ea10e3199e727cd7308da" exitCode=0
Sep 18 17:51:26.242662 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:26.242632    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-74lfl"
Sep 18 17:51:26.242662 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:26.242646    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-74lfl" event={"ID":"edf79def-78af-4176-8a84-e4743b413dd6","Type":"ContainerDied","Data":"494322e37768d9dc65bf6297379e8cd03ce2c217f32ea10e3199e727cd7308da"}
Sep 18 17:51:26.242957 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:26.242722    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-74lfl" event={"ID":"edf79def-78af-4176-8a84-e4743b413dd6","Type":"ContainerDied","Data":"f2edf86e8e5bf542029d0bdaf896a73ce691dd68e34dcf219444d4569def61bc"}
Sep 18 17:51:26.242957 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:26.242756    2881 scope.go:117] "RemoveContainer" containerID="e8cd6c41e639b2b456fa8013c0f67661371423bac7044a1f2cbe0a4e3aa1924f"
Sep 18 17:51:26.251423 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:26.251404    2881 scope.go:117] "RemoveContainer" containerID="494322e37768d9dc65bf6297379e8cd03ce2c217f32ea10e3199e727cd7308da"
Sep 18 17:51:26.262563 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:26.262539    2881 scope.go:117] "RemoveContainer" containerID="e6cc8133b7bfbee3fdd902a06a0236f6fda6d0dd15f25033c1e4b4b6b1f06553"
Sep 18 17:51:26.272132 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:26.272104    2881 scope.go:117] "RemoveContainer" containerID="e8cd6c41e639b2b456fa8013c0f67661371423bac7044a1f2cbe0a4e3aa1924f"
Sep 18 17:51:26.272363 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:26.272345    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-74lfl"]
Sep 18 17:51:26.272504 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:51:26.272482    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8cd6c41e639b2b456fa8013c0f67661371423bac7044a1f2cbe0a4e3aa1924f\": container with ID starting with e8cd6c41e639b2b456fa8013c0f67661371423bac7044a1f2cbe0a4e3aa1924f not found: ID does not exist" containerID="e8cd6c41e639b2b456fa8013c0f67661371423bac7044a1f2cbe0a4e3aa1924f"
Sep 18 17:51:26.272550 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:26.272515    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8cd6c41e639b2b456fa8013c0f67661371423bac7044a1f2cbe0a4e3aa1924f"} err="failed to get container status \"e8cd6c41e639b2b456fa8013c0f67661371423bac7044a1f2cbe0a4e3aa1924f\": rpc error: code = NotFound desc = could not find container \"e8cd6c41e639b2b456fa8013c0f67661371423bac7044a1f2cbe0a4e3aa1924f\": container with ID starting with e8cd6c41e639b2b456fa8013c0f67661371423bac7044a1f2cbe0a4e3aa1924f not found: ID does not exist"
Sep 18 17:51:26.272550 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:26.272534    2881 scope.go:117] "RemoveContainer" containerID="494322e37768d9dc65bf6297379e8cd03ce2c217f32ea10e3199e727cd7308da"
Sep 18 17:51:26.272788 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:51:26.272772    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"494322e37768d9dc65bf6297379e8cd03ce2c217f32ea10e3199e727cd7308da\": container with ID starting with 494322e37768d9dc65bf6297379e8cd03ce2c217f32ea10e3199e727cd7308da not found: ID does not exist" containerID="494322e37768d9dc65bf6297379e8cd03ce2c217f32ea10e3199e727cd7308da"
Sep 18 17:51:26.272837 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:26.272794    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"494322e37768d9dc65bf6297379e8cd03ce2c217f32ea10e3199e727cd7308da"} err="failed to get container status \"494322e37768d9dc65bf6297379e8cd03ce2c217f32ea10e3199e727cd7308da\": rpc error: code = NotFound desc = could not find container \"494322e37768d9dc65bf6297379e8cd03ce2c217f32ea10e3199e727cd7308da\": container with ID starting with 494322e37768d9dc65bf6297379e8cd03ce2c217f32ea10e3199e727cd7308da not found: ID does not exist"
Sep 18 17:51:26.272837 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:26.272808    2881 scope.go:117] "RemoveContainer" containerID="e6cc8133b7bfbee3fdd902a06a0236f6fda6d0dd15f25033c1e4b4b6b1f06553"
Sep 18 17:51:26.273085 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:51:26.273067    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6cc8133b7bfbee3fdd902a06a0236f6fda6d0dd15f25033c1e4b4b6b1f06553\": container with ID starting with e6cc8133b7bfbee3fdd902a06a0236f6fda6d0dd15f25033c1e4b4b6b1f06553 not found: ID does not exist" containerID="e6cc8133b7bfbee3fdd902a06a0236f6fda6d0dd15f25033c1e4b4b6b1f06553"
Sep 18 17:51:26.273134 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:26.273088    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6cc8133b7bfbee3fdd902a06a0236f6fda6d0dd15f25033c1e4b4b6b1f06553"} err="failed to get container status \"e6cc8133b7bfbee3fdd902a06a0236f6fda6d0dd15f25033c1e4b4b6b1f06553\": rpc error: code = NotFound desc = could not find container \"e6cc8133b7bfbee3fdd902a06a0236f6fda6d0dd15f25033c1e4b4b6b1f06553\": container with ID starting with e6cc8133b7bfbee3fdd902a06a0236f6fda6d0dd15f25033c1e4b4b6b1f06553 not found: ID does not exist"
Sep 18 17:51:26.277706 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:26.277681    2881 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-74lfl"]
Sep 18 17:51:26.397648 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:26.397607    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d15df05b-a9de-45d5-8142-a0547993ad0f-proxy-tls\") pod \"isvc-primary-56241a-predictor-5449ff77bd-jlptd\" (UID: \"d15df05b-a9de-45d5-8142-a0547993ad0f\") " pod="kserve-ci-e2e-test/isvc-primary-56241a-predictor-5449ff77bd-jlptd"
Sep 18 17:51:26.400123 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:26.400101    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d15df05b-a9de-45d5-8142-a0547993ad0f-proxy-tls\") pod \"isvc-primary-56241a-predictor-5449ff77bd-jlptd\" (UID: \"d15df05b-a9de-45d5-8142-a0547993ad0f\") " pod="kserve-ci-e2e-test/isvc-primary-56241a-predictor-5449ff77bd-jlptd"
Sep 18 17:51:26.655532 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:26.655485    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-56241a-predictor-5449ff77bd-jlptd"
Sep 18 17:51:26.934317 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:26.934287    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-56241a-predictor-5449ff77bd-jlptd"]
Sep 18 17:51:26.937420 ip-10-0-143-5 kubenswrapper[2881]: W0918 17:51:26.937383    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd15df05b_a9de_45d5_8142_a0547993ad0f.slice/crio-bdfea447a77a289acdf99b900d5fd388a57e8f6484f1a56beab08bdd6212b6ed WatchSource:0}: Error finding container bdfea447a77a289acdf99b900d5fd388a57e8f6484f1a56beab08bdd6212b6ed: Status 404 returned error can't find the container with id bdfea447a77a289acdf99b900d5fd388a57e8f6484f1a56beab08bdd6212b6ed
Sep 18 17:51:27.250587 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:27.250489    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-56241a-predictor-5449ff77bd-jlptd" event={"ID":"d15df05b-a9de-45d5-8142-a0547993ad0f","Type":"ContainerStarted","Data":"a95b3ea2a1fd729e4e0a52432a8076107c524e82e2540b14c4484387d4c6ac9e"}
Sep 18 17:51:27.250587 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:27.250562    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-56241a-predictor-5449ff77bd-jlptd" event={"ID":"d15df05b-a9de-45d5-8142-a0547993ad0f","Type":"ContainerStarted","Data":"bdfea447a77a289acdf99b900d5fd388a57e8f6484f1a56beab08bdd6212b6ed"}
Sep 18 17:51:27.893490 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:27.893456    2881 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edf79def-78af-4176-8a84-e4743b413dd6" path="/var/lib/kubelet/pods/edf79def-78af-4176-8a84-e4743b413dd6/volumes"
Sep 18 17:51:28.888263 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:28.888204    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-sqp78" podUID="3e434de9-e182-41b8-9a4a-e6dd17684bf4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.73:8080: connect: connection refused"
Sep 18 17:51:30.512970 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:30.512934    2881 scope.go:117] "RemoveContainer" containerID="8f552cb4eb6975bd61313870a84e44cc8c45aee043afea1fee55a295ad0ccbcd"
Sep 18 17:51:31.273687 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:31.273650    2881 generic.go:356] "Generic (PLEG): container finished" podID="d15df05b-a9de-45d5-8142-a0547993ad0f" containerID="a95b3ea2a1fd729e4e0a52432a8076107c524e82e2540b14c4484387d4c6ac9e" exitCode=0
Sep 18 17:51:31.273870 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:31.273718    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-56241a-predictor-5449ff77bd-jlptd" event={"ID":"d15df05b-a9de-45d5-8142-a0547993ad0f","Type":"ContainerDied","Data":"a95b3ea2a1fd729e4e0a52432a8076107c524e82e2540b14c4484387d4c6ac9e"}
Sep 18 17:51:32.280199 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:32.280159    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-56241a-predictor-5449ff77bd-jlptd" event={"ID":"d15df05b-a9de-45d5-8142-a0547993ad0f","Type":"ContainerStarted","Data":"47a0753200cef0e597aa9a91051b76f52ba06fed638ef661cab76badcf7b5466"}
Sep 18 17:51:32.280750 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:32.280252    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-56241a-predictor-5449ff77bd-jlptd" event={"ID":"d15df05b-a9de-45d5-8142-a0547993ad0f","Type":"ContainerStarted","Data":"5380cb867027a9bae07f02d26ebf247d4734d5a0d211e59a9e1d24bd3cf8c07c"}
Sep 18 17:51:32.280750 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:32.280559    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-56241a-predictor-5449ff77bd-jlptd"
Sep 18 17:51:32.280750 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:32.280618    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-56241a-predictor-5449ff77bd-jlptd"
Sep 18 17:51:32.281915 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:32.281887    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-56241a-predictor-5449ff77bd-jlptd" podUID="d15df05b-a9de-45d5-8142-a0547993ad0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.74:8080: connect: connection refused"
Sep 18 17:51:32.305771 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:32.305697    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-56241a-predictor-5449ff77bd-jlptd" podStartSLOduration=3.414031625 podStartE2EDuration="7.305678594s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.891646969s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:51:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:51:32.304165541 +0000 UTC m=+3482.504020838" watchObservedRunningTime="2026-09-18 17:51:32.305678594 +0000 UTC m=+3482.505533881"
Sep 18 17:51:33.284994 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:33.284952    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-56241a-predictor-5449ff77bd-jlptd" podUID="d15df05b-a9de-45d5-8142-a0547993ad0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.74:8080: connect: connection refused"
Sep 18 17:51:38.289389 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:38.289351    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-56241a-predictor-5449ff77bd-jlptd"
Sep 18 17:51:38.289884 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:38.289857    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-56241a-predictor-5449ff77bd-jlptd" podUID="d15df05b-a9de-45d5-8142-a0547993ad0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.74:8080: connect: connection refused"
Sep 18 17:51:38.888839 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:38.888796    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-sqp78" podUID="3e434de9-e182-41b8-9a4a-e6dd17684bf4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.73:8080: connect: connection refused"
Sep 18 17:51:48.290120 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:48.290079    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-56241a-predictor-5449ff77bd-jlptd" podUID="d15df05b-a9de-45d5-8142-a0547993ad0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.74:8080: connect: connection refused"
Sep 18 17:51:48.888397 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:48.888365    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-sqp78"
Sep 18 17:51:58.166861 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:58.166823    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-sqp78"]
Sep 18 17:51:58.167333 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:58.167073    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-sqp78" podUID="3e434de9-e182-41b8-9a4a-e6dd17684bf4" containerName="kserve-container" containerID="cri-o://fb98b7ebb399f7611dd7a2af7c6628eed08ed86f214c1f37fe7fb20a7934e8ce" gracePeriod=30
Sep 18 17:51:58.167333 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:58.167120    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-sqp78" podUID="3e434de9-e182-41b8-9a4a-e6dd17684bf4" containerName="kube-rbac-proxy" containerID="cri-o://a52ad0718337c662cce339808d605fdf5b9b196e351e0e3209341f05494e25af" gracePeriod=30
Sep 18 17:51:58.290466 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:58.290430    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-56241a-predictor-5449ff77bd-jlptd" podUID="d15df05b-a9de-45d5-8142-a0547993ad0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.74:8080: connect: connection refused"
Sep 18 17:51:58.404711 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:58.404675    2881 generic.go:356] "Generic (PLEG): container finished" podID="3e434de9-e182-41b8-9a4a-e6dd17684bf4" containerID="a52ad0718337c662cce339808d605fdf5b9b196e351e0e3209341f05494e25af" exitCode=2
Sep 18 17:51:58.404890 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:58.404728    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-sqp78" event={"ID":"3e434de9-e182-41b8-9a4a-e6dd17684bf4","Type":"ContainerDied","Data":"a52ad0718337c662cce339808d605fdf5b9b196e351e0e3209341f05494e25af"}
Sep 18 17:51:58.888323 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:51:58.888275    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-sqp78" podUID="3e434de9-e182-41b8-9a4a-e6dd17684bf4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.73:8080: connect: connection refused"
Sep 18 17:52:00.856975 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:00.856928    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-sqp78" podUID="3e434de9-e182-41b8-9a4a-e6dd17684bf4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.73:8643/healthz\": dial tcp 10.134.0.73:8643: connect: connection refused"
Sep 18 17:52:02.916130 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:02.916105    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-sqp78"
Sep 18 17:52:02.934087 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:02.934057    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e434de9-e182-41b8-9a4a-e6dd17684bf4-kserve-provision-location\") pod \"3e434de9-e182-41b8-9a4a-e6dd17684bf4\" (UID: \"3e434de9-e182-41b8-9a4a-e6dd17684bf4\") "
Sep 18 17:52:02.934284 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:02.934120    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-55s5n\" (UniqueName: \"kubernetes.io/projected/3e434de9-e182-41b8-9a4a-e6dd17684bf4-kube-api-access-55s5n\") pod \"3e434de9-e182-41b8-9a4a-e6dd17684bf4\" (UID: \"3e434de9-e182-41b8-9a4a-e6dd17684bf4\") "
Sep 18 17:52:02.934284 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:02.934144    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e434de9-e182-41b8-9a4a-e6dd17684bf4-kube-rbac-proxy-sar-config\") pod \"3e434de9-e182-41b8-9a4a-e6dd17684bf4\" (UID: \"3e434de9-e182-41b8-9a4a-e6dd17684bf4\") "
Sep 18 17:52:02.934284 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:02.934186    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e434de9-e182-41b8-9a4a-e6dd17684bf4-cabundle-cert\") pod \"3e434de9-e182-41b8-9a4a-e6dd17684bf4\" (UID: \"3e434de9-e182-41b8-9a4a-e6dd17684bf4\") "
Sep 18 17:52:02.934284 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:02.934239    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e434de9-e182-41b8-9a4a-e6dd17684bf4-proxy-tls\") pod \"3e434de9-e182-41b8-9a4a-e6dd17684bf4\" (UID: \"3e434de9-e182-41b8-9a4a-e6dd17684bf4\") "
Sep 18 17:52:02.934519 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:02.934427    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e434de9-e182-41b8-9a4a-e6dd17684bf4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3e434de9-e182-41b8-9a4a-e6dd17684bf4" (UID: "3e434de9-e182-41b8-9a4a-e6dd17684bf4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:52:02.935079 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:02.935050    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e434de9-e182-41b8-9a4a-e6dd17684bf4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3e434de9-e182-41b8-9a4a-e6dd17684bf4" (UID: "3e434de9-e182-41b8-9a4a-e6dd17684bf4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:52:02.935214 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:02.935115    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e434de9-e182-41b8-9a4a-e6dd17684bf4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3e434de9-e182-41b8-9a4a-e6dd17684bf4" (UID: "3e434de9-e182-41b8-9a4a-e6dd17684bf4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:52:02.936523 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:02.936484    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e434de9-e182-41b8-9a4a-e6dd17684bf4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3e434de9-e182-41b8-9a4a-e6dd17684bf4" (UID: "3e434de9-e182-41b8-9a4a-e6dd17684bf4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:52:02.936684 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:02.936640    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e434de9-e182-41b8-9a4a-e6dd17684bf4-kube-api-access-55s5n" (OuterVolumeSpecName: "kube-api-access-55s5n") pod "3e434de9-e182-41b8-9a4a-e6dd17684bf4" (UID: "3e434de9-e182-41b8-9a4a-e6dd17684bf4"). InnerVolumeSpecName "kube-api-access-55s5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:52:03.035069 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:03.035022    2881 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e434de9-e182-41b8-9a4a-e6dd17684bf4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:52:03.035069 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:03.035055    2881 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e434de9-e182-41b8-9a4a-e6dd17684bf4-cabundle-cert\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:52:03.035069 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:03.035067    2881 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e434de9-e182-41b8-9a4a-e6dd17684bf4-proxy-tls\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:52:03.035069 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:03.035075    2881 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e434de9-e182-41b8-9a4a-e6dd17684bf4-kserve-provision-location\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:52:03.035069 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:03.035085    2881 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55s5n\" (UniqueName: \"kubernetes.io/projected/3e434de9-e182-41b8-9a4a-e6dd17684bf4-kube-api-access-55s5n\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:52:03.372604 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:03.372513    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-69pvz"]
Sep 18 17:52:03.373051 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:03.373033    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e434de9-e182-41b8-9a4a-e6dd17684bf4" containerName="storage-initializer"
Sep 18 17:52:03.373097 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:03.373054    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e434de9-e182-41b8-9a4a-e6dd17684bf4" containerName="storage-initializer"
Sep 18 17:52:03.373097 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:03.373065    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e434de9-e182-41b8-9a4a-e6dd17684bf4" containerName="kube-rbac-proxy"
Sep 18 17:52:03.373097 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:03.373071    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e434de9-e182-41b8-9a4a-e6dd17684bf4" containerName="kube-rbac-proxy"
Sep 18 17:52:03.373097 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:03.373090    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e434de9-e182-41b8-9a4a-e6dd17684bf4" containerName="kserve-container"
Sep 18 17:52:03.373097 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:03.373095    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e434de9-e182-41b8-9a4a-e6dd17684bf4" containerName="kserve-container"
Sep 18 17:52:03.373292 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:03.373152    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e434de9-e182-41b8-9a4a-e6dd17684bf4" containerName="kserve-container"
Sep 18 17:52:03.373292 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:03.373160    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e434de9-e182-41b8-9a4a-e6dd17684bf4" containerName="kube-rbac-proxy"
Sep 18 17:52:03.376536 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:03.376519    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-69pvz"
Sep 18 17:52:03.379361 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:03.379332    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 18 17:52:03.379542 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:03.379529    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-predictor-serving-cert\""
Sep 18 17:52:03.388508 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:03.388483    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-69pvz"]
Sep 18 17:52:03.427961 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:03.427925    2881 generic.go:356] "Generic (PLEG): container finished" podID="3e434de9-e182-41b8-9a4a-e6dd17684bf4" containerID="fb98b7ebb399f7611dd7a2af7c6628eed08ed86f214c1f37fe7fb20a7934e8ce" exitCode=0
Sep 18 17:52:03.428145 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:03.427994    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-sqp78" event={"ID":"3e434de9-e182-41b8-9a4a-e6dd17684bf4","Type":"ContainerDied","Data":"fb98b7ebb399f7611dd7a2af7c6628eed08ed86f214c1f37fe7fb20a7934e8ce"}
Sep 18 17:52:03.428145 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:03.428014    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-sqp78"
Sep 18 17:52:03.428145 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:03.428058    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-sqp78" event={"ID":"3e434de9-e182-41b8-9a4a-e6dd17684bf4","Type":"ContainerDied","Data":"081a89f972c8d15f27a7292b25cd3359e6259ac2c92f6124532d784c33a09faa"}
Sep 18 17:52:03.428145 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:03.428095    2881 scope.go:117] "RemoveContainer" containerID="a52ad0718337c662cce339808d605fdf5b9b196e351e0e3209341f05494e25af"
Sep 18 17:52:03.437668 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:03.437642    2881 scope.go:117] "RemoveContainer" containerID="fb98b7ebb399f7611dd7a2af7c6628eed08ed86f214c1f37fe7fb20a7934e8ce"
Sep 18 17:52:03.438983 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:03.438961    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85d978fc-d3e2-4c7a-9871-494ba0bb52d5-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-69pvz\" (UID: \"85d978fc-d3e2-4c7a-9871-494ba0bb52d5\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-69pvz"
Sep 18 17:52:03.439051 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:03.439007    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85d978fc-d3e2-4c7a-9871-494ba0bb52d5-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-69pvz\" (UID: \"85d978fc-d3e2-4c7a-9871-494ba0bb52d5\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-69pvz"
Sep 18 17:52:03.439143 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:03.439122    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85d978fc-d3e2-4c7a-9871-494ba0bb52d5-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-69pvz\" (UID: \"85d978fc-d3e2-4c7a-9871-494ba0bb52d5\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-69pvz"
Sep 18 17:52:03.439186 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:03.439169    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99dtg\" (UniqueName: \"kubernetes.io/projected/85d978fc-d3e2-4c7a-9871-494ba0bb52d5-kube-api-access-99dtg\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-69pvz\" (UID: \"85d978fc-d3e2-4c7a-9871-494ba0bb52d5\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-69pvz"
Sep 18 17:52:03.439285 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:03.439262    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85d978fc-d3e2-4c7a-9871-494ba0bb52d5-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-69pvz\" (UID: \"85d978fc-d3e2-4c7a-9871-494ba0bb52d5\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-69pvz"
Sep 18 17:52:03.446020 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:03.446000    2881 scope.go:117] "RemoveContainer" containerID="b683f1449c23a7b2b393b0a8686cb6f566a368c5f7fd99bc9099e1543f354bb8"
Sep 18 17:52:03.454449 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:03.454428    2881 scope.go:117] "RemoveContainer" containerID="a52ad0718337c662cce339808d605fdf5b9b196e351e0e3209341f05494e25af"
Sep 18 17:52:03.454743 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:52:03.454719    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a52ad0718337c662cce339808d605fdf5b9b196e351e0e3209341f05494e25af\": container with ID starting with a52ad0718337c662cce339808d605fdf5b9b196e351e0e3209341f05494e25af not found: ID does not exist" containerID="a52ad0718337c662cce339808d605fdf5b9b196e351e0e3209341f05494e25af"
Sep 18 17:52:03.454822 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:03.454751    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a52ad0718337c662cce339808d605fdf5b9b196e351e0e3209341f05494e25af"} err="failed to get container status \"a52ad0718337c662cce339808d605fdf5b9b196e351e0e3209341f05494e25af\": rpc error: code = NotFound desc = could not find container \"a52ad0718337c662cce339808d605fdf5b9b196e351e0e3209341f05494e25af\": container with ID starting with a52ad0718337c662cce339808d605fdf5b9b196e351e0e3209341f05494e25af not found: ID does not exist"
Sep 18 17:52:03.454822 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:03.454770    2881 scope.go:117] "RemoveContainer" containerID="fb98b7ebb399f7611dd7a2af7c6628eed08ed86f214c1f37fe7fb20a7934e8ce"
Sep 18 17:52:03.455035 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:52:03.455012    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb98b7ebb399f7611dd7a2af7c6628eed08ed86f214c1f37fe7fb20a7934e8ce\": container with ID starting with fb98b7ebb399f7611dd7a2af7c6628eed08ed86f214c1f37fe7fb20a7934e8ce not found: ID does not exist" containerID="fb98b7ebb399f7611dd7a2af7c6628eed08ed86f214c1f37fe7fb20a7934e8ce"
Sep 18 17:52:03.455146 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:03.455046    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb98b7ebb399f7611dd7a2af7c6628eed08ed86f214c1f37fe7fb20a7934e8ce"} err="failed to get container status \"fb98b7ebb399f7611dd7a2af7c6628eed08ed86f214c1f37fe7fb20a7934e8ce\": rpc error: code = NotFound desc = could not find container \"fb98b7ebb399f7611dd7a2af7c6628eed08ed86f214c1f37fe7fb20a7934e8ce\": container with ID starting with fb98b7ebb399f7611dd7a2af7c6628eed08ed86f214c1f37fe7fb20a7934e8ce not found: ID does not exist"
Sep 18 17:52:03.455146 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:03.455073    2881 scope.go:117] "RemoveContainer" containerID="b683f1449c23a7b2b393b0a8686cb6f566a368c5f7fd99bc9099e1543f354bb8"
Sep 18 17:52:03.455408 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:52:03.455371    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b683f1449c23a7b2b393b0a8686cb6f566a368c5f7fd99bc9099e1543f354bb8\": container with ID starting with b683f1449c23a7b2b393b0a8686cb6f566a368c5f7fd99bc9099e1543f354bb8 not found: ID does not exist" containerID="b683f1449c23a7b2b393b0a8686cb6f566a368c5f7fd99bc9099e1543f354bb8"
Sep 18 17:52:03.455508 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:03.455414    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b683f1449c23a7b2b393b0a8686cb6f566a368c5f7fd99bc9099e1543f354bb8"} err="failed to get container status \"b683f1449c23a7b2b393b0a8686cb6f566a368c5f7fd99bc9099e1543f354bb8\": rpc error: code = NotFound desc = could not find container \"b683f1449c23a7b2b393b0a8686cb6f566a368c5f7fd99bc9099e1543f354bb8\": container with ID starting with b683f1449c23a7b2b393b0a8686cb6f566a368c5f7fd99bc9099e1543f354bb8 not found: ID does not exist"
Sep 18 17:52:03.456606 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:03.456583    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-sqp78"]
Sep 18 17:52:03.458381 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:03.458359    2881 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-sqp78"]
Sep 18 17:52:03.540343 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:03.540306    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85d978fc-d3e2-4c7a-9871-494ba0bb52d5-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-69pvz\" (UID: \"85d978fc-d3e2-4c7a-9871-494ba0bb52d5\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-69pvz"
Sep 18 17:52:03.540545 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:03.540359    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85d978fc-d3e2-4c7a-9871-494ba0bb52d5-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-69pvz\" (UID: \"85d978fc-d3e2-4c7a-9871-494ba0bb52d5\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-69pvz"
Sep 18 17:52:03.540545 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:03.540409    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85d978fc-d3e2-4c7a-9871-494ba0bb52d5-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-69pvz\" (UID: \"85d978fc-d3e2-4c7a-9871-494ba0bb52d5\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-69pvz"
Sep 18 17:52:03.540545 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:03.540430    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99dtg\" (UniqueName: \"kubernetes.io/projected/85d978fc-d3e2-4c7a-9871-494ba0bb52d5-kube-api-access-99dtg\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-69pvz\" (UID: \"85d978fc-d3e2-4c7a-9871-494ba0bb52d5\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-69pvz"
Sep 18 17:52:03.540545 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:03.540452    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85d978fc-d3e2-4c7a-9871-494ba0bb52d5-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-69pvz\" (UID: \"85d978fc-d3e2-4c7a-9871-494ba0bb52d5\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-69pvz"
Sep 18 17:52:03.540798 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:03.540775    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85d978fc-d3e2-4c7a-9871-494ba0bb52d5-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-69pvz\" (UID: \"85d978fc-d3e2-4c7a-9871-494ba0bb52d5\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-69pvz"
Sep 18 17:52:03.541076 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:03.541060    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85d978fc-d3e2-4c7a-9871-494ba0bb52d5-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-69pvz\" (UID: \"85d978fc-d3e2-4c7a-9871-494ba0bb52d5\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-69pvz"
Sep 18 17:52:03.541204 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:03.541182    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85d978fc-d3e2-4c7a-9871-494ba0bb52d5-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-69pvz\" (UID: \"85d978fc-d3e2-4c7a-9871-494ba0bb52d5\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-69pvz"
Sep 18 17:52:03.543021 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:03.543003    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85d978fc-d3e2-4c7a-9871-494ba0bb52d5-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-69pvz\" (UID: \"85d978fc-d3e2-4c7a-9871-494ba0bb52d5\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-69pvz"
Sep 18 17:52:03.550257 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:03.550216    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99dtg\" (UniqueName: \"kubernetes.io/projected/85d978fc-d3e2-4c7a-9871-494ba0bb52d5-kube-api-access-99dtg\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-69pvz\" (UID: \"85d978fc-d3e2-4c7a-9871-494ba0bb52d5\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-69pvz"
Sep 18 17:52:03.687594 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:03.687482    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-69pvz"
Sep 18 17:52:03.892869 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:03.892833    2881 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e434de9-e182-41b8-9a4a-e6dd17684bf4" path="/var/lib/kubelet/pods/3e434de9-e182-41b8-9a4a-e6dd17684bf4/volumes"
Sep 18 17:52:03.967577 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:03.967515    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-69pvz"]
Sep 18 17:52:03.969975 ip-10-0-143-5 kubenswrapper[2881]: W0918 17:52:03.969945    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85d978fc_d3e2_4c7a_9871_494ba0bb52d5.slice/crio-7b8d0a7d1b30308d5de50b97e61106689bef1d62f7e20e64f8e3c345bf2a5ae7 WatchSource:0}: Error finding container 7b8d0a7d1b30308d5de50b97e61106689bef1d62f7e20e64f8e3c345bf2a5ae7: Status 404 returned error can't find the container with id 7b8d0a7d1b30308d5de50b97e61106689bef1d62f7e20e64f8e3c345bf2a5ae7
Sep 18 17:52:04.436092 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:04.436048    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-69pvz" event={"ID":"85d978fc-d3e2-4c7a-9871-494ba0bb52d5","Type":"ContainerStarted","Data":"87101fe5d782bc8c2d2e5958a842c2c2029030ffb22a3d7ecd491641ac1ecba3"}
Sep 18 17:52:04.436320 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:04.436125    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-69pvz" event={"ID":"85d978fc-d3e2-4c7a-9871-494ba0bb52d5","Type":"ContainerStarted","Data":"7b8d0a7d1b30308d5de50b97e61106689bef1d62f7e20e64f8e3c345bf2a5ae7"}
Sep 18 17:52:08.290308 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:08.290272    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-56241a-predictor-5449ff77bd-jlptd" podUID="d15df05b-a9de-45d5-8142-a0547993ad0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.74:8080: connect: connection refused"
Sep 18 17:52:08.458555 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:08.458463    2881 generic.go:356] "Generic (PLEG): container finished" podID="85d978fc-d3e2-4c7a-9871-494ba0bb52d5" containerID="87101fe5d782bc8c2d2e5958a842c2c2029030ffb22a3d7ecd491641ac1ecba3" exitCode=0
Sep 18 17:52:08.458555 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:08.458527    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-69pvz" event={"ID":"85d978fc-d3e2-4c7a-9871-494ba0bb52d5","Type":"ContainerDied","Data":"87101fe5d782bc8c2d2e5958a842c2c2029030ffb22a3d7ecd491641ac1ecba3"}
Sep 18 17:52:09.465297 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:09.465260    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-69pvz" event={"ID":"85d978fc-d3e2-4c7a-9871-494ba0bb52d5","Type":"ContainerStarted","Data":"88269f02b709957eaa22467f06835229cb4eea9579f731d312a06962f69cc13c"}
Sep 18 17:52:09.465694 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:09.465328    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-69pvz" event={"ID":"85d978fc-d3e2-4c7a-9871-494ba0bb52d5","Type":"ContainerStarted","Data":"c953eeac10219ab1637dbe4cca3a4636b5ce7de1334345306e33bd16f9339c3c"}
Sep 18 17:52:09.465694 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:09.465588    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-69pvz"
Sep 18 17:52:09.465765 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:09.465722    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-69pvz"
Sep 18 17:52:09.467169 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:09.467143    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-69pvz" podUID="85d978fc-d3e2-4c7a-9871-494ba0bb52d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.75:8080: connect: connection refused"
Sep 18 17:52:09.493661 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:09.493608    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-69pvz" podStartSLOduration=2.343005745 podStartE2EDuration="6.493591378s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.150585633s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:52:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:52:09.491317306 +0000 UTC m=+3519.691172591" watchObservedRunningTime="2026-09-18 17:52:09.493591378 +0000 UTC m=+3519.693446665"
Sep 18 17:52:10.470643 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:10.470603    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-69pvz" podUID="85d978fc-d3e2-4c7a-9871-494ba0bb52d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.75:8080: connect: connection refused"
Sep 18 17:52:15.475727 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:15.475700    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-69pvz"
Sep 18 17:52:15.476451 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:15.476424    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-69pvz" podUID="85d978fc-d3e2-4c7a-9871-494ba0bb52d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.75:8080: connect: connection refused"
Sep 18 17:52:18.290112 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:18.290075    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-56241a-predictor-5449ff77bd-jlptd" podUID="d15df05b-a9de-45d5-8142-a0547993ad0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.74:8080: connect: connection refused"
Sep 18 17:52:25.477369 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:25.477318    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-69pvz" podUID="85d978fc-d3e2-4c7a-9871-494ba0bb52d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.75:8080: connect: connection refused"
Sep 18 17:52:28.290480 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:28.290423    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-56241a-predictor-5449ff77bd-jlptd" podUID="d15df05b-a9de-45d5-8142-a0547993ad0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.74:8080: connect: connection refused"
Sep 18 17:52:30.534392 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:30.534311    2881 scope.go:117] "RemoveContainer" containerID="acf93b439856898d14f8303c78f01cd6f4bf7a64d9b1c4b58a589d757e72e133"
Sep 18 17:52:30.543161 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:30.543140    2881 scope.go:117] "RemoveContainer" containerID="08ea7bbca0d56fea748a3bbfde00715f54046e1619baa8058a6339fdc3896743"
Sep 18 17:52:35.476370 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:35.476330    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-69pvz" podUID="85d978fc-d3e2-4c7a-9871-494ba0bb52d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.75:8080: connect: connection refused"
Sep 18 17:52:38.290074 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:38.290031    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-56241a-predictor-5449ff77bd-jlptd" podUID="d15df05b-a9de-45d5-8142-a0547993ad0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.74:8080: connect: connection refused"
Sep 18 17:52:45.476310 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:45.476271    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-69pvz" podUID="85d978fc-d3e2-4c7a-9871-494ba0bb52d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.75:8080: connect: connection refused"
Sep 18 17:52:48.290121 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:48.290080    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-56241a-predictor-5449ff77bd-jlptd" podUID="d15df05b-a9de-45d5-8142-a0547993ad0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.74:8080: connect: connection refused"
Sep 18 17:52:53.893328 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:53.893295    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-56241a-predictor-5449ff77bd-jlptd"
Sep 18 17:52:55.476400 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:52:55.476349    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-69pvz" podUID="85d978fc-d3e2-4c7a-9871-494ba0bb52d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.75:8080: connect: connection refused"
Sep 18 17:53:05.476449 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:05.476407    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-69pvz" podUID="85d978fc-d3e2-4c7a-9871-494ba0bb52d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.75:8080: connect: connection refused"
Sep 18 17:53:10.987644 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:10.987598    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-56241a-predictor-7975b48799-njsrg"]
Sep 18 17:53:10.991600 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:10.991574    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-56241a-predictor-7975b48799-njsrg"
Sep 18 17:53:10.995186 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:10.995156    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-56241a-dockercfg-7rdlv\""
Sep 18 17:53:10.995364 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:10.995273    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-56241a\""
Sep 18 17:53:10.995609 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:10.995590    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-56241a-kube-rbac-proxy-sar-config\""
Sep 18 17:53:10.996460 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:10.996446    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-56241a-predictor-serving-cert\""
Sep 18 17:53:11.014200 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:11.014160    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-56241a-predictor-7975b48799-njsrg"]
Sep 18 17:53:11.030412 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:11.030372    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5779729f-7ad6-4038-8c61-4a7a8cc2a325-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-56241a-predictor-7975b48799-njsrg\" (UID: \"5779729f-7ad6-4038-8c61-4a7a8cc2a325\") " pod="kserve-ci-e2e-test/isvc-secondary-56241a-predictor-7975b48799-njsrg"
Sep 18 17:53:11.030591 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:11.030428    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpt2k\" (UniqueName: \"kubernetes.io/projected/5779729f-7ad6-4038-8c61-4a7a8cc2a325-kube-api-access-mpt2k\") pod \"isvc-secondary-56241a-predictor-7975b48799-njsrg\" (UID: \"5779729f-7ad6-4038-8c61-4a7a8cc2a325\") " pod="kserve-ci-e2e-test/isvc-secondary-56241a-predictor-7975b48799-njsrg"
Sep 18 17:53:11.030591 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:11.030482    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5779729f-7ad6-4038-8c61-4a7a8cc2a325-proxy-tls\") pod \"isvc-secondary-56241a-predictor-7975b48799-njsrg\" (UID: \"5779729f-7ad6-4038-8c61-4a7a8cc2a325\") " pod="kserve-ci-e2e-test/isvc-secondary-56241a-predictor-7975b48799-njsrg"
Sep 18 17:53:11.030591 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:11.030504    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5779729f-7ad6-4038-8c61-4a7a8cc2a325-cabundle-cert\") pod \"isvc-secondary-56241a-predictor-7975b48799-njsrg\" (UID: \"5779729f-7ad6-4038-8c61-4a7a8cc2a325\") " pod="kserve-ci-e2e-test/isvc-secondary-56241a-predictor-7975b48799-njsrg"
Sep 18 17:53:11.030709 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:11.030597    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5779729f-7ad6-4038-8c61-4a7a8cc2a325-kserve-provision-location\") pod \"isvc-secondary-56241a-predictor-7975b48799-njsrg\" (UID: \"5779729f-7ad6-4038-8c61-4a7a8cc2a325\") " pod="kserve-ci-e2e-test/isvc-secondary-56241a-predictor-7975b48799-njsrg"
Sep 18 17:53:11.131279 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:11.131204    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5779729f-7ad6-4038-8c61-4a7a8cc2a325-cabundle-cert\") pod \"isvc-secondary-56241a-predictor-7975b48799-njsrg\" (UID: \"5779729f-7ad6-4038-8c61-4a7a8cc2a325\") " pod="kserve-ci-e2e-test/isvc-secondary-56241a-predictor-7975b48799-njsrg"
Sep 18 17:53:11.131501 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:11.131301    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5779729f-7ad6-4038-8c61-4a7a8cc2a325-kserve-provision-location\") pod \"isvc-secondary-56241a-predictor-7975b48799-njsrg\" (UID: \"5779729f-7ad6-4038-8c61-4a7a8cc2a325\") " pod="kserve-ci-e2e-test/isvc-secondary-56241a-predictor-7975b48799-njsrg"
Sep 18 17:53:11.131501 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:11.131351    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5779729f-7ad6-4038-8c61-4a7a8cc2a325-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-56241a-predictor-7975b48799-njsrg\" (UID: \"5779729f-7ad6-4038-8c61-4a7a8cc2a325\") " pod="kserve-ci-e2e-test/isvc-secondary-56241a-predictor-7975b48799-njsrg"
Sep 18 17:53:11.131501 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:11.131387    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpt2k\" (UniqueName: \"kubernetes.io/projected/5779729f-7ad6-4038-8c61-4a7a8cc2a325-kube-api-access-mpt2k\") pod \"isvc-secondary-56241a-predictor-7975b48799-njsrg\" (UID: \"5779729f-7ad6-4038-8c61-4a7a8cc2a325\") " pod="kserve-ci-e2e-test/isvc-secondary-56241a-predictor-7975b48799-njsrg"
Sep 18 17:53:11.131501 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:11.131418    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5779729f-7ad6-4038-8c61-4a7a8cc2a325-proxy-tls\") pod \"isvc-secondary-56241a-predictor-7975b48799-njsrg\" (UID: \"5779729f-7ad6-4038-8c61-4a7a8cc2a325\") " pod="kserve-ci-e2e-test/isvc-secondary-56241a-predictor-7975b48799-njsrg"
Sep 18 17:53:11.131769 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:11.131745    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5779729f-7ad6-4038-8c61-4a7a8cc2a325-kserve-provision-location\") pod \"isvc-secondary-56241a-predictor-7975b48799-njsrg\" (UID: \"5779729f-7ad6-4038-8c61-4a7a8cc2a325\") " pod="kserve-ci-e2e-test/isvc-secondary-56241a-predictor-7975b48799-njsrg"
Sep 18 17:53:11.131979 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:11.131949    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5779729f-7ad6-4038-8c61-4a7a8cc2a325-cabundle-cert\") pod \"isvc-secondary-56241a-predictor-7975b48799-njsrg\" (UID: \"5779729f-7ad6-4038-8c61-4a7a8cc2a325\") " pod="kserve-ci-e2e-test/isvc-secondary-56241a-predictor-7975b48799-njsrg"
Sep 18 17:53:11.132146 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:11.132123    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5779729f-7ad6-4038-8c61-4a7a8cc2a325-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-56241a-predictor-7975b48799-njsrg\" (UID: \"5779729f-7ad6-4038-8c61-4a7a8cc2a325\") " pod="kserve-ci-e2e-test/isvc-secondary-56241a-predictor-7975b48799-njsrg"
Sep 18 17:53:11.133969 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:11.133949    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5779729f-7ad6-4038-8c61-4a7a8cc2a325-proxy-tls\") pod \"isvc-secondary-56241a-predictor-7975b48799-njsrg\" (UID: \"5779729f-7ad6-4038-8c61-4a7a8cc2a325\") " pod="kserve-ci-e2e-test/isvc-secondary-56241a-predictor-7975b48799-njsrg"
Sep 18 17:53:11.142754 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:11.142725    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpt2k\" (UniqueName: \"kubernetes.io/projected/5779729f-7ad6-4038-8c61-4a7a8cc2a325-kube-api-access-mpt2k\") pod \"isvc-secondary-56241a-predictor-7975b48799-njsrg\" (UID: \"5779729f-7ad6-4038-8c61-4a7a8cc2a325\") " pod="kserve-ci-e2e-test/isvc-secondary-56241a-predictor-7975b48799-njsrg"
Sep 18 17:53:11.303248 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:11.303188    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-56241a-predictor-7975b48799-njsrg"
Sep 18 17:53:11.611072 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:11.611044    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-56241a-predictor-7975b48799-njsrg"]
Sep 18 17:53:11.613720 ip-10-0-143-5 kubenswrapper[2881]: W0918 17:53:11.613688    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5779729f_7ad6_4038_8c61_4a7a8cc2a325.slice/crio-6c65800dc586bdb8d36f9d24224d30058e36470870c7e80968d7a59fa6781237 WatchSource:0}: Error finding container 6c65800dc586bdb8d36f9d24224d30058e36470870c7e80968d7a59fa6781237: Status 404 returned error can't find the container with id 6c65800dc586bdb8d36f9d24224d30058e36470870c7e80968d7a59fa6781237
Sep 18 17:53:11.751981 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:11.751943    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-56241a-predictor-7975b48799-njsrg" event={"ID":"5779729f-7ad6-4038-8c61-4a7a8cc2a325","Type":"ContainerStarted","Data":"b9cee1f53d97dfda8fe9036cf14f86c804fc2e85afcac60e3f80536abcb3ce60"}
Sep 18 17:53:11.752172 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:11.752018    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-56241a-predictor-7975b48799-njsrg" event={"ID":"5779729f-7ad6-4038-8c61-4a7a8cc2a325","Type":"ContainerStarted","Data":"6c65800dc586bdb8d36f9d24224d30058e36470870c7e80968d7a59fa6781237"}
Sep 18 17:53:15.476635 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:15.476589    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-69pvz" podUID="85d978fc-d3e2-4c7a-9871-494ba0bb52d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.75:8080: connect: connection refused"
Sep 18 17:53:18.786753 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:18.786725    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-56241a-predictor-7975b48799-njsrg_5779729f-7ad6-4038-8c61-4a7a8cc2a325/storage-initializer/0.log"
Sep 18 17:53:18.787135 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:18.786763    2881 generic.go:356] "Generic (PLEG): container finished" podID="5779729f-7ad6-4038-8c61-4a7a8cc2a325" containerID="b9cee1f53d97dfda8fe9036cf14f86c804fc2e85afcac60e3f80536abcb3ce60" exitCode=1
Sep 18 17:53:18.787135 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:18.786842    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-56241a-predictor-7975b48799-njsrg" event={"ID":"5779729f-7ad6-4038-8c61-4a7a8cc2a325","Type":"ContainerDied","Data":"b9cee1f53d97dfda8fe9036cf14f86c804fc2e85afcac60e3f80536abcb3ce60"}
Sep 18 17:53:19.793306 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:19.793280    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-56241a-predictor-7975b48799-njsrg_5779729f-7ad6-4038-8c61-4a7a8cc2a325/storage-initializer/0.log"
Sep 18 17:53:19.793696 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:19.793395    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-56241a-predictor-7975b48799-njsrg" event={"ID":"5779729f-7ad6-4038-8c61-4a7a8cc2a325","Type":"ContainerStarted","Data":"0c89c95bc5d49bf278ab79a397bb11a1e692c2a04dfffafb99a128c65de7be66"}
Sep 18 17:53:21.806010 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:21.805975    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-56241a-predictor-7975b48799-njsrg_5779729f-7ad6-4038-8c61-4a7a8cc2a325/storage-initializer/1.log"
Sep 18 17:53:21.806425 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:21.806412    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-56241a-predictor-7975b48799-njsrg_5779729f-7ad6-4038-8c61-4a7a8cc2a325/storage-initializer/0.log"
Sep 18 17:53:21.806475 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:21.806444    2881 generic.go:356] "Generic (PLEG): container finished" podID="5779729f-7ad6-4038-8c61-4a7a8cc2a325" containerID="0c89c95bc5d49bf278ab79a397bb11a1e692c2a04dfffafb99a128c65de7be66" exitCode=1
Sep 18 17:53:21.806552 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:21.806528    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-56241a-predictor-7975b48799-njsrg" event={"ID":"5779729f-7ad6-4038-8c61-4a7a8cc2a325","Type":"ContainerDied","Data":"0c89c95bc5d49bf278ab79a397bb11a1e692c2a04dfffafb99a128c65de7be66"}
Sep 18 17:53:21.806661 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:21.806647    2881 scope.go:117] "RemoveContainer" containerID="b9cee1f53d97dfda8fe9036cf14f86c804fc2e85afcac60e3f80536abcb3ce60"
Sep 18 17:53:21.807102 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:21.807085    2881 scope.go:117] "RemoveContainer" containerID="b9cee1f53d97dfda8fe9036cf14f86c804fc2e85afcac60e3f80536abcb3ce60"
Sep 18 17:53:21.821482 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:53:21.821444    2881 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-56241a-predictor-7975b48799-njsrg_kserve-ci-e2e-test(5779729f-7ad6-4038-8c61-4a7a8cc2a325)\"" pod="kserve-ci-e2e-test/isvc-secondary-56241a-predictor-7975b48799-njsrg" podUID="5779729f-7ad6-4038-8c61-4a7a8cc2a325"
Sep 18 17:53:21.821627 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:53:21.821558    2881 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-56241a-predictor-7975b48799-njsrg_kserve-ci-e2e-test_5779729f-7ad6-4038-8c61-4a7a8cc2a325_0 in pod sandbox 6c65800dc586bdb8d36f9d24224d30058e36470870c7e80968d7a59fa6781237 from index: no such id: 'b9cee1f53d97dfda8fe9036cf14f86c804fc2e85afcac60e3f80536abcb3ce60'" containerID="b9cee1f53d97dfda8fe9036cf14f86c804fc2e85afcac60e3f80536abcb3ce60"
Sep 18 17:53:21.821627 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:21.821597    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9cee1f53d97dfda8fe9036cf14f86c804fc2e85afcac60e3f80536abcb3ce60"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-56241a-predictor-7975b48799-njsrg_kserve-ci-e2e-test_5779729f-7ad6-4038-8c61-4a7a8cc2a325_0 in pod sandbox 6c65800dc586bdb8d36f9d24224d30058e36470870c7e80968d7a59fa6781237 from index: no such id: 'b9cee1f53d97dfda8fe9036cf14f86c804fc2e85afcac60e3f80536abcb3ce60'"
Sep 18 17:53:22.813131 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:22.813101    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-56241a-predictor-7975b48799-njsrg_5779729f-7ad6-4038-8c61-4a7a8cc2a325/storage-initializer/1.log"
Sep 18 17:53:25.476368 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:25.476328    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-69pvz" podUID="85d978fc-d3e2-4c7a-9871-494ba0bb52d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.75:8080: connect: connection refused"
Sep 18 17:53:30.944558 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:30.944527    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-56241a-predictor-7975b48799-njsrg"]
Sep 18 17:53:30.977750 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:30.977713    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-56241a-predictor-5449ff77bd-jlptd"]
Sep 18 17:53:30.978335 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:30.978279    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-56241a-predictor-5449ff77bd-jlptd" podUID="d15df05b-a9de-45d5-8142-a0547993ad0f" containerName="kserve-container" containerID="cri-o://5380cb867027a9bae07f02d26ebf247d4734d5a0d211e59a9e1d24bd3cf8c07c" gracePeriod=30
Sep 18 17:53:30.978581 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:30.978549    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-56241a-predictor-5449ff77bd-jlptd" podUID="d15df05b-a9de-45d5-8142-a0547993ad0f" containerName="kube-rbac-proxy" containerID="cri-o://47a0753200cef0e597aa9a91051b76f52ba06fed638ef661cab76badcf7b5466" gracePeriod=30
Sep 18 17:53:31.101304 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:31.101282    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-56241a-predictor-7975b48799-njsrg_5779729f-7ad6-4038-8c61-4a7a8cc2a325/storage-initializer/1.log"
Sep 18 17:53:31.103433 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:31.103415    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-56241a-predictor-7975b48799-njsrg"
Sep 18 17:53:31.220950 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:31.220854    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5779729f-7ad6-4038-8c61-4a7a8cc2a325-cabundle-cert\") pod \"5779729f-7ad6-4038-8c61-4a7a8cc2a325\" (UID: \"5779729f-7ad6-4038-8c61-4a7a8cc2a325\") "
Sep 18 17:53:31.220950 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:31.220945    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mpt2k\" (UniqueName: \"kubernetes.io/projected/5779729f-7ad6-4038-8c61-4a7a8cc2a325-kube-api-access-mpt2k\") pod \"5779729f-7ad6-4038-8c61-4a7a8cc2a325\" (UID: \"5779729f-7ad6-4038-8c61-4a7a8cc2a325\") "
Sep 18 17:53:31.221163 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:31.221045    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5779729f-7ad6-4038-8c61-4a7a8cc2a325-kserve-provision-location\") pod \"5779729f-7ad6-4038-8c61-4a7a8cc2a325\" (UID: \"5779729f-7ad6-4038-8c61-4a7a8cc2a325\") "
Sep 18 17:53:31.221163 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:31.221109    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5779729f-7ad6-4038-8c61-4a7a8cc2a325-kube-rbac-proxy-sar-config\") pod \"5779729f-7ad6-4038-8c61-4a7a8cc2a325\" (UID: \"5779729f-7ad6-4038-8c61-4a7a8cc2a325\") "
Sep 18 17:53:31.221163 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:31.221132    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5779729f-7ad6-4038-8c61-4a7a8cc2a325-proxy-tls\") pod \"5779729f-7ad6-4038-8c61-4a7a8cc2a325\" (UID: \"5779729f-7ad6-4038-8c61-4a7a8cc2a325\") "
Sep 18 17:53:31.221357 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:31.221334    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5779729f-7ad6-4038-8c61-4a7a8cc2a325-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5779729f-7ad6-4038-8c61-4a7a8cc2a325" (UID: "5779729f-7ad6-4038-8c61-4a7a8cc2a325"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:53:31.221423 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:31.221356    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5779729f-7ad6-4038-8c61-4a7a8cc2a325-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5779729f-7ad6-4038-8c61-4a7a8cc2a325" (UID: "5779729f-7ad6-4038-8c61-4a7a8cc2a325"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:53:31.221482 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:31.221458    2881 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5779729f-7ad6-4038-8c61-4a7a8cc2a325-cabundle-cert\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:53:31.221482 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:31.221472    2881 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5779729f-7ad6-4038-8c61-4a7a8cc2a325-kserve-provision-location\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:53:31.221590 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:31.221549    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5779729f-7ad6-4038-8c61-4a7a8cc2a325-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5779729f-7ad6-4038-8c61-4a7a8cc2a325" (UID: "5779729f-7ad6-4038-8c61-4a7a8cc2a325"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:53:31.223093 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:31.223073    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5779729f-7ad6-4038-8c61-4a7a8cc2a325-kube-api-access-mpt2k" (OuterVolumeSpecName: "kube-api-access-mpt2k") pod "5779729f-7ad6-4038-8c61-4a7a8cc2a325" (UID: "5779729f-7ad6-4038-8c61-4a7a8cc2a325"). InnerVolumeSpecName "kube-api-access-mpt2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:53:31.223189 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:31.223120    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5779729f-7ad6-4038-8c61-4a7a8cc2a325-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5779729f-7ad6-4038-8c61-4a7a8cc2a325" (UID: "5779729f-7ad6-4038-8c61-4a7a8cc2a325"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:53:31.322266 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:31.322206    2881 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpt2k\" (UniqueName: \"kubernetes.io/projected/5779729f-7ad6-4038-8c61-4a7a8cc2a325-kube-api-access-mpt2k\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:53:31.322266 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:31.322258    2881 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5779729f-7ad6-4038-8c61-4a7a8cc2a325-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:53:31.322266 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:31.322269    2881 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5779729f-7ad6-4038-8c61-4a7a8cc2a325-proxy-tls\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:53:31.861332 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:31.861300    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-56241a-predictor-7975b48799-njsrg_5779729f-7ad6-4038-8c61-4a7a8cc2a325/storage-initializer/1.log"
Sep 18 17:53:31.861665 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:31.861636    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-56241a-predictor-7975b48799-njsrg" event={"ID":"5779729f-7ad6-4038-8c61-4a7a8cc2a325","Type":"ContainerDied","Data":"6c65800dc586bdb8d36f9d24224d30058e36470870c7e80968d7a59fa6781237"}
Sep 18 17:53:31.861823 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:31.861792    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-56241a-predictor-7975b48799-njsrg"
Sep 18 17:53:31.862028 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:31.862006    2881 scope.go:117] "RemoveContainer" containerID="0c89c95bc5d49bf278ab79a397bb11a1e692c2a04dfffafb99a128c65de7be66"
Sep 18 17:53:31.865059 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:31.865035    2881 generic.go:356] "Generic (PLEG): container finished" podID="d15df05b-a9de-45d5-8142-a0547993ad0f" containerID="47a0753200cef0e597aa9a91051b76f52ba06fed638ef661cab76badcf7b5466" exitCode=2
Sep 18 17:53:31.865166 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:31.865113    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-56241a-predictor-5449ff77bd-jlptd" event={"ID":"d15df05b-a9de-45d5-8142-a0547993ad0f","Type":"ContainerDied","Data":"47a0753200cef0e597aa9a91051b76f52ba06fed638ef661cab76badcf7b5466"}
Sep 18 17:53:31.902679 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:31.902644    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-56241a-predictor-7975b48799-njsrg"]
Sep 18 17:53:31.907181 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:31.907151    2881 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-56241a-predictor-7975b48799-njsrg"]
Sep 18 17:53:33.285916 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:33.285871    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-56241a-predictor-5449ff77bd-jlptd" podUID="d15df05b-a9de-45d5-8142-a0547993ad0f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.74:8643/healthz\": dial tcp 10.134.0.74:8643: connect: connection refused"
Sep 18 17:53:33.888920 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:33.888875    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-56241a-predictor-5449ff77bd-jlptd" podUID="d15df05b-a9de-45d5-8142-a0547993ad0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.74:8080: connect: connection refused"
Sep 18 17:53:33.893684 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:33.893654    2881 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5779729f-7ad6-4038-8c61-4a7a8cc2a325" path="/var/lib/kubelet/pods/5779729f-7ad6-4038-8c61-4a7a8cc2a325/volumes"
Sep 18 17:53:35.476881 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:35.476840    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-69pvz" podUID="85d978fc-d3e2-4c7a-9871-494ba0bb52d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.75:8080: connect: connection refused"
Sep 18 17:53:36.425634 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:36.425603    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-56241a-predictor-5449ff77bd-jlptd"
Sep 18 17:53:36.572880 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:36.572839    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d15df05b-a9de-45d5-8142-a0547993ad0f-cabundle-cert\") pod \"d15df05b-a9de-45d5-8142-a0547993ad0f\" (UID: \"d15df05b-a9de-45d5-8142-a0547993ad0f\") "
Sep 18 17:53:36.573323 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:36.572911    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qtksc\" (UniqueName: \"kubernetes.io/projected/d15df05b-a9de-45d5-8142-a0547993ad0f-kube-api-access-qtksc\") pod \"d15df05b-a9de-45d5-8142-a0547993ad0f\" (UID: \"d15df05b-a9de-45d5-8142-a0547993ad0f\") "
Sep 18 17:53:36.573323 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:36.572955    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d15df05b-a9de-45d5-8142-a0547993ad0f-kube-rbac-proxy-sar-config\") pod \"d15df05b-a9de-45d5-8142-a0547993ad0f\" (UID: \"d15df05b-a9de-45d5-8142-a0547993ad0f\") "
Sep 18 17:53:36.573323 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:36.573011    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d15df05b-a9de-45d5-8142-a0547993ad0f-kserve-provision-location\") pod \"d15df05b-a9de-45d5-8142-a0547993ad0f\" (UID: \"d15df05b-a9de-45d5-8142-a0547993ad0f\") "
Sep 18 17:53:36.573323 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:36.573068    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d15df05b-a9de-45d5-8142-a0547993ad0f-proxy-tls\") pod \"d15df05b-a9de-45d5-8142-a0547993ad0f\" (UID: \"d15df05b-a9de-45d5-8142-a0547993ad0f\") "
Sep 18 17:53:36.573479 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:36.573347    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d15df05b-a9de-45d5-8142-a0547993ad0f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d15df05b-a9de-45d5-8142-a0547993ad0f" (UID: "d15df05b-a9de-45d5-8142-a0547993ad0f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:53:36.573479 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:36.573351    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d15df05b-a9de-45d5-8142-a0547993ad0f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d15df05b-a9de-45d5-8142-a0547993ad0f" (UID: "d15df05b-a9de-45d5-8142-a0547993ad0f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:53:36.573479 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:36.573376    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d15df05b-a9de-45d5-8142-a0547993ad0f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d15df05b-a9de-45d5-8142-a0547993ad0f" (UID: "d15df05b-a9de-45d5-8142-a0547993ad0f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:53:36.575294 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:36.575260    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d15df05b-a9de-45d5-8142-a0547993ad0f-kube-api-access-qtksc" (OuterVolumeSpecName: "kube-api-access-qtksc") pod "d15df05b-a9de-45d5-8142-a0547993ad0f" (UID: "d15df05b-a9de-45d5-8142-a0547993ad0f"). InnerVolumeSpecName "kube-api-access-qtksc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:53:36.575294 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:36.575284    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d15df05b-a9de-45d5-8142-a0547993ad0f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d15df05b-a9de-45d5-8142-a0547993ad0f" (UID: "d15df05b-a9de-45d5-8142-a0547993ad0f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:53:36.674492 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:36.674445    2881 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d15df05b-a9de-45d5-8142-a0547993ad0f-cabundle-cert\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:53:36.674492 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:36.674481    2881 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtksc\" (UniqueName: \"kubernetes.io/projected/d15df05b-a9de-45d5-8142-a0547993ad0f-kube-api-access-qtksc\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:53:36.674492 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:36.674496    2881 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d15df05b-a9de-45d5-8142-a0547993ad0f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:53:36.674492 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:36.674507    2881 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d15df05b-a9de-45d5-8142-a0547993ad0f-kserve-provision-location\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:53:36.674786 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:36.674520    2881 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d15df05b-a9de-45d5-8142-a0547993ad0f-proxy-tls\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:53:36.890591 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:36.890505    2881 generic.go:356] "Generic (PLEG): container finished" podID="d15df05b-a9de-45d5-8142-a0547993ad0f" containerID="5380cb867027a9bae07f02d26ebf247d4734d5a0d211e59a9e1d24bd3cf8c07c" exitCode=0
Sep 18 17:53:36.890591 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:36.890562    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-56241a-predictor-5449ff77bd-jlptd"
Sep 18 17:53:36.890786 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:36.890592    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-56241a-predictor-5449ff77bd-jlptd" event={"ID":"d15df05b-a9de-45d5-8142-a0547993ad0f","Type":"ContainerDied","Data":"5380cb867027a9bae07f02d26ebf247d4734d5a0d211e59a9e1d24bd3cf8c07c"}
Sep 18 17:53:36.890786 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:36.890657    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-56241a-predictor-5449ff77bd-jlptd" event={"ID":"d15df05b-a9de-45d5-8142-a0547993ad0f","Type":"ContainerDied","Data":"bdfea447a77a289acdf99b900d5fd388a57e8f6484f1a56beab08bdd6212b6ed"}
Sep 18 17:53:36.890786 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:36.890687    2881 scope.go:117] "RemoveContainer" containerID="47a0753200cef0e597aa9a91051b76f52ba06fed638ef661cab76badcf7b5466"
Sep 18 17:53:36.899462 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:36.899438    2881 scope.go:117] "RemoveContainer" containerID="5380cb867027a9bae07f02d26ebf247d4734d5a0d211e59a9e1d24bd3cf8c07c"
Sep 18 17:53:36.908044 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:36.908025    2881 scope.go:117] "RemoveContainer" containerID="a95b3ea2a1fd729e4e0a52432a8076107c524e82e2540b14c4484387d4c6ac9e"
Sep 18 17:53:36.915368 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:36.915337    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-56241a-predictor-5449ff77bd-jlptd"]
Sep 18 17:53:36.916424 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:36.916405    2881 scope.go:117] "RemoveContainer" containerID="47a0753200cef0e597aa9a91051b76f52ba06fed638ef661cab76badcf7b5466"
Sep 18 17:53:36.916777 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:53:36.916758    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47a0753200cef0e597aa9a91051b76f52ba06fed638ef661cab76badcf7b5466\": container with ID starting with 47a0753200cef0e597aa9a91051b76f52ba06fed638ef661cab76badcf7b5466 not found: ID does not exist" containerID="47a0753200cef0e597aa9a91051b76f52ba06fed638ef661cab76badcf7b5466"
Sep 18 17:53:36.916832 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:36.916787    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47a0753200cef0e597aa9a91051b76f52ba06fed638ef661cab76badcf7b5466"} err="failed to get container status \"47a0753200cef0e597aa9a91051b76f52ba06fed638ef661cab76badcf7b5466\": rpc error: code = NotFound desc = could not find container \"47a0753200cef0e597aa9a91051b76f52ba06fed638ef661cab76badcf7b5466\": container with ID starting with 47a0753200cef0e597aa9a91051b76f52ba06fed638ef661cab76badcf7b5466 not found: ID does not exist"
Sep 18 17:53:36.916832 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:36.916807    2881 scope.go:117] "RemoveContainer" containerID="5380cb867027a9bae07f02d26ebf247d4734d5a0d211e59a9e1d24bd3cf8c07c"
Sep 18 17:53:36.917076 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:53:36.917056    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5380cb867027a9bae07f02d26ebf247d4734d5a0d211e59a9e1d24bd3cf8c07c\": container with ID starting with 5380cb867027a9bae07f02d26ebf247d4734d5a0d211e59a9e1d24bd3cf8c07c not found: ID does not exist" containerID="5380cb867027a9bae07f02d26ebf247d4734d5a0d211e59a9e1d24bd3cf8c07c"
Sep 18 17:53:36.917139 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:36.917079    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5380cb867027a9bae07f02d26ebf247d4734d5a0d211e59a9e1d24bd3cf8c07c"} err="failed to get container status \"5380cb867027a9bae07f02d26ebf247d4734d5a0d211e59a9e1d24bd3cf8c07c\": rpc error: code = NotFound desc = could not find container \"5380cb867027a9bae07f02d26ebf247d4734d5a0d211e59a9e1d24bd3cf8c07c\": container with ID starting with 5380cb867027a9bae07f02d26ebf247d4734d5a0d211e59a9e1d24bd3cf8c07c not found: ID does not exist"
Sep 18 17:53:36.917139 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:36.917096    2881 scope.go:117] "RemoveContainer" containerID="a95b3ea2a1fd729e4e0a52432a8076107c524e82e2540b14c4484387d4c6ac9e"
Sep 18 17:53:36.917486 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:53:36.917414    2881 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a95b3ea2a1fd729e4e0a52432a8076107c524e82e2540b14c4484387d4c6ac9e\": container with ID starting with a95b3ea2a1fd729e4e0a52432a8076107c524e82e2540b14c4484387d4c6ac9e not found: ID does not exist" containerID="a95b3ea2a1fd729e4e0a52432a8076107c524e82e2540b14c4484387d4c6ac9e"
Sep 18 17:53:36.917486 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:36.917451    2881 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a95b3ea2a1fd729e4e0a52432a8076107c524e82e2540b14c4484387d4c6ac9e"} err="failed to get container status \"a95b3ea2a1fd729e4e0a52432a8076107c524e82e2540b14c4484387d4c6ac9e\": rpc error: code = NotFound desc = could not find container \"a95b3ea2a1fd729e4e0a52432a8076107c524e82e2540b14c4484387d4c6ac9e\": container with ID starting with a95b3ea2a1fd729e4e0a52432a8076107c524e82e2540b14c4484387d4c6ac9e not found: ID does not exist"
Sep 18 17:53:36.920403 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:36.920379    2881 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-56241a-predictor-5449ff77bd-jlptd"]
Sep 18 17:53:37.893406 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:37.893373    2881 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d15df05b-a9de-45d5-8142-a0547993ad0f" path="/var/lib/kubelet/pods/d15df05b-a9de-45d5-8142-a0547993ad0f/volumes"
Sep 18 17:53:39.893611 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:39.893584    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-69pvz"
Sep 18 17:53:45.413213 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:45.413176    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-69pvz"]
Sep 18 17:53:45.413752 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:45.413488    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-69pvz" podUID="85d978fc-d3e2-4c7a-9871-494ba0bb52d5" containerName="kserve-container" containerID="cri-o://c953eeac10219ab1637dbe4cca3a4636b5ce7de1334345306e33bd16f9339c3c" gracePeriod=30
Sep 18 17:53:45.413752 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:45.413490    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-69pvz" podUID="85d978fc-d3e2-4c7a-9871-494ba0bb52d5" containerName="kube-rbac-proxy" containerID="cri-o://88269f02b709957eaa22467f06835229cb4eea9579f731d312a06962f69cc13c" gracePeriod=30
Sep 18 17:53:45.471033 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:45.470986    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-69pvz" podUID="85d978fc-d3e2-4c7a-9871-494ba0bb52d5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.75:8643/healthz\": dial tcp 10.134.0.75:8643: connect: connection refused"
Sep 18 17:53:45.929693 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:45.929659    2881 generic.go:356] "Generic (PLEG): container finished" podID="85d978fc-d3e2-4c7a-9871-494ba0bb52d5" containerID="88269f02b709957eaa22467f06835229cb4eea9579f731d312a06962f69cc13c" exitCode=2
Sep 18 17:53:45.929875 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:45.929728    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-69pvz" event={"ID":"85d978fc-d3e2-4c7a-9871-494ba0bb52d5","Type":"ContainerDied","Data":"88269f02b709957eaa22467f06835229cb4eea9579f731d312a06962f69cc13c"}
Sep 18 17:53:49.891887 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:49.891842    2881 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-69pvz" podUID="85d978fc-d3e2-4c7a-9871-494ba0bb52d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.75:8080: connect: connection refused"
Sep 18 17:53:49.949379 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:49.949347    2881 generic.go:356] "Generic (PLEG): container finished" podID="85d978fc-d3e2-4c7a-9871-494ba0bb52d5" containerID="c953eeac10219ab1637dbe4cca3a4636b5ce7de1334345306e33bd16f9339c3c" exitCode=0
Sep 18 17:53:49.949556 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:49.949431    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-69pvz" event={"ID":"85d978fc-d3e2-4c7a-9871-494ba0bb52d5","Type":"ContainerDied","Data":"c953eeac10219ab1637dbe4cca3a4636b5ce7de1334345306e33bd16f9339c3c"}
Sep 18 17:53:50.059263 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:50.059217    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-69pvz"
Sep 18 17:53:50.095646 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:50.095560    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-99dtg\" (UniqueName: \"kubernetes.io/projected/85d978fc-d3e2-4c7a-9871-494ba0bb52d5-kube-api-access-99dtg\") pod \"85d978fc-d3e2-4c7a-9871-494ba0bb52d5\" (UID: \"85d978fc-d3e2-4c7a-9871-494ba0bb52d5\") "
Sep 18 17:53:50.095646 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:50.095606    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85d978fc-d3e2-4c7a-9871-494ba0bb52d5-cabundle-cert\") pod \"85d978fc-d3e2-4c7a-9871-494ba0bb52d5\" (UID: \"85d978fc-d3e2-4c7a-9871-494ba0bb52d5\") "
Sep 18 17:53:50.095646 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:50.095626    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85d978fc-d3e2-4c7a-9871-494ba0bb52d5-proxy-tls\") pod \"85d978fc-d3e2-4c7a-9871-494ba0bb52d5\" (UID: \"85d978fc-d3e2-4c7a-9871-494ba0bb52d5\") "
Sep 18 17:53:50.095959 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:50.095669    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85d978fc-d3e2-4c7a-9871-494ba0bb52d5-kserve-provision-location\") pod \"85d978fc-d3e2-4c7a-9871-494ba0bb52d5\" (UID: \"85d978fc-d3e2-4c7a-9871-494ba0bb52d5\") "
Sep 18 17:53:50.095959 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:50.095725    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85d978fc-d3e2-4c7a-9871-494ba0bb52d5-kube-rbac-proxy-sar-config\") pod \"85d978fc-d3e2-4c7a-9871-494ba0bb52d5\" (UID: \"85d978fc-d3e2-4c7a-9871-494ba0bb52d5\") "
Sep 18 17:53:50.096080 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:50.096046    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85d978fc-d3e2-4c7a-9871-494ba0bb52d5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "85d978fc-d3e2-4c7a-9871-494ba0bb52d5" (UID: "85d978fc-d3e2-4c7a-9871-494ba0bb52d5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:53:50.096139 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:50.096074    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85d978fc-d3e2-4c7a-9871-494ba0bb52d5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "85d978fc-d3e2-4c7a-9871-494ba0bb52d5" (UID: "85d978fc-d3e2-4c7a-9871-494ba0bb52d5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:53:50.096236 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:50.096190    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85d978fc-d3e2-4c7a-9871-494ba0bb52d5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "85d978fc-d3e2-4c7a-9871-494ba0bb52d5" (UID: "85d978fc-d3e2-4c7a-9871-494ba0bb52d5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:53:50.097864 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:50.097835    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85d978fc-d3e2-4c7a-9871-494ba0bb52d5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "85d978fc-d3e2-4c7a-9871-494ba0bb52d5" (UID: "85d978fc-d3e2-4c7a-9871-494ba0bb52d5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:53:50.097864 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:50.097853    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85d978fc-d3e2-4c7a-9871-494ba0bb52d5-kube-api-access-99dtg" (OuterVolumeSpecName: "kube-api-access-99dtg") pod "85d978fc-d3e2-4c7a-9871-494ba0bb52d5" (UID: "85d978fc-d3e2-4c7a-9871-494ba0bb52d5"). InnerVolumeSpecName "kube-api-access-99dtg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:53:50.196471 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:50.196427    2881 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85d978fc-d3e2-4c7a-9871-494ba0bb52d5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:53:50.196471 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:50.196468    2881 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99dtg\" (UniqueName: \"kubernetes.io/projected/85d978fc-d3e2-4c7a-9871-494ba0bb52d5-kube-api-access-99dtg\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:53:50.196693 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:50.196481    2881 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85d978fc-d3e2-4c7a-9871-494ba0bb52d5-cabundle-cert\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:53:50.196693 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:50.196496    2881 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85d978fc-d3e2-4c7a-9871-494ba0bb52d5-proxy-tls\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:53:50.196693 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:50.196508    2881 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85d978fc-d3e2-4c7a-9871-494ba0bb52d5-kserve-provision-location\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:53:50.955733 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:50.955696    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-69pvz" event={"ID":"85d978fc-d3e2-4c7a-9871-494ba0bb52d5","Type":"ContainerDied","Data":"7b8d0a7d1b30308d5de50b97e61106689bef1d62f7e20e64f8e3c345bf2a5ae7"}
Sep 18 17:53:50.955733 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:50.955715    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-69pvz"
Sep 18 17:53:50.956266 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:50.955773    2881 scope.go:117] "RemoveContainer" containerID="88269f02b709957eaa22467f06835229cb4eea9579f731d312a06962f69cc13c"
Sep 18 17:53:50.965422 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:50.965365    2881 scope.go:117] "RemoveContainer" containerID="c953eeac10219ab1637dbe4cca3a4636b5ce7de1334345306e33bd16f9339c3c"
Sep 18 17:53:50.973034 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:50.973013    2881 scope.go:117] "RemoveContainer" containerID="87101fe5d782bc8c2d2e5958a842c2c2029030ffb22a3d7ecd491641ac1ecba3"
Sep 18 17:53:50.980291 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:50.980263    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-69pvz"]
Sep 18 17:53:50.984029 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:50.984003    2881 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-69pvz"]
Sep 18 17:53:51.892734 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:51.892693    2881 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85d978fc-d3e2-4c7a-9871-494ba0bb52d5" path="/var/lib/kubelet/pods/85d978fc-d3e2-4c7a-9871-494ba0bb52d5/volumes"
Sep 18 17:53:52.171802 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:52.171711    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bz2w9/must-gather-sl6xd"]
Sep 18 17:53:52.172192 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:52.172042    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5779729f-7ad6-4038-8c61-4a7a8cc2a325" containerName="storage-initializer"
Sep 18 17:53:52.172192 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:52.172054    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="5779729f-7ad6-4038-8c61-4a7a8cc2a325" containerName="storage-initializer"
Sep 18 17:53:52.172192 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:52.172062    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85d978fc-d3e2-4c7a-9871-494ba0bb52d5" containerName="storage-initializer"
Sep 18 17:53:52.172192 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:52.172068    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="85d978fc-d3e2-4c7a-9871-494ba0bb52d5" containerName="storage-initializer"
Sep 18 17:53:52.172192 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:52.172076    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85d978fc-d3e2-4c7a-9871-494ba0bb52d5" containerName="kube-rbac-proxy"
Sep 18 17:53:52.172192 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:52.172082    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="85d978fc-d3e2-4c7a-9871-494ba0bb52d5" containerName="kube-rbac-proxy"
Sep 18 17:53:52.172192 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:52.172094    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d15df05b-a9de-45d5-8142-a0547993ad0f" containerName="storage-initializer"
Sep 18 17:53:52.172192 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:52.172099    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="d15df05b-a9de-45d5-8142-a0547993ad0f" containerName="storage-initializer"
Sep 18 17:53:52.172192 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:52.172110    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d15df05b-a9de-45d5-8142-a0547993ad0f" containerName="kserve-container"
Sep 18 17:53:52.172192 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:52.172115    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="d15df05b-a9de-45d5-8142-a0547993ad0f" containerName="kserve-container"
Sep 18 17:53:52.172192 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:52.172124    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85d978fc-d3e2-4c7a-9871-494ba0bb52d5" containerName="kserve-container"
Sep 18 17:53:52.172192 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:52.172129    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="85d978fc-d3e2-4c7a-9871-494ba0bb52d5" containerName="kserve-container"
Sep 18 17:53:52.172192 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:52.172134    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5779729f-7ad6-4038-8c61-4a7a8cc2a325" containerName="storage-initializer"
Sep 18 17:53:52.172192 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:52.172139    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="5779729f-7ad6-4038-8c61-4a7a8cc2a325" containerName="storage-initializer"
Sep 18 17:53:52.172192 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:52.172149    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d15df05b-a9de-45d5-8142-a0547993ad0f" containerName="kube-rbac-proxy"
Sep 18 17:53:52.172192 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:52.172153    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="d15df05b-a9de-45d5-8142-a0547993ad0f" containerName="kube-rbac-proxy"
Sep 18 17:53:52.172730 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:52.172236    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="85d978fc-d3e2-4c7a-9871-494ba0bb52d5" containerName="kserve-container"
Sep 18 17:53:52.172730 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:52.172247    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="5779729f-7ad6-4038-8c61-4a7a8cc2a325" containerName="storage-initializer"
Sep 18 17:53:52.172730 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:52.172253    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="85d978fc-d3e2-4c7a-9871-494ba0bb52d5" containerName="kube-rbac-proxy"
Sep 18 17:53:52.172730 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:52.172260    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="d15df05b-a9de-45d5-8142-a0547993ad0f" containerName="kserve-container"
Sep 18 17:53:52.172730 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:52.172265    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="d15df05b-a9de-45d5-8142-a0547993ad0f" containerName="kube-rbac-proxy"
Sep 18 17:53:52.172730 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:52.172350    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="5779729f-7ad6-4038-8c61-4a7a8cc2a325" containerName="storage-initializer"
Sep 18 17:53:52.176896 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:52.176876    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bz2w9/must-gather-sl6xd"
Sep 18 17:53:52.179404 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:52.179379    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bz2w9\"/\"openshift-service-ca.crt\""
Sep 18 17:53:52.179557 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:52.179407    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bz2w9\"/\"kube-root-ca.crt\""
Sep 18 17:53:52.185900 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:52.185873    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bz2w9/must-gather-sl6xd"]
Sep 18 17:53:52.214747 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:52.214707    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/bb0e84cb-3717-40d6-9156-59d43497a89c-must-gather-output\") pod \"must-gather-sl6xd\" (UID: \"bb0e84cb-3717-40d6-9156-59d43497a89c\") " pod="openshift-must-gather-bz2w9/must-gather-sl6xd"
Sep 18 17:53:52.214930 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:52.214782    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r84wz\" (UniqueName: \"kubernetes.io/projected/bb0e84cb-3717-40d6-9156-59d43497a89c-kube-api-access-r84wz\") pod \"must-gather-sl6xd\" (UID: \"bb0e84cb-3717-40d6-9156-59d43497a89c\") " pod="openshift-must-gather-bz2w9/must-gather-sl6xd"
Sep 18 17:53:52.316081 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:52.316025    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/bb0e84cb-3717-40d6-9156-59d43497a89c-must-gather-output\") pod \"must-gather-sl6xd\" (UID: \"bb0e84cb-3717-40d6-9156-59d43497a89c\") " pod="openshift-must-gather-bz2w9/must-gather-sl6xd"
Sep 18 17:53:52.316297 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:52.316113    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r84wz\" (UniqueName: \"kubernetes.io/projected/bb0e84cb-3717-40d6-9156-59d43497a89c-kube-api-access-r84wz\") pod \"must-gather-sl6xd\" (UID: \"bb0e84cb-3717-40d6-9156-59d43497a89c\") " pod="openshift-must-gather-bz2w9/must-gather-sl6xd"
Sep 18 17:53:52.316400 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:52.316378    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/bb0e84cb-3717-40d6-9156-59d43497a89c-must-gather-output\") pod \"must-gather-sl6xd\" (UID: \"bb0e84cb-3717-40d6-9156-59d43497a89c\") " pod="openshift-must-gather-bz2w9/must-gather-sl6xd"
Sep 18 17:53:52.325443 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:52.325420    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r84wz\" (UniqueName: \"kubernetes.io/projected/bb0e84cb-3717-40d6-9156-59d43497a89c-kube-api-access-r84wz\") pod \"must-gather-sl6xd\" (UID: \"bb0e84cb-3717-40d6-9156-59d43497a89c\") " pod="openshift-must-gather-bz2w9/must-gather-sl6xd"
Sep 18 17:53:52.497759 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:52.497655    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bz2w9/must-gather-sl6xd"
Sep 18 17:53:52.770470 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:52.770445    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bz2w9/must-gather-sl6xd"]
Sep 18 17:53:52.772568 ip-10-0-143-5 kubenswrapper[2881]: W0918 17:53:52.772539    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb0e84cb_3717_40d6_9156_59d43497a89c.slice/crio-9278b5b641c691816c3460e0c35ffb514106941e66f64eedd2a99440245c26c6 WatchSource:0}: Error finding container 9278b5b641c691816c3460e0c35ffb514106941e66f64eedd2a99440245c26c6: Status 404 returned error can't find the container with id 9278b5b641c691816c3460e0c35ffb514106941e66f64eedd2a99440245c26c6
Sep 18 17:53:52.967623 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:52.967584    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bz2w9/must-gather-sl6xd" event={"ID":"bb0e84cb-3717-40d6-9156-59d43497a89c","Type":"ContainerStarted","Data":"9278b5b641c691816c3460e0c35ffb514106941e66f64eedd2a99440245c26c6"}
Sep 18 17:53:59.007496 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:53:59.007453    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bz2w9/must-gather-sl6xd" event={"ID":"bb0e84cb-3717-40d6-9156-59d43497a89c","Type":"ContainerStarted","Data":"b65edcb974e814fd74316cfa2681b20a682153740f91ebb73a8380d7e0dd2a85"}
Sep 18 17:54:00.015423 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:00.015330    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bz2w9/must-gather-sl6xd" event={"ID":"bb0e84cb-3717-40d6-9156-59d43497a89c","Type":"ContainerStarted","Data":"38eb8e97f069c5adc1aaaa63a50ae14a274903126c12c71c434ed0de735ef6f4"}
Sep 18 17:54:00.035096 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:00.035037    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bz2w9/must-gather-sl6xd" podStartSLOduration=1.951702655 podStartE2EDuration="8.035019349s" totalImagesPullingTime="6.083316694s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:53:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:54:00.033290478 +0000 UTC m=+3630.233145767" watchObservedRunningTime="2026-09-18 17:54:00.035019349 +0000 UTC m=+3630.234874632"
Sep 18 17:54:30.171708 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:30.171669    2881 generic.go:356] "Generic (PLEG): container finished" podID="bb0e84cb-3717-40d6-9156-59d43497a89c" containerID="b65edcb974e814fd74316cfa2681b20a682153740f91ebb73a8380d7e0dd2a85" exitCode=0
Sep 18 17:54:30.172165 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:30.171741    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bz2w9/must-gather-sl6xd" event={"ID":"bb0e84cb-3717-40d6-9156-59d43497a89c","Type":"ContainerDied","Data":"b65edcb974e814fd74316cfa2681b20a682153740f91ebb73a8380d7e0dd2a85"}
Sep 18 17:54:30.172165 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:30.172100    2881 scope.go:117] "RemoveContainer" containerID="b65edcb974e814fd74316cfa2681b20a682153740f91ebb73a8380d7e0dd2a85"
Sep 18 17:54:30.514107 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:30.514067    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-bz2w9_must-gather-sl6xd_bb0e84cb-3717-40d6-9156-59d43497a89c/gather/0.log"
Sep 18 17:54:31.213884 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:31.213847    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-97c2p/must-gather-vmbkr"]
Sep 18 17:54:31.218958 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:31.218936    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-97c2p/must-gather-vmbkr"
Sep 18 17:54:31.221814 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:31.221794    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-97c2p\"/\"kube-root-ca.crt\""
Sep 18 17:54:31.222033 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:31.222018    2881 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-97c2p\"/\"default-dockercfg-h4984\""
Sep 18 17:54:31.222275 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:31.222257    2881 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-97c2p\"/\"openshift-service-ca.crt\""
Sep 18 17:54:31.227619 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:31.227599    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-97c2p/must-gather-vmbkr"]
Sep 18 17:54:31.273689 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:31.273644    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vsvb\" (UniqueName: \"kubernetes.io/projected/d0be2ca1-bbf1-49bb-8771-a4f7735e9bac-kube-api-access-7vsvb\") pod \"must-gather-vmbkr\" (UID: \"d0be2ca1-bbf1-49bb-8771-a4f7735e9bac\") " pod="openshift-must-gather-97c2p/must-gather-vmbkr"
Sep 18 17:54:31.273879 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:31.273750    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d0be2ca1-bbf1-49bb-8771-a4f7735e9bac-must-gather-output\") pod \"must-gather-vmbkr\" (UID: \"d0be2ca1-bbf1-49bb-8771-a4f7735e9bac\") " pod="openshift-must-gather-97c2p/must-gather-vmbkr"
Sep 18 17:54:31.374607 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:31.374566    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vsvb\" (UniqueName: \"kubernetes.io/projected/d0be2ca1-bbf1-49bb-8771-a4f7735e9bac-kube-api-access-7vsvb\") pod \"must-gather-vmbkr\" (UID: \"d0be2ca1-bbf1-49bb-8771-a4f7735e9bac\") " pod="openshift-must-gather-97c2p/must-gather-vmbkr"
Sep 18 17:54:31.374805 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:31.374635    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d0be2ca1-bbf1-49bb-8771-a4f7735e9bac-must-gather-output\") pod \"must-gather-vmbkr\" (UID: \"d0be2ca1-bbf1-49bb-8771-a4f7735e9bac\") " pod="openshift-must-gather-97c2p/must-gather-vmbkr"
Sep 18 17:54:31.374938 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:31.374923    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d0be2ca1-bbf1-49bb-8771-a4f7735e9bac-must-gather-output\") pod \"must-gather-vmbkr\" (UID: \"d0be2ca1-bbf1-49bb-8771-a4f7735e9bac\") " pod="openshift-must-gather-97c2p/must-gather-vmbkr"
Sep 18 17:54:31.384116 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:31.384092    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vsvb\" (UniqueName: \"kubernetes.io/projected/d0be2ca1-bbf1-49bb-8771-a4f7735e9bac-kube-api-access-7vsvb\") pod \"must-gather-vmbkr\" (UID: \"d0be2ca1-bbf1-49bb-8771-a4f7735e9bac\") " pod="openshift-must-gather-97c2p/must-gather-vmbkr"
Sep 18 17:54:31.529112 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:31.529063    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-97c2p/must-gather-vmbkr"
Sep 18 17:54:31.808210 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:31.808059    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-97c2p/must-gather-vmbkr"]
Sep 18 17:54:31.810598 ip-10-0-143-5 kubenswrapper[2881]: W0918 17:54:31.810571    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0be2ca1_bbf1_49bb_8771_a4f7735e9bac.slice/crio-d51799c487c95a46ea1f2cf84b84db502e5c8643e95806f9fd3a08a1c02af39e WatchSource:0}: Error finding container d51799c487c95a46ea1f2cf84b84db502e5c8643e95806f9fd3a08a1c02af39e: Status 404 returned error can't find the container with id d51799c487c95a46ea1f2cf84b84db502e5c8643e95806f9fd3a08a1c02af39e
Sep 18 17:54:32.182862 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:32.182771    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-97c2p/must-gather-vmbkr" event={"ID":"d0be2ca1-bbf1-49bb-8771-a4f7735e9bac","Type":"ContainerStarted","Data":"d51799c487c95a46ea1f2cf84b84db502e5c8643e95806f9fd3a08a1c02af39e"}
Sep 18 17:54:33.190688 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:33.190651    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-97c2p/must-gather-vmbkr" event={"ID":"d0be2ca1-bbf1-49bb-8771-a4f7735e9bac","Type":"ContainerStarted","Data":"75c62daa1d00b2644d481de173f7197cbfac1299798db0b9b41b1c66235fcbd4"}
Sep 18 17:54:33.191123 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:33.190781    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-97c2p/must-gather-vmbkr" event={"ID":"d0be2ca1-bbf1-49bb-8771-a4f7735e9bac","Type":"ContainerStarted","Data":"4191439bfca87b38f16062489867b7f83e07deaacf191bb7ad3c24467224a0dd"}
Sep 18 17:54:33.212092 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:33.212021    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-97c2p/must-gather-vmbkr" podStartSLOduration=1.4337374920000001 podStartE2EDuration="2.211998998s" totalImagesPullingTime="778.261506ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:54:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:54:33.208805563 +0000 UTC m=+3663.408660851" watchObservedRunningTime="2026-09-18 17:54:33.211998998 +0000 UTC m=+3663.411854286"
Sep 18 17:54:34.491822 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:34.491775    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-fx8nf_a4df48ac-dd04-4504-b828-66d7e2e986e1/global-pull-secret-syncer/0.log"
Sep 18 17:54:34.791149 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:34.791107    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-g7d7f_fb327355-e5c6-48fd-abf3-24fa50a8e0b5/konnectivity-agent/0.log"
Sep 18 17:54:35.136379 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:35.136301    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-5.ec2.internal_640796da303be256712f04b36adf26a6/haproxy/0.log"
Sep 18 17:54:36.077685 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:36.077638    2881 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-bz2w9/must-gather-sl6xd"]
Sep 18 17:54:36.078181 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:36.077879    2881 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-bz2w9/must-gather-sl6xd" podUID="bb0e84cb-3717-40d6-9156-59d43497a89c" containerName="copy" containerID="cri-o://38eb8e97f069c5adc1aaaa63a50ae14a274903126c12c71c434ed0de735ef6f4" gracePeriod=2
Sep 18 17:54:36.083357 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:36.083322    2881 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-bz2w9/must-gather-sl6xd"]
Sep 18 17:54:36.083588 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:54:36.083558    2881 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-sl6xd\" is forbidden: User \"system:node:ip-10-0-143-5.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-bz2w9\": no relationship found between node 'ip-10-0-143-5.ec2.internal' and this object" podUID="bb0e84cb-3717-40d6-9156-59d43497a89c" pod="openshift-must-gather-bz2w9/must-gather-sl6xd"
Sep 18 17:54:36.210174 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:36.210151    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-bz2w9_must-gather-sl6xd_bb0e84cb-3717-40d6-9156-59d43497a89c/copy/0.log"
Sep 18 17:54:36.210810 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:36.210785    2881 generic.go:356] "Generic (PLEG): container finished" podID="bb0e84cb-3717-40d6-9156-59d43497a89c" containerID="38eb8e97f069c5adc1aaaa63a50ae14a274903126c12c71c434ed0de735ef6f4" exitCode=143
Sep 18 17:54:36.479455 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:36.479418    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-bz2w9_must-gather-sl6xd_bb0e84cb-3717-40d6-9156-59d43497a89c/copy/0.log"
Sep 18 17:54:36.483003 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:36.482668    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bz2w9/must-gather-sl6xd"
Sep 18 17:54:36.486591 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:54:36.486537    2881 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-sl6xd\" is forbidden: User \"system:node:ip-10-0-143-5.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-bz2w9\": no relationship found between node 'ip-10-0-143-5.ec2.internal' and this object" podUID="bb0e84cb-3717-40d6-9156-59d43497a89c" pod="openshift-must-gather-bz2w9/must-gather-sl6xd"
Sep 18 17:54:36.548889 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:36.546876    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r84wz\" (UniqueName: \"kubernetes.io/projected/bb0e84cb-3717-40d6-9156-59d43497a89c-kube-api-access-r84wz\") pod \"bb0e84cb-3717-40d6-9156-59d43497a89c\" (UID: \"bb0e84cb-3717-40d6-9156-59d43497a89c\") "
Sep 18 17:54:36.548889 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:36.546987    2881 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/bb0e84cb-3717-40d6-9156-59d43497a89c-must-gather-output\") pod \"bb0e84cb-3717-40d6-9156-59d43497a89c\" (UID: \"bb0e84cb-3717-40d6-9156-59d43497a89c\") "
Sep 18 17:54:36.548889 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:36.548521    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb0e84cb-3717-40d6-9156-59d43497a89c-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "bb0e84cb-3717-40d6-9156-59d43497a89c" (UID: "bb0e84cb-3717-40d6-9156-59d43497a89c"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:54:36.551394 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:36.551360    2881 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb0e84cb-3717-40d6-9156-59d43497a89c-kube-api-access-r84wz" (OuterVolumeSpecName: "kube-api-access-r84wz") pod "bb0e84cb-3717-40d6-9156-59d43497a89c" (UID: "bb0e84cb-3717-40d6-9156-59d43497a89c"). InnerVolumeSpecName "kube-api-access-r84wz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:54:36.648109 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:36.648045    2881 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/bb0e84cb-3717-40d6-9156-59d43497a89c-must-gather-output\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:54:36.648109 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:36.648082    2881 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r84wz\" (UniqueName: \"kubernetes.io/projected/bb0e84cb-3717-40d6-9156-59d43497a89c-kube-api-access-r84wz\") on node \"ip-10-0-143-5.ec2.internal\" DevicePath \"\""
Sep 18 17:54:37.219968 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:37.219933    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-bz2w9_must-gather-sl6xd_bb0e84cb-3717-40d6-9156-59d43497a89c/copy/0.log"
Sep 18 17:54:37.220593 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:37.220529    2881 scope.go:117] "RemoveContainer" containerID="38eb8e97f069c5adc1aaaa63a50ae14a274903126c12c71c434ed0de735ef6f4"
Sep 18 17:54:37.220765 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:37.220735    2881 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bz2w9/must-gather-sl6xd"
Sep 18 17:54:37.226134 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:54:37.226102    2881 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-sl6xd\" is forbidden: User \"system:node:ip-10-0-143-5.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-bz2w9\": no relationship found between node 'ip-10-0-143-5.ec2.internal' and this object" podUID="bb0e84cb-3717-40d6-9156-59d43497a89c" pod="openshift-must-gather-bz2w9/must-gather-sl6xd"
Sep 18 17:54:37.233724 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:37.233390    2881 scope.go:117] "RemoveContainer" containerID="b65edcb974e814fd74316cfa2681b20a682153740f91ebb73a8380d7e0dd2a85"
Sep 18 17:54:37.250405 ip-10-0-143-5 kubenswrapper[2881]: E0918 17:54:37.246312    2881 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-sl6xd\" is forbidden: User \"system:node:ip-10-0-143-5.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-bz2w9\": no relationship found between node 'ip-10-0-143-5.ec2.internal' and this object" podUID="bb0e84cb-3717-40d6-9156-59d43497a89c" pod="openshift-must-gather-bz2w9/must-gather-sl6xd"
Sep 18 17:54:37.897976 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:37.897935    2881 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb0e84cb-3717-40d6-9156-59d43497a89c" path="/var/lib/kubelet/pods/bb0e84cb-3717-40d6-9156-59d43497a89c/volumes"
Sep 18 17:54:38.988404 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:38.988317    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cdcc3b80-4274-48f4-bf97-09412758b028/alertmanager/0.log"
Sep 18 17:54:39.033860 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:39.033834    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cdcc3b80-4274-48f4-bf97-09412758b028/config-reloader/0.log"
Sep 18 17:54:39.082403 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:39.082238    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cdcc3b80-4274-48f4-bf97-09412758b028/kube-rbac-proxy-web/0.log"
Sep 18 17:54:39.116091 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:39.116058    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cdcc3b80-4274-48f4-bf97-09412758b028/kube-rbac-proxy/0.log"
Sep 18 17:54:39.163721 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:39.163675    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cdcc3b80-4274-48f4-bf97-09412758b028/kube-rbac-proxy-metric/0.log"
Sep 18 17:54:39.196247 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:39.196196    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cdcc3b80-4274-48f4-bf97-09412758b028/prom-label-proxy/0.log"
Sep 18 17:54:39.235418 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:39.235352    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cdcc3b80-4274-48f4-bf97-09412758b028/init-config-reloader/0.log"
Sep 18 17:54:39.332042 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:39.332006    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-tblfl_8c199118-475a-4607-ba80-9a8027f1b047/kube-state-metrics/0.log"
Sep 18 17:54:39.367353 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:39.367268    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-tblfl_8c199118-475a-4607-ba80-9a8027f1b047/kube-rbac-proxy-main/0.log"
Sep 18 17:54:39.407716 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:39.407682    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-tblfl_8c199118-475a-4607-ba80-9a8027f1b047/kube-rbac-proxy-self/0.log"
Sep 18 17:54:39.448647 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:39.448584    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-67b6f56d48-lvpbv_0d71645f-8eff-473b-b150-d623c873cf57/metrics-server/0.log"
Sep 18 17:54:39.491584 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:39.491546    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5984459ff8-j8k2q_62a6075e-7f67-4a49-80f6-a8f59132bb8a/monitoring-plugin/0.log"
Sep 18 17:54:39.644573 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:39.644527    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pbhzd_bea5387e-1e26-4be9-bd38-71c49eb4225d/node-exporter/0.log"
Sep 18 17:54:39.722075 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:39.722047    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pbhzd_bea5387e-1e26-4be9-bd38-71c49eb4225d/kube-rbac-proxy/0.log"
Sep 18 17:54:39.770709 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:39.770683    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pbhzd_bea5387e-1e26-4be9-bd38-71c49eb4225d/init-textfile/0.log"
Sep 18 17:54:39.907102 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:39.907005    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-dn7sr_69a62b4d-4c8d-4dc4-81a8-35628f51c837/kube-rbac-proxy-main/0.log"
Sep 18 17:54:39.935302 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:39.935261    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-dn7sr_69a62b4d-4c8d-4dc4-81a8-35628f51c837/kube-rbac-proxy-self/0.log"
Sep 18 17:54:39.969206 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:39.969167    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-dn7sr_69a62b4d-4c8d-4dc4-81a8-35628f51c837/openshift-state-metrics/0.log"
Sep 18 17:54:40.027500 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:40.027461    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_465d3c10-1648-4252-87c0-9b27987d8de4/prometheus/0.log"
Sep 18 17:54:40.063105 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:40.063073    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_465d3c10-1648-4252-87c0-9b27987d8de4/config-reloader/0.log"
Sep 18 17:54:40.099305 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:40.099279    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_465d3c10-1648-4252-87c0-9b27987d8de4/thanos-sidecar/0.log"
Sep 18 17:54:40.158439 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:40.158359    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_465d3c10-1648-4252-87c0-9b27987d8de4/kube-rbac-proxy-web/0.log"
Sep 18 17:54:40.197399 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:40.197367    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_465d3c10-1648-4252-87c0-9b27987d8de4/kube-rbac-proxy/0.log"
Sep 18 17:54:40.237034 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:40.236992    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_465d3c10-1648-4252-87c0-9b27987d8de4/kube-rbac-proxy-thanos/0.log"
Sep 18 17:54:40.270296 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:40.270263    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_465d3c10-1648-4252-87c0-9b27987d8de4/init-config-reloader/0.log"
Sep 18 17:54:40.314792 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:40.314753    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-skzgp_809dca2e-a794-48bd-a35c-eafd66d3a19b/prometheus-operator/0.log"
Sep 18 17:54:40.342602 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:40.342570    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-skzgp_809dca2e-a794-48bd-a35c-eafd66d3a19b/kube-rbac-proxy/0.log"
Sep 18 17:54:40.509819 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:40.509792    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-96b4948d4-zsn4k_0de2e307-fa16-46a0-bc95-2a59d4fa9d44/thanos-query/0.log"
Sep 18 17:54:40.540170 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:40.540139    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-96b4948d4-zsn4k_0de2e307-fa16-46a0-bc95-2a59d4fa9d44/kube-rbac-proxy-web/0.log"
Sep 18 17:54:40.573564 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:40.573499    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-96b4948d4-zsn4k_0de2e307-fa16-46a0-bc95-2a59d4fa9d44/kube-rbac-proxy/0.log"
Sep 18 17:54:40.612758 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:40.612724    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-96b4948d4-zsn4k_0de2e307-fa16-46a0-bc95-2a59d4fa9d44/prom-label-proxy/0.log"
Sep 18 17:54:40.663065 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:40.663031    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-96b4948d4-zsn4k_0de2e307-fa16-46a0-bc95-2a59d4fa9d44/kube-rbac-proxy-rules/0.log"
Sep 18 17:54:40.734243 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:40.733667    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-96b4948d4-zsn4k_0de2e307-fa16-46a0-bc95-2a59d4fa9d44/kube-rbac-proxy-metrics/0.log"
Sep 18 17:54:42.930508 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:42.930462    2881 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-97c2p/perf-node-gather-daemonset-9qnks"]
Sep 18 17:54:42.931085 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:42.930997    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb0e84cb-3717-40d6-9156-59d43497a89c" containerName="gather"
Sep 18 17:54:42.931085 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:42.931012    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb0e84cb-3717-40d6-9156-59d43497a89c" containerName="gather"
Sep 18 17:54:42.931085 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:42.931022    2881 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb0e84cb-3717-40d6-9156-59d43497a89c" containerName="copy"
Sep 18 17:54:42.931085 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:42.931030    2881 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb0e84cb-3717-40d6-9156-59d43497a89c" containerName="copy"
Sep 18 17:54:42.931335 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:42.931105    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb0e84cb-3717-40d6-9156-59d43497a89c" containerName="gather"
Sep 18 17:54:42.931335 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:42.931115    2881 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb0e84cb-3717-40d6-9156-59d43497a89c" containerName="copy"
Sep 18 17:54:42.937194 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:42.937161    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-97c2p/perf-node-gather-daemonset-9qnks"
Sep 18 17:54:42.946184 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:42.945873    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-97c2p/perf-node-gather-daemonset-9qnks"]
Sep 18 17:54:43.011874 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:43.011839    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nqrj\" (UniqueName: \"kubernetes.io/projected/02b68a54-64be-4676-a650-8aaeed3281ae-kube-api-access-5nqrj\") pod \"perf-node-gather-daemonset-9qnks\" (UID: \"02b68a54-64be-4676-a650-8aaeed3281ae\") " pod="openshift-must-gather-97c2p/perf-node-gather-daemonset-9qnks"
Sep 18 17:54:43.011874 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:43.011877    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/02b68a54-64be-4676-a650-8aaeed3281ae-proc\") pod \"perf-node-gather-daemonset-9qnks\" (UID: \"02b68a54-64be-4676-a650-8aaeed3281ae\") " pod="openshift-must-gather-97c2p/perf-node-gather-daemonset-9qnks"
Sep 18 17:54:43.012139 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:43.011909    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/02b68a54-64be-4676-a650-8aaeed3281ae-podres\") pod \"perf-node-gather-daemonset-9qnks\" (UID: \"02b68a54-64be-4676-a650-8aaeed3281ae\") " pod="openshift-must-gather-97c2p/perf-node-gather-daemonset-9qnks"
Sep 18 17:54:43.012139 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:43.012039    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/02b68a54-64be-4676-a650-8aaeed3281ae-lib-modules\") pod \"perf-node-gather-daemonset-9qnks\" (UID: \"02b68a54-64be-4676-a650-8aaeed3281ae\") " pod="openshift-must-gather-97c2p/perf-node-gather-daemonset-9qnks"
Sep 18 17:54:43.012139 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:43.012085    2881 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02b68a54-64be-4676-a650-8aaeed3281ae-sys\") pod \"perf-node-gather-daemonset-9qnks\" (UID: \"02b68a54-64be-4676-a650-8aaeed3281ae\") " pod="openshift-must-gather-97c2p/perf-node-gather-daemonset-9qnks"
Sep 18 17:54:43.113052 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:43.113008    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02b68a54-64be-4676-a650-8aaeed3281ae-sys\") pod \"perf-node-gather-daemonset-9qnks\" (UID: \"02b68a54-64be-4676-a650-8aaeed3281ae\") " pod="openshift-must-gather-97c2p/perf-node-gather-daemonset-9qnks"
Sep 18 17:54:43.113282 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:43.113090    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nqrj\" (UniqueName: \"kubernetes.io/projected/02b68a54-64be-4676-a650-8aaeed3281ae-kube-api-access-5nqrj\") pod \"perf-node-gather-daemonset-9qnks\" (UID: \"02b68a54-64be-4676-a650-8aaeed3281ae\") " pod="openshift-must-gather-97c2p/perf-node-gather-daemonset-9qnks"
Sep 18 17:54:43.113282 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:43.113122    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/02b68a54-64be-4676-a650-8aaeed3281ae-proc\") pod \"perf-node-gather-daemonset-9qnks\" (UID: \"02b68a54-64be-4676-a650-8aaeed3281ae\") " pod="openshift-must-gather-97c2p/perf-node-gather-daemonset-9qnks"
Sep 18 17:54:43.113282 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:43.113140    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02b68a54-64be-4676-a650-8aaeed3281ae-sys\") pod \"perf-node-gather-daemonset-9qnks\" (UID: \"02b68a54-64be-4676-a650-8aaeed3281ae\") " pod="openshift-must-gather-97c2p/perf-node-gather-daemonset-9qnks"
Sep 18 17:54:43.113282 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:43.113153    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/02b68a54-64be-4676-a650-8aaeed3281ae-podres\") pod \"perf-node-gather-daemonset-9qnks\" (UID: \"02b68a54-64be-4676-a650-8aaeed3281ae\") " pod="openshift-must-gather-97c2p/perf-node-gather-daemonset-9qnks"
Sep 18 17:54:43.113536 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:43.113342    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/02b68a54-64be-4676-a650-8aaeed3281ae-proc\") pod \"perf-node-gather-daemonset-9qnks\" (UID: \"02b68a54-64be-4676-a650-8aaeed3281ae\") " pod="openshift-must-gather-97c2p/perf-node-gather-daemonset-9qnks"
Sep 18 17:54:43.113536 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:43.113358    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/02b68a54-64be-4676-a650-8aaeed3281ae-podres\") pod \"perf-node-gather-daemonset-9qnks\" (UID: \"02b68a54-64be-4676-a650-8aaeed3281ae\") " pod="openshift-must-gather-97c2p/perf-node-gather-daemonset-9qnks"
Sep 18 17:54:43.113536 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:43.113371    2881 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/02b68a54-64be-4676-a650-8aaeed3281ae-lib-modules\") pod \"perf-node-gather-daemonset-9qnks\" (UID: \"02b68a54-64be-4676-a650-8aaeed3281ae\") " pod="openshift-must-gather-97c2p/perf-node-gather-daemonset-9qnks"
Sep 18 17:54:43.113536 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:43.113482    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/02b68a54-64be-4676-a650-8aaeed3281ae-lib-modules\") pod \"perf-node-gather-daemonset-9qnks\" (UID: \"02b68a54-64be-4676-a650-8aaeed3281ae\") " pod="openshift-must-gather-97c2p/perf-node-gather-daemonset-9qnks"
Sep 18 17:54:43.128617 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:43.128583    2881 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nqrj\" (UniqueName: \"kubernetes.io/projected/02b68a54-64be-4676-a650-8aaeed3281ae-kube-api-access-5nqrj\") pod \"perf-node-gather-daemonset-9qnks\" (UID: \"02b68a54-64be-4676-a650-8aaeed3281ae\") " pod="openshift-must-gather-97c2p/perf-node-gather-daemonset-9qnks"
Sep 18 17:54:43.235988 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:43.235901    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7dd6748f4f-46p69_87e97870-514e-482f-b4bc-c62cfe89dc39/console/0.log"
Sep 18 17:54:43.251298 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:43.251262    2881 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-97c2p/perf-node-gather-daemonset-9qnks"
Sep 18 17:54:43.538785 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:43.538760    2881 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-97c2p/perf-node-gather-daemonset-9qnks"]
Sep 18 17:54:43.541204 ip-10-0-143-5 kubenswrapper[2881]: W0918 17:54:43.541178    2881 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod02b68a54_64be_4676_a650_8aaeed3281ae.slice/crio-c782ff77c444bd8f98d913f1c68e44889e879d4c3c85af741d1ae874685204cc WatchSource:0}: Error finding container c782ff77c444bd8f98d913f1c68e44889e879d4c3c85af741d1ae874685204cc: Status 404 returned error can't find the container with id c782ff77c444bd8f98d913f1c68e44889e879d4c3c85af741d1ae874685204cc
Sep 18 17:54:44.263778 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:44.263724    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-97c2p/perf-node-gather-daemonset-9qnks" event={"ID":"02b68a54-64be-4676-a650-8aaeed3281ae","Type":"ContainerStarted","Data":"645294b0349d9df2207cfe60698986e5fc68f28276ee41e49911c3a5e56e405c"}
Sep 18 17:54:44.264271 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:44.263812    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-97c2p/perf-node-gather-daemonset-9qnks"
Sep 18 17:54:44.264271 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:44.263836    2881 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-97c2p/perf-node-gather-daemonset-9qnks" event={"ID":"02b68a54-64be-4676-a650-8aaeed3281ae","Type":"ContainerStarted","Data":"c782ff77c444bd8f98d913f1c68e44889e879d4c3c85af741d1ae874685204cc"}
Sep 18 17:54:44.289249 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:44.289169    2881 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-97c2p/perf-node-gather-daemonset-9qnks" podStartSLOduration=2.289122485 podStartE2EDuration="2.289122485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 17:54:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:54:44.28878889 +0000 UTC m=+3674.488644177" watchObservedRunningTime="2026-09-18 17:54:44.289122485 +0000 UTC m=+3674.488977773"
Sep 18 17:54:45.175088 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:45.175055    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-c8l26_47f2a06a-1509-44ba-a107-56212158df3d/dns/0.log"
Sep 18 17:54:45.231895 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:45.231857    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-c8l26_47f2a06a-1509-44ba-a107-56212158df3d/kube-rbac-proxy/0.log"
Sep 18 17:54:45.458712 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:45.458629    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nmhbw_8925ea6c-2280-428c-bb7f-1dc4e8bab99b/dns-node-resolver/0.log"
Sep 18 17:54:46.066769 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:46.066719    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-sghwn_7295391c-ebd9-4f37-9f70-a86b0bab3fcf/node-ca/0.log"
Sep 18 17:54:47.365687 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:47.365657    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-b66z2_7b5b1755-3465-43cd-b0d1-750ef9bce496/serve-healthcheck-canary/0.log"
Sep 18 17:54:48.006900 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:48.006874    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kjcm9_b7fad33b-7ced-4268-8aab-5e98a1831c7a/kube-rbac-proxy/0.log"
Sep 18 17:54:48.032853 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:48.032814    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kjcm9_b7fad33b-7ced-4268-8aab-5e98a1831c7a/exporter/0.log"
Sep 18 17:54:48.060324 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:48.060282    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kjcm9_b7fad33b-7ced-4268-8aab-5e98a1831c7a/extractor/0.log"
Sep 18 17:54:50.280348 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:50.280319    2881 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-97c2p/perf-node-gather-daemonset-9qnks"
Sep 18 17:54:51.663370 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:51.663327    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-x2g9p_21ce4cae-ef77-4050-8d30-e0522e6c7116/s3-init/0.log"
Sep 18 17:54:51.684666 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:51.684629    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-x2g9p_21ce4cae-ef77-4050-8d30-e0522e6c7116/download-hf-model/0.log"
Sep 18 17:54:51.722750 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:51.722723    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-6zcfm_18be7a4a-ca58-4ce2-b81e-e07710a62746/s3-tls-init-custom/0.log"
Sep 18 17:54:51.799919 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:51.799890    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-6zcfm_18be7a4a-ca58-4ce2-b81e-e07710a62746/download-hf-model/0.log"
Sep 18 17:54:51.864425 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:51.864387    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-m5bq2_9037cfd0-1f18-4bc6-b35a-f920c082823e/s3-tls-init-serving/0.log"
Sep 18 17:54:51.919486 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:51.919458    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-m5bq2_9037cfd0-1f18-4bc6-b35a-f920c082823e/download-hf-model/0.log"
Sep 18 17:54:59.003429 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:59.003388    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-8rtmt_b6ec1dac-2d47-4156-bb74-b7055bedbc2b/kube-multus/0.log"
Sep 18 17:54:59.450653 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:59.450627    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n4nk2_d019d1b3-e6ee-4b27-9fc1-a6675650d42a/kube-multus-additional-cni-plugins/0.log"
Sep 18 17:54:59.480181 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:59.480151    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n4nk2_d019d1b3-e6ee-4b27-9fc1-a6675650d42a/egress-router-binary-copy/0.log"
Sep 18 17:54:59.506608 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:59.506569    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n4nk2_d019d1b3-e6ee-4b27-9fc1-a6675650d42a/cni-plugins/0.log"
Sep 18 17:54:59.533035 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:59.533001    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n4nk2_d019d1b3-e6ee-4b27-9fc1-a6675650d42a/bond-cni-plugin/0.log"
Sep 18 17:54:59.559338 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:59.559309    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n4nk2_d019d1b3-e6ee-4b27-9fc1-a6675650d42a/routeoverride-cni/0.log"
Sep 18 17:54:59.589033 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:59.589002    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n4nk2_d019d1b3-e6ee-4b27-9fc1-a6675650d42a/whereabouts-cni-bincopy/0.log"
Sep 18 17:54:59.615946 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:59.615910    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n4nk2_d019d1b3-e6ee-4b27-9fc1-a6675650d42a/whereabouts-cni/0.log"
Sep 18 17:54:59.682582 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:59.682549    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-lpt5n_89faf7f5-d633-4219-b3fc-323f8c23b599/network-metrics-daemon/0.log"
Sep 18 17:54:59.704738 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:54:59.704655    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-lpt5n_89faf7f5-d633-4219-b3fc-323f8c23b599/kube-rbac-proxy/0.log"
Sep 18 17:55:00.732934 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:55:00.732899    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7dglz_a03f2d01-68c8-4498-927e-c8722e1829b9/ovn-controller/0.log"
Sep 18 17:55:00.767893 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:55:00.767856    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7dglz_a03f2d01-68c8-4498-927e-c8722e1829b9/ovn-acl-logging/0.log"
Sep 18 17:55:00.811867 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:55:00.811831    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7dglz_a03f2d01-68c8-4498-927e-c8722e1829b9/kube-rbac-proxy-node/0.log"
Sep 18 17:55:00.840803 ip-10-0-143-5 kubenswrapper[2881]: I0918 17:55:00.840772    2881 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7dglz_a03f2d01-68c8-4498-927e-c8722e1829b9/kube-rbac-proxy-ovn-metrics/0.log"